Device, method and graphical user interface for providing computer-generated experience

Abstract

To provide a computer-generated experience to make interaction with a computing system more efficient and intuitive for a user.SOLUTION: A computing system displays, via a first display generation component, a first computer-generated environment and simultaneously displays, via a second display generation component, a visual representation of a portion of a user of the computing system who is in a position to view the first computer-generated environment via the first display generation component and one or more graphical elements that provide a visual indication of content in the first computer-generated environment. The computing system alters the visual representation of a portion of the user to represent a change in the appearance of the user over a discrete period and alters one or more graphical elements to represent a change in the first computer-generated environment over a discrete period.SELECTED DRAWING: None

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] (Priority and related applications) This application is a continuation of U.S. Patent Application No. 17 / 200,676, filed March 12, 2021, entitled "DEVICES, METHODS, AND GRAPHICAL USER INTERFACES FOR PROVIDING COMPUTER-GENERATED EXPERIENCES," which claims priority to U.S. Provisional Application No. 62 / 990,408, filed March 16, 2020. [Technical Field]

[0002] The present disclosure generally relates to computing systems having one or more display generation components and one or more input devices that provide computer-generated experiences, including, but not limited to, electronic devices that provide virtual reality and mixed reality experiences through one or more displays. [Background technology]

[0003] The development of computing systems for augmented reality has progressed significantly in recent years. Exemplary augmented reality environments include at least some virtual elements that replace or augment the representation of the physical world. Input devices such as cameras, controllers, joysticks, touch-sensitive surfaces, and touchscreen displays for computing systems and other electronic computing devices are used to interact with the virtual / augmented reality environment. Exemplary virtual elements include virtual three-dimensional objects, digital images, video, text, icons, and control elements such as buttons and other graphics.

[0004] However, methods and interfaces for interacting with environments (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) that include at least some virtual elements are cumbersome, inefficient, and limited. For example, systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to achieve a desired result in an augmented reality environment, and systems in which manipulating virtual objects is complex and error-prone create a significant cognitive burden for users and detract from the experience of the virtual / augmented reality environment. Additionally, these methods are unnecessarily time-consuming, thereby wasting energy. This latter consideration is particularly important in battery-operated devices. Additionally, many systems that provide virtual reality and / or mixed reality experiences use head-mounted display devices that physically shield the user's face from the user's surroundings when the user engages in the virtual reality and mixed reality experience, preventing social interaction and exchange of information with the outside world. Summary of the Invention

[0005] Thus, there is a need for a computing system with improved methods and interfaces for providing users with computer-generated experiences that make interaction with the computing system more efficient and intuitive for the user. There is also a need for a computing system with improved methods and interfaces for providing users with computer-generated experiences that facilitate better social interaction, etiquette, and information exchange with the surrounding environment while the user is engaging in various virtual reality and mixed reality experiences. Such methods and interfaces optionally complement or replace conventional methods of providing users with computer-generated reality experiences. Such methods and interfaces reduce the number, extent, and / or type of inputs from the user by helping the user understand the connection between the inputs provided and the device responses to those inputs, thereby creating a more efficient human-machine interface. Such methods and interfaces also improve the user's experience by, for example, reducing errors, interruptions, and time delays due to a lack of social cues and visual information on the part of the user and others present in the same physical environment when the user engages in virtual reality and / or mixed reality experiences provided by the computing system.

[0006] The above-mentioned deficiencies and other problems associated with user interfaces for computing systems having a display generating component and one or more input devices are reduced or eliminated by the disclosed system. In some embodiments, the computing system is a desktop computer with one or more associated displays. In some embodiments, the computing system is a portable device (e.g., a notebook computer, a tablet computer, or a handheld device). In some embodiments, the computing system is a personal electronic device (e.g., a wearable electronic device such as a watch or a head-mounted device). In some embodiments, the computing system has a touchpad. In some embodiments, the computing system has one or more cameras. In some embodiments, the computing system has a touch-sensitive display (also known as a "touch screen" or "touchscreen display"). In some embodiments, the computing system has one or more eye-tracking components. In some embodiments, the computing system has one or more hand-tracking components. In some embodiments, the computing system has one or more output devices, in addition to the one or more display generating components, the output devices including one or more tactile output generators and one or more audio output devices. In some embodiments, a computing system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or instruction sets stored in the memory for performing functions. In some embodiments, a user interacts with the GUI through stylus and / or finger contacts and gestures on a touch-sensitive surface, the movement of the user's eyes and hands in space relative to the GUI or the user's body as captured by cameras and other movement sensors, and voice input as captured by one or more audio input devices.In some embodiments, the functions performed through the interactions optionally include image editing, drawing, presenting, word processing, spreadsheet creation, game playing, making phone calls, video conferencing, emailing, instant messaging, training support, digital photography, digital videography, web browsing, digital music playback, note taking, and / or digital video playback, and executable instructions to perform those functions are optionally contained on a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0007] As disclosed herein, a computing system includes at least two display generating components, a first display generating component facing a user and providing a three-dimensional computer-generated experience to the user, and a second display generating component facing away from the user and providing state information related to the user (e.g., the user's eye movement) and / or state information related to the computer-generated experience currently viewed by the user (e.g., metadata related to content viewed by the user and an immersion level associated with the content). The first display generating component and the second display generating component are optionally enclosed in the same housing of a head-mounted display device (HMD) and are two displays facing inward toward a user wearing the HMD and outward toward the physical environment surrounding the user, respectively. The second display generation component optionally provides real-time state information, including a visual representation of a portion of the user that is occluded behind the first display generation component, and including metadata related to and / or an immersion level associated with the content currently being shown to the user via the first display generation component, so that another person(s) in the physical environment around the user can see the visual information and metadata provided by the second display generation component and act accordingly, e.g., engage with the user when appropriate while the user is viewing computer-generated content via the first display generation component, as opposed to unnecessarily avoiding interaction with the user or inappropriately interrupting the user.In some embodiments, a user of the computing system optionally activates different modes of the computing system to suit the user's intended level of engagement and privacy needs when engaging with the computer-generated environment provided via the first display generation component, and the computing system provides status information associated with the various modes to alert people in the surrounding physical environment of the user's intentions and needs, thereby reducing unintended, undesired, and / or unnecessary interruptions and interactions by people in the surrounding physical environment.

[0008] As disclosed herein, when a computing system includes at least two display generating components within the same housing, a second (e.g., outward-facing) display generating component optionally displays contextual information indicating the availability of a computer-generated experience based on a current context. In response to detecting that a first (e.g., inward-facing) display generating component is positioned in front of a user's eyes (e.g., a user is wearing an HMD with an inward-facing display facing the user's eyes, or a user is holding an HMD with an inward-facing display in front of the user's eyes), the computing system provides a computer-generated experience to the user via the first display generating component. By automatically alerting the user to computer-generated experiences available via the outward-facing display based on the current context (e.g., automatically displaying computer-generated experiences corresponding to the current context while the user is in a position to view the outward-facing display (e.g., while the user is not wearing the HMD on their head, while the HMD is placed on a table, etc.)) and / or when the inward-facing display is placed in front of the user (e.g., when the user is wearing the HMD on their head or holding the HMD with the internal display facing the user's face or eyes), the number, complexity, and range of inputs required for users to discover what computer-generated experiences are available in various contexts and selectively view desired computer-generated experiences (e.g., without having to always wear the HMD and / or without having to browse through selectable options to locate a desired CGR content item and / or activate displayed controls while wearing the HMD to start a desired CGR experience).In some embodiments, depending on whether the first display generation component is actually worn by the user (e.g., strapped to the user's head or body as opposed to being held in front of the user in the user's hand(s)), the computing system optionally provides different computer-generated experiences corresponding to the wearing state of the first display generation component (e.g., displaying previews (e.g., abbreviated, two-dimensional or three-dimensional, interactive, etc.) of available computer-generated experiences when the first display generation component is not actually worn by the user, and displaying full versions of the available computer-generated experiences when the first display generation component is worn by the user). Selectively displaying different versions of a computer-generated experience or different computer-generated experiences depending not only on the position of the display generation component relative to the user (e.g., based on whether the position allows the user to view the CGR experience), but also on whether the display generation component is securely worn by the user (e.g., based on whether the user's hands are free or are needed to hold the display generation component in its current position) reduces the number of inputs required to trigger an intended result and avoids unnecessarily activating the full computer-generated experience, thereby saving the user time when the user only wants to briefly preview the computer-generated experience and conserving battery power of the display generation component and the computing system when battery-powered.

[0009] As disclosed herein, in some embodiments, a computing system includes a first display generating component and a second display generating component within the same housing or mounted to the same physical support structure. The first display generating component and the second display generating component are optionally non-transparent and have their respective display surfaces facing in opposite directions. The display generating components, together with the housing or support structure, can be quite bulky and cumbersome to attach to and remove from a user's head / body. The display generating components also together form a significant physical barrier between the user and others in the surrounding physical environment. By utilizing the external display (e.g., the second display generation component) to indicate status information (e.g., title, progress, type, etc.) related to metadata of the CGR content displayed on the internal display (e.g., the first display generation component), the immersion level associated with the displayed CGR content (e.g., full pass-through, mixed reality, virtual reality, etc.), and / or visual characteristics of the displayed CGR content (e.g., varying color, brightness, etc.), the current display mode of the computing system (e.g., privacy mode, parental control mode, silent mode, etc.), and / or characteristics of the user (e.g., the user's eye appearance, the user's identifier, etc.), without requiring the user to physically remove the display generation component, the impact of the presence of a physical barrier between the user and others in the surrounding environment is reduced, reducing unnecessary obstacles to desired social interaction and unnecessary interruptions to the user's engagement with the computer-generated experience. Furthermore, by utilizing the external display to indicate contextual information and indications of contextually relevant computer-generated experiences, the user does not need to constantly pick up the HMD and place the internal display in front of the user to find out what CGR content is available. Users also do not need to be fully strapped into an HMD to preview available CGR experiences.The user only needs to fully don the HMD when they want to fully engage in the CGR experience (e.g., interact with the CGR environment using air gestures and microgestures). In this way, the number of times the user needs to place the HMD's internal display in front of the user's eyes and / or fully strap the HMD onto the user's head is reduced without compromising the user's need to know what CGR experiences are available and / or without interfering with the user's ability to enjoy the desired CGR experience.

[0010] As disclosed herein, a computer-generated experience is provided via a display generation component of a computing system (e.g., a single display generation component of a device, an internal display of an HMD, etc.) in response to a user's physical interaction with a real-world physical object. Specifically, the computing system displays a visual indication that a computer-generated experience is available at a location within the three-dimensional environment displayed via the display generation component, the location of the visual indication corresponding to the location of a representation of the physical object within the three-dimensional environment. In response to detecting a physical interaction with the physical object in a first manner that satisfies preset criteria associated with the physical object, the computing system displays a computer-generated experience associated with the physical object and, optionally, the first manner of physical interaction. For example, the computing system displays a pass-through view of the user's hand and the physical object before the preset criteria are met by the user's manipulation of the physical object, and the computing system displays a computer-augmented representation of the user's hand(s) manipulating the physical object after the preset criteria are met. Automatically initiating a computer-generated experience in response to detecting a preset physical interaction with a real-world physical object improves the user's experience interacting with the physical object, making the interaction more intuitive and reducing user errors when interacting with the physical object.

[0011] As disclosed herein, a computing system includes a display generating component within a housing (e.g., a single display generating component of a device, an internal display of an HMD, etc.) and provides a user interface (e.g., buttons, a touch-sensitive surface, etc.) on the housing of the display generating component. When an input is detected via the user interface, the computing system determines whether to perform an action associated with the detected input via the user interface on the housing of the display generating component or to refrain from performing the action depending on whether a preset configuration of a user's hands (e.g., both hands) touching the housing is detected on the housing of the display generating component. By selecting to perform or not perform an action depending on the detected input in conjunction with the configuration of the hands on the housing of the display generating component, the computing system automatically distinguishes between a user's intentional input and other touches on the housing of the display generating component for purposes other than providing input to trigger a specific action, helping to avoid unintended consequences, reducing user confusion, and making it faster and easier for users to operate the display generating component.

[0012] According to some embodiments, a method is executed in a computing system including a first display generating component, a second display generating component, and one or more input devices, and includes displaying a first computer-generated environment via the first display generating component, and while displaying the first computer-generated environment via the first display generating component, simultaneously displaying via the second display generating component a visual representation of a portion of a user of the computing system positioned to view the first computer-generated environment via the first display generating component and one or more graphic elements that provide a visual indication of content within the first computer-generated environment, wherein simultaneously displaying the visual representation of the portion of the user and the one or more graphic elements includes modifying the visual representation of the portion of the user to represent changes in the user's appearance over discrete periods of time, and modifying the one or more graphic elements that provide a visual indication of content within the first computer-generated environment to represent changes in the first computer-generated environment over discrete periods of time.

[0013] According to some embodiments, a method is executed in a computing system including a first display generating component, a second display generating component, and one or more input devices, and includes: displaying a computer-generated environment via the first display generating component; displaying state information corresponding to the computing system via the second display generating component, including simultaneously displaying, while displaying the computer-generated environment via the first display generating component, a visual representation of a portion of a user of the computing system positioned to view the computer-generated environment via the first display generating component and one or more graphical elements providing a visual indication of content within the computer-generated environment; detecting a discrete event; and in response to detecting the discrete event, altering an immersion level of the computer-generated environment displayed via the first display generating component; and altering the state information displayed via the second display generating component, including altering an appearance of the visual representation of the portion of the user of the computing system.

[0014] According to some embodiments, a method is performed in a computing system including a first display generating component, a second display generating component, and one or more input devices, the method comprising: displaying one or more user interface elements via the second display generating component; detecting, while displaying the one or more user interface elements via the second display generating component, that the first display generating component has been moved in a predetermined orientation relative to a distinct portion of a user; and, in response to detecting that the first display generating component has been moved in the predetermined orientation relative to the distinct portion of the user, generating a first state corresponding to displaying the first user interface element via the second display generating component when the first display generating component is moved in the predetermined orientation relative to the distinct portion of the user. displaying a first user interface via the first display generating component while the first display generating component is in a predetermined orientation relative to the respective portion of the user in accordance with a determination that the computing system was in a first state corresponding to, instead of displaying the first user interface element via the second display generating component when the first display generating component is moved to the predetermined orientation relative to the respective portion of the user, displaying a second user interface different from the first user interface via the first display generating component while the first display generating component is in a predetermined orientation relative to the respective portion of the user in accordance with a determination that the computing system was in a second state corresponding to, instead of displaying the first user interface element via the second display generating component when the first display generating component is moved to the predetermined orientation relative to the respective portion of the user, displaying a second user interface different from the first user interface via the first display generating component while the first display generating component is in a predetermined orientation relative to the respective portion of the user.

[0015] According to some embodiments, a method is executed in a computing system including a first display generating component and one or more input devices, and includes detecting a first trigger event corresponding to the first display generating component being placed in a first default configuration for a user; in response to detecting the first trigger event, providing a first computer-generated experience via the first display generating component in accordance with a determination that the computing system including the first display generating component is worn by the user while in the first default configuration for the user; and providing a second computer-generated experience separate from the first computer-generated experience via the first display generating component in accordance with a determination that the computing system including the first display generating component is not worn by the user while in the first default configuration for the user.

[0016] According to some embodiments, a method is executed in a computing system including a first display generation component and one or more input devices, and includes: displaying a visual indication that a computer-generated experience corresponding to the physical object is available for display via the first display generation component while displaying a representation of the physical object at a position in a three-dimensional environment corresponding to the position of the physical object in the physical environment; detecting an interaction with the physical object in the physical environment while displaying the visual indication that the computer-generated experience is available for display via the first display generation component; in response to detecting the interaction with the physical object in the physical environment, displaying a computer-generated experience corresponding to the physical object via the first display generation component in accordance with a determination that the interaction with the physical object in the physical environment satisfies first criteria corresponding to the physical object; and ceasing to display the computer-generated experience corresponding to the physical object in accordance with a determination that the interaction with the physical object in the physical environment does not satisfy the first criteria.

[0017] According to some embodiments, a method is executed in a computing system including a housing, a first display generating component housed in the housing, and one or more input devices, and includes detecting a first hand on the housing housed in the first display generating component; in response to detecting the first hand on the housing housed in the first display generating component, aborting performance of an action associated with the first hand in accordance with a determination that the first hand has been detected in conjunction with a second hand being detected on the housing; and performing the action associated with the first hand in accordance with a determination that the first hand has been detected on the housing without detecting another hand on the housing.

[0018] According to some embodiments, a computing system includes one or more display generating components (e.g., one or more displays, projectors, head-mounted displays, etc., enclosed in the same or different housings), one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), optionally one or more tactile output generators, one or more processors, and a memory that stores one or more programs, the one or more programs configured to be executed by the one or more processors, the one or more programs including instructions to perform or cause to be performed any of the operations of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium has instructions stored therein that, when executed by a computing system having one or more display generating components, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), and optionally one or more tactile output generators, cause the device to perform or cause to be performed any of the operations of the methods described herein. According to some embodiments, a graphical user interface on a computing system having one or more display generation components, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), optionally one or more tactile output generators, memory, and one or more processors executing one or more programs stored in the memory, includes one or more of the elements displayed in any of the methods described herein, which elements are updated in response to input as described in any of the methods described herein.According to some embodiments, a computing system includes one or more display generation components, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), optionally one or more tactile output generators, and means for performing or causing to be performed any of the operations of the methods described herein. According to some embodiments, an information processing apparatus for use in a computing system having one or more display generation components, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), and optionally one or more tactile output generators includes means for performing or causing to be performed any of the operations of the methods described herein.

[0019] Thus, a computing system having one or more display generation components is provided with improved methods and interfaces for providing a user with computer-generated experiences that make interaction with the computing system more efficient and intuitive for the user. Also provided is a computing system having improved methods and interfaces for providing a user with computer-generated experiences that facilitate better social interaction, etiquette, and information exchange with the surrounding environment while the user is engaging in various virtual reality and mixed reality experiences. Such methods and interfaces optionally complement or replace conventional methods of providing a user with computer-generated reality experiences. Such methods and interfaces reduce the number, extent, and / or type of inputs from the user by helping the user understand the connection between the inputs provided and the device responses to those inputs, thereby creating a more efficient human-machine interface. Such methods and interfaces also improve the user's experience by, for example, reducing errors, interruptions, and time delays due to a lack of social cues and visual information on the part of the user and others present in the same physical environment when the user engages in virtual reality and / or mixed reality experiences provided by the computing system.

[0020] It should be noted that the various embodiments described above can be combined with any other embodiment described herein. The features and advantages described herein are not exhaustive, and many additional features and advantages will become apparent to those skilled in the art, particularly in light of the drawings, specification, and claims. Furthermore, it should be noted that the language used in this specification has been selected solely for the purposes of readability and explanation, and not to define or limit the subject matter of the present invention. [Brief explanation of the drawings]

[0021] For a better understanding of the various described embodiments, reference should be made to the following Detailed Description of the Invention in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout:

[0022] [Figure 1] FIG. 1 is a block diagram illustrating an operating environment of a computing system for providing a CGR experience, according to some embodiments.

[0023] [Figure 2] FIG. 1 is a block diagram illustrating a controller of a computing system configured to manage and coordinate a user's CGR experience, according to some embodiments.

[0024] [Figure 3] FIG. 1 is a block diagram illustrating display generation components of a computing system configured to provide a user with visual components of a CGR experience, according to some embodiments.

[0025] [Figure 4] FIG. 1 is a block diagram illustrating a hand tracking unit of a computing system configured to capture a user's gesture input, according to some embodiments.

[0026] [Figure 5] FIG. 1 is a block diagram illustrating an eye-tracking unit of a computing system configured to capture a user's gaze input, according to some embodiments.

[0027] [Figure 6] 1 is a flowchart illustrating a glint-assisted gaze tracking pipeline, according to some embodiments.

[0028] [Figure 7A]1 illustrates a computing system including a first display generating component and a second display generating component (e.g., separate displays facing different directions, displays enclosed in the same housing but facing different directions (e.g., facing opposite directions back to back, facing at different angles so that they cannot be viewed by the same user at the same time, etc.)). In some embodiments, the computing system displays content to a user via the first display generating component while displaying dynamically updated state information associated with the user and / or content via the second display generating component (e.g., a representation corresponding to changes in the user's appearance behind the first display generating component, metadata for the content, changes in immersion levels associated with content playback, etc.), according to some embodiments. [Figure 7B] 1 illustrates a computing system including a first display generating component and a second display generating component (e.g., separate displays facing different directions, displays enclosed in the same housing but facing different directions (e.g., facing opposite directions back to back, facing at different angles so that they cannot be viewed by the same user at the same time, etc.)). In some embodiments, the computing system displays content to a user via the first display generating component while displaying dynamically updated state information associated with the user and / or content via the second display generating component (e.g., a representation corresponding to changes in the user's appearance behind the first display generating component, metadata for the content, changes in immersion levels associated with content playback, etc.), according to some embodiments. [Figure 7C]1 illustrates a computing system including a first display generating component and a second display generating component (e.g., separate displays facing different directions, displays enclosed in the same housing but facing different directions (e.g., facing opposite directions back to back, facing at different angles so that they cannot be viewed by the same user at the same time, etc.)). In some embodiments, the computing system displays content to a user via the first display generating component while displaying dynamically updated state information associated with the user and / or content via the second display generating component (e.g., a representation corresponding to changes in the user's appearance behind the first display generating component, metadata for the content, changes in immersion levels associated with content playback, etc.), according to some embodiments. [Figure 7D] 1 illustrates a computing system including a first display generating component and a second display generating component (e.g., separate displays facing different directions, displays enclosed in the same housing but facing different directions (e.g., facing opposite directions back to back, facing at different angles so that they cannot be viewed by the same user at the same time, etc.)). In some embodiments, the computing system displays content to a user via the first display generating component while displaying dynamically updated state information associated with the user and / or content via the second display generating component (e.g., a representation corresponding to changes in the user's appearance behind the first display generating component, metadata for the content, changes in immersion levels associated with content playback, etc.), according to some embodiments. [Figure 7E]1 illustrates a computing system including a first display generating component and a second display generating component (e.g., separate displays facing different directions, displays enclosed in the same housing but facing different directions (e.g., facing opposite directions back to back, facing at different angles so that they cannot be viewed by the same user at the same time, etc.)). In some embodiments, the computing system displays content to a user via the first display generating component while displaying dynamically updated state information associated with the user and / or content via the second display generating component (e.g., a representation corresponding to changes in the user's appearance behind the first display generating component, metadata for the content, changes in immersion levels associated with content playback, etc.), according to some embodiments.

[0029] [Figure 7F]1 illustrates a computing system including a first display generating component and a second display generating component (e.g., separate displays facing different directions, displays enclosed in the same housing but facing different directions (e.g., facing opposite directions back to back, facing at different angles so that they cannot be viewed by the same user at the same time, etc.)). In some embodiments, the computing system displays indications of different computer-generated experiences via the second display generating component based on contextual information associated with the computing system (e.g., positions of the first and second display generating components, a user's identity, the current time, etc.), according to some embodiments, and triggers the display of different computer-generated experiences corresponding to the contextual information via the first display generating component in response to changes in spatial relationship (e.g., from not facing the user's eyes to facing the user's eyes, from resting on a table or in a bag to being lifted to the user's eye level, etc.) and / or changes in the wearing state of the first display generating component relative to the user (e.g., from being supported by the user's hands to being supported by the user's head / nose / ears, from not being worn on the user's head / body to being worn on the user's head / body, etc.). In some embodiments, the computing system includes only a single display generation component and / or is arranged in a pre-configured configuration for the user and does not display an indication of available computer-generated experiences before beginning to display different computer-generated experiences based on the wear state of the display generation component. [Figure 7G]1 illustrates a computing system including a first display generating component and a second display generating component (e.g., separate displays facing different directions, displays enclosed in the same housing but facing different directions (e.g., facing opposite directions back to back, facing at different angles so that they cannot be viewed by the same user at the same time, etc.)). In some embodiments, the computing system displays indications of different computer-generated experiences via the second display generating component based on contextual information associated with the computing system (e.g., positions of the first and second display generating components, a user's identity, the current time, etc.), according to some embodiments, and triggers the display of different computer-generated experiences corresponding to the contextual information via the first display generating component in response to changes in spatial relationship (e.g., from not facing the user's eyes to facing the user's eyes, from resting on a table or in a bag to being lifted to the user's eye level, etc.) and / or changes in the wearing state of the first display generating component relative to the user (e.g., from being supported by the user's hands to being supported by the user's head / nose / ears, from not being worn on the user's head / body to being worn on the user's head / body, etc.). In some embodiments, the computing system includes only a single display generation component and / or is arranged in a pre-configured configuration for the user and does not display an indication of available computer-generated experiences before beginning to display different computer-generated experiences based on the wear state of the display generation component. [Figure 7H]1 illustrates a computing system including a first display generating component and a second display generating component (e.g., separate displays facing different directions, displays enclosed in the same housing but facing different directions (e.g., facing opposite directions back to back, facing at different angles so that they cannot be viewed by the same user at the same time, etc.)). In some embodiments, the computing system displays indications of different computer-generated experiences via the second display generating component based on contextual information associated with the computing system (e.g., positions of the first and second display generating components, a user's identity, the current time, etc.), according to some embodiments, and triggers the display of different computer-generated experiences corresponding to the contextual information via the first display generating component in response to changes in spatial relationship (e.g., from not facing the user's eyes to facing the user's eyes, from resting on a table or in a bag to being lifted to the user's eye level, etc.) and / or changes in the wearing state of the first display generating component relative to the user (e.g., from being supported by the user's hands to being supported by the user's head / nose / ears, from not being worn on the user's head / body to being worn on the user's head / body, etc.). In some embodiments, the computing system includes only a single display generation component and / or is arranged in a pre-configured configuration for the user and does not display an indication of available computer-generated experiences before beginning to display different computer-generated experiences based on the wear state of the display generation component. [Figure 7I]1 illustrates a computing system including a first display generating component and a second display generating component (e.g., separate displays facing different directions, displays enclosed in the same housing but facing different directions (e.g., facing opposite directions back to back, facing at different angles so that they cannot be viewed by the same user at the same time, etc.)). In some embodiments, the computing system displays indications of different computer-generated experiences via the second display generating component based on contextual information associated with the computing system (e.g., positions of the first and second display generating components, a user's identity, the current time, etc.), according to some embodiments, and triggers the display of different computer-generated experiences corresponding to the contextual information via the first display generating component in response to changes in spatial relationship (e.g., from not facing the user's eyes to facing the user's eyes, from resting on a table or in a bag to being lifted to the user's eye level, etc.) and / or changes in the wearing state of the first display generating component relative to the user (e.g., from being supported by the user's hands to being supported by the user's head / nose / ears, from not being worn on the user's head / body to being worn on the user's head / body, etc.). In some embodiments, the computing system includes only a single display generation component and / or is arranged in a pre-configured configuration for the user and does not display an indication of available computer-generated experiences before beginning to display different computer-generated experiences based on the wear state of the display generation component. [Figure 7J]1 illustrates a computing system including a first display generating component and a second display generating component (e.g., separate displays facing different directions, displays enclosed in the same housing but facing different directions (e.g., facing opposite directions back to back, facing at different angles so that they cannot be viewed by the same user at the same time, etc.)). In some embodiments, the computing system displays indications of different computer-generated experiences via the second display generating component based on contextual information associated with the computing system (e.g., positions of the first and second display generating components, a user's identity, the current time, etc.), according to some embodiments, and triggers the display of different computer-generated experiences corresponding to the contextual information via the first display generating component in response to changes in spatial relationship (e.g., from not facing the user's eyes to facing the user's eyes, from resting on a table or in a bag to being lifted to the user's eye level, etc.) and / or changes in the wearing state of the first display generating component relative to the user (e.g., from being supported by the user's hands to being supported by the user's head / nose / ears, from not being worn on the user's head / body to being worn on the user's head / body, etc.). In some embodiments, the computing system includes only a single display generation component and / or is arranged in a pre-configured configuration for the user and does not display an indication of available computer-generated experiences before beginning to display different computer-generated experiences based on the wear state of the display generation component.

[0030] [Figure 7K] According to some embodiments, an indication of the availability of a computer-generated experience is displayed at a location corresponding to a representation of a physical object in a mixed reality environment, and in response to detecting a pre-configured physical interaction with the real-world physical object, the display of a computer-generated experience corresponding to the physical object is triggered. [Figure 7L] According to some embodiments, an indication of the availability of a computer-generated experience is displayed at a location corresponding to a representation of a physical object in a mixed reality environment, and in response to detecting a pre-configured physical interaction with the real-world physical object, the display of a computer-generated experience corresponding to the physical object is triggered. [Figure 7M] According to some embodiments, an indication of the availability of a computer-generated experience is displayed at a location corresponding to a representation of a physical object in a mixed reality environment, and in response to detecting a pre-configured physical interaction with the real-world physical object, the display of a computer-generated experience corresponding to the physical object is triggered.

[0031] [Figure 7N] 10 illustrates selecting to perform or not perform an action in response to an input detected on a housing of a display generating component depending on whether one or both hands are detected on the housing at the time the input is detected, according to some embodiments. [Figure 7O] 10 illustrates selecting to perform or not perform an action in response to an input detected on a housing of a display generating component depending on whether one or both hands are detected on the housing at the time the input is detected, according to some embodiments. [Figure 7P] 10 illustrates selecting to perform or not perform an action in response to an input detected on a housing of a display generating component depending on whether one or both hands are detected on the housing at the time the input is detected, according to some embodiments. [Figure 7Q] 10 illustrates selecting to perform or not perform an action in response to an input detected on a housing of a display generating component depending on whether one or both hands are detected on the housing at the time the input is detected, according to some embodiments.

[0032] [Figure 8]1 is a flowchart of a method for displaying a computer-generated environment and state information associated with the computer-generated environment and a user located at a viewing location of the computer-generated environment, according to some embodiments.

[0033] [Figure 9] 1 is a flowchart of a method for displaying a computer-generated environment and state information associated with the computer-generated environment and a user located at a viewing location of the computer-generated environment, according to some embodiments.

[0034] [Figure 10] 1 is a flowchart of a method for providing a computer-generated experience based on contextual information, according to some embodiments.

[0035] [Figure 11] 1 is a flowchart of a method for providing a computer-generated experience based on a wearable state of a display generation component, according to some embodiments.

[0036] [Figure 12] 1 is a flowchart of a method for triggering the display of a computer-generated experience based on detecting a pre-configured physical interaction with a real-world physical object, according to some embodiments.

[0037] [Figure 13] 1 is a flowchart of a method for performing an action in response to an input on a housing of a display generating component, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0038] The present disclosure relates to a user interface that provides a computer-generated reality (CGR) experience to a user, according to some embodiments.

[0039] The systems, methods, and GUIs described herein improve user interface interaction with virtual / augmented reality environments in several ways.

[0040] In some embodiments, a computing system includes a first display generation component and a second display generation component (e.g., separate displays, displays enclosed in the same housing but facing different directions (e.g., facing opposite directions back to back, facing at different angles so that they cannot be viewed by the same user at the same time, etc.)). The first display generation component displays a computer-generated environment (e.g., a user faces a display surface of the display generation component (e.g., a surface of a physical environment illuminated by a projector, a surface of a display that emits light that forms an image on the user's retina)) that provides a computer-generated experience to a user positioned to view content presented via the first display generation component. The first display generation component optionally provides computer-generated experiences with different levels of immersion corresponding to different amounts of visual and audio information from the surrounding physical environment that are still perceptible through the first display generation component when the computer-generated experience is being provided by the first display generation component. During normal operation (e.g., when a user is wearing an HMD including a first display generating component and / or facing the display surface of the first display generating component), the first display generating component blocks the user's direct view of the surrounding physical environment and simultaneously blocks others' views of the user's face or eyes when the user is positioned to view content presented via the first display generating component. In some embodiments, the first display generating component is an internal display of the HMD that faces the user's eyes when the HMD is positioned on the user's head. Traditionally, when a user is positioned to view content presented via the display generating component, the user has the option of viewing the physical environment or not viewing the physical environment by switching between displaying a computer-generated environment with different levels of appearance (e.g., switching between a full pass-through mode, a mixed reality mode, or a virtual reality mode).However, others in the surrounding environment facing the back of the display generating component have little or no visual clues about the user's attention state, what content is being shown on the display generating component, and / or whether the user can see the surrounding environment and the people therein. This imbalance of visual information (and optionally audio information) on the two sides of the display generating component makes social interaction between the user and others in the surrounding environment unnatural and inefficient. Many considerations could benefit from a computing system that uses a second display generating component to display to people in the surrounding environment an appropriate amount of visual information that conveys status information related to the user and / or the content displayed to the user via the first display generating component. The display of status information by the second display generating component is optionally displayed as long as the first display generating component is in use, or optionally triggered in response to detecting the presence of other people in the same physical environment and / or in response to detecting an indication that others may wish to engage the user in a social conversation (e.g., by entering the same room, looking in the user's direction, waving to the user, etc.). In some embodiments, showing the state information on the second display generating component includes displaying a representation of a portion of the user (e.g., a portion of the user that is obscured by the first display generating component when the user is in a position to view the content displayed via the first display generating component) that is dynamically updated according to changes in the user's appearance (e.g., changes in the portion of the user that is obscured by the first display generating component). In some embodiments, showing the state information also includes displaying a graphical element that provides a visual indication of the content currently being displayed via the first display generating component (e.g., simultaneously with showing the representation of the portion of the user).This method and system, which uses a second display generation component to display updated status information related to a user viewing content shown via a first display generation component and metadata associated with the state of the content (e.g., title, progress, immersion level, display mode, etc.), allows others in the user's surrounding environment to gain useful insight into the user's current state while the user is engaging in a computer-generated experience, but without fully revealing the computer-generated experience to the surrounding environment. In some embodiments, a representation of a portion of the user (e.g., the user's eyes or face) occluded by the first display generation component and graphic elements indicating the state of the content displayed via the first display generation component are respectively displayed on different display layers of the second display generation component and updated independently of each other. In some embodiments, updates to the representation of a portion of the user and the graphic elements indicating the state of the content on different display layers of the second display generation component provide a more realistic view of the user's state behind a head-mounted display device that houses both the first and second display generation components. The status information shown on the second display generation component allows the user to remain socially connected to people in the surrounding environment when engaging in a computer-generated experience via the first display generation component. Dynamically updated status information on the second display generation component indicating the state of the user's eyes and the state of the content shown to the user improves the user's engagement with the computer-generated experience when the user is in a public or semi-public environment, for example, by encouraging appropriate social interaction when such interaction is desired by the user, reducing unnecessary avoidance of social interaction by others in the surrounding environment due to a lack of visual cues about the user's permission to engage socially, and informing others of an appropriate time to interrupt the user's engagement with the computer-generated experience, reducing unwelcome interruptions to the user's engagement experience due to a lack of visual cues about the user's desire to remain undisturbed.

[0041] As discussed above, many considerations could benefit from a computing system that uses a second display generation component to display to others in the surrounding environment an appropriate amount of visual information that conveys status information related to the user and the content displayed to the user via the first display generation component. In some embodiments, the status information is displayed on the second display generation component as long as the first display generation component is in use. In some embodiments, the status information is displayed only in response to detecting the presence of other people in the same physical environment and / or in response to detecting some indication that others in the same physical environment may wish the user to engage in a social conversation (e.g., by entering the same room, looking in the user's direction, waving to the user, etc.). Showing the status information on the second display generation component optionally includes displaying a representation of a portion of the user (e.g., a portion of the user that is obstructed by the first display generation component when the user is in a position to view the content displayed via the first display generation component) and displaying a graphical element that provides a visual indication of the content currently being shown via the first display generation component. Additionally, in some embodiments, a representation of a portion of the user is updated in conjunction with changes in the immersion level of the computer-generated experience displayed via the first display generation component. This method and system for updating the state information using a second display generation component to display state information related to a user viewing content shown via the first display generation component and related to the content viewed by the user, including updating the appearance of a representation of a portion of the user according to changes in the immersion level associated with the presentation of the content, allows others in the user's surrounding environment to gain useful insight into the user's current state while the user is engaging with the computer-generated experience without fully revealing the computer-generated experience to the surrounding environment.In some embodiments, updates to a representation of a portion of the user occluded by the first display generating component (e.g., the user's eyes or face) and updates to graphic elements indicating the state of the content displayed by the first display generating component are shown on different display layers and updated independently of each other. Displaying the representation of a portion of the user and the graphic elements indicating the state of the content on different display layers provides a more realistic view of the user's state behind the head-mounted display device that houses both the first and second display generating components. In some embodiments, the state information shown via the second display generating component (e.g., including the graphic elements indicating the user's representation and the state of the content) optionally provides a visual indication of many different modes of use of the computing system that correspond to different needs of the user and others in the same physical environment as the user. This allows the user to remain socially connected to people in the surrounding environment when engaging in a computer-generated experience. Dynamically updated status information on the second display generation component indicating the state of the user's eyes and the state of the content shown to the user improves the user's engagement with the computer-generated experience when the user is in a public or semi-public environment, for example, by encouraging appropriate social interaction when such interaction is desired by the user, reducing unnecessary avoidance of social interaction by others in the surrounding environment due to a lack of visual cues about the user's permission to engage socially, and informing others of an appropriate time to interrupt the user's engagement with the computer-generated experience, reducing unwelcome interruptions to the user's engagement experience due to a lack of visual cues about the user's desire to remain undisturbed.

[0042] In some embodiments, a computing system includes a first display generating component and a second display generating component (e.g., separate displays, displays enclosed in the same housing but facing different directions (e.g., facing opposite directions back to back, facing at different angles so that they cannot be viewed by the same user at the same time, etc.)) that face in two different directions. The first display generating component displays a computer-generated environment that provides a computer-generated experience to the user when the user is positioned to view the content presented via the first display generating component (e.g., facing in front of a physical environment illuminated by a projector, facing in front of a display that emits light that forms an image on the user's retina, etc.). The user may be positioned to view the content presented on the second display generating component before the user (e.g., by moving the display generating component, the user themselves, or both) positions the first display generating component in a position and orientation relative to the user to view the content displayed thereon. In an exemplary scenario, the first display generating component is an internal display of the HMD that faces the user's eyes when the HMD is placed on the user's head, and the second display generating component is an external display of the HMD that the user can see when the HMD is on a table or in the user's hand extended away from the user's face and not placed on the user's head or held close to the user's eyes. As disclosed herein, a computing system utilizes the second display generating component to display indications of the availability of different computer-generated experiences based on contextual information (e.g., location, time, user identification information, user permission level, etc.) and triggers the display of a selected computer-generated experience in response to detecting that the first display generating component has been moved to a predetermined position and orientation relative to the user (e.g., the first display generating component faces the user's eyes as a result of the movement) that allows the user to view the content shown via the first display generating component.The computer-generated experience to be displayed is optionally selected based on the state of the second display-generating component at a time corresponding to the first display-generating component being moved to a predetermined location and orientation relative to the user. Indicating the availability of a computer-generated experience on the second display-generating component based on contextual information and automatically triggering the display of a selected computer-generated experience on the first display-generating component based on changes in the state of the second display-generating component (and its contextual information) and the orientation of the first display-generating component relative to the user reduces the time and number of inputs required to achieve a desired result (e.g., obtain information related to available experiences relevant to the current context and launch the desired computer-generated experience) and reduces user error and time spent browsing through and launching available computer-generated experiences using a conventional user interface.

[0043] In some embodiments, a user can position and orient the first display generation component relative to the user to view content displayed thereon in different ways, for example, in an impromptu or temporary manner (e.g., held in front of the user's eyes at a distance or held in the hand in close proximity to the user's eyes), or in a more formal and established manner (e.g., strapped to the user's head or face or otherwise worn without being supported by the user's hand). The computing system selectively displays different computer-generated experiences (e.g., different versions of the computer-generated experience, different computer-generated experiences corresponding to different characteristics of the user or context properties, previews of an experience versus the actual experience, etc.) depending on how the first display generation component is positioned and oriented relative to the user to enable the user to view content displayed on the first display generation component. By selectively displaying different computer-generated experiences (e.g., automatically starting the display of computer-generated experiences via the first display generating component without any additional user input within a user interface provided by the first display generating component) in response to a trigger event corresponding to the first display generating component being placed in a predetermined configuration relative to the user, and according to how the first display generating component is held in its position and orientation (e.g., with or without support from the user's hand, with or without support from another mechanism other than the user's hand, etc.), the time and number of inputs required to achieve a desired result (e.g., starting a desired computer-generated experience) is reduced, and user error and time wasted compared to browsing through and starting available computer-generated experiences using a conventional user interface are reduced.

[0044] In some embodiments, displaying information on the second display generating component (e.g., status information related to the user's eyes, the status of content being displayed via the first display generating component, the display mode of the computing system, an indication of available computer-generated experiences, etc.) helps reduce the number of times a user needs to put on and take off an HMD including both the first and second display generating components and / or activate or deactivate computer-generated experiences, for example, to accommodate others in the surrounding physical environment and / or to find a desired computer-generated experience. This helps save the user time, reduce power usage, and reduce user errors when using the display generating components, improving the user experience. In some embodiments, a preset manner of physical manipulation of a real-world physical object is detected and used as a trigger for launching a computer-generated experience associated with the physical object. In some embodiments, prior to launching a computer-generated experience associated with the physical object, visual indication(s) of available computer-generated experiences, and optionally a visual guide (e.g., preview and animation) on how to launch the computer-generated experience, are displayed in a location within the mixed reality environment corresponding to the location of a representation of the physical object within the mixed reality environment. In addition to displaying a visual indication regarding the availability of a computer-generated experience and / or a visual guide regarding the physical manipulation required to trigger the computer-generated experience, using pre-configured physical manipulations of a physical object to trigger the display of a computer-generated experience associated with the physical object allows a user to achieve a desired result (e.g., enter a desired computer-generated experience) more intuitively, more quickly, and with less input. This user interaction heuristic also helps reduce user errors when users interact with physical objects, thereby making the human-machine interface more efficient and thereby conserving power on battery-operated computing systems.

[0045] In some embodiments, the display generation component is housed in a housing that includes sensors (or has external sensors otherwise associated) for detecting touch or hover input near or on various portions of the housing. Touch and / or hover input of different types (e.g., based on movement pattern (e.g., tap, swipe, etc.), duration (e.g., long, short, etc.), intensity (e.g., light, deep, etc.), etc.) and different locations on or near the exterior of the housing are used to trigger different actions associated with the display generation component or the computer-generated environment displayed by the display generation component. Interaction heuristics are used to determine whether an action should be performed depending on whether one or both hands are detected on the housing at the time the input is detected. Using the number of hands detected on the housing as an indicator of whether the user intends to provide input or is simply adjusting the position of the display generation component with the user's hands helps reduce inadvertent or unintended operation of the display generation component, thereby making the human-machine interface more efficient and thereby conserving power on battery-powered computing systems.

[0046] 1-6 illustrate an exemplary computing system for providing a CGR experience to a user. FIGS. 7A-7E illustrate a computing system displaying content to a user via a first display generation component while displaying dynamically updated state information associated with the user and / or the content via a second display generation component, according to some embodiments. FIGS. 7F-7J illustrate a computing system displaying an indication of different computer-generated experiences via a second display generation component based on contextual information, and triggering the display of different computer-generated experiences corresponding to the contextual information via the first display generation component in response to detecting a change in spatial relationship to the user, and optionally according to the wear state of the first display generation component relative to the user, according to some embodiments. FIGS. 7K-7M illustrate displaying an indication of the availability of computer-generated experiences associated with physical objects in an augmented reality environment, and triggering the display of a computer-generated experience corresponding to the physical object in response to detecting a preset physical interaction with the physical object, according to some embodiments. 7N-7Q illustrate selecting to perform or not perform an action associated with an input detected on a housing of a display generation component according to some embodiments, depending on whether one or both hands are detected on the housing at the time the input is detected on the housing. FIG. 8 is a flowchart of a method for displaying a computer-generated environment and state information, according to some embodiments. FIG. 9 is a flowchart of a method for displaying a computer-generated environment and state information, according to some embodiments. FIG. 10 is a flowchart of a method for providing a computer-generated experience based on contextual information, according to some embodiments. FIG. 11 is a flowchart of a method for providing a computer-generated experience based on a wearable state of a display generation component, according to some embodiments. FIG. 12 is a flowchart of a method for triggering the display of a computer-generated experience based on a physical interaction with a physical object, according to some embodiments.13 is a flowchart of a method for performing an action in response to an input detected on a housing of a display generating component, according to some embodiments. The user interfaces of FIGS. 7A-7Q are used to illustrate the processes of FIGS. 8-13, respectively.

[0047] In some embodiments, the CGR experience is provided to a user via an operating environment 100 that includes a computing system 101, as shown in FIG. The computing system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), one or more display generation components 120 (e.g., one or more head-mounted devices (HMDs), HMDs with internal and external displays, one or more displays, one or more projectors, one or more touchscreens, etc., enclosed in the same housing and facing different directions or enclosed in separate housings), one or more input devices 125 (e.g., an eye-tracking device 130, a hand-tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a tactile output generator 170, and other output devices 180), one or more sensors 190 (e.g., an image sensor, a light sensor, a depth sensor, a tactile sensor, an orientation sensor, a proximity sensor, a temperature sensor, a position sensor, a motion sensor, a speed sensor, etc.), and optionally one or more peripheral devices 195 (e.g., a consumer electronics device, a wearable device, etc.). In some embodiments, one or more of the input device(s) 125, output device(s) 155, sensors 190, and peripheral devices 195 are integrated with the display generation component 120 (e.g., in a head-mounted device (e.g., on the housing of the HMD or on a display facing outward from the HMD) or in a handheld device).

[0048] When describing a CGR experience, various terms are used to individually refer to several related, but distinct, environments that a user can sense and / or interact with (e.g., using inputs detected by the computing system 101 that generates the CGR experience, causing the computing system that generates the CGR experience to generate audio, visual, and / or haptic feedback corresponding to various inputs provided to the computing system 101). The following is a subset of these terms:

[0049] Physical Environment: The physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic systems. A physical environment, such as a physical park, includes physical objects such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment through their senses of sight, touch, hearing, taste, smell, etc.

[0050] Computer-Generated Reality: In contrast, a computer-generated reality (CGR) environment refers to a wholly or partially mimicked environment that people sense and / or interact with via electronic systems. In a CGR, a subset of a person's body movements or representations thereof are tracked, and in response, one or more properties of one or more virtual objects simulated within the CGR environment are adjusted to behave according to at least one law of physics. For example, a CGR system may detect a person's head rotation and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. In some circumstances (e.g., for accessibility reasons), adjustments to property(ies) of virtual object(s) in a CGR environment may be made in response to representations of body movements (e.g., voice commands). A person may sense and / or interact with a CGR object using any one of these senses, including sight, hearing, touch, taste, and smell. For example, a person may sense and / or interact with audio objects that create a 3D or spatially expansive audio environment that provides the perception of a point sound source in 3D space. In another example, audio objects may enable audio transparency that selectively incorporates ambient sounds from the physical environment, with or without computer-generated audio. In some CGR environments, a person may sense and / or interact with only audio objects.

[0051] Examples of CGR include virtual reality and mixed reality.

[0052] Virtual Reality: A virtual reality (VR) environment refers to an emulated environment designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes multiple virtual objects that a person can sense and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence in the computer-generated environment and / or through a simulation of a subset of the person's physical movements in the computer-generated environment.

[0053] Mixed Reality: A mixed reality (MR) environment refers to a mimetic environment designed to incorporate sensory input from or representations of a physical environment in addition to including computer-generated sensory input (e.g., virtual objects), as opposed to a VR environment designed to be based entirely on computer-generated sensory input. On the virtuality continuum, a mixed reality environment is anywhere between, but not including, a complete physical environment at one end and a virtual reality environment at the other. In some MR environments, computer-generated sensory input may respond to changes in sensory input from the physical environment. Some electronic systems for presenting MR environments may also track the position and / or orientation relative to the physical environment to allow virtual objects to interact with real objects (i.e., physical items or representations thereof from the physical environment). For example, the system may account for movement so that a virtual tree appears stationary relative to the physical ground.

[0054] Examples of mixed reality include augmented reality and augmented virtuality.

[0055] Augmented reality: An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on the transparent or translucent display, whereby a person using the system perceives the virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system composites the images or videos with virtual objects and presents the composite on the opaque display. A person uses the system to indirectly view the physical environment through the images or videos of the physical environment and perceive the virtual objects superimposed on the physical environment. As used herein, videos of a physical environment shown on an opaque display are referred to as "pass-through video," meaning that the system captures images of the physical environment using one or more image sensors and uses those images in presenting the AR environment on the opaque display. Alternatively, the system may include a projection system that projects virtual objects, e.g., as holograms, into the physical environment or onto a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment. An augmented reality environment also refers to a simulated environment in which a representation of the physical environment is transformed by computer-generated sensory information. For example, when providing pass-through video, the system may distort one or more sensor images to impose a selected perspective (e.g., viewpoint) different from the perspective captured by the image sensor. As another example, the representation of the physical environment may be distorted by graphically altering (e.g., enlarging) a portion thereof, thereby rendering the altered portion a non-photorealistic, altered version of the originally captured image. As a further example, the representation of the physical environment may be distorted by graphically removing or obscuring a portion thereof.

[0056] Augmented Virtuality: An augmented virtuality (AV) environment refers to a mimicking environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. The sensory inputs may be representations of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, while people with faces are realistically recreated from images taken of physical people. As another example, virtual objects may adopt the shape or color of physical items imaged by one or more imaging sensors. As a further example, virtual objects may adopt shadows that match the position of the sun in the physical environment.

[0057] Hardware: There are many different types of electronic systems that enable a person to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed over a person's eyes (e.g., similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have one or more speaker(s) and an integrated opaque display. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. A head-mounted system may have a transparent or translucent display rather than an opaque display. A transparent or translucent display may have a medium through which light representing an image is directed toward a person's eye. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser-scanned light source, or any combination of these technologies. The medium may be a light guide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, the transparent or translucent display may be configured to be selectively opaque. A projection-based system may employ retinal projection technology that projects graphical images onto a person's retina. The projection system may also be configured to project virtual objects into the physical environment, for example, as holograms or as physical surfaces. In some embodiments, the controller 110 is configured to manage and coordinate the user's CGR experience.In some embodiments, controller 110 includes a suitable combination of software, firmware, and / or hardware. Controller 110 is described in more detail below with reference to FIG. 2. In some embodiments, controller 110 is a computing device that is local or remote to scene 105 (e.g., physical setting / environment). For example, controller 110 is a local server located within scene 105. In another example, controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside scene 105. In some embodiments, controller 110 is communicatively coupled to display generation component 120 (e.g., one or more HMDs, displays, projectors, touchscreens, etc., enclosed in the same or different housings) via one or more wired or wireless communication channels 144 (e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is contained within or shares the same physical housing or support structure as one or more of the display generation component(s) 120 (e.g., an HMD or a portable electronic device including a display and one or more processors), one or more of the input devices 125, one or more of the output devices 155, one or more of the sensors 190, and / or the peripheral devices 195.

[0058] In some embodiments, at least one of display generation component(s) 120 is configured to provide a CGR experience (e.g., at least a visual component of the CGR experience) to a user. In some embodiments, display generation component(s) 120 includes a suitable combination of software, firmware, and / or hardware. One example of display generation component 120 is described in more detail below with reference to FIG. 3. In some embodiments, functionality of controller 110 is provided by and / or combined with display generation component(s) 120.

[0059] According to some embodiments, at least one of the display generation components 120 provides a CGR experience to the user while the user is virtually and / or physically present within the scene 105 .

[0060] In some embodiments, the display generation component(s) are worn on a part of the user's body (e.g., the user's head, the user's hand, etc.). Thus, at least one of the display generation component(s) 120 includes one or more CGR displays provided for displaying CGR content. For example, at least one of the display generation component(s) 120 surrounds the user's field of view. In some embodiments, at least one of the display generation component(s) 120 is a handheld device (e.g., a smartphone or tablet) configured to present CGR content, where the user holds the device with a display pointed toward the user's field of view and a camera pointed toward the scene 105. In some embodiments, the handheld device is optionally located in a housing worn on the user's head. In some embodiments, the handheld device is optionally located on a support (e.g., a tripod) in front of the user. In some embodiments, at least one of the display generation component(s) 120 is a CGR chamber, enclosure, or room configured to present CGR content without the user wearing or holding the display generation component(s) 120. Many user interfaces described with reference to one type of hardware for displaying CGR content (e.g., a handheld device or a device on a tripod) may be implemented on another type of hardware for displaying CGR content (e.g., an HMD or other wearable computing device). For example, a user interface showing interactions with CGR content that are triggered based on interactions occurring in the space in front of a handheld or tripod-mounted device may be implemented similarly to an HMD, where the interactions occur in the space in front of the HMD and the CGR content responses are displayed via the HMD.Similarly, a user interface showing interactions with CGR content triggered based on movement of a handheld or tripod-mounted device relative to the physical environment (e.g., scene 105 or a part of the user's body (e.g., the user's eye(s), head, or hand)) may be implemented similarly to an HMD triggered by movement of the HMD relative to the physical environment (e.g., scene 105 or a part of the user's body (e.g., the user's eye(s), head, or hand)).

[0061] While relevant features of operating environment 100 are shown in FIG. 1, those skilled in the art will understand from this disclosure that various other features have not been shown for the sake of brevity so as not to obscure more pertinent aspects of the exemplary embodiments disclosed herein.

[0062] 2 is a block diagram of an example controller 110, according to some embodiments. While certain features are shown, those skilled in the art will understand from this disclosure that various other features are not shown for the sake of brevity so as not to obscure more pertinent aspects of the embodiments disclosed herein. Thus, by way of non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., Universal Serial Bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), BLUETOOTH, ZIGBEE, or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these and various other components.

[0063] In some embodiments, one or more communication buses 204 include circuitry that interconnects and controls communications between system components. In some embodiments, one or more I / O devices 206 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0064] Memory 220 includes high-speed random-access memory, such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDRRAM), or other random-access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile storage devices. Memory 220 optionally includes one or more storage devices remotely located from one or more processing units 202. Memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, memory 220, or the non-transitory computer-readable storage medium of memory 220, stores the following programs, modules, and data structures, or a subset thereof, including optional operating system 230 and CGR experience module 240:

[0065] Operating system 230 includes instructions for handling various basic system services and for performing hardware-dependent tasks. In some embodiments, CGR experience module 240 is configured to manage and coordinate one or more CGR experiences for one or more users (e.g., a single CGR experience for one or more users, or multiple CGR experiences for respective groups of one or more users). To that end, in various embodiments, CGR experience module 240 includes a data acquisition unit 241, a tracking unit 242, an adjustment unit 246, and a data transmission unit 248.

[0066] 1 , and optionally one or more of input devices 125, output devices 155, sensors 190, and / or peripheral devices 195. To that end, in various embodiments, data acquisition unit 241 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0067] In some embodiments, tracking unit 242 is configured to map scene 105 and track the position / location of at least one or more of display generation component(s) 120 relative to scene 105 of FIG. 1 and, optionally, one or more of input device(s) 125, output device(s) 155, sensors 190, and / or peripheral devices 195. To that end, in various embodiments, tracking unit 242 includes instructions and / or logic therefor, as well as heuristics and metadata therefor. In some embodiments, tracking unit 242 includes hand tracking unit 244 and / or eye tracking unit 243. In some embodiments, hand tracking unit 244 is configured to track the position of one or more parts of a user's hand and / or the movement of one or more parts of a user's hand relative to scene 105 of FIG. 1, relative to at least one of display generation component(s) 120, and / or relative to a coordinate system defined relative to the user's hand. Hand tracking unit 244 is described in more detail below with reference to FIG. 4. In some embodiments, eye tracking unit 243 is configured to track the position and movement of a user's gaze (or, more broadly, the user's eyes, face, or head) relative to scene 105 (e.g., relative to the physical environment and / or the user (e.g., the user's hands)) or relative to CGR content displayed via at least one of display generation component(s) 120. Eye tracking unit 243 is described in more detail below with reference to FIG. 5.

[0068] In some embodiments, coordination unit 246 is configured to manage and coordinate the CGR experience presented to the user by at least one of display generation component(s) 120, and optionally by one or more of output device(s) 155 and / or peripheral device(s) 195. To that end, in various embodiments, coordination unit 246 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0069] In some embodiments, data transmission unit 248 is configured to transmit data (e.g., presentation data, location data, etc.) to at least one or more of display generation component(s) 120, and optionally to one or more of input device(s) 125, output device(s) 155, sensors 190, and / or peripheral devices 195. To that end, in various embodiments, data transmission unit 248 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0070] Although the data acquisition unit 241, the tracking unit 242 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the adjustment unit 246, and the data transmission unit 248 are shown as being present on a single device (e.g., the controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 241, the tracking unit 242 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the adjustment unit 246, and the data transmission unit 248 may be located within separate computing devices.

[0071] Furthermore, Figure 2 is intended more to illustrate the functionality of various features that may be present in particular embodiments, as opposed to a structural overview of the embodiments described herein. As will be recognized by those skilled in the art, items shown separately may be combined and some items may be separated. For example, some functional modules shown separately in Figure 2 may be embodied within a single module, and various functions of a single functional block may be performed by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of specific functions and how functions are allocated among them, will vary from embodiment to embodiment and, in some embodiments, will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0072] 3 is a block diagram of an example of at least one of the display generation component(s) 120, according to some embodiments. While certain features are shown, those skilled in the art will understand from this disclosure that, for the sake of brevity, various other features are not shown so as to not obscure more pertinent aspects of the embodiments disclosed herein. To that end, by way of non-limiting example, in some embodiments, a computing system (e.g., an HMD) including display generation component(s) 120 also includes, within the same housing, one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more CGR displays 312, one or more optional inward-facing and / or outward-facing image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these and various other components.

[0073] In some embodiments, the one or more communication buses 304 include circuitry that interconnects and controls communication between the system components. In some embodiments, the one or more I / O devices and sensors 306 include at least one of an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0074] In some embodiments, one or more CGR displays 312 are configured to provide a user with a CGR experience and, optionally, state information related to the CGR experience. In some embodiments, one or more CGR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conduction electron-emissive element display (SED), field-emission display (FED), quantum dot light-emitting diode (QD-LED), MEMS, and / or similar display types. In some embodiments, one or more CGR displays 312 correspond to a waveguide display, such as a diffractive, reflective, polarized, holographic, etc. For example, an HMD includes a single CGR display. In another example, an HMD includes a CGR display for each eye of the user. In some embodiments, one or more CGR displays 312 are capable of presenting MR or VR content. In some embodiments, one or more CGR displays 312 are capable of presenting MR or VR content. In some embodiments, the HMD includes one or more CGR displays that face the user's eyes and one or more CGR displays that face away from the user's eyes (e.g., towards the external environment) when the HMD is placed on the user's head. In some embodiments, the computing system is a CGR room or CGR enclosure that includes an interior CGR display that provides CGR content to a user within the CGR room or enclosure, and optionally one or more exterior peripheral displays that display status information related to the CGR content and the status of the user within.

[0075] In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as eye-tracking cameras). In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's hand(s) and optionally the user's arm(s) (and may be referred to as hand-tracking cameras). In some embodiments, the one or more image sensors 314 are configured to face forward to acquire image data corresponding to a scene as the user would view it if the display generation component(s) 120 were not present (and may be referred to as scene cameras). The one or more optional image sensors 314 may include one or more RGB cameras (e.g., with a complementary metal-oxide semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and / or the like.

[0076] Memory 320 includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 320 optionally includes one or more storage devices located remotely from one or more processing units 302. Memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, memory 320, or its non-transitory computer-readable storage medium, stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 330 and a CGR presentation module 340:

[0077] The operating system 330 includes instructions for handling various basic system services and for performing hardware-dependent tasks. In some embodiments, the CGR presentation module 340 is configured to present CGR content to a user via one or more CGR displays 312. To that end, in various embodiments, the CGR presentation module 340 includes a data acquisition unit 342, a CGR presentation unit 344, a CGR map generation unit 346, a data transmission unit 348, and optionally other operating units for displaying state information related to the user and the CGR content.

[0078] In some embodiments, the data acquisition unit 342 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the controller 110 of Figure 1. To that end, in various embodiments, the data acquisition unit 342 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0079] In some embodiments, the CGR presentation unit 344 is configured to present CGR content and associated state information via one or more CGR displays 312. To that end, in various embodiments, the CGR presentation unit 344 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0080] In some embodiments, the CGR map generation unit 346 is configured to generate a CGR map (e.g., a 3D map of a mixed reality scene or a map of a physical environment in which computer-generated objects can be placed to generate a computer-generated reality) based on the media content data. To that end, in various embodiments, the CGR map generation unit 346 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0081] In some embodiments, data transmission unit 348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least controller 110, and optionally to one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To that end, in various embodiments, data transmission unit 348 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0082] Although the data acquisition unit 342, the CGR presentation unit 344, the CGR map generation unit 346, and the data transmission unit 348 are shown as residing on a single device (e.g., the display generation component 120 of FIG. 1), it should be understood that in other embodiments, any combination of the data acquisition unit 342, the CGR presentation unit 344, the CGR map generation unit 346, and the data transmission unit 348 may be located within separate computing devices.

[0083] Furthermore, Figure 3 is intended more to illustrate the functionality of various features that may be present in particular embodiments, as opposed to a structural overview of the embodiments described herein. As will be recognized by those skilled in the art, items shown separately may be combined and some items may be separated. For example, some functional modules shown separately in Figure 3 may be implemented within a single module, and various functions of a single functional block may be performed by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of specific functions and how functions are allocated among them, will vary from embodiment to embodiment and, in some embodiments, will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0084] 4 is a schematic diagram of an example embodiment of a hand tracking device 140. In some embodiments, hand tracking device 140 ( FIG. 1 ) is controlled by hand tracking unit 244 ( FIG. 2 ) to track the position and / or movement of one or more portions of a user's hand relative to scene 105 of FIG. 1 (e.g., relative to a portion of the physical environment surrounding the user, relative to at least one of display generating component(s) 120, or relative to a coordinate system defined relative to a portion of the user (e.g., the user's face, eyes, or head), and / or relative to the user's hand). In some embodiments, hand tracking device 140 is part of at least one of display generating component(s) 120 (e.g., embedded in or attached to the same housing as the display generating component(s) (e.g., in a head-mounted device)). In some embodiments, hand tracking device 140 is separate from display generating component(s) 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0085] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras) that captures three-dimensional scene information including at least the hand 406 of a human user. The image sensor 404 captures hand images with sufficient resolution to allow for differentiation of the fingers and their respective positions. The image sensor 404 typically captures images of other parts of the user's body, or all of the body, and can have either zoom capabilities or a dedicated sensor with high magnification to capture hand images at a desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with or functions as an image sensor that captures the physical environment of the scene 105. In some embodiments, the image sensor 404 is positioned relative to the user or the user's environment such that the field of view of the image sensor, or a portion thereof, is used to define an interaction space in which hand movements captured by the image sensor are processed as inputs to the controller 110.

[0086] In some embodiments, image sensor 404 outputs a sequence of frames containing 3D map data (and possibly color image data) to controller 110, which extracts high-level information from the map data. This high-level information is typically provided via an application program interface (API) to an application running on the controller, which drives display generation component(s) 120 accordingly. For example, a user can interact with software running on controller 110 by moving hand 408 and changing hand posture.

[0087] In some embodiments, the image sensor 404 projects a spot pattern onto a scene containing the hand 406 and captures an image of the projected pattern. In some embodiments, the controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on the lateral shift of the pattern's spots. This approach is advantageous in that it does not require the user to hold or wear any type of beacon, sensor, or other marker. This provides depth coordinates of points in the scene relative to a predetermined reference plane at a specific distance from the image sensor 404. In this disclosure, the image sensor 404 is assumed to define an orthogonal set of x, y, and z axes such that the depth coordinate of a point in the scene corresponds to the z-component measured by the image sensor. Alternatively, the hand tracking device 440 can use other 3D mapping methods, such as stereoscopic imaging or time-of-flight measurements, based on single or multiple cameras or other types of sensors.

[0088] In some embodiments, the hand tracking device 140 captures and processes a time sequence of depth maps containing the user's hand while the user moves the hand (e.g., the entire hand or one or more fingers). Software running on the image sensor 404 and / or a processor in the controller 110 processes the 3D map data to extract patch descriptors of the hand in these depth maps. The software matches these descriptors with patch descriptors stored in the database 408, based on a previous training process, to estimate the pose of the hand in each frame. The pose typically includes the 3D positions of the user's wrist joints and fingertips.

[0089] The software can also analyze hand and / or finger trajectories across multiple frames in a sequence to identify gestures. The pose estimation functionality described herein may be interleaved with motion tracking functionality, whereby patch-based pose estimation is performed only once every two (or more) frames, while tracking is used to discover pose changes that occur across the remaining frames. The pose, motion, and gesture information is provided to an application program running on the controller 110 via the API described above. This program can, for example, move and modify an image presented on the display generation component 120 or perform other functions in response to the pose and / or gesture information.

[0090] In some embodiments, the software may be downloaded to the controller 110 in electronic form, for example, over a network, or alternatively may be provided on a tangible, non-transitory medium, such as an optical, magnetic, or electronic memory medium. In some embodiments, the database 408 is similarly stored in memory associated with the controller 110. Alternatively, or additionally, some or all of the described functions of the computer may be performed in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). While the controller 110 is shown in FIG. 4 as, by way of example, a separate unit from the image sensor 440, some or all of the controller's processing functions may be performed by a suitable microprocessor and software, or by dedicated circuitry within the housing of the hand tracking device 402, or otherwise associated with the image sensor 404. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component(s) 120 (e.g., in a television set, handheld device, or head-mounted device), or using any other suitable computerized device, such as a game console or media player. The sensing function of the image sensor 404 may likewise be integrated into a computer or other computerized device that is controlled by the sensor output.

[0091] FIG. 4 also includes a schematic diagram of a depth map 410 captured by the image sensor 404, according to some embodiments. The depth map includes a matrix of pixels having respective depth values, as described above. A pixel 412 corresponding to the hand 406 is segmented from the background and wrist in this map. The intensity of each pixel in the depth map 410 is inversely proportional to the depth value, i.e., the measured z-distance from the image sensor 404, with increasing depth resulting in darker shades. The controller 110 processes these depth values ​​to identify and segment components of the image (i.e., groups of adjacent pixels) that have characteristics of a human hand. These characteristics can include, for example, the overall size, shape, and frame-to-frame motion of the depth map sequence.

[0092] Figure 4 also schematically illustrates a hand skeleton 414 that the controller 110 ultimately extracts from the depth map 410 of the hand 406, according to some embodiments. In Figure 4, the skeleton 414 is superimposed on a hand background 416 that was segmented from the original depth map. In some embodiments, key feature points on the hand (e.g., knuckles, fingertips, center of the palm, end of the hand where it connects to the wrist, etc.), and optionally the wrist or arm connected to the hand, are identified and located on the hand skeleton 414. In some embodiments, the location and movement of these key feature points over multiple image frames are used by the controller 110 to determine hand gestures performed by the hand or the current state of the hand, according to some embodiments.

[0093] FIG. 5 illustrates an exemplary embodiment of eye tracking device 130 (FIG. 1). In some embodiments, eye tracking device 130 is controlled by eye tracking unit 244 (FIG. 2) to track the position and movement of a user's gaze relative to scene 105 or relative to CGR content displayed via at least one of display generation component(s) 120. In some embodiments, eye tracking device 130 is integrated with at least one of display generation component(s) 120. For example, in some embodiments, if display generation component(s) 120 are part of a head-mounted device, such as a headset, helmet, goggles, or glasses, or a handheld device disposed in a wearable frame, the head-mounted device includes both components for generating CGR content for viewing by the user and components for tracking the user's gaze relative to the CGR content. In some embodiments, eye tracking device 130 is separate from display generation component(s) 120. For example, if the display generation component(s) are provided by a handheld device or a CGR chamber, the eye tracking device 130 is optionally a separate device from the handheld device or the CGR chamber. In some embodiments, the eye tracking device 130 is a head-mounted device or part of a head-mounted device. In some embodiments, the head-mounted eye tracking device 130 is optionally also used in conjunction with at least one of the display generation component(s) that is / are mounted on the head or at least one of the display generation component(s) that is / are not mounted on the head. In some embodiments, the eye tracking device 130 is not a head-mounted device, but is optionally used in combination with a head-mounted display generation component. In some embodiments, the eye tracking device 130 is not a head-mounted device, but is optionally part of a non-head-mounted display generation component.

[0094] In some embodiments, at least one of the display generation component(s) 120 uses a display mechanism (e.g., display panels near the left and right eyes) that displays frames including left and right images in front of the user's eyes, thus providing the user with a 3D virtual view. For example, a head-mounted display generation component may include left and right optical lenses (referred to herein as eyepieces) positioned between the display and the user's eyes. In some embodiments, at least one of the display generation component(s) 120 may include or be coupled to one or more external video cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generation component may have a transparent or translucent display that allows the user to view the physical environment directly and display virtual objects on the transparent or translucent display. In some embodiments, at least one of the display generation component(s) 120 projects virtual objects into the physical environment. The virtual objects are projected, for example, onto a physical surface or as a hologram, allowing an individual using the system to observe virtual objects superimposed on the physical environment. In such cases, separate display panels and image frames for the left and right eyes may not be required.

[0095] As shown in FIG. 5 , in some embodiments, the eye tracking device 130 includes at least one eye tracking camera (e.g., an infrared (IR) or near-IR (NIR) camera) and an illumination source (e.g., an IR or NIR light source such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eyes. The eye tracking camera may be aimed at the user's eyes to receive reflected IR or NIR light from the light source directly from the eyes, or alternatively, it may be aimed at a “hot” mirror positioned between the user's eyes and a display panel that reflects the IR or NIR light from the eyes to the eye tracking camera while allowing visual light to pass through. The eye tracking device 130 optionally captures images of the user's eyes (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes the images to generate eye tracking information, and communicates the eye tracking information to the controller 110. In some embodiments, the user's eyes are tracked separately by their respective eye tracking cameras and illumination sources. In some embodiments, only one eye of the user is tracked by a corresponding eye-tracking camera and lighting source.

[0096] In some embodiments, the eye tracking device 130 is calibrated using a device-specific calibration process to determine the eye tracking device's parameters for the particular operating environment 100, such as the 3D geometric relationships and parameters of the LEDs, camera, hot mirror (if present), eyepiece, and display screen. The device-specific calibration process may be performed at a factory or another facility prior to delivery of the AR / VR equipment to the end user. The device-specific calibration process may be an automatic or manual calibration process. The user-specific calibration process may include estimation of a particular user's eye parameters, such as pupil position, central vision position, optical axis, visual axis, eye spacing, etc. According to some embodiments, once the device-specific and user-specific parameters for the eye tracking device 130 have been determined, images captured by the eye tracking camera can be processed using glint-assisted methods to determine the user's current visual axis and viewpoint relative to the display.

[0097] As shown in FIG. 5, eye tracking device 130 (e.g., 130A or 130B) includes an eyepiece(s) 520 and an eye tracking system including at least one eye tracking camera 540 (e.g., an infrared (IR) or near-IR (NIR) camera) positioned on the side of the user's face where eye tracking occurs and an illumination source 530 (e.g., an IR or NIR light source such as an array or ring of NIR light emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye(s) 592. The eye tracking camera 540 may be pointed at a mirror 550 located between the user's eye(s) 592 and the display 510 (e.g., a left or right display panel of a head-mounted display, an internal display of a head-mounted device, or a display of a handheld device, a projector, etc.) that reflects IR or NIR light from the eye(s) 592 while transmitting visible light (e.g., as shown in the top of FIG. 5), or may be pointed at the user's eye(s) 592 to receive reflected IR or NIR light from the user's eye(s) 592 (e.g., as shown in the bottom of FIG. 5).

[0098] In some embodiments, controller 110 renders AR or VR frames 562 (e.g., left and right frames for left and right display panels) and provides frames 562 to display 510. Controller 110 uses gaze tracking input 542 from eye tracking camera 540 for various purposes, such as in processing frames 562 for display. Controller 110 optionally estimates the user's viewpoint on display 510 based on gaze tracking input 542 obtained from eye tracking camera 540, using a glint-assisted method or other suitable method. The viewpoint estimated from gaze tracking input 542 is optionally used to determine the user's current looking direction.

[0099] Several possible use cases of the user's current gaze direction are described below, but are not intended to be limiting. As an exemplary use case, the controller 110 can render virtual content differently based on the determined user's gaze direction. For example, the controller 110 can generate virtual content with higher resolution in a central visual area determined from the user's current gaze direction than in a peripheral area. As another example, the controller can position or move virtual content within a view based at least in part on the user's current gaze direction. As another example, the controller can display particular virtual content within a view based at least in part on the user's current gaze direction. As another exemplary use case in an AR application, the controller 110 can orient an external camera to capture the physical environment of the CGR experience and focus in the determined direction. The external camera's autofocus mechanism can then focus on an object or surface within the environment the user is currently viewing on the display 510. As another exemplary use case, eyepiece 520 may be a focusable lens, and eye-tracking information is used by the controller to adjust the focus of eyepiece 520 so that the virtual object the user is currently looking at has the proper binocular coordination to match the convergence of the user's eyes 592. Controller 110 can utilize the eye-tracking information to orient and focus eyepiece 520 so that close objects the user is looking at appear at the correct distance.

[0100] In some embodiments, the eye tracking device is part of a head-mounted device that includes a display (e.g., display 510), two eyepieces (e.g., eyepiece(s) 520), an eye tracking camera (e.g., eye tracking camera(s) 540), and a light source (e.g., light source 530 (e.g., IR or NIR LED)) attached to the wearable housing. The light source emits light (e.g., IR or NIR light) toward the user's eye(s) 592. In some embodiments, the light sources may be arranged in a ring or circle around each lens, as shown in FIG. 5. In some embodiments, eight light sources 530 (e.g., LEDs) are arranged around each lens 520, as an example. However, more or fewer light sources 530 may be used, and other arrangements and positions of the light sources 530 may be used.

[0101] In some embodiments, the display 510 emits light in the visible light range and not in the IR or NIR range, and therefore does not introduce noise into the eye-tracking system. Note that the positions and angles of the eye-tracking camera(s) 540 are given by way of example and are not intended to be limiting. In some embodiments, a single eye-tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user's face. In some embodiments, a camera 540 with a wide field of view (FOV) and a camera 540 with a narrow FOV may be used on each side of the user's face. In some embodiments, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

[0102] Embodiments of an eye tracking system such as that shown in FIG. 5 may be used, for example, in computer-generated reality (e.g., including virtual reality and / or mixed reality) applications to provide a user with a computer-generated reality (e.g., including virtual reality, augmented reality, and / or augmented virtuality) experience.

[0103] Figure 6 shows a glint-assisted eye tracking pipeline, according to some embodiments. In some embodiments, the eye tracking pipeline is implemented by a glint-assisted eye tracking system (e.g., eye tracking device 130 as shown in Figures 1 and 5). The glint-assisted eye tracking system can maintain a tracking state. Initially, the tracking state is off or "no." When in the tracking state, the glint-assisted eye tracking system tracks the pupil contour and glint in the current frame using prior information from the previous frame when analyzing the current frame. When not in the tracking state, the glint-assisted eye tracking system attempts to detect the pupil and glint in the current frame, and if successful, initializes the tracking state to "yes" and continues to the next frame in the tracking state.

[0104] As shown in FIG. 6, an eye-tracking camera can capture left and right images of a user's left and right eyes. The captured images are then input into an eye-tracking pipeline for processing beginning at 610. As indicated by the arrow returning to element 600, the eye-tracking system can continue to capture images of the user's eyes at a rate of, for example, 60-120 frames per second. In some embodiments, each set of captured images may be input into the pipeline for processing. However, in some embodiments, or under some conditions, not all captured frames are processed by the pipeline.

[0105] At 610, if the tracking status is yes for the currently captured image, the method proceeds to element 640. If the tracking status is no at 610, the image is analyzed to detect the user's pupil and glint in the image, as shown at 620. If the pupil and glint are successfully detected at 630, the method proceeds to element 640. If not, the method returns to element 610 to process the next image of the user's eyes.

[0106] At 640, proceeding from element 410, the current frame is analyzed to track pupils and glints based in part on prior information from the previous frame. At 640, proceeding from element 630, a tracking state is initialized based on the detected pupils and glints in the current frame. The results of the processing at element 640 are checked to ensure that the tracking or detection results are reliable. For example, the results can be checked to determine whether a sufficient number of glints are successfully tracked or detected in the current frame to perform pupil and gaze estimation. At 650, if the results are not reliable, the tracking state is set to no and the method returns to element 610 to process the next image of the user's eyes. At 650, if the results are reliable, the method proceeds to element 670. At 670, the tracking state is set to yes (if not already yes) and pupil and glint information is passed to element 680 to estimate the user's gaze point.

[0107] 6 is intended to serve as an example of eye-tracking technology that may be used in particular implementations. As will be recognized by those skilled in the art, other eye-tracking technologies, now existing or developed in the future, may be used in place of or in combination with the glint-assisted viewing eye-tracking technology described herein in computing system 101 to provide a user with a CGR experience according to various embodiments.

[0108] In this disclosure, various input methods are described with respect to interaction with a computing system. Where one example is provided using one input device or input method and another example is provided using a different input device or input method, it should be understood that each example is compatible with, and may optionally be utilized with, the input device or input method described with respect to the other example. Similarly, various output methods are described with respect to interaction with a computing system. Where one example is provided using one output device or output method and another example is provided using a different output device or output method, it should be understood that each example is compatible with, and may optionally be utilized with, the output device or output method described with respect to the other example. Similarly, various methods are described with respect to interaction with a virtual environment or a mixed reality environment via a computing system. Where an example is provided using interaction with a virtual environment and another example is provided using a mixed reality environment, it should be understood that each example is compatible with, and may optionally be utilized with, the method described with respect to the other example. Thus, this disclosure discloses embodiments that are combinations of features of multiple examples, without exhaustively listing all features of the embodiments in the description of each embodiment. User Interface and Related Processes

[0109] Attention is now directed to embodiments of user interfaces ("UI") and associated processes that may be executed on a computing system, such as a portable multifunction device or head-mounted device, equipped with one or more display generating components, one or more input devices, and (optionally) one or more cameras.

[0110] 7A-7E illustrate a computing system (such as computing system 101 of FIG. 1 or computing system 140 of FIG. 4) including at least a first display generation component (e.g., display 7100) and a second display generation component (e.g., display 7102), according to some embodiments, where the computing system displays computer-generated content to a user via the first display generation component (e.g., display 7100) while displaying dynamically updated state information associated with the user and / or the content via the second display generation component (e.g., display 7102). FIGS. 7A-7E are used to illustrate processes described below, including the processes of FIGS. 8-13.

[0111] As shown in the left portion of Figure 7A, a first display generating component (e.g., display 7100) resides at location A7000-a and is displaying CGR content (e.g., a three-dimensional movie, a virtual reality game, a video, a three-dimensional environment including user interface objects, etc.) A first user 7202 also resides at location A7000-a.

[0112] As shown in the right portion of Figure 7A, a second display generation component (e.g., display 7102) is present at location B 7000-b and is displaying status information corresponding to the first user 7202 and / or the CGR content presented via the first display generation component (e.g., display 7100). In the exemplary scenario shown in Figure 7A, a second user 7204 is also present at location B 7000-b.

[0113] As shown in FIG. 7A , the spatial relationship between a first display generating component (e.g., display 7100) and a first user 7202 is such that the first user 7202 is positioned to view the CGR content presented via the first display generating component. For example, the first user 7202 faces the display surface of the first display generating component. In some embodiments, the first display generating component is an internal display of an HMD, and the spatial relationship represented by the simultaneous presence of the display 7100 and the first user 7202 at the same position A7000-a corresponds to the first user wearing or holding the HMD with the internal display of the HMD facing the user's eyes. In some embodiments, the first user is positioned to view the CGR content presented via the first display generating component when the first user faces a portion of the physical environment illuminated by the projection system of the first display generating component. For example, the virtual content is projected onto a portion of the physical environment, and the virtual content and the portion of the physical environment are seen by the user through a camera view of the portion of the physical environment or a transparent portion of the first display generating component when the user faces the display surface of the first display generating component. In some embodiments, the first display generating component emits light that forms an image on the user's retina when the user faces the display surface of the first display generating component. For example, the virtual content is displayed by an LCD or LED display, overlaying or replacing a portion of the view of the physical environment displayed by the LCD or LED display, such that a user facing the display surface of the LCD or LED display can see the virtual content along with a view of the portion of the physical environment. In some embodiments, the first display generating component displays a camera view of the physical environment in front of the first user or includes a transparent or translucent portion through which a portion of the physical environment in front of the first user is visible to the first user.In some embodiments, the portion of the physical environment made visible to the first user via the first display generating component is a portion of the physical environment corresponding to a display surface of the second display generating component 7102 (e.g., location B7000-b including the display surface of the second display generating component and optionally the second user 7204). In some embodiments, the display surface of the second display generating component is a surface of the second display generating component that faces away from the first user when the first user is in a position to view the content shown by the first display generating component (e.g., when the first user is facing the display surface of the first display generating component), and is a surface that emits light to form an image viewable by others facing a predetermined portion of the first user (e.g., the second user 7204 or others facing the first user's face or eyes in the physical environment).

[0114] 7A , the spatial relationship between the second display generating component (e.g., display 7102) and the second user 7204 is such that the second user 7204 is positioned to view the state information presented by the second display generating component. For example, the second user 7204 is in front of and / or facing the display surface of the second display generating component. In some embodiments, the second display generating component is an external display of an HMD that also includes an internal display (e.g., represented by display 7100) that is presenting CGR content to the first user 7202. In such embodiments, the spatial relationship represented by the simultaneous presence of the display 7102 and the second user 7204 at the same location B7000-b corresponds to the second user being in a portion of the physical environment facing the external display of the HMD (e.g., the physical environment is also hosting the first display generating component and the first user 7202). In some embodiments, the first display generating component displays a camera view of the physical environment in front of the first user, or includes a transparent or semi-transparent pass-through portion through which a portion of the physical environment in front of the first user is visible to the first user, and the portion of the physical environment included in the camera view or pass-through portion is also a portion of the physical environment in front of the display surface of the second display generating component. In some embodiments, the second display generating component is positioned back-to-back with the first display generating component such that the portion of the physical environment in front of the display surface of the second display generating component 7102 (e.g., position B7000-b including the display surface of the second display generating component and optionally the second user 7204) would also be in front of the first user and within the first user's field of view if the first and second display generating components were not occluding the first user's face.

[0115] As mentioned above and repeated herein, while FIG. 7A (and FIGS. 7B-7J) shows the first display generating component (e.g., display 7100) and the second display generating component (e.g., display 7102) as being located in two separate and distinct parts of the physical environment, it should be understood that the first display generating component and the second display generating component are, optionally, two display generating components contained in the same housing (e.g., the housing of a single HMD) or mounted on the same support structure (e.g., mounted back to back to each other or on two sides of a single wall or surface) and facing in different (e.g., substantially opposite) directions. Thus, location A7000-a represents a first portion of a physical environment in which content (e.g., CGR content) presented via a first display generation component can be seen by a first user (e.g., first user 7202) facing the display surface of the first display generation component, and content (e.g., status information) presented via a second display generation component cannot be seen by the first user (e.g., first user 7202), and location B7000-b represents a second portion of the same physical environment in which content (e.g., CGR content) presented via the first display generation component cannot be seen by another user (e.g., second user 7204) facing the display surface of the second display generation component, and content (e.g., status information) presented via the second display generation component can be seen by that other user (e.g., second user 7204).In the disclosure presented herein, the first display generation component and the second display generation component are controlled by the same computing system (e.g., an HMD, a portable electronic device housed separately from the display generation component, a portable electronic device with two displays facing in different directions, a remote server computer, etc.), and a user of the computing system, unless otherwise specified, generally refers to a person who has control of at least the first display generation component and positions it or themselves in a position that allows them to view the CGR content shown via the first display generation component.

[0116] 7A , a computing system controlling the first and second display generating components is in communication with a first image sensor (e.g., camera 7104) and a second image sensor (e.g., camera 7106). The first image sensor faces a display surface of the first display generating component (e.g., display 7100) and is configured to capture an image of a portion of the physical environment (e.g., location A7000-a) including at least a portion of the first user (e.g., the face and / or eyes of first user 7202) but not including second user 7204 (or any other user, if the first display generating component is an internal display of an HMD worn by the first user). The second image sensor is configured to capture an image of a portion of the physical environment (e.g., location B7000-b) that does not include a portion of the first user (e.g., the face or eyes of the first user 7202), but includes at least a portion of the second user (e.g., a portion of the second user 7204 that is within the field of view of the first user 7202 provided by the first display generating component). As mentioned above, in some embodiments, the portion of the physical environment captured by the second image sensor 7106 includes a portion of the physical environment that would be within the field of view of the first user if the first user's eyes were not physically obstructed by the presence of the second display generating component (and optionally, the presence of the first display generating component). Similarly, in some embodiments, the portion of the physical environment captured by the first image sensor 7104 includes a portion of the user (e.g., the user's face or eyes) that is physically obstructed by the presence of the first display generating component (and optionally, the presence of the second display generating component). In some embodiments, the computing system also communicates with the first image sensor, the second image sensor, and / or other image sensors to receive images of the hands and wrists of the first user and / or the second user to identify gesture inputs provided by the first user and / or the second user. In some embodiments, the first image sensor 7104 is also used to capture gaze inputs provided by the first user.In some embodiments, the first image sensor and the second image sensor optionally function as image sensors for capturing gesture input of the first user and / or the second user.

[0117] In some embodiments, the computing system optionally controls one or more audio output devices that respectively provide audio output (e.g., sounds of CGR content) to a first user located at location A 7000-a, and optionally audio output (e.g., status indication sounds or alerts, sounds of CGR content, etc.) to a second user located at location B 7000-b. In some embodiments, the computing system optionally partially or completely shields location A and the first user from sounds propagating from location B (e.g., via one or more active or passive noise suppression or cancellation components), and optionally partially or completely shields location B and the second user from sounds propagating from location A. In some embodiments, the amount of active sound insulation or sound pass-through is determined by the computing system based on the current immersion level associated with the CGR content shown via the first display generation component (e.g., no sound insulation when in pass-through mode, or partial sound insulation when in mixed reality mode, full sound insulation when in virtual reality mode, etc.), and optionally based on whether another user is present at location B (e.g., no sound insulation when no one is present at location B, sound insulation when people are present at location B or the noise level is above a threshold level, etc.).

[0118] In some embodiments, as shown in FIG. 7A , a computing system displays CGR content 7002 (e.g., shown as 7002-a in FIG. 7A ) via a first display generating component (e.g., display 7100 or an internal display of an HMD) while a first user 7202 is in a position to view the CGR content (e.g., the first user 7202 is collocated with and at least partially facing the display surface of the first display generating component at position A, the first user is wearing an HMD on their head, the first user is holding the HMD with the internal display in front of the user's eyes, etc.). At the moment shown in FIG. 7A , the computing system is displaying movie X (e.g., a three-dimensional movie, a two-dimensional movie, an interactive computer-generated experience, etc.). The movie is displayed in a mixed reality mode in which the content of the movie is visible via the first display generating component simultaneously with a representation of the physical environment (e.g., a representation of position B (e.g., a portion of the physical environment in front of the first user that is obstructed by the presence of the first display generating component)). In some embodiments, this mixed reality mode corresponds to an intermediate level of immersion associated with the CGR content presented via the first display generating component. In some embodiments, the intermediate level of immersion also corresponds to partial occlusion or partial pass-through of sound propagating from the physical environment (e.g., position B (e.g., a portion of the physical environment surrounding the first user)). In this example, the representation of the physical environment includes a representation 7010 (e.g., shown as 7010-a in FIG. 7A ) of a second user 7204 located at position B 7000-b in front of the second display generating component 7102 (e.g., in front of the back of the first display generating component 7100). In some embodiments, the representation of the physical environment includes a camera view of the portion of the physical environment that would be in the first user's field of view if the user's eyes were not occluded by the presence of the first display generating component and the second display generating component (e.g., if the first user were not wearing or holding an HMD in front of the user's eyes).In mixed reality mode, CGR content 7002 (e.g., movie X, a three-dimensional augmented reality environment, a user interface, a virtual object, etc.) is displayed to overlay or replace at least a portion, if not all, of the representation of the physical environment. In some embodiments, the first display generating component includes a transparent portion through which a portion of the physical environment is visible to the first user. In some embodiments, in mixed reality mode, CGR content 7002 (e.g., movie X, a three-dimensional augmented reality environment, a user interface, a virtual object, etc.) is projected onto a physical surface or empty space within the physical environment, is visible through the transparent portion along with the physical environment, is viewable through the transparent portion of the first display generating component, or is viewable through a camera view of the physical environment provided by the first display generating component. In some embodiments, CGR content 7002 is displayed to overlay a portion of the display, blocking the view of at least a portion, if not all, of the physical environment visible through the transparent or semi-transparent portion of the first display generating component. In some embodiments, the first display generation component 7100 does not provide a view of the physical environment, but rather provides a complete virtual environment (e.g., without a camera view or transparent pass-through portions) augmented with a real-time visual representation(s) (e.g., a stylized representation or segmented camera image) of the physical environment as currently captured by one or more sensors (e.g., a camera, motion sensors, other orientation sensors, etc.). In mixed reality modes (e.g., augmented reality based on a camera view or transparent display, or augmented virtuality based on a virtualized representation of the physical environment), the first user is not fully immersed in the computer-generated environment, but still receives sensory information (e.g., visual and audio) that directly corresponds to the physical environment surrounding the first user and the first display generation component.

[0119] As shown in Figure 7A, while a computing system displays CGR content 7002-a (e.g., movie X, a three-dimensional augmented reality environment, a user interface, virtual objects, etc.) in a mixed reality mode via a first display generating component 7100, the computing system simultaneously displays state information related to the first user and the CGR content via a second display generating component (e.g., display 7102 or an external display of an HMD). As shown in the right portion of Figure 7A, the second display generating component (e.g., display 7102 or an external display of an HMD) displays one or more graphical elements representing the state of the CGR content 7002 being displayed via the first display generating component (e.g., display 7100 or an internal display of an HMD), as well as a representation 7006 (e.g., shown as 7006-a in Figure 7A) of at least a portion of a first user 7202 in front of the display surface of the first display generating component (e.g., display 7100 or an internal display of an HMD). In this example, the one or more graphical elements representing the status of the CGR content being displayed via the first display generation component optionally include an identifier for the CGR content (e.g., the title of Movie X), a progress bar 7004 indicating the current progress of the CGR content (e.g., shown as 7004-a in FIG. 7A ), and a visual representation 7008 of the CGR content (e.g., shown as 7008-a in FIG. 7A ). In some embodiments, the visual representation 7008 of the CGR content obscures portions of the CGR content (e.g., via blurring, distortion, etc.) and merely conveys the sense of change and color or gradation of the CGR content. As shown in the right portion of FIG. 7A , the representation 7006 of a portion of the first user 7202 optionally includes a camera view of the first user's face or a graphical representation generated based on a camera view of the first user's face. In some embodiments, the representation of the portion of the first user 7202 optionally includes a graphical representation generated based on an eye camera view of the first user or an eye camera view of the first user in front of the display surface of the first display generation component.In some embodiments, the representation of the portion of the first user 7202 is displayed on a display layer different from the display layer(s) of one or more graphical elements representing the state of the CGR content being displayed via the first display generating component. In some embodiments, the simultaneous display of the representation of the portion of the first user by the second display generating component with the representation of the state of the CGR content provides an indication that the CGR content is being displayed via the first display generating component in a mixed reality mode and that the first user is being provided with a view of the physical environment along with the CGR content. By displaying a visual representation of the first user's face and / or eyes, as well as a representation of the state of the CGR content being viewed by the first user, on the second display generating component while the first user's face and / or eyes are obscured by the presence of the first display generating component (and optionally the presence of the second display generating component) (e.g., by the presence of an HMD including an internal display and an external display), other users in the physical environment around the first user are provided with more information to initiate or refrain from interacting with the first user or to act in an appropriate manner in the first user's presence.

[0120] 7B continues from FIG. 7A, showing that the CGR content then progresses further on the first display generating component (e.g., display 7100 or an internal display of an HMD) and the first user's appearance changes. For example, the change in the first user's appearance is due to movement of at least a portion of the first user 7202 (e.g., the first user's eyes or face) relative to the first display generating component in position A7000-a (e.g., while the first user is still wearing the HMD and / or facing the internal display of the HMD) (e.g., movement includes lateral movement of the first user's eyes, blinking of the user's eyes, opening and closing of the user's eyes, moving the user's eyes up and down, etc., and / or movement includes movement of the user's face or head relative to the display surface of the first display generating component (e.g., moving away from or towards the first display generating component). At this point, the CGR content 7002 is still displayed in mixed reality mode, and the representation 7010 (e.g., shown as 7010-b) of the physical environment (e.g., position B including the second user 7204) remains displayed simultaneously with the CGR content 7002 via the first display generation component. In some embodiments, any changes in the appearance of the physical environment (e.g., movement of the second user 7204 relative to the first display generation component, the second display generation component, and / or the first user) are similarly reflected by the representation 7010 of the physical environment shown by the first display generation component. In some embodiments, in accordance with movement of a portion of the first user (e.g., movement of the first user's eyes or face) relative to the first display generation component, the computing system updates the representation 7006 (e.g., shown as 7006-b in FIG. 7B ) displayed via the second display generation component 7102.For example, when the first user 7202 or a portion thereof (e.g., the user's face or eyes) moves in a direction toward a first edge of the first display generating component 7100 (e.g., the left edge of the display surface of the first display generating component, the top edge of the display surface of the first display generating component, etc.), the representation 7006 of the portion of the first user shown on the display surface of the second display generating component also moves toward a corresponding second edge of the second display generating component (e.g., the right edge of the display surface of the second display generating component (e.g., corresponding to the left edge of the display surface of the first display generating component), the top edge of the display surface of the second display generating component (e.g., corresponding to the top edge of the display surface of the first display generating component), etc.). In addition to updating the representation 7006 of the portion of the first user, the computing system also updates the representation of the state of the CGR content on the second display generating component. For example, the progress bar 7004 (e.g., shown as 7004-b in FIG. 7B ) is updated to indicate that the playback of the CGR content has progressed a first amount from the time shown in FIG. 7A . In some embodiments, the representation 7008 (e.g., shown as 7008-b in FIG. 7B ) of the CGR content as shown on the second display generating component (e.g., display 7102, an external display of an HMD, etc.) is also updated according to the current appearance of the CGR content 7002 as shown on the first display generating component (e.g., display 7100, an internal display of an HMD, etc.). In some embodiments, showing real-time updates of the appearance of a portion of the first user (e.g., showing changes and movements of the first user's face and eyes behind the first display generating component) and showing real-time or periodic updates of the state of the CGR content shown by the first display generating component allows others in the physical environment around the first user (e.g., at location B) to obtain information regarding the first user's attention state and whether it is appropriate to engage or interrupt the first user at this time.In some embodiments, while changes in the first user's appearance and CGR content are reflected by updates to the state information shown by the second display generating component, any changes in the appearance of the physical environment (e.g., movement of the second user 7204 relative to the first display generating component, the second display generating component, and / or the first user) are similarly reflected by the representation 7010 of the physical environment shown by the first display generating component.

[0121] Figure 7C continues from Figure 7A, showing that the CGR content then progresses further on the first display generation component 7100 and the second user 7204 moves relative to the second display generation component 7204, viewing the second display generation component from a different angle compared to the scenario shown in Figure 7A. At this point, the CGR content 7002 is still displayed in mixed reality mode, and a representation 7010 (e.g., shown as 7010-c) of the physical environment (e.g., position B including the second user 7204) remains simultaneously displayed via the first display generation component 7100. As the second user 7204 moves relative to the second display generating component (and relative to the first display generating component if the first and second display generating components have a fixed spatial relationship to each other (e.g., fixed back-to-back within the same housing of the HMD), the computing system updates the representation 7010 (e.g., shown as 7010-c in FIG. 7C ) displayed via the first display generating component (e.g., display 7100 or the internal display of the HMD). For example, when the second user 7204 or a portion thereof moves in a direction toward a third edge (e.g., a right edge, a top edge, etc.) of the display surface of the second display generating component 7102, the representation 7010 of the portion of the second user shown on the display surface of the first display generating component 7100 also moves toward a corresponding fourth edge of the first display generating component (e.g., a left edge of the display surface of the first display generating component (e.g., corresponding to a right edge of the display surface of the second display generating component), a top edge of the display surface of the first display generating component (e.g., corresponding to a top edge of the display surface of the second display generating component)). In accordance with changes in the CGR content shown on the first display generating component (e.g., display 7100, an internal display of an HMD, etc.), the computing system also updates the representation of the state of the CGR content shown on the second display generating component (e.g., display 7102, an external display of an HMD, etc.).For example, progress bar 7004 (e.g., shown as 7004-c in FIG. 7C ) is updated to indicate that playback of the CGR content has progressed a second amount from the time shown in FIG. 7A and a third amount from the time shown in FIG. 7B . In some embodiments, representation 7008 (e.g., shown as 7008-c in FIG. 7C ) of CGR content 7002 as shown on a second display generation component (e.g., display 7102, an external display of an HMD, etc.) is also updated according to the current appearance of the CGR content 7002 shown on the first display generation component (e.g., display 7100, an internal display of an HMD, etc.). 7A and 7B illustrate that for the same relative spatial location between the first display generating component and the portion of the first user 7202 represented in the state information shown by the second display generating component, the representation 7006 of the portion of the first user 7202 is displayed at a different depth than the representation 7008 of the CGR content, and optionally at a different depth than other state information (e.g., the progress bar 7004). The difference in display depth from the display surface of the second display generating component 7102 or from the position of the second user 7204 creates a visual parallax effect. For example, as the second user 7204 moves relative to the second display generating component (e.g., the display 7102, the external display of an HMD, etc.), the representation 7006 of the portion of the first user 7202 and the representation 7008 of the CGR content appear to move by different amounts (and appear to move relative to each other) on the display surface of the second display generating component. In some embodiments, the representation 7008 of the CGR content appears as a diffuse layer between the representation 7006 of the portion of the first user and the representation of other state information (e.g., the title of the CGR content, the progress bar 7004, etc.). In some embodiments, the representation 7006 of the portion of the first user appears on a display layer that is farthest from the display surface of the second display generating component compared to the display layers of the representations of the CGR content and other state information shown by the second display generating component.

[0122] In some embodiments, the first display generating component and the second display generating component are arranged back-to-back (e.g., with their respective display surfaces facing different (e.g., substantially opposite) directions) in an HMD worn on the first user's head or positioned in front of the user's face. In some embodiments, the second display generating component shows a visual representation of the first user's eyes, generated based on actual images of the first user's eyes using one or more image processing filters. For example, the visual representation of the first user's eyes is optionally generated by reducing opacity, increasing transparency, reducing color saturation levels, reducing luminance levels, reducing pixel resolution, reducing color resolution, etc. In some embodiments, the amount of modification applied to various display characteristics of the individual camera images of the first user's eyes is optionally specified relative to values ​​of various display characteristics of the representation 7008 of the CGR content simultaneously shown by the second display generating component 7102. For example, if the representation of the CGR content is relatively dark (e.g., having a first range of luminance values), the representation of the eyes is also made darker, more translucent, and / or less color saturated (e.g., having a second range of luminance values, a second range of transparency values, and a second range of color saturation values ​​selected based on the first range of luminance values), and if the representation of the CGR content is lighter (e.g., having a second range of luminance values ​​greater than the first range of luminance values), the representation of the eyes is made lighter, less translucent, and / or more color saturated (e.g., a third range of luminance values, a third range of transparency values, and a third range of color saturation values ​​selected based on the second range of luminance values). In some embodiments, other display characteristics (e.g., color saturation, pixel resolution, color resolution, depth, etc.) are used as a basis for selecting value ranges for display characteristics of representations of a user's portion (e.g., the user's face or eyes).In some embodiments, the representation of the first user's eyes is generated by applying one or more pre-configured image filters, such as a bokeh filter, a color filter, a luminance filter, etc., that modify the original appearance of the first user's eyes when the representation is displayed by the second display generation component.

[0123] In some embodiments, the representation of the CGR content (e.g., representation 7008) shown by the second display generation component is generated by applying a diffusion filter to the CGR content displayed by the first display generation component (e.g., all visible content, only media content, or optionally, visible content excluding a pass-through view of the physical environment). For example, the color and tonality of the scene are preserved by the representation of the CGR content 7008, but the outlines of objects in the CGR content are blurred and not clearly defined in the representation of the CGR content 7008. In some embodiments, the representation of the CGR content is semi-transparent, through which the representation of the first user's portion 7006 is visible. In some embodiments, graphical user interface elements representing metadata associated with the CGR content (e.g., progress bar 7004, title of the CGR content, etc.) are displayed by the second display generation component (e.g., on the same display layer as the representation of the CGR content 7008 or a different display layer and / or on the same display layer as the representation of the first user's portion 7006 or a different display layer). In some embodiments, graphical user interface elements representing metadata associated with the CGR content are displayed with higher pixel resolution, higher color resolution, higher color saturation, higher opacity, higher luminance, and / or better defined contours compared to the representation 7008 of the CGR content and / or the representation 7006 of the portion of the first user.

[0124] In some embodiments, a portion of the first user (e.g., the first user's face or eyes) moves relative to the first display generating component (e.g., display 7100, an internal display of an HMD), while the CGR content 7002 presented by the first display generating component does not change. In such a case, the representation 7006 of the portion of the user is updated on the second display generating component 7102, optionally without updating the representation 7006 of the CGR content and the progress bar 7004. In some embodiments, the CGR content is not displayed or paused, and the first user is viewing a pass-through view of the physical environment through the first display generating component without a simultaneous display of the CGR content, and the second display generating component optionally updates the representation of the portion of the first user according to changes in the appearance of the portion of the first user (e.g., due to movement or other changes in the portion of the user) without displaying any representation of the CGR content or showing a representation of the CGR content in a stationary or paused state.

[0125] In some embodiments, the CGR content changes on the first display generating component, while the first user's portion does not change its appearance (e.g., moves or does not change for other reasons). Thus, the representation 7006 of the first user's portion remains unchanged, and the second display generating component only updates the representation 7008 of the CGR content and other indicators of the CGR content's status (e.g., progress bar 7004) in accordance with the changes in the CGR content indicated by the first display generating component.

[0126] In some embodiments, if changes in both the CGR content and the appearance of the first user's portion are detected during the same period (e.g., simultaneously and / or within a preset time window of each other), the second display generation component updates both the visual representation of the user's portion and one or more graphical elements indicating the status of the CGR content (e.g., representation of the CGR content 7008 and progress bar 7004) according to the detected changes.

[0127] 7A-7C illustrate updating, via the second display generating component 7102, the appearance of a visual representation (e.g., representation 7006) of the first user 7202 (e.g., the first user's eyes or face) in accordance with changes in the first user's appearance (e.g., by the first user's eye movement or changing facial expression, changes in lighting, etc.), and updating, via the second display generating component 7102, one or more graphical elements (e.g., progress bar 7004, representation 7008 of CGR content 7002, etc.) that provide a visual indication of content within the CGR environment shown via the first display generating component 7100 in accordance with changes in the CGR environment. In the exemplary scenario illustrated in FIGS. 7A-7C, the immersion level associated with the CGR content and the attention state of the first user 7202 do not change, corresponding to an intermediate immersion level associated with the presentation of the CGR content. In some embodiments, the immersion level and corresponding attention state of the first user associated with the presentation of the CGR content are optionally changed over a period of time, e.g., increasing to a greater level of immersion and a more engaged user attention state, or decreasing to a lesser level of immersion and a less engaged user attention state. In some embodiments, the state information shown by the second display generation component is updated based on changes in the immersion level at which the CGR content is presented by the first display generation component. In some embodiments, the updates to the state information include updates to a representation of a portion of the first user (e.g., updating the visibility of a representation of a portion of the first user, updating the appearance of a representation of a portion of the first user, etc.).

[0128] In some embodiments, the computing system is configured to display the CGR content 7002 at at least a first immersion level, a second immersion level, and a third immersion level. In some embodiments, the computing system transitions the CGR content displayed via the first display generation component between different immersion levels in response to a series of one or more events (e.g., natural termination or progression of an application or experience; starting, stopping, and / or pausing an experience in response to user input; changing the immersion level of an experience in response to user input; changing the state of the computing device; changing the external environment; etc.). In some embodiments, the first, second, and third immersion levels correspond to increasing amounts of virtual content present in the CGR environment and / or decreasing amounts of representations of the surrounding physical environment (e.g., representations of position B or a portion of the physical environment in front of the display surface of the second display generation component 7102) present in the CGR environment. In some embodiments, the first, second, and third immersion levels correspond to different modes of content display having increasing image fidelity (e.g., increasing pixel resolution, increasing color resolution, increasing color saturation, increasing luminance, increasing opacity, increasing image detail, etc.) and / or spatial extent (e.g., angular extent, spatial depth, etc.) of the computer-generated content and / or decreasing image fidelity and / or spatial extent of the representation of the surrounding physical environment (e.g., a representation of a portion of the physical environment in front of the display surface of position B or the second display generating component). In some embodiments, the first immersion level is a pass-through mode in which the physical environment (e.g., a portion of the physical environment in front of the display surface of position B or the second display generating component) is fully visible to the first user through the first display generating component (e.g., as a camera view of the physical environment or through a transparent portion of the first display generating component).In some embodiments, CGR content presented in pass-through mode includes a pass-through view of the physical environment with a minimal amount of virtual elements simultaneously visible as the view of the physical environment, or with only virtual elements peripheral to the user's view of the physical environment (e.g., indicators and controls displayed in a peripheral area of ​​the display). Figure 7E shows an example of a first immersion level associated with CGR content 7002, according to some embodiments. For example, the view of the physical environment (e.g., a portion of the physical environment in front of the display surface of the second display generation component (e.g., a portion of the physical environment also in front of the first user)) occupies a central and majority area of ​​the field of view provided by the first display generation component, and only a few controls (e.g., movie title, progress bar, playback controls (e.g., play button), etc.) are displayed in the peripheral area of ​​the field of view provided by the first display generation component. In some embodiments, the second immersion level is a mixed reality mode in which a pass-through view of the physical environment is augmented with virtual elements generated by the computing system, with the virtual elements occupying a central and / or majority area of ​​the user's field of view (e.g., virtual content is integrated with the physical environment within the view of the computer-generated environment). Examples of the second immersion level associated with CGR content 7002, according to some embodiments, are shown in Figures 7A-7C. In some embodiments, the third immersion level is a virtual reality mode in which the user's view of the physical environment is completely replaced or blocked by a view of the virtual content provided by the first display generation component. Figure 7D shows an example of the third immersion level associated with CGR content 7002, according to some embodiments.

[0129] As shown in FIG. 7D following FIG. 7C, the computing system has a switch from displaying the CGR content 7002 in mixed reality mode to displaying the CGR content 7002 (e.g., movie X shown as 7002-d) in virtual reality mode without a representation of the physical environment (e.g., location B (e.g., a portion of the physical environment in front of the display surface of the second display generation component 7102) including the second user 7204). In some embodiments, the switch performed by the computing system is in response to a request from the first user (e.g., a gesture input (e.g., lifting the first user's hand away from the HMD) that meets pre-set criteria for changing the immersion level of the CGR content). In conjunction with the switch from displaying the CGR content 7002 in mixed reality mode via the first display generation component 7100 to displaying the CGR content 7002 in virtual reality mode, the computing system changes the state information displayed via the second display generation component 7102. As shown in FIG. 7D, one or more graphical elements representing the CGR content 7002 (e.g., a title, a progress bar 7004 (e.g., shown as 7004-d in FIG. 7D), and a representation 7008 (e.g., shown as 7008-d in FIG. 7D)) are still displayed and continue to be updated according to changes in the CGR content 7002 (e.g., shown as 7002-d in FIG. 7D) displayed by the first display generation component 7100, but the representation 7006 of a portion of the first user (e.g., the first user's eyes or face) is no longer displayed by the second display generation component 7100. In some embodiments, instead of completely ceasing to display the representation of the portion of the first user, the computing system reduces the visibility of the representation of the portion of the first user relative to the visibility of other state information on the second display generation component (e.g., representations of CGR content, representations of metadata related to the CGR content or the user, etc.) (e.g., reduce luminance, reduce color resolution, reduce opacity, reduce pixel resolution, etc.).In some embodiments, the representation 7006 of the first user's portion is optionally displayed with reduced visibility relative to its previous appearance (e.g., not visible at all or with reduced luminance, increased transparency, reduced opacity, reduced color saturation, increased blur level, etc.) to indicate an increased level of immersion associated with the CGR content shown by the first display generation component. Other state information (e.g., the representation 7008 of the CGR content and the progress bar 7004) is continuously or periodically updated according to changes in the CGR content 7002 shown by the first display generation component, while such other state information remains displayed by the second display generation component 7102 (e.g., without a reduction in visibility relative to its previous appearance, unless the reduction is due to a change in the appearance of the CGR content).

[0130] In some embodiments, the switch from mixed reality mode to virtual reality mode is triggered by the second user 7204 moving out of the estimated field of view the first user would have had if the first user's eyes were not obstructed by the presence of the first and / or second display generation components. In some embodiments, the switch from mixed reality mode to virtual reality mode is triggered by the second user 7204 moving out of the physical environment surrounding the first user (e.g., out of a room occupied by the first user). In some embodiments, the computing system stops displaying a representation of the physical environment (e.g., a representation of position B (e.g., a representation of a portion of the physical environment in front of the first user)) if there are no other users present in the physical environment. In some embodiments, the movement of a second user 7204 into the estimated field of view that the first user would have if the first user's view at position B was not obstructed by the presence of the first and / or second display generation components and / or into the physical environment surrounding the first user (e.g., into a room occupied by the first user), a pre-defined gesture performed by the second user (e.g., the second user waving to the first user), the second user moving within a threshold distance range of the first user, etc. are optionally used as conditions for triggering a switch from virtual reality mode to mixed reality mode. In some embodiments, in conjunction with switching the display mode from virtual reality mode to mixed reality mode, the computing system restores a level of visibility of the representation 7006 of the first user's portion among the elements of the state information indicated by the second display generation component 7102 (e.g., restoring the display of the representation of the first user's portion if the representation was not visible, or increasing the luminance, color saturation, pixel resolution, opacity, and / or color resolution of the representation of the user's portion, etc.).Accordingly, in mixed reality mode, a first display generating component (e.g., display 7100, an internal display of an HMD, etc.) displays a representation (e.g., representation 7010) of a portion of the physical environment in front of the display surface of the second display generating component (and accordingly in front of the first user, if the first display generating component and the second display generating component are enclosed back-to-back within the same housing of an HMD worn by the first user) along with computer-generated virtual content (e.g., movie X).

[0131] As shown in FIG. 7E (e.g., after FIG. 7C or 7D), the computing system switches from displaying CGR content in mixed reality mode (e.g., as shown in FIGS. 7A-7C) or virtual reality mode (e.g., as shown in FIG. 7D) to displaying the CGR content (e.g., CGR content 7002) in a full pass-through or reality mode (e.g., movie X has completed and is no longer shown) with only a representation (e.g., shown as 7010-e in FIG. 7E) of the physical environment (e.g., position B (e.g., a portion of the physical environment in front of the display surface of the second display generation component 7102)) including the second user 7204 on the peripheral region of the display (e.g., the top and bottom edges of the display), and optionally only a minimal amount of virtual content (e.g., only indicators (e.g., the title of movie X, a progress bar, playback controls, etc.) and no playback content). In some embodiments, the display mode switch is performed by the computing system in response to the termination or pausing of playback of the CGR content and / or a request from the first user (e.g., a gesture input that meets pre-set criteria for changing the immersion level of the CGR content (e.g., placing a hand over the first user's eyebrows, pulling down with pinched fingers on the surface of the HMD, etc.)). In some embodiments, in conjunction with switching from displaying the CGR content in mixed reality or virtual reality mode via the first display generation component 7100 to displaying the CGR content in full pass-through or reality mode, the computing system changes the state information displayed via the second display generation component 7102.As shown in FIG. 7E , one or more graphical elements indicating the appearance and status of the CGR content (e.g., title, progress bar 7004 (e.g., shown as 7004-c in FIG. 7C and 7004-d in FIG. 7D , etc.), and representation 7008 (e.g., shown as 7008-c in FIG. 7C and 7008-d in FIG. 7D , etc.)) are no longer displayed by the second display generation component, and first user portion representation 7006 (e.g., shown as 7006-e in FIG. 7E ) is fully displayed by the second display generation component with increased visibility (e.g., becoming visible if previously invisible, or displayed with increased luminance, decreased transparency, increased opacity, increased color saturation, increased pixel resolution, increased color resolution, reduced blur level, etc.). In some embodiments, first user portion representation 7006 is optionally displayed with increased visibility relative to its previous appearance to indicate a decreased level of immersion associated with the CGR content shown by the first display generation component. In some embodiments, the representation 7006 of the portion of the first user is continuously updated according to changes in the appearance of the first user 7202 while the representation 7006 of the portion of the first user is being displayed by the second display generation component.

[0132] In some embodiments, switching from mixed reality or virtual reality mode to full pass-through or reality mode is triggered by the second user 7204 moving into an estimated field of view that the first user would have had if the first user's eyes were not obstructed by the presence of the first and / or second display generation components. In some embodiments, switching from mixed reality or virtual reality mode to full pass-through or reality mode is triggered by the second user 7204 moving into a personal space within a threshold distance of the first user 7202 (e.g., within an arm's length of the first user, within three feet of the first user, etc.). In some embodiments, upon entering full pass-through or reality mode (e.g., when a preset condition is met, e.g., when a preset person (e.g., spouse, teacher, teammate, child, etc.) enters the first user's 7202's estimated field of view), the computing system stops displaying CGR content via the first display generation component and displays only a representation of the physical environment (e.g., position B, the physical environment in front of the first user, etc.). In some embodiments, movement of a second user 7204 leaving the estimated field of view of the first user 7202 and / or leaving a personal space within a threshold distance from the first user 7202, and / or other conditions, is used to trigger an automatic switch from full pass-through mode or reality mode back to mixed reality mode or virtual reality mode (e.g., a preset mode or a previous mode). In some embodiments, in conjunction with switching the display mode from full pass-through mode to virtual reality mode or mixed reality mode, the computing system restores the level of visibility (e.g., reduces its visibility without ceasing to display entirely or making it completely invisible) of a representation 7006 of the first user's portion in the element of state information indicated by the second display generation component 7102, and restores the level of visibility (e.g., increases its visibility) of a representation 7008 of the CGR content.

[0133] 7C-7E illustrate transitions from a second immersion level (e.g., mixed reality mode) down to a first immersion level (e.g., pass-through mode) and up to a third immersion level (e.g., virtual reality mode), according to some embodiments, and corresponding changes in the information shown by the first display generation component 7100 and the information shown by the second display generation component 7102. In some embodiments, direct transitions between any two of the three levels of immersion are possible in response to different events that meet the respective criteria for such direct transitions. In response, the information displayed by the first display generation component and the information displayed by the second display generation component are updated (e.g., changing the visibility of different components of the information (e.g., CGR content, representation of the physical environment, representation of the CGR content, representation of a portion of the first user, representation of static metadata associated with the CGR content, etc.)) to reflect the current immersion level at which the CGR content is being displayed by the first display generation component 7100.

[0134] In some embodiments, as shown in FIG. 7E, a representation 7006 of a portion of a first user (e.g., a representation of the first user's face or eyes) is displayed without simultaneous display of a representation of the CGR content (e.g., without overlaying a diffused version of the CGR content, title, progress bar, etc.) or with increased visibility relative to the representation of the CGR content (e.g., the visibility of representation 7006 is increased relative to its previous level, the visibility of the representation of the CGR content is decreased relative to its previous level, and / or some of the graphical elements for representing the CGR content are no longer displayed, etc.) as a result of the computing system switching from displaying the CGR content using a mixed reality mode to displaying the CGR content using a full pass-through mode or reality mode.

[0135] In some embodiments, a representation 7006 of a portion of the first user (e.g., a representation of the first user's face or eyes) is displayed together with the representation 7008 of the CGR content (e.g., with an overlay of a diffused version of the CGR content) (e.g., with equal visibility relative to the representation 7008 of the CGR content (e.g., increased or decreased visibility of representation 7006 and / or visibility of representation 7008 relative to their respective previous levels)) as a result of the computing system switching from displaying the CGR content using a virtual reality mode or a pass-through mode to displaying the CGR content using a mixed reality mode.

[0136] In some embodiments, when a computing system switches from displaying CGR content using a mixed reality mode to displaying CGR content using a virtual reality mode, a representation 7006 of a portion of the first user (e.g., a representation of the first user's face or eyes) is not displayed with the representation 7008 of the CGR content (e.g., is not displayed with a diffused version of the CGR content) or is displayed with reduced visibility relative to the representation 7008 of the CGR content.

[0137] In some embodiments, the computing system may display CGR content using other special display modes, such as a private mode, a do-not-disturb (DND) mode (DND mode), a parental control mode, etc. When one or more of these special display modes are turned on, the manner in which state information is displayed and / or updated on the second display generating component is adjusted from the manner in which state information is displayed and / or updated on the second display generating component when such special modes are not turned on (e.g., as described above with reference to Figures 7A-7E).

[0138] For example, the private mode is optionally activated by the computing system or the first user to hide state information associated with the CGR content currently being displayed by the first display generation component and / or state information associated with the first user's attention state. In some embodiments, while the private mode is turned on (e.g., in response to a request from the first user), the representation 7006 of the portion of the first user and / or the representation 7008 of the CGR content are no longer updated, cease to be displayed, and / or are replaced with other placeholder content on the second display generation component, such that they no longer reflect changes detected in the appearance of the portion of the first user and / or changes detected in the CGR content being displayed by the first display generation component. In some embodiments, the private mode is activated in response to a user request (e.g., a preset gesture input by the first user on the HMD, a preset voice command, etc.) detected by the computing system (e.g., when the computing system is displaying CGR content to the first user using a mixed reality mode or a virtual reality mode and / or before the CGR content is started, etc.). In some embodiments, the private mode is activated in response to the user accessing particular CGR content associated with a preset privacy level that exceeds a first threshold privacy level (e.g., a default privacy level, a privacy level associated with the first user, etc.). In some embodiments, while the privacy mode is turned on, the representation 7006 of the portion of the first user and / or the representation 7008 of the CGR content are no longer updated, cease to be displayed, and / or are replaced with other placeholder content, such that they no longer reflect changes in the immersion level at which the CGR content is displayed by the first display generation component.The private mode allows the first user to enjoy more privacy and share less information about the user's own attention state, immersion level, and the content the user is viewing using the first display generation component through the content displayed by the second display generation component.

[0139] In some embodiments, the DND mode is turned on in advance by the first user and / or automatically by the computing system based on preset conditions to indicate to the external environment that the first user does not want to be interrupted or disturbed by others in the external environment during the user's engagement with the CGR content. In some embodiments, the DND mode is optionally applicable to other intrusive events occurring within the computing system and / or the surrounding environment. For example, in some embodiments, in response to activation of the DND mode, the computing system optionally activates noise cancellation to block out sounds from the surrounding environment, stops / pauses presentation of notifications and / or alerts on the first display generation component, reduces the intrusiveness of how notifications and / or alerts are presented in the CGR environment shown by the first display generation component (e.g., selects visual alerts over audio alerts, selects short alert sounds over speech outputs, reduces the visual prominence of notifications and alerts, etc.), automatically forwards calls to voicemail without notifying the first user, and / or displays a silent mode sign on the second display generation component, etc. In some embodiments, one or more methods used by the computing system to reduce the annoyance of the event to the first user involve changing the way representations of the physical environment (e.g., representation 7010, a representation of position B, a representation of a portion of the physical environment in front of the first user, etc.) are displayed on the first display generation component and / or changing the way state information is displayed by the second display generation component. In some embodiments, DND mode is optionally turned on while the computing system is displaying CGR content using a mixed reality mode or a virtual reality mode. In some embodiments, in response to DND mode being turned on, the computing system optionally displays a visual indicator via the second display generation component (e.g., a text label “DND” on the HMD's external display, a lit red border around the HMD's external display, etc.) to indicate that DND mode is active.In some embodiments, while DND mode is active on the computing system, the representation of the CGR content is optionally updated according to changes in the CGR content displayed by the first display generation component, but the representation of the portion of the first user is no longer updated, is replaced by placeholder content, or ceases to be displayed by the second display generation component (e.g., regardless of changes in the appearance of the portion of the first user (e.g., changes in the first user's eyes) and / or changes in the immersion level at which the CGR content is displayed by the first display generation component).

[0140] In some embodiments, parental mode is turned on to override the normal display of status information by the second display generating component (e.g., as described with reference to FIGS. 7A-7E). Parental mode is turned on so that a parent, teacher, or supervisor can view and monitor the CGR content presented to the first user and, optionally, input provided by the first user to modify and / or interact with the CGR content. In some embodiments, parental mode is optionally turned on by the second user (e.g., via preset gesture input, touch input on the second display generating component or housing of the HMD, voice command, etc.) while the CGR content is being presented by the first display generating component. In some embodiments, parental mode is optionally turned on before the particular CGR content is initiated on the first display generation component (e.g., via interaction with a user interface presented by the first display generation component, interaction with a user interface presented by the second display generation component, interaction with the computing system housing or other input device, etc.) and remains turned on while the particular CGR content is displayed by the first display generation component. In some embodiments, while parental mode is turned on, the computing system displays the same CGR content simultaneously via both the first display generation component and the second display generation component, regardless of changes in immersion level and / or whether private mode is turned on. In some embodiments, the computing system displays only the virtual content portion of the CGR content shown by the first display generation component on the second display generation component.In some embodiments, while parental mode is turned on, the computing system does not display the representation 7006 of the portion of the first user as part of the state information shown using the second display generation component (e.g., if parental mode is used simply to monitor content shown to the first user, and not to monitor the first user himself / herself). In some embodiments, while parental mode is turned on, the computing system displays the representation 7006 of the portion of the user and the CGR content with visibility equivalent to that of the CGR content (e.g., the visibility of the representation 7006 is increased compared to the previous level of visibility it had when parental mode was not turned on) (e.g., if parental mode is used to monitor content shown to the first user, as well as the attention state and appearance of the first user). In some embodiments, whether the representation 7006 of the portion of the user is displayed by the second display generation component during parental mode is determined according to how parental mode is activated (e.g., using a first type of input vs. using a second type of input, using a first control vs. using a second control, etc.). In some embodiments, whether a representation of the user's portion is displayed by the second display generating component during parental mode is determined according to whether private mode is turned on. For example, if private mode is turned on, the representation of the user's portion is not displayed with the CGR content by the second display generating component, and if private is not turned on, the representation of the user's portion is displayed with the CGR content by the second display generating component.In some embodiments, changes in the immersion level at which CGR content is displayed by the first display generation component while parental mode is turned on do not change the information shown by the second display generation component (e.g., the same CGR content is still shown in both the first display generation component and the second display generation component, optionally with the current visibility level of the representation of the first user's portion unchanged or without the display of the first user's representation).

[0141] In some embodiments, the visibility and information density of the status information shown by the second display generating component is dynamically adjusted by the computing system according to the distance of the second user, who is present in a position that allows the second user to view the content shown by the second display generating component (e.g., directly or partially in front of the display surface of the second display generating component). For example, when the second user moves closer to (e.g., moves within a threshold distance, moves within a threshold viewing angle, etc.) the display surface of the second display generating component (e.g., moves closer to the first user and the first display generating component when the first and second display generating components are positioned back-to-back in the same housing of an HMD worn by the first user), the computing system changes (e.g., increases) the amount of information detail (e.g., detail of graphical features, amount of text characters per unit display area, color resolution, pixel resolution, etc.) provided on the second display generating component to inform the second user of the first user's status and the status and metadata of the CGR content. Accordingly, when the second user moves farther away from the display surface of the second display generating component (e.g., moves beyond a threshold distance, moves outside a threshold viewing angle, etc.), the computing system changes the amount of information detail provided on the second display generating component in the opposite direction (e.g., decreases the amount of information detail).

[0142] In some embodiments, in response to detecting changes in the surrounding physical environment that meet pre-set criteria (e.g., people entering the room or coming within a threshold distance of the first user, other users waving or gesturing towards the first user, etc.), the computing system automatically transitions from displaying the computer-generated experience in a fully immersive mode (e.g., displaying a virtual reality environment or displaying CGR content at a third immersion level) to displaying the computer-generated experience in a less immersive mode (e.g., displaying indications of the physical environment within the virtual reality environment (e.g., displaying outlines of people and objects in the physical environment as visual distortions, shadows, etc.), displaying a pass-through portion (e.g., a camera view of the physical environment) in the view of the computer-generated environment, etc.). In some embodiments, in conjunction with automatically altering the immersion level of the computer-generated environment displayed via the first display generation component, the computing system also alters the state information displayed via the second display generation component, including increasing the visibility of a visual representation of a portion of the user of the computing system (e.g., increasing the visibility of the visual representation of the user includes switching from not displaying a visual representation of the portion of the user to displaying a visual representation of the portion of the user, or increasing the luminance, sharpness, opacity, and / or resolution of the visual representation of the portion of the user). In this way, a visual barrier separating the first user from others in the surrounding environment (e.g., the presence of the display generation component on the first user's face) is simultaneously reduced, facilitating more informed interaction between the first user and surrounding users.In some embodiments, if the computing system reduces the immersive level of the content shown on the first display generation component in response to an action of the second user (e.g., in response to the second user waving at the first user and / or in response to the second user moving too close to the first user, etc.), the computing system stops displaying a representation of the CGR content, or does not display a representation of the CGR content and displays only a representation of a portion of the first user (e.g., the first user's face or eyes) on the second display generation component (e.g., to inform the second user that the first user can see the second user through the first display generation component). In some embodiments, if the computing system increases the immersive level of the content shown on the first display generation component in response to an action of the second user (e.g., in response to the second user putting on an HMD and / or in response to the second user moving away from the first user, etc.), the computing system re-displays a representation of the CGR content on the second display generation component and stops displaying (or reduces the luminance, sharpness, opacity, color, pixel resolution, etc.) a representation of a portion of the first user (e.g., the first user's face or eyes).

[0143] Further details regarding the user interface and modes of operation of the computing system are provided with reference to Figures 7F-7Q and 8-13 and the accompanying description.

[0144] 7F-7I illustrate a computing system (e.g., computing system 101 of FIG. 1 or computing system 140 of FIG. 4) including at least a first display generating component (e.g., display 7100, an internal display of an HMD, etc.) and a second display generating component (e.g., display 7102, an external display of an HMD, etc.), where the first display generating component of the computing system is configured to display visual content (e.g., a user interface, a computer-generated experience, media content, etc.) to a user (e.g., first user 7202) when the computing system determines that the first display generating component is positioned in a predetermined spatial relationship (e.g., a predetermined orientation (e.g., with the display surface facing the first user's face or eyes) with respect to the user (e.g., first user 7202) or a portion of the user (e.g., the first user's face or eyes)). In some embodiments, before displaying visual content to a user via a first display generation component (e.g., display 7100, an internal display of an HMD, etc.), the computing system uses a second display generation component (e.g., display 7102, an external display of an HMD, etc.) to display one or more user interface elements (e.g., a user interface, a computer-generated experience (e.g., AR content, VR content, etc.), media content, etc.) that prompt the user regarding visual content available for display via the first display generation component when a condition for triggering the display of the content (e.g., a condition related to a spatial relationship between the first display generation component and the user) is met. In some embodiments, the one or more user interface elements include user interface objects that convey contextual information (e.g., current time, location, external conditions, user identification information, and / or the current state of the computing system, or other prompts, notifications, etc.) based on which available content is currently available for display via the first display generation component.Figures 7F-7G and 7H-7I show two parallel examples illustrating different computer-generated experiences being displayed by a first display generation component based on different states of the computing system as reflected by different user interface objects shown by a second display generation component before a pre-set spatial relationship between the first display generation component and the user is met.

[0145] As shown in the left portions of each of Figures 7F and 7H, a first display generating component (e.g., display 7100) resides in position A7000-a without a user facing the display surface of the first display generating component. As a result, the first display generating component is not currently displaying any CGR content. As shown in the right portions of each of Figures 7F and 7H, a second display generating component (e.g., display 7102) resides in position B7000-b and is displaying one or more user interface elements, including a first user interface element (e.g., circle 7012) corresponding to a first computer-generated experience 7024 available for display via the first display generating component given the current contextual state of the computing system shown in Figure 7F, and a second user interface element (e.g., square 7026) corresponding to a second computer-generated experience 7030 available for display via the first display generating component given the current contextual state of the computing system shown in Figure 7H. The context state of the computing system may be determined based on, for example, the current time, the current location of the first display generating component (which in some embodiments is also the location of the first user and the second display generating component), the user identification or permission level of the first user 7202 present in front of the display surface of the second display generating component, receipt or generation of notifications of individual applications by the computing system, the occurrence of pre-configured events on the computing system, and / or other context information.

[0146] 7F and 7H , a first user 7202 is shown as being at location B 7000-b. In some embodiments, the spatial relationship between a second display generating component (e.g., display 7102, an external display of an HMD, etc.) and the first user 7202 is such that the first user 7202 is positioned to view one or more user interface elements (e.g., user interface element 7012 and user interface element 7026, respectively) presented by the second display generating component. For example, the first user 7202 faces the display surface of the second display generating component when the one or more user interface elements are displayed. In some embodiments, the second display generating component is an external display of an HMD that also includes an internal display (e.g., the first display generating component represented by display 7100) configured to present CGR content corresponding to the user interface elements shown on the external display of the HMD. In such embodiments, the spatial relationship represented by the simultaneous presence of the display 7102 and the first user 7202 at the same location B7000-b corresponds to the first user being in a portion of the physical environment facing the HMD's external display (e.g., a physical environment that also hosts the second display generating component and the first user 7202). In some embodiments, the second display generating component is positioned back-to-back with the first display generating component such that a portion of the physical environment in front of the display surface of the second display generating component 7102 (e.g., location B7000-b including the display surface of the second display generating component) is also within the pass-through view provided by the first display generating component.For example, physical object 7014 (or both physical object 7014 and physical object 7028 in FIG. 7H ) located in a portion of the physical environment in front of the display surface of second display generating component 7102 will be within the pass-through view provided by the first display generating component when first user 7102 moves to the display surface of the first display generating component (e.g., moves to position A7000-a and / or faces the display surface of the first display generating component, etc.). In some embodiments, the computing system displays one or more user interface elements only in response to detecting that the first user is in a position to view content displayed by the second display generating component, and ceases displaying the one or more user interface elements using the second display generating component when the user is not in a position relative to the second display generating component that allows the user to view content presented by the second display generating component. In some embodiments, the computing system displays one or more user interface elements in response to detecting an event indicating availability of the first computer-generated experience based on the current state or context of the computing system (e.g., a time-based alert or a location-based notification is generated on the computing system, a user removes the HMD from a bag, a user turns on the HMD, a user plugs the HMD into a charging station, etc.). In some embodiments, the computing system discloses one or more user interface objects on the second display generation component only when the first display generation component is not displaying any CGR content (e.g., the first display generation component is inactive, in a power-saving state, etc.).7F and 7H showing that the first user 7202 is not simultaneously present at the first display generating component and at Position A or in a position to view the content displayed by the first display generating component corresponds to the first user not having the internal display of the HMD in front of the user's face or eyes (e.g., by either holding the HMD up with the internal display facing the user's face or by wearing the HMD on the user's head). In some embodiments, the one or more user interface objects are displayed on the second display generating component regardless of whether the first user is present at the second display generating component or in a position to view the content shown on the second display generating component.

[0147] As shown in Figure 7G, which follows Figure 7F, and Figure 7I, which follows Figure 7H, while the computing system is using the second display generating component to display one or more user interface objects (e.g., circle 7012 or square 7026, respectively), the computing system detects (e.g., due to movement of the first user 7202, movement of the first display generating component, or both) that the first display generating component (e.g., display 7100 or the internal display of the HMD) is currently in a preset spatial relationship with the first user 7202. In the example shown in Figures 7F and 7G, a first user 7202 moves to a position A7000-a in front of a display surface of a first display generation component (e.g., display 7100, an internal display of an HMD, etc.), and in response to detecting that the first user has moved to position A7000-a in front of the display surface of the first display generation component, the computing system displays, via the first display generation component, an individual computer-generated experience (e.g., a first computer-generated experience 7024 or a second computer-generated experience 7030) corresponding to one or more user interface objects previously shown on the second display generation component (e.g., a circle 7012 or a square 7026, respectively). For example, as shown in FIG. 7G, the computing system displays a first computer-generated experience 7024 in response to an event indicating that relative movement of the first display generating component and the first user has positioned the first display generating component and the first user (or the first user's face or eyes) in a predetermined spatial relationship or configuration (e.g., the first user is facing the display surface of the first display generating component, or the HMD is positioned in front of the user's eyes, the HMD is positioned on the user's head, etc.) while a first user interface object (e.g., circle 7012) is being displayed by a second display generating component (FIG. 7F).As shown in FIG. 7I, the computing system displays a second computer-generated experience 7030 in response to an event indicating that relative movement of the first display generating component and the first user has positioned the first display generating component and the first user (or the first user's face or eyes) in a predetermined spatial relationship or configuration (e.g., the first user is facing the display surface of the first display generating component, or the HMD is positioned in front of the user's eyes, the HMD is positioned on the user's head, etc.) while a second user interface object (e.g., square 7026) is being displayed by the second display generating component ( FIG. 7H ).

[0148] As shown in FIG. 7G , which follows FIG. 7F , an example of the first computer-generated experience 7024 is an augmented reality experience that shows a portion of the physical environment in front of the display surface of the second display generation component (e.g., a portion of the physical environment in front of the HMD's external display that is also a portion of the physical environment in front of the first user wearing the HMD and / or facing the HMD's internal display). The first computer-generated experience optionally includes a representation 7014′ of the physical object 7014, augmented with some virtual content (e.g., a virtual ball 7020 positioned on top of the representation 7014′ of the physical object 7014, and / or some other virtual object). In some embodiments, the first computer-generated experience is a purely virtual experience and does not include a representation of the physical environment surrounding the first display generation component and / or the second display generation component.

[0149] As shown in Figure 7I, which follows Figure 7H, an example of the second computer-generated experience 7030 is an augmented reality experience that shows a portion of the physical environment in front of the display surface of the second display generation component (e.g., a portion of the physical environment in front of the HMD's external display that is also a portion of the physical environment in front of a first user wearing an HMD and / or facing the HMD's internal display). The second computer-generated experience includes physical objects 7014 and 7028, optionally stacked together and augmented with some virtual content (e.g., a virtual box 7032 placed next to representations 7014' and 7028' of physical objects 7014 and 7028, or some other virtual object). In some embodiments, the second computer-generated experience is a purely virtual experience and does not include a representation of the physical environment surrounding the first display generation component and / or the second display generation component.

[0150] In some embodiments, as previously described in this disclosure, the first display generating component is an internal display of the HMD and the second display generating component is an external display of the HMD, and the spatial relationship represented by the simultaneous presence of the display 7100 and the first user 7202 at the same location A7000-a corresponds to the first user wearing or holding the HMD with the internal display of the HMD facing the user's eyes or face. In some embodiments, the first display generating component displays a camera view of the physical environment in front of the first user or includes a transparent or semi-transparent portion through which a portion of the physical environment in front of the first user is visible to the first user. In some embodiments, the physical environment made visible to the first user via the first display generating component is a portion of the physical environment in front of the display surface of the second display generating component (e.g., location B7000-b including the area in front of the display surface of the second display generating component and the area in front of physical object 7014 (and optionally physical object 7028), the HMD's external display, etc.). In some embodiments, the computing system requires that the first display generation component be moved to a predetermined orientation relative to the first user or a distinct portion of the first user (e.g., the internal display of the HMD is oriented to face the user's eyes or face, the first user moves to face the display surface of the first display generation component, and / or the internal display of the HMD is upright relative to the user's face, etc.) to trigger the display of the computer-generated experience via the first display generation component.In some embodiments, the distinct computer-generated experience is selected according to the current state of the computing system (e.g., one or more states determined based on contextual information (e.g., time, location, what physical objects are in front of the user, the user's identity, new notifications or alerts generated on the computing system, etc.)) and / or which user interface element(s) (e.g., one or more of user interface elements conveying the identity and characteristics of the selected computer-generated experience and / or user interface elements conveying contextual information used for the selected computer-generated experience, etc.) are displayed by the second display generating component when movement to the preset spatial relationship between the user and the first display generating component begins and / or is completed. In the examples shown in FIGS. 7F and 7G , the current state of the computing system is determined based on the current position of the display generating component(s) and / or the physical object(s) present in front of the display surface of the second display generating component (and optionally also the current position and physical objects present in front of the external display of the HMD housing both the first and second display generating components).

[0151] 7A-7E, and reiterated herein, while the first display generating component (e.g., display 7100) and the second display generating component (e.g., display 7102) are shown in FIGS. 7F-7I as being located in two distinct and separate portions of the physical environment, it should be understood that the first display generating component and the second display generating component are, optionally, two display generating components housed in the same housing (e.g., the housing of a single HMD) or mounted on the same support structure (e.g., back-to-back with each other or mounted on two sides of a single wall or surface) and facing in different directions (e.g., facing in opposite directions, facing at different angles, etc.). According to some embodiments, a user can move (e.g., rotate, change orientation, or flip vertically or horizontally) the housings or support structures of the first display generating component and the second display generating component to move the first display generating component into a preset spatial configuration relative to the user or the user's face or eyes. According to some embodiments, a user can insert their head into the housing of the display generating component or attach the support structure of the display generating component to a part of the user's body (e.g., head, shoulder, nose, ear, etc.) to move the first display generating component into a preset spatial configuration relative to the user or the user's face or eyes. Thus, the simultaneous presence of the first user and the second display generating component at location B7000-b and the simultaneous presence of the first user and the first display generating component at location A7000-a represent a first time before the preset spatial relationship between the user and the first display generating component is satisfied to trigger the display of a computer-generated experience and when the second display generating component is displaying user interface elements indicative of available CGR experiences, and a second time after the preset spatial relationship is satisfied by movement of the user and / or the display generating component(s) and when the available CGR experiences are being displayed via the first display generating component, respectively.

[0152] In some embodiments, the second display generating component is a lower-resolution, smaller, simpler, monostereoscopic, monochrome, low-power, and / or secondary display, and the first display generating component is a higher-resolution, larger, more complex, stereoscopic, full-color, full-power, and / or primary display of the computing system. In some embodiments, the second display generating component is used by the computing system to display status information, event information, state information related to the computing system, and in particular notifications and prompts to the user to position the first display generating component in a pre-set spatial relationship with the user's eyes to view additional available content related to the current context. In some embodiments, the second display generating component is used by the computing system when the first display generating component is not positioned in front of the user's eyes (or more generally, when the user is not in a position to fully enjoy the CGR content displayed on the first display generating component) and / or when the display generating component of the computing system (e.g., as part of a single HMD) is positioned on a desk, in the user's hand, in a container (e.g., a backpack, holder, case, etc.), or in a standby state (e.g., plugged in at a charging station, set to a low-power mode, etc.), etc. In some embodiments, while displaying information using the second display generating component, the computing system continues to monitor the spatial relationship between a user (e.g., the first user or any user) and the first display generating component (e.g., using sensors attached to or surrounded by the housing of the first display generating component (e.g., motion sensors, orientation sensors, image sensors, touch sensors, etc.) and / or external sensors (e.g., motion sensors, orientation sensors, image sensors, etc.)).In some embodiments, in response to detecting relative movement between the first display generating component and the user (e.g., in response to the user picking up a display generating component enclosed in the same housing or mounted on the same support structure and reorienting the display surface of the first display generating component toward the user's eyes or face, and / or in response to the user donning an HMD including the first and second display generating components on the user's head, etc.), in accordance with a determination that a preset spatial relationship between the user and the first display generating component has been satisfied, the computing system displays a computer-generated experience corresponding to the state of the computing system at the time the preset spatial relationship was satisfied (e.g., optionally, this is the same state the computing system had at the time information (e.g., one or more user interface objects indicating the availability of a computer-generated experience) was being displayed by the second display generating component at the start of the relative movement).

[0153] In some embodiments, as shown in Figures 7G and 7I, when a computing system is displaying a distinct computer-generated experience corresponding to a current context (e.g., the respective state of the computing system as indicated by user interface objects 7012 and 7026 in Figures 7F and 7H) via a first display generation component, the computing system optionally displays state information (state information 7022 and 7034, respectively) using a second display generation component. In some embodiments, the displayed state information conveys information about the computer-generated content displayed via the first display generation component and, optionally, the state of a user viewing the computer-generated content via the first display generation component (e.g., the appearance of the user's face or eyes). Other aspects and details related to displaying state information using a second display generation component while a computing system is displaying computer-generated content using a first display generation component are described with reference to Figures 7A-7E and the accompanying description, as well as the processes described with reference to Figures 8-13. In some embodiments, the second display generating component stops displaying any content (e.g., user interface element 7012 or 7026) once the first display generating component begins displaying content and / or once a preset spatial relationship between the user and the first display generating component is satisfied. In some embodiments, the computing system does not display status information or any other content while using the first display generating component to display the computer-generated experience. In some embodiments, the computing system uses the second display generating component to display other information (e.g., a digital clock, a weather forecast, a countdown timer based on the duration of the computer-generated experience, or the time allotted to the first user to use the first display generating component) while using the first display generating component to display the computer-generated experience.

[0154] In some embodiments, the separate computer-generated experience displayed via the first display generation component is a mixed reality experience in which virtual content is viewed simultaneously with a representation of the physical environment (e.g., position B, a portion of the physical environment in front of the first user, etc.). In some embodiments, the representation of the physical environment includes a camera view of a portion of the physical environment that would be in the first user's field of view if the user's eyes were not obstructed by the presence of the first display generation component and the second display generation component (e.g., if the first user were not wearing an HMD or holding an HMD in front of the user's eyes). In mixed reality mode, CGR content (e.g., a movie, a three-dimensional augmented reality environment, a user interface, and / or a virtual object, etc.) is displayed to overlay or replace at least a portion, but not all, of the representation of the physical environment. In some embodiments, the first display generation component includes a transparent portion through which a portion of the physical environment is visible to the first user, and in mixed reality mode, the CGR content (e.g., a movie, a three-dimensional augmented reality environment, a user interface, a virtual object, etc.) is projected onto a physical surface or empty space within the physical environment and is visible through the transparent portion along with the physical environment. In some embodiments, the CGR content is displayed on a portion of the display, blocking the view of at least a portion, but not all, of the physical environment visible through a transparent or semi-transparent portion of the first display generation component. In some embodiments, the first display generation component 7100 does not provide a view of the physical environment, but rather provides a complete virtual environment (e.g., without a camera view or transparent pass-through portion) augmented with a real-time visual representation(s) of the physical environment (e.g., a stylized representation or segmented camera image) as currently captured by one or more sensors (e.g., a camera, motion sensor, other orientation sensor, etc.).In some embodiments, in mixed reality modes (e.g., augmented reality based on a camera view or a see-through display, or augmented virtuality based on a virtualized representation of the physical environment), the first user is not fully immersed in the computer-generated environment, but is still provided with sensory information (e.g., visual, audio, etc.) that directly corresponds to the physical environment surrounding the first user and the first display generation component. In some embodiments, while the first display generation component displays a fully immersive environment, the second display generation component optionally displays state information without information about the state of the user's eyes (e.g., only state information related to CGR content) or without displaying any state information at all.

[0155] In some embodiments, the computing system optionally has any number of different states corresponding to the availability of different computer-generated experiences for display via the first display generation component. Each different state of the computing system optionally has a corresponding set of one or more user interface elements that are displayed by the second display generation component when the computing system enters and / or remains in that state. Each different state of the computing system is optionally triggered by a corresponding event or set of events and / or the satisfaction of a corresponding set of one or more preset criteria. Although only two states of the computing system, two user interface objects corresponding to the two states, and two different computer-generated experiences are shown in the example described with reference to FIGS. 7F-7I, a third state, a third user interface element, and a third computer-generated experience are optionally implemented by the computing system in a manner similar to that described with respect to the two states, user interface elements, and computer-generated experiences described in the example. In some embodiments, any finite number of states, user interface elements, and computer-generated experiences are optionally implemented.

[0156] In some embodiments, the computer-generated experience provided by the first display generation component is an immersive experience (e.g., an AR or VR experience) that takes into account the first user's actions (e.g., gestures, movements, speech, and / or gaze, etc.) within the physical environment. For example, as the user's hands move within the physical environment or as the user moves (e.g., turns or walks) within the physical environment, the user interface and / or the user's view of the computer-generated three-dimensional environment are updated to reflect the user's hand movements (e.g., pressing open a virtual window within the AR environment, activating a user interface element within a home screen or menu presented within the AR environment, etc.) or the user's movements (e.g., the user's viewpoint moves relative to the AR environment or virtual three-dimensional game world, etc.).

[0157] In some embodiments, the different computer-generated experiences (e.g., a first computer-generated experience, a second computer-generated experience, etc.) are AR experiences that include representations of the same physical environment but different virtual elements (e.g., as indicated by one or more user interface elements (e.g., circle 7012, square 7026, etc.) displayed by the second display generation component) that are selected based on the state of the computing system. For example, in some embodiments, the computer-generated experiences optionally include a view of the same room in which the first user is located. Pursuant to a determination that the computing system displayed a respective one of several different event reminders on the second display generation component immediately before the user positioned the display surface of the first display generation component in front of the user, the computing system displays a representation of the room with a respective one of several differently themed virtual wallpapers on the representation of the room's walls while displaying a respective introductory video for the event corresponding to the respective event reminder.

[0158] In some embodiments, the different computer-generated experiences are either augmented reality experiences or virtual experiences, depending on the context (e.g., the state of the computing system as determined based on relevant contextual information (e.g., location, time, user identification, receipt of a notification or alert, etc.) and / or what is shown on the second display generation component). In some embodiments, after the computer-generated experience begins in one of the AR and VR modes, the experience can transition to the other of the AR and VR modes (e.g., in response to a user request, in response to other preset conditions being met, etc.).

[0159] In some embodiments, the computing system is configured to display respective user interfaces and / or user interface objects of various applications using a second display generation component based on a state of the computing system. For example, in some embodiments, one or more user interface elements displayed on the second display generation component include elements of an electronic calendar (e.g., a social calendar, a work calendar, a daily planner, a weekly planner, a monthly calendar, a standard calendar showing dates and weeks by month, etc.) with scheduled events, appointments, holidays, reminders, etc. In some embodiments, the computing system displays different computer-generated experiences via the first display generation component when a preset spatial configuration between the first display generation component and the first user is met (e.g., the first user or the user's eyes are facing the display surface of the first display generation component, the first user is in a position that allows the first user to view the content displayed by the first display generation component, etc.), and the particular computer-generated experience displayed is based on what calendar content was shown on the second display generation component immediately before the movement that placed the first display generation component and the first user in the preset spatial configuration began and / or completed. For example, pursuant to determining that one or more user interface elements presented on the second display generation component correspond to the first calendar event (e.g., the user interface elements present event information, alerts, notifications, calendar data, notes, etc. for the first calendar event), the computing system displays a first computer-generated experience (e.g., detailed information and / or interactive information (e.g., previews, videos, models of the venue and attendees, etc.)) corresponding to the first calendar event.In accordance with determining that one or more user interface elements illustrated on the second display generation component correspond to a second calendar event (e.g., the user interface elements show event information, alerts, notifications, calendar data, notes, etc. for the second calendar event), the computing system displays a second computer-generated experience (e.g., detailed information and / or interactive information (e.g., previews, videos, models of the venue and attendees, etc.)) corresponding to the second calendar event. In some embodiments, when the dual-sided HMD is not being worn by a user (e.g., placed on a desk with the external display facing the user), the external display of the HMD is used to display a calendar including the current date, time, weather information, geographic location, and / or a list of upcoming tasks or scheduled appointments that are due that day or within a preset time period (e.g., in the next two hours, the next five minutes, etc.). When a user picks up the HMD and places the internal display of the HMD in front of the user (e.g., by lifting the HMD or by wearing the HMD on the user's head, etc.), the internal display of the HMD displays calendar details (e.g., showing a more complete calendar including the current week or current month, showing all scheduled events for the day, showing more details of upcoming events, etc.) In some embodiments, the one or more user interface elements corresponding to a first calendar event include a notification of the first calendar event, and the user interface elements corresponding to a second calendar event are a notification of the second calendar event.

[0160] In some embodiments, the computing system uses the second display generation component to display media objects, such as photographs and / or video clips with two-dimensional images, and the first display generation component to display three-dimensional experiences or unabridged media content corresponding to the media objects displayed on the second display generation component. For example, user interface elements shown on the second display generation component optionally include snapshots or clips from a longer video, a reduced-resolution or two-dimensional version of a three-dimensional video, a non-interactive user interface corresponding to an interactive computer environment, etc., and when criteria for triggering the display of such augmented content are met (e.g., when the first display generation component and the first user are placed in a pre-set spatial configuration, and optionally other conditions (e.g., the user is seated, the HMD has sufficient power, etc.)), the first display generation component displays the longer video, the three-dimensional video, the interactive computer environment, etc. In some embodiments, when a dual-sided HMD is not being worn by a user (e.g., placed on a desk with the external display facing the user), the external display of the HMD is used to display visual representations of available media items that can be displayed via the HMD's internal display. In some embodiments, the available media items change depending on the current location of the HMD and / or the availability of media items as specified by the media item provider, etc. When a user picks up the HMD and places the internal display in front of the user's eyes, the first display generation component displays the actual content of the media items (e.g., showing a more complete movie, a more immersive experience, and / or enabling more interactive features of the media items, etc.).

[0161] In some embodiments, the computing system uses the second display generation component to display an alert of an incoming communication request (e.g., an incoming phone call, an audio / video chat request, a video conference request, etc.), and the computing system uses the first display generation component to display a corresponding communication environment when the first display generation component is placed in a pre-configured physical configuration for the first user (e.g., by movement of the first user, the first display generation component, or both). In some embodiments, the communication environment displayed via the first display generation component shows a mimicked environment in which virtual avatars or images of each participant are present (e.g., avatars seated around a representation of a table in front of the first user, or as speakers on a table surface in front of the first user, etc.). In some embodiments, the computing system, in response to detecting placement of the first display generation component in a pre-configured physical configuration for the first user, accepts the incoming communication request and initiates a corresponding communication session (e.g., using the first display generation component and other components of the computing system). In some embodiments, the computing system, in response to detecting placement of the first display generating component in a preset physical configuration for the first user, initiates an application corresponding to the incoming communication request and displays (e.g., using the first display generating component) a user interface of the application from which the first user can select to accept the incoming communication request. In some embodiments, when the dual-sided HMD is not being worn by a user (e.g., placed on a desk with the external display facing the user), the external display of the HMD is used to display a notification of the incoming communication request when such a request is received by the computing system. In some embodiments, the notification provides an identification of the caller and an indication of the type of communication session being requested.When a user picks up the HMD and places the HMD's internal display in front of the user (e.g., by lifting the HMD using the user's hands or by wearing the HMD on the user's head), the HMD's internal display displays a communication interface corresponding to the received communication request, and the user can begin communicating with the caller using the HMD's internal display. In some embodiments, the computing system launches different applications (or launches different modes of the same application) depending on characteristics of the incoming communication request (e.g., the requester's identity, the time of day, the subject of the call, etc.). For example, for an incoming request from a colleague, the computing system displays a user interface on the first display generation component that waits for pre-configured input from the first user before initiating the communication session; for an incoming request from a family member, the computing system initiates the communication session without displaying a user interface and / or without requiring pre-configured input from the first user. In another example, for an incoming request arriving at the user's home, the computing system initiates the communication session with an avatar of the first user in casual attire; and for an incoming request arriving at the user's office, the computing system initiates the communication session with an avatar of the first user in work attire. In another example, for an incoming request for a phone call, the computing system displays detailed speaker representations of each participant, and for an incoming request for a video chat, the computing system displays full-body representations of each participant showing the participants' actual body movements. In some embodiments, one or more user interface elements displayed on the second display generation component visually indicate certain characteristics of the incoming communication request that are used by the computing system to determine characteristics of the computer-generated experience (e.g., characteristics of the user interface or environment of the communication session).In some embodiments, selected characteristics of the computer-generated experience are also visually represented by one or more user interface elements shown by the second display generating component before the computer-generated experience is displayed by the first display generating component. In some embodiments, the computing system modifies the characteristics of the computer-generated experience according to user input (e.g., a touch gesture on the second display generating component, a touch gesture on a housing of the first and / or second display generating component, an air gesture, a voice command, etc.) received before displaying the computer-generated experience using the first display generating component.

[0162] In some embodiments, the computing system modifies the content (e.g., one or more user interface elements) displayed on the second display generation component in response to various parameters (e.g., user distance, user identity, user gestures, etc.). For example, in response to detecting a first user at a first distance away from the second display generation component (e.g., the first distance is less than a first threshold distance but greater than a second threshold distance), the computing system displays a first version of one or more user interface elements (e.g., large simple icons or text) to indicate the availability of a personalized computer-generated experience, and in response to detecting a first user at a second distance away from the second display generation component (e.g., the second distance is less than the second threshold distance), the computing system displays a second version of the one or more user interface elements (e.g., graphics, more details, etc.) to indicate the availability of a personalized computer-generated experience (e.g., replacing the display of the first version of the one or more user interface elements as the first user moves closer to the second display generation component). In another example, upon detecting a user within a threshold distance of the second display generation component, the computing system displays generic versions of one or more user interface elements (e.g., large simple icons or text) to indicate the availability of a personalized computer-generated experience, and upon detecting a user's identity (e.g., upon detecting a user's fingerprint when picking up the first / second display generation component (e.g., HMD) or upon the user moving near the second display generation component, etc.), the computing system displays user-specific versions of one or more user interface elements corresponding to the user's identity (e.g., customized based on the user's preferences, usage history, demographics, etc.).

[0163] In some embodiments, the computing system displays a user interface including selectable options (e.g., one or more user interface elements and / or one or more user interface objects other than user interface elements) before detecting that the first display generating component has been placed in a preset physical configuration for the first user, and detects user input selecting one or more of the selectable options, the selectable options including preferences for customizing a computer-generated experience corresponding to the one or more user interface elements available for display via the first display generating component. Once the first display generating component has been placed in the preset physical configuration for the first user, the computing system displays a customized computer-generated experience based on the option(s) selected by the user. In some embodiments, the selectable options correspond to a set of two or more modes of the computing system (e.g., an AR mode, a VR mode, a 2D mode, a private mode, a parental control mode, a DND mode, etc.) in which the computer-generated experience can be presented via the first display generating component.

[0164] In some embodiments, the one or more user interface elements displayed by the second display generating component include a preview of the three-dimensional experience available for display by the first display generating component. In some embodiments, the preview provided by the second display generating component is a three-dimensional preview that simulates a viewport into the three-dimensional experience. The user can move their head relative to the second display generating component to view different portions of the three-dimensional environment represented in the three-dimensional experience. In some embodiments, the preview is initiated when the user picks up the second display generating component (e.g., picks up a dual-sided HMD) and / or places the second display generating component in a preset spatial configuration with respect to the first user (e.g., holds the HMD with the external display facing the user's eyes). In some embodiments, in response to detecting that the user places the first display generating component in a preset spatial relationship with the first user (e.g., holds the HMD with the internal display facing the user's face or eyes, wears the HMD on the user's head, etc.) after the preview is initiated on the first display generating component, the computing system launches the computer-generated experience on the first display generating component.

[0165] 7H-7J illustrate displaying different computer-generated experiences depending on how the first display generation component is maintained in a preset spatial relationship or configuration with respect to the first user during presentation of the computer-generated experiences, according to some embodiments. In some embodiments, the different computer-generated experiences are related to one another. For example, the different computer-generated experiences may be a preview of a three-dimensional computer-generated experience and the three-dimensional computer-generated experience itself, respectively, or a segment or edited version of a computer-generated experience and the complete version of the computer-generated experience, respectively, etc. In some embodiments, the computing system determines that the first display generation component is positioned in a preset spatial relationship or configuration with respect to the first user when the display surface of the first display generation component faces the first user or a preset portion of the first user (e.g., the user's eyes or face) and / or when the user is in a position to view content shown on the first display generation component (e.g., the user is wearing an HMD, holding an HMD, raising an HMD in their hands, placing the HMD at a preset viewing station, connecting the HMD to a preset output device, etc.). In some embodiments, the computing system determines which of the computer-generated experiences to display on the first display generating component based on whether the first display generating component (or optionally the entire HMD surrounding the first display generating component (and optional second display generating component)) is being worn by the first user (e.g., secured with straps, held in front of the user's eyes without support from the user's hand(s)). In some embodiments, the computing system determines whether the first display generating component is being worn by the first user based on the state of devices or sensors other than the first display generating component (e.g., straps or buckles on the housing of the first display generating component, touch or position sensors attached to the housing of the first display generating component, etc.).For example, the strap or buckle optionally has an open state and a closed state, with the strap or buckle being in the closed state when the first indication-generating component is being worn by the first user, and the strap or buckle being in the open state when the first indication-generating component is only temporarily positioned in front of the user (e.g., held up to eye level by the user's hand) and not being worn by the user. In some embodiments, a touch or position sensor on the housing of the first indication-generating component switches to a first state (a "yes" state) when the housing of the first indication-generating component is resting and supported against the user's nose, ear, head, and / or other part of the user's body other than the user's hand, and the touch or position sensor switches to a second state (a "no" state) when the housing of the first indication-generating component is supported by the user's hand(s). In some embodiments, by distinguishing how the first display generating component (or an HMD including the first display generating component) is maintained in a pre-set configuration for the first user so that the user can view the computer-generated experience displayed on the first display generating component, the computing system can better adjust the interaction model and depth of content presented to the user.For example, the computing system enables a first interaction model that requires movement of the user's hand as input (e.g., air gestures, microgestures, input provided on a control device separate from the housing of the first display generating component, etc.) only when the first display generating component is not being maintained in a preset configuration for the first user by the user's hand(s), and while the first display generating component is being maintained in a preset configuration for the first user by the user's hand(s), the computing system only enables other types of interaction models that do not require the user's hand(s) to move away from the housing of the first display generating component (e.g., speech interaction, gaze interaction, touch interaction on the housing of the first display generating component, etc.).

[0166] In Figure 7I, the first user 7202 is collocated with the first display generating component (e.g., display 7100, an internal display of an HMD, etc.) at position A7000-a, facing the display surface of the first display generating component. This is to illustrate an exemplary scenario in which the first display generating component is in a preset configuration relative to the first user or a preset portion of the first user (e.g., the first display generating component is positioned in front of the user's face or eyes), according to some embodiments. In the exemplary scenario shown in Figure 7I, the first user 7202 is holding a sensor object 7016 in the user's hand. This position of the sensor object 7016 relative to the user's hand corresponds to the state of the first display generating component when the first display generating component is not being worn by the first user 7202. Another example scenario corresponding to the state of the first display generating component when the first display generating component is not being worn by the first user is when the first display generating component is a display of an HMD (e.g., an internal display of the HMD, a single display of the HMD, etc.) and the display of the HMD is held or held up by the first user's hand to face the first user's eyes (e.g., as opposed to being supported by the user's head, nose, ears, or other part of the user that is not the user's hand). In some embodiments, in a scenario when the sensor object 7016 is held in the first user's hand (which also corresponds to the first display generating component being supported by the user's hand or not being worn by the first user when placed in front of the user's face or eye), the computing system displays a second computer-generated experience 7030 (e.g., an experience corresponding to the second state of the computing system shown in FIG. 7H ).

[0167] In Figure 7J, the first user is collocated with the first display generating component at position A7000-a, facing the display surface of the first display generating component 7100. This is to illustrate another exemplary scenario in which the first display generating component 7100 is in a preset configuration relative to the first user or a preset portion of the first user (e.g., the first display generating component is positioned on the user's face or in front of the user's eyes), according to some embodiments. In the scenario shown in Figure 7J, the first user 7202 is no longer holding the sensor object 7016 in the user's hand, but has placed the sensor object on the user's body (e.g., the user's back), so that the first user 7202 is now wearing the sensor object 7016. This position of the sensor object 7016 relative to the user's hand and body corresponds to the state of the first display generating component 7100 when the first display generating component is being worn by the first user. Another example scenario corresponding to the state of the first display generating component when it is being worn by the first user is when the first display generating component is a display of an HMD (e.g., an internal display of the HMD) and the HMD is worn normally on the first user's head (e.g., fully strapped, buckled, or stationary around the user's nose, ears, and / or head, as opposed to being supported by the user's hand(s)).In some embodiments, in a scenario where the sensor object 7016 is worn by a user and not held in the first user's hand (which also corresponds to the first display generating component being worn by the user and not supported by the user's hand when placed in front of the user's face or eye), the computing system displays a third computer-generated experience 7036 (e.g., a different experience that also corresponds to the second state of the computing system shown in FIG. 7H (e.g., showing square 7026), but that is different from the second computer-generated experience (e.g., experience 7030 of FIG. 7I) due to the state of the sensor object 7016 (and accordingly, the wearing state of the first display generating component (e.g., display 7100 or the internal display of the HMD)).

[0168] 7J , the computing system detects (e.g., using camera 7104 and / or other sensors) that a first user 7202 is moving their hand in the air to provide an air gesture, and in response, moves virtual object 7032 on top of representation 7028′ of physical object 7028. In some embodiments, the computing system disables at least some of the input devices (e.g., touch-sensitive surface, buttons, switches, etc.) provided on the housing of the first display generating component (e.g., display 7100, an internal display of an HMD, a single display of an HMD, etc.) while displaying the third computer-generated experience 7036. In some embodiments, the computing system enables at least one interaction model (e.g., an interaction model supporting air hand gestures, microgestures, and / or input detected via an input device separate from the housing of the first display generating component, etc.) that was not enabled when the first display generating component was not being worn by the first user 7202 (e.g., as determined based on the state of sensor object 7016 or other sensors, etc.). In some embodiments, the third computer-generated experience 7036 and the second computer-generated experience 7030 are related experiences that have corresponding content (e.g., the same content, or different versions of the same content) but different interaction models (e.g., different interaction models, or overlapping but different sets of interaction models, etc.).

[0169] As shown in FIGS. 7I and 7J , in some embodiments, the computing system optionally includes a second display generating component (e.g., a display included in a different housing than the first display generating component (e.g., display 7102) or included in the same housing as the first display generating component (e.g., back-to-back or otherwise facing in a different direction) (e.g., an HMD with an internal display and an external display)). In some embodiments, the second display generating component optionally displays state information related to the content shown via the first display generating component (e.g., an altered visual representation of the content), and optionally displays state information related to the state of the first user (e.g., an image or representation of the user's face or eyes) and / or the operating mode of the computing system (e.g., mixed reality mode, virtual reality mode, full pass-through mode, parental control mode, private mode, DND mode, etc.). In some embodiments, the second display generating component also displays user interface elements corresponding to computer-generated experiences available for display by the first display generating component. More details of the operation of the second display generating component and the corresponding operation of the first display generating component are described with reference to the processes described with reference to Figures 7A-7E and 7F-7I, as well as Figures 8-13. In some embodiments, a computing system that displays different computer-generated experiences based on whether the first display generating component is worn by a user when the first display generating component is arranged in a preset configuration for the user does not have another display generating component other than the first display generating component, and therefore does not display status information and / or user interface elements indicating the availability of the computer-generated experiences described herein.

[0170] In some embodiments, when the first display generating component is arranged in a preset configuration for the first user (e.g., with the display surface of the first display generating component facing the user's eyes or face and / or within a threshold distance from the user's face, etc.), depending on whether the first display generating component is being worn by the first user (e.g., whether the HMD is strapped or buckled to the user's head and can remain in front of the user's eyes without the support of the user's hand(s), or whether it is simply held in front of the user's eyes by the user's hand(s) and would fall off without the support of the user's hand(s), etc.), the computing system optionally uses the first display generating component to display different types of user interfaces (e.g., a system user interface (e.g., an application launch user interface, a home screen, a multitasking user interface, a configuration user interface, etc.) versus an application user interface (e.g., a camera user interface, an infrared scanner user interface (e.g., showing a heat map of the current physical environment), an augmented reality measurement application (e.g., automatically displaying measurements of physical objects in the camera view), etc.)). In some embodiments, the computing system takes photographs or videos of the physical environment captured within the camera view in response to user input detected via an input device (e.g., a touch sensor, a contact intensity sensor, a button, a switch, etc.) located on a housing of the first display generating component when the computing system is displaying an application user interface using the first display generating component.

[0171] In some embodiments, the computing system prioritizes touch input detected on a touch-based input device located on the housing of the first display generating component over microgesture input and / or air gesture input detected in front of the first user (e.g., the microgesture input and air gesture input are performed by the user's hands not holding the housing of the first display generating component) when determining a response to a user input detected while the user is holding the first display generating component in front of the user's eyes but is not wearing the first display generating component. In some embodiments, the computing system prioritizes microgesture input and / or air gesture input detected in front of the first user over touch input detected on a touch-based input device located on the housing of the first display generating component when determining a response to a user input detected when the user is wearing the first display generating component (e.g., when the user's hands are not required to support the first display generating component). In some embodiments, in response to detecting multiple types of input simultaneously (e.g., input performed by a hand away from the first display generating component, input performed by a hand touching the first display generating component or its housing, etc.), and in accordance with determining that the first display generating component is being worn by a user while in a configuration preset for the user (e.g., an HMD including the first display generating component is strapped to the user's head, buckled, not supported by the user's hand, etc.), the computing system enables interaction with the displayed computer-generated experience based on gestures (e.g., air gestures, microgestures, etc.) performed by hands located away from the first display generating component and its housing (e.g., gestures captured by a camera on the HMD, a mechanical or touch-sensitive input device, a sensor worn on the user's hand, etc.).Pursuant to a determination that the first display generating component is not being worn by the user while in a configuration preset for the user (e.g., not strapped to the user's head, not buckled, supported by the user's hand, etc.), the computing system enables interaction with the displayed computer-generated experience based on gestures (e.g., touch gestures, manipulation of physical controls, etc.) performed by the first display generating component or the hand on its housing (e.g., gestures captured by a touch-sensitive surface of the HMD's housing, a button or switch on the HMD's housing, etc.).

[0172] 7K-7M illustrate a computing system (e.g., computing system 101 of FIG. 1 or computing system 140 of FIG. 4 ) including at least a first display generating component (e.g., display 7100, an internal display of an HMD, a single display of an HMD, etc.) and optionally a second display generating component (e.g., display 7102, an external display of an HMD, etc.), where the computing system displays a computer-generated experience (e.g., an augmented reality experience, an augmented virtual experience, a virtual reality experience, etc.) to a user via the first display generating component (e.g., display 7100, an internal display of an HMD, a single display of an HMD, etc.) according to physical interactions between the user and physical objects in a physical environment (e.g., picking up an instrument and playing the instrument, picking up a book and opening the book, holding a box and opening the box, etc.). In some embodiments, only certain physical interactions that meet preset criteria corresponding to the physical objects can trigger the display of the computer-generated experience. In some embodiments, different computer-generated experiences are optionally displayed depending on which of multiple sets of criteria is met by the physical interaction with the physical object. In some embodiments, the different computer-generated experiences include different augmented reality experiences that correspond to different modes of moving a physical object (e.g., tapping, flicking, banging, opening, shaking, etc.) Figures 7K-7M are used to illustrate the processes described below, including the processes of Figures 8-13.

[0173] As shown in FIG. 7K , a user (e.g., user 7202) is present in scene 105 including physical objects (e.g., objects including a box lid 7042 and box body 7040, a book, a musical instrument, etc.) in a room having walls and a floor. In FIG. 7K , the user is holding a first display generating component (e.g., display 7100, an HMD, a handheld device, etc.) in hand 7036. In some embodiments, the first display generating component is not held in the user's hand 7036, but is supported by a housing or support structure placed on the user's body (e.g., head, ear, nose, etc.). In some embodiments, the first display generating component (e.g., a head-up display, projector, etc.) is placed in front of the first user's eyes or face and is supported by another support structure (e.g., a tabletop, TV stand, etc.) that is not part of the user's body.

[0174] 7K, a computing system provides an augmented reality view 105′ of a physical environment (e.g., a room including physical objects) via a first display generation component (e.g., display 7100, a display of an HMD, etc.). In the augmented reality view of a portion of the physical environment, the view includes representations of the physical objects (e.g., including a representation 7042′ of a box's lid 7042 and a representation 7040′ of a box's body 7040), and optionally, a representation of the surrounding environment (e.g., a representation 7044′ of a support structure 7044 that supports the physical objects and a representation of the room's walls and floor). In addition to the representations of the physical objects in the environment, the computing system also displays some virtual content (e.g., user interface objects, visual extensions of the physical objects, etc.), including a visual indication (e.g., label 7046 or other visual indication, etc.) that one or more computer-generated experiences corresponding to the physical objects (e.g., the box including lid 7042 and body 7040, another physical object in the environment, etc.) are available for display via the first display generation component. 7K(B), in some embodiments, a visual indication (e.g., label 7046) is displayed in the view of the physical environment at a location corresponding to the location of the physical object (e.g., at the location of box lid 7042). For example, label 7046 appears to be located on top of box lid 7042 in the view of the physical environment displayed via the first display generation component.

[0175] In some embodiments, the visual indication (e.g., label 7046, or other visual indication, etc.) includes descriptive information (e.g., icons, graphics, text, animations, video clips, images, etc.) about the computer-generated experience available for display by the first display generation component. In some embodiments, when the first display generation component or one or more cameras of the computing system move within the physical environment, and / or when a physical object moves within the physical environment, such that the augmented reality view of the physical environment shown by the first display generation component includes only a representation of less than a threshold portion of the physical object (e.g., less than 50% of the physical object, or without including a major portion of the physical object (e.g., lid 7042 of a box, title text of a book, sound-producing portion of a musical instrument, etc.), etc.), the computing system stops displaying (or ceases displaying) the visual indication within the view of the augmented reality environment.

[0176] In some embodiments, the visual indication includes prompts or guidance information regarding the physical interaction required to trigger the display of the computer-generated experience (e.g., an animated diagram, an indicator pointing to a particular portion of a representation of a physical object, etc.). In some embodiments, the computing system displays prompts or guidance information regarding the physical interaction required to trigger the display of the computer-generated experience only in response to detecting some user input that meets a pre-set first criterion (e.g., a criterion used to assess a user's interest in viewing the computer-generated experience, a criterion used to detect the presence of a user, a criterion for detecting contact of a user's hand on a physical object, etc.). As shown in FIG. 7L , in some embodiments, when the computing system detects that a user is touching a physical object with the user's hand but has not performed the interaction required to trigger the display of a computer-generated experience (e.g., hand 7038 is touching a physical object's box lid 7042 or box body 7040 but has not opened the box lid 7042), the computing system displays prompt or guidance information (e.g., an animated arrow 7048 or other visual effect or virtual object, etc.) at a location within the view of the augmented reality environment that corresponds to the location of the box lid 7042 to prompt the user to open the box lid 7042. In some embodiments, the prompt and guidance information (e.g., arrow direction, animation sequence, etc.) are updated depending on how the user interacts with the physical object. In some embodiments, a representation of the user's hand (e.g., representation 7038′ of hand 7038) is shown in the augmented reality view 105′ of the physical environment as the user uses their hand to manipulate physical objects in the physical environment.Note that the prompts and guidance information are distinct from the actual computer-generated experience available for display via the first display generation component upon detection of a required physical interaction with the physical object (e.g., opening lid 7042, some other interaction (e.g., picking up box 7040 from support 7044 after lid 7042 is removed), etc.). In some embodiments, the computing system does not display any prompts or guidance information in response to a physical interaction with the physical object that does not meet the criteria for triggering the display of a computer-generated experience (e.g., the computing system does not display the computer-generated experience and does not display the prompts and guidance information, but optionally maintains the display of a visual indication (e.g., label 7046) to indicate that a computer-generated experience is available for display).

[0177] As shown in Figure 7M, in some embodiments, when the computing system detects that a user has performed a physical interaction with a physical object required to trigger the display of the computer-generated experience, the computing system displays the computer-generated experience. For example, in response to detecting that a user's hand 7038 has lifted a box lid 7042 from a box body 7040, the computing system determines that the required physical interaction with the physical object has met the preset criteria for triggering the display of the computer-generated experience and displays the computer-generated experience using a first display generation component (e.g., display 7100, an internal display of an HMD, a single display of an HMD, etc.). In Figure 7M, the computer-generated experience is an augmented reality experience that shows a representation 7038' of the user's hand 7038, a representation 7042' of a box lid 7042, and a representation 7040' of a box body 7040 in positions and orientations that correspond to their physical positions and orientations in the physical environment. Additionally, in some embodiments, the augmented reality experience also shows virtual content (e.g., a virtual ball 7050 that appears to protrude from the body of the box 7040 and casts a virtual shadow on the inside of the body of the box, a virtual platform 7052 that replaces a representation of the physical support 7044 below the box 7040, etc.) simultaneously with the representation of the physical environment. Additionally, in some embodiments, representations of walls in the physical environment are replaced with virtual overlays in the augmented reality experience. In some embodiments, once the computer-generated experience is initiated, as the user continues to interact with the physical objects, the computing system displays changes in the augmented reality environment according to the user's physical manipulation of the physical objects (e.g., the body of the box 7040, the box lid 7042, etc.) and, optionally, according to other inputs detected via various input devices of the computing system (e.g., gesture input, touch input, gaze input, voice input, etc.). In some embodiments, the computer-generated experience progresses in a manner determined according to continued physical interaction with the physical objects.For example, in response to detecting that a user has moved a box lid 7042 within the physical environment, the computing system moves a representation 7042′ of the box lid 7042 and pushes the virtual ball 7050 into the empty space above the representation 7040′ of the box body 7040, and in response to detecting that the user has placed the box lid 7042 back on the box body 7040, the computing system displays the representation 7042′ of the box lid 7042 back on the representation 7040′ of the box body 7040 and stops displaying the virtual ball 7050. In some embodiments, the computing system requires that the user be in physical contact with the physical object when performing the required physical interaction to trigger the display of the computer-generated experience, and the computing system stops displaying the computer-generated experience pursuant to determining that the user has stopped physical contact with the physical object for longer than a threshold time. For example, in some embodiments, the computing system stops displaying the computer-generated experience immediately after detecting that the physical object has been released from the user's hand. In some embodiments, the computing system stops displaying the computer-generated experience when it detects that the physical object has landed and come to rest on another physical surface after being released from the user's hand.

[0178] In some embodiments, the visual feedback provided in response to detecting a user's physical interaction with a physical object prior to a criterion for triggering the display of the computer-generated experience includes a preview of the computer-generated experience, with visual characteristics that are dynamically updated according to characteristics of the physical interaction as the physical interaction is detected. For example, the animation, visual effects, and / or range (e.g., size, dimensions, angular range, etc.) of the virtual object, the amount of detail in the visual feedback, the brightness, color saturation, visual clarity, etc. of the visual feedback, are optionally adjusted (e.g., dynamically in real time, periodically, etc.) according to characteristic values ​​of the interaction with the physical object in the physical environment (e.g., the characteristic values ​​include movement distance, angular range of movement, movement speed, type of interaction, distance to a predetermined reference point, etc.). For example, in some embodiments, if the physical object is a book, as the cover of the book is slowly opened by the user in the physical environment, the color and light of the computer-generated experience emerge from the gap between the cover and the first page, becoming brighter and more saturated as more and more of the cover is opened. The complete computer-generated experience is optionally initiated within the three-dimensional environment once the book cover has been opened beyond a threshold amount and a first criterion has been met. In another example, when the user lifts a corner of the box lid 7042, a virtual light is shown emerging from the representation 7040′ of the box body 7040, and a virtual ball 7050 is shown glimpsed. As the user lifts the corner of the box lid 7042 higher, more virtual light is shown emerging from the representation 7040′ of the box body 7040, and the virtual ball 7050 begins to move around within the representation 7040′ of the box body 7040. When the user finally lifts the box lid 7042 away from the box body 7040, the computer-generated experience begins, and the entire three-dimensional environment changes, the representation of the room is replaced with a virtual platform 7052, and the virtual ball 7050 pops out of the box representation 7040′.

[0179] In some embodiments, the computer-generated experience is optionally triggered by more than one type of physical interaction. In other words, the criteria for triggering a computer-generated experience associated with a physical object are optionally satisfied by a first way of interacting with the physical object and a second way of interacting with the physical object. For example, a computer-generated experience associated with a book is optionally initiated in response to a user picking up the book and placing it against a bookstand with the cover upright against the user's face, and in response to a user picking up the book and opening the cover with the user's hand. In some embodiments, the computer-generated experience is optionally initiated from different parts of the computer-generated experience. For example, the criteria for triggering a computer-generated experience associated with a physical object are optionally satisfied by the same way of interacting with the physical object but with different parameter values ​​(e.g., different pages, different speeds, different times, etc.). For example, a computer-generated experience associated with a book optionally begins with a first portion of the computer-generated experience in response to a user picking up the book and opening it to the first page, and optionally begins with a second, different portion of the computer-generated experience in response to a user picking up the book and opening it to a previously bookmarked page. In another example, slowly opening the book triggers a computer-generated experience with calming background music and / or more subdued colors, while quickly opening the book triggers a computer-generated experience with livelier background music and brighter colors. The book example is merely illustrative. The same principles apply to other computer-generated experiences associated with other types of physical objects and other triggering physical interactions. In some embodiments, different computer-generated experiences are associated with the same physical object and are triggered by different ways of interacting with the physical object.For example, a box may be associated with two different computer-generated experiences: a first computer-generated experience is triggered when a user opens the lid of the box (e.g., a virtual ball pops out of the box when the user presses on the lid), and a second computer-generated experience is triggered when a user flips the box upside down (e.g., a virtual bug emerges from the bottom of the box and chases the user's finger as it moves across the bottom of the box). In some embodiments, different ways of interacting with a physical object trigger different versions of the computer-generated experience that allow for different input modalities. For example, when a book is held in one hand and opened with the other, air gestures (e.g., air tap gestures, hand waving, sign language gestures, etc.) and microgestures of one hand are enabled to interact with the computer-generated experience, and touch gestures are not enabled to interact with the computer-generated experience. When the book is held open with both hands, air gestures are disabled and touch gestures (e.g., taps, swipes, etc.) on the back, front, and / or sides of the book are enabled to interact with the computer-generated experience.

[0180] 7N-7Q illustrate selecting to perform or not perform an action in response to an input detected on the housing of a display generating component according to whether one or both hands are detected on the housing at the time of the input, according to some embodiments. In some embodiments, touch input as an input modality is disabled when both hands are simultaneously detected on the housing of the display generating component (e.g., on the housing of an HMD housing the display generating component, on the frame of the display generating component, etc.). In some embodiments, the computing system will respond to touch input performed by either hand of the user on the housing of the display generating component (e.g., on a touch-sensitive surface or other touch sensor disposed on or around the structure of the display generating component) as long as only one hand (e.g., the hand providing the touch input) is touching the housing of the display generating component; if the computing detects that an additional hand is also touching the housing of the display generating component at the time the touch input is detected, the computing system ignores the touch input and refrains from performing an action corresponding to the touch input (e.g., activating a control, interacting with a computer-generated experience, etc.). In some embodiments, the computing system detects the presence of both hands using the same sensors and input devices used to detect touch input provided by one hand. In some embodiments, a camera is used to capture the position of the user's hands, and images from the camera are used by a computing system to determine whether both of the user's hands are touching the housing of the display generating component when touch input is detected by one or more touch sensors present on the housing of the display generating component. In some embodiments, other means of detecting touch input and / or whether one or both of the user's hands are touching or supporting the housing of the display generating component are implemented.For example, according to various embodiments, position sensors, proximity sensors, mechanical sensors, etc. may be used to detect the presence and posture of a user's hand(s) on the housing of the display generating component.

[0181] 7N , a user (e.g., user 7202) is present in a physical scene 105 that includes physical objects (e.g., box 7052), walls 7054 and 7056, and a floor 7058. User 7202 is standing in front of a display generation component 7100 (e.g., a tablet display device, a projector, an HMD, an internal display of an HMD, a head-up display, etc.) that is supported by a stand. Display generation component 7100 displays a view of physical environment 105. For example, in some embodiments, the view of the physical environment is a camera view of the physical environment. In some embodiments, the view of the physical environment is provided through a transparent portion of the display generation component. In some embodiments, display generation component 7100 displays an augmented reality environment having both a view of the physical environment and virtual objects and content displayed overlaid on or replacing portions of the view of the physical environment. As shown in Figure 7N, the view of the physical environment includes a representation 7052' of box 7052, representations 7054' and 7056' of walls 7054 and 7056, and a representation 7058' of floor 7058. In Figure 7N, user 7202 is not touching the housing of the display generation component, and the computing system does not detect any touch input on the housing of the display generation component. In some embodiments, a representation 7038' of the user's hand 7038 is shown via display generation component 7100 as part of the representation of the physical environment. In some embodiments, display generation component 7100 displays the virtual environment or displays nothing before any touch input is detected on its housing.

[0182] In some embodiments, in addition to touch input, the computing system is optionally configured to detect hover input near the housing of the display generating component. In some embodiments, a proximity sensor disposed on the housing of the display generating component is configured to detect a user's finger or hand approaching the housing of the display generating component and generate an input signal based on the proximity of the finger or hand to the housing of the display generating component (e.g., proximity to a portion of the housing configured to detect touch input, other portions of the housing, etc.). In some embodiments, the computing system is configured to detect respective hover inputs at different positions near the housing of the display generating component (e.g., using proximity sensors located on different portions of the housing of the display generating component) and provide different feedback according to the position of the hover input. In some embodiments, the computing system adjusts values ​​of various characteristics of the visual feedback based on the hover distance of the detected hover input (e.g., the distance of the fingertip(s) from the surface of the housing or a touch-sensitive portion of the housing).

[0183] 7O , in some embodiments, the computing system detects that a hand 7038 or a portion thereof (e.g., a finger of the hand 7038, or two fingers of the hand 7038, etc.) has moved within a threshold distance from a first touch-sensitive portion of a housing of the display generating component 7100 (e.g., an upper portion of the left edge of the housing of the display generating component 7100, a left or top edge portion of an HMD, etc.). In response to detecting that the hand 7038 or a portion thereof has moved within the threshold distance from the first touch-sensitive portion of the housing of the display generating component 7100, the computing system optionally displays one or more user interface objects at a position corresponding to the position of the hand 7038 or portion thereof. For example, user interface object 7060 is displayed near an upper portion of the left edge of the housing of the display generating component 7100 (or a left or top edge portion of an HMD, etc.), next to the hand 7038 or a lifted finger of the hand 7038. In some embodiments, the computing system dynamically updates the position of one or more displayed user interface objects (e.g., moving user interface object 7060 closer to or farther away from the left edge of the housing and / or moving up and down the left edge of the housing) according to changes in the position of the hand 7038 or a portion thereof near the housing of the display generating component (e.g., moving closer / farther away and / or moving up and down along the left edge, etc.). In some embodiments, the computing system dynamically updates the appearance of one or more displayed user interface objects (e.g., changing size, shape, color, opacity, resolution, content, etc.) according to changes in the position of the hand or a portion thereof near the housing of the display generating component.In some embodiments, the computing system optionally changes the type of user interface object(s) displayed near a first touch-sensitive portion of the housing of the display generating component (e.g., changing from displaying a representation of a first control (e.g., volume control, power control, etc.) to displaying a representation of a second control (e.g., display brightness control, WiFi control, etc.), changing from displaying a first type of affordance (e.g., scroll bar, button, etc.) to displaying a second type of affordance (e.g., scroll wheel, switch, etc.), etc.) in accordance with changes in the posture of the hand 7038 near the housing of the display generating component (e.g., lifting two fingers instead of one finger, lifting a different finger, etc.). In some embodiments, one or more user interface objects grow from a mere indication of a control (e.g., a small dot, a faint shadow, etc.) to a more specific and clear representation of the control (e.g., a button or switch with graphics and / or text thereon that indicate the state of the computing system or the display generating component) as the hand 7038 moves near the first touch-sensitive portion of the housing of the display generating component. In some embodiments, one or more user interface objects (e.g., user interface object 7060, or other user interface objects) are displayed simultaneously with the view of the physical environment. In some embodiments, the one or more user interface objects are displayed simultaneously with the virtual environment, for example, if the virtual environment was displayed prior to detecting the hover input by hand 7038. In some embodiments, the one or more user interface objects are displayed without the simultaneous display of either the view of the physical environment or any other virtual content, for example, if the display generation component was not displaying either the view of the physical environment or any other virtual content prior to detecting the hover input.

[0184] 7O , the computing system detects a touch input at a first touch-sensitive portion on the housing of the display generation component, e.g., after detecting a hover input near a first touch-sensitive portion on the housing of the display generation component. In some embodiments, in response to detecting the touch input at a first portion of the touch-sensitive portion of the housing of the display generation component, the computing system performs a first action corresponding to the touch input in accordance with a determination that the touch input was detected without another hand being detected on the housing of the display generation component. For example, as shown in FIG. 7O , in response to detecting the touch input on a top portion of the left edge of the housing of the display generation component, the computing system activated an augmented reality experience that includes some virtual content (e.g., virtual overlay 7062) in combination with a view of the physical environment (e.g., virtual overlay 7062 is displayed at a position corresponding to the position of box 7052, such that overlay 7062 appears to be positioned in front of representation 7052′ of box 7052 in the augmented reality environment). In some embodiments, the initial touchdown of the user's hand 7038 or portion of a hand causes only visual feedback that a touch has been detected on the housing of the display generating component; the touch does not yet meet the criteria for initiating any particular action on the computing system (e.g., user interface objects (e.g., user interface object 7060) are displayed in their fully functional state if they were not displayed or were displayed merely as an indication, user interface objects (e.g., user interface object 7060) change their appearance to indicate that a touch has been detected, etc.).In some embodiments, the computing system evaluates the touch input provided by the hand 7038 against criteria for detecting one or more valid touch inputs, and in response to a determination that the touch input satisfies the criteria (e.g., criteria for detecting an upward swipe, a double-tap input, a single finger tap, etc.) for triggering a first action (e.g., decrease display brightness, decrease volume, switch to AR mode, etc.) associated with one or more user interface objects (e.g., user interface object 7060 or another user interface object displayed in place of user interface object 7060, etc.), the computing system performs the first action associated with the one or more user interface objects, and in response to a determination that the touch input satisfies the criteria (e.g., criteria for detecting a downward swipe, a single tap, a two-finger tap, etc.) for triggering a second action (e.g., increase display brightness, increase volume, switch to VR mode, etc.) associated with one or more user interface objects (e.g., user interface object 7060 or another user interface object displayed in place of user interface object 7060, etc.), the computing system performs a second action associated with the one or more user interface objects.

[0185] 7P , the computing system detects a user's hand 7038 approaching a second touch-sensitive portion of the housing of the display generating component (e.g., a lower portion of the left edge of the housing of the display generating component, a lower left edge of the HMD, a upper right edge of the HMD, etc.), and optionally displays one or more user interface objects (e.g., user interface object 7064 or other user interface objects) in response. When the computing system detects a touch input on the second touch-sensitive portion of the housing of the display generating component, in accordance with a determination that only one hand (e.g., hand 7038) was touching the housing of the display generating component when the touch input was detected, the computing system performs a second operation corresponding to the touch input performed by hand 7038 or a portion thereof. In this example, activation of control 7064 initiates an augmented reality experience that includes some virtual content (e.g., a virtual ball 7066, other virtual objects, etc.) displayed in combination with a view of the physical environment (e.g., virtual ball 7066 is displayed in a position corresponding to the position of floor 7058, such that virtual ball 7066 appears to be resting on the surface of representation 7058' of floor 7058). Other features associated with this illustrative scenario are similar to those described with respect to FIG. 7O, the only difference being that hand 7038 was detected near and / or at the first touch-sensitive portion of the housing of the display generating component, and the particular visual feedback and action(s) performed corresponded to the touch input detected on or near the first touch-sensitive portion of the housing of the display generating component. For the sake of brevity, similar features will not be repeated herein.

[0186] Figure 7Q illustrates that another hand (e.g., hand 7036) is touching (e.g., holding, supporting, otherwise contacting, etc.) the housing of a display generating component (e.g., display 7100, HMD, etc.) when touch input by hand 7038 is detected on the housing of the display generating component. In Figure 7Q, hand 7038 is touching a first touch-sensitive portion of the housing of the display generating component and a second touch-sensitive portion of the housing, respectively, providing the same touch input that previously triggered the execution of a first action (e.g., displaying virtual overlay 7062 of Figure 7O) and a second action (e.g., displaying virtual ball 7066 of Figure 7P), respectively. However, upon detecting one of the touch inputs by hand 7038, pursuant to a determination that the touch input was detected while another hand (e.g., hand 7036) was also touching the housing of the display generation component (e.g., pursuant to a determination that the other hand was touching the opposite side of the housing from the side where the touch input was detected, a determination that the other hand was touching somewhere on the housing, etc.), the computing system refrains from performing the action corresponding to the detected touch input. For example, as shown in FIG. 7Q , the computing system still optionally displays user interface objects (e.g., user interface object 7060, user interface object 7064, etc.) near the location of hand 7038 that provided the valid touch input, but does not initiate the corresponding computer-generated experience when the other hand 7036 is touching the housing of the display generation component. In some embodiments, the computing system does not display any user interface objects if hover input from two hands is detected simultaneously near a touch-sensitive portion of the housing of the display generation component and / or if touch input is detected simultaneously on two sides of the housing of the display generation component.In some embodiments, when two hands are detected simultaneously approaching and / or touching a display generating component (e.g., an HMD, other type of display generating component, etc.), it is more likely that the user will want to remove the display generating component rather than provide input to it. Thus, it is more advantageous to ignore such hover or touch input (e.g., to reduce user confusion, conserve power, etc.) without the user's explicit command.

[0187] In some embodiments, the operation performed in accordance with the touch input detected on the housing of the display generating component changes the state of the computing system. For example, in accordance with determining that the touch input satisfies a first criterion, the computing system switches to a first state, and in accordance with determining that the touch input satisfies a second criterion different from the first criterion, the computing system switches to a second state different from the first state. In some embodiments, the first criterion and the second criterion have different position-based criteria that require the touch input to be detected at different positions on the housing. In some embodiments, the first criterion and the second criterion have different intensity-based criteria that require the touch input to satisfy different intensity thresholds. In some embodiments, the first criterion and the second criterion have different duration-based criteria that require the touch input to be detected less than a threshold movement amount for different threshold times on the housing. In some embodiments, the first criterion and the second criterion have different distance-based criteria that require the touch input to move by more than different threshold distances. In some embodiments, the first criterion and the second criterion have different touch pattern criteria that require the touch input to include different numbers of contacts and / or contact movement patterns. In some embodiments, the first criterion and the second criterion have different movement criteria that require the touch input to include movement of the c...

Claims

1. 1. A computing system including a first display generating component, a second display generating component, and one or more input devices, displaying a first computer-generated environment via the first display generation component; and while displaying the first computer-generated environment via the first display generation component, via the second display generation component; a visual representation of a portion of a user of the computing system positioned to view the first computer-generated environment via the first display generation component; and one or more graphical elements that provide a visual indication of content within the first computer-generated environment; and simultaneously displaying Simultaneously displaying the visual representation of the portion of the user and the one or more graphical elements; modifying the visual representation of the portion of the user to represent a change in appearance of the user over a discrete period of time; modifying the one or more graphical elements that provide the visual indication of content within the first computer-generated environment to represent changes in the first computer-generated environment over the discrete time periods; and A method comprising:

2. modifying the visual representation of the portion of the user to represent the change in appearance of the user over the discrete time periods; displaying movement of the visual representation of the portion of the user in the first direction on the second display generating component in accordance with movement of the portion of the user in the first direction relative to the first display generating component; displaying movement of the visual representation of the portion of the user in a third direction on the second display generating component substantially opposite to the second direction in accordance with movement of the portion of the user in a second direction substantially perpendicular to the first direction relative to the first display generating component; The method of claim 1 , comprising:

3. 2. The method of claim 1 , wherein simultaneously displaying the visual representation of the portion of the user and the one or more graphical elements includes generating the visual representation of the portion of the user according to a predefined characteristic value specified for a first display characteristic of the visual representation.

4. The method of claim 3 , wherein the predefined characteristic value specified for the first display characteristic of the visual representation comprises a predefined amount of reduction in opacity of the visual representation of the portion of the user.

5. 5. The method of claim 1, wherein simultaneously displaying the visual representation of the portion of the user and the one or more graphical elements comprises applying a predefined visual effect to the visual representation of the portion of the user.

6. 6. The method of claim 1 , wherein simultaneously displaying the visual representation of the portion of the user and the one or more graphic elements via the second display generation component includes displaying a diffuse colored layer between the visual representation of the portion of the user and the one or more graphic elements that provide the visual indication of the content of the first computer-generated environment.

7. 7. The method of claim 1, wherein simultaneously displaying the visual representation of the portion of the user and the one or more graphical elements via the second display generation component comprises displaying the visual representation of the portion of the user in a first graphical resolution and displaying the one or more graphical elements providing the visual indication of the content of the first computer-generated environment in a second graphical resolution different from the first graphical resolution.

8. Simultaneously displaying the visual representation of the portion of the user and the one or more graphical elements; 8. The method of claim 1 , further comprising: modifying the visual representation of the portion of the user in accordance with the first change in the user's appearance without modifying the one or more graphical elements that provide the visual indication of the content within the first computer-generated environment in accordance with a determination that a first change in the user's appearance has been detected.

9. Simultaneously displaying the visual representation of the portion of the user and the one or more graphical elements; 9. The method of claim 1, further comprising: modifying at least one of the one or more graphical elements providing the visual indication of the content in the first computer-generated environment in accordance with the second change in the content in the first computer-generated environment without modifying the visual representation of the portion of the user in accordance with a determination that a second change in the content in the first computer-generated environment has been detected while the appearance of the user remains unchanged.

10. Simultaneously displaying the visual representation of the portion of the user and the one or more graphical elements; 10. The method of claim 1, further comprising: modifying the visual representation of the portion of the user in accordance with the third change in the user's appearance in conjunction with modifying the one or more graphical elements that provide the visual indication of the content in the first computer-generated environment in accordance with the fourth change in the content in the first computer-generated environment in accordance with a determination that a third change in the appearance of the user has been detected together with a fourth change in the content in the first computer-generated environment.

11. concurrently displaying the visual representation of the portion of the user and the one or more graphical elements includes displaying a modified version of the content concurrently with the visual representation of the portion of the user, the modified version of the content being generated using one or more image filters that, when applied to an image of the content, reduce image fidelity of the image; modifying the one or more graphical elements providing the visual indication of content within the first computer-generated environment to represent changes in the first computer-generated environment over the discrete time periods includes updating the modified version of the content by applying the one or more image filters to an image of the content as the content changes.

11. The method according to any one of claims 1 to 10.

12. The method of claim 1 , wherein the visual representation of the portion of the user includes an image of an eye of the user.

13. 12. The method of claim 1, wherein the visual representation of the portion of the user includes a virtual object generated based on one or more images of the user's eyes, the appearance and movement of the virtual object corresponding to the appearance and movement of the user's eyes captured by one or more sensors of the computing system.

14. modifying the visual representation of the portion of the user to represent the change in appearance of the user over the discrete time periods; acquiring eye tracking data corresponding to the user; updating the visual representation of the portion of the user based on the eye tracking data corresponding to the user; and 14. The method of any one of claims 1 to 13, comprising:

15. 1. A computing system including a first display generating component, a second display generating component, and one or more input devices, displaying a computer-generated environment via the first display generation component; and displaying state information corresponding to the computing system via the second display generation component while displaying the computer-generated environment via the first display generation component; a visual representation of a portion of a user of the computing system positioned to view the computer-generated environment via the first display generation component; and one or more graphical elements that provide a visual indication of content within the computer-generated environment; and Detecting individual events; In response to detecting the individual event, Varying an immersion level of the computer-generated environment displayed via the first display generation component; and modifying the state information displayed via the second display generation component, including modifying an appearance of the visual representation of the portion of the user of the computing system; A method comprising:

16. the computing system is configured to display the computer-generated environment at least a first immersive level, a second immersive level, and a third immersive level. The method of claim 15.

17. modifying the immersion level of the computer-generated environment displayed via the first display generation component in response to detecting the distinct event. switch from displaying the computer-generated environment at the second level of immersion to displaying the computer-generated environment at the first level of immersion in response to determining that the individual event is an event that satisfies a first criterion; switch from displaying the computer-generated environment at the second level of immersion to displaying the computer-generated environment at the third level of immersion in response to a determination that the individual event is an event that satisfies second criteria different from the first criteria; 17. The method of claim 16, comprising:

18. changing the immersion level of the computer-generated environment displayed via the first view generation component includes switching from displaying the computer-generated environment at a second immersion level to displaying the computer-generated environment at a first immersion level, wherein the computer-generated environment displayed at the first immersion level provides a view of the physical environment having less than a threshold amount of computer-generated content; modifying the state information displayed via the second display generation component includes switching from displaying the visual representation of the portion of the user of the computing system with the one or more graphical elements providing the visual indication of the content within the computer-generated environment to displaying the visual representation of the portion of the user without the one or more graphical elements in conjunction with switching from displaying the computer-generated environment at the second immersive level to displaying the computer-generated environment at the first immersive level.

18. The method according to any one of claims 15 to 17.

19. 19. The method of claim 15, wherein the computing system simultaneously displays the visual representation of the portion of the user and the one or more graphical elements providing a visual indication of content within the computer-generated environment in accordance with a determination that the computer-generated environment is a mixed reality environment including a representation of a physical environment surrounding the first display generation component and at least a threshold amount of virtual objects.

20. changing the immersion level of the computer-generated environment displayed via the first view generation component includes switching from displaying the computer-generated environment at a second immersion level to displaying the computer-generated environment at a third immersion level, wherein the computer-generated environment displayed at the third immersion level provides a virtual environment having a representation of less than a threshold amount of the physical environment; modifying the state information displayed via the second display generation component includes switching from displaying the visual representation of the portion of the user of the computing system with the one or more graphical elements providing the visual indication of the content within the computer-generated environment to displaying the one or more graphical elements without the visual representation of the portion of the user in conjunction with switching from displaying the computer-generated environment at the second immersive level to displaying the computer-generated environment at the third immersive level.

20. The method according to any one of claims 15 to 19.

21. detecting a first user request to activate a privacy mode of the computing device while displaying the computer-generated environment via the first display generation component and displaying the state information corresponding to the computing system via the second display generation component, the request requiring the one or more graphical elements displayed via the second display generation component to have visibility below a first threshold visibility; In response to detecting the first user request, pursuant to a determination that the first user request was received while visibility of the one or more graphical elements providing a visual indication of content within the computer-generated environment was above the first threshold visibility corresponding to the privacy mode, reducing the visibility of the one or more graphical elements on the second display generation component below the first threshold visibility corresponding to the privacy mode; maintaining the visibility of the one or more graphical elements providing a visual indication of content within the computer-generated environment below the first threshold visibility corresponding to the privacy mode pursuant to a determination that the first user request has been received while the visibility of the one or more graphical elements does not exceed the first threshold visibility corresponding to the privacy mode; 21. The method of any one of claims 15 to 20, comprising:

22. The privacy mode requires that the visual representation of the portion of the user displayed via the second display generation component have visibility below a second threshold visibility, and the method further comprises: In response to detecting the first user request, reducing the visibility of the visual representation of the portion of the user on the second display generation component below the second threshold visibility corresponding to the privacy mode in accordance with a determination that the first user request was received while visibility of the visual representation of the portion of the user was above the second threshold visibility corresponding to the privacy mode; maintaining the visibility of the visual representation of the portion of the user below the second threshold visibility corresponding to the privacy mode in accordance with a determination that the first user request was received while the visibility of the visual representation of the portion of the user did not exceed the second threshold visibility corresponding to the privacy mode; 22. The method of claim 21 , comprising:

23. detecting a second discrete event that changes an immersive level of the computer-generated environment displayed via the first display generation component while the privacy mode is active on the computing system; and in response to detecting the second discrete event and the corresponding change in the immersion level of the computer-generated environment displayed via the first display generation component, ceasing to modify the state information displayed via the second display generation component; 23. The method of claim 21 or 22, comprising:

24. detecting a second user request to activate a silent (DND) mode of the computing device while displaying the computer-generated environment via the first display generation component; In response to detecting the second user request, displaying a visual indicator via the second display generation component to indicate that the DND mode is active; and 24. The method of any one of claims 15 to 23, comprising:

25. detecting a third distinct event that alters an immersion level of the computer-generated environment displayed via the first display generation component while the DND mode is active on the computing system; and in response to detecting the third discrete event and the corresponding change in the immersion level of the computer-generated environment displayed via the first display generation component, ceasing to modify the state information displayed via the second display generation component; 25. The method of claim 24, comprising:

26. detecting, while displaying the computer-generated environment via the first display generation component, a third user request to activate a parental control mode of the computing device, the parental control mode requiring the one or more graphical elements displayed via the second display generation component to have visibility greater than a third threshold visibility; In response to detecting the third user request, increasing the visibility of the one or more graphical elements on the second display generation component above the third threshold visibility corresponding to the parental control mode in accordance with a determination that the third user request was received while visibility of the one or more graphical elements providing a visual indication of content within the computer-generated environment was less than the third threshold visibility corresponding to the parental control mode; 26. The method of any one of claims 15 to 25, comprising:

27. detecting a fourth discrete event that alters an immersive level of the computer-generated environment displayed via the first display generation component while the parental control mode is active on the computing system; and in response to detecting the fourth discrete event and the corresponding change in the immersion level of the computer-generated environment displayed via the first display generation component, ceasing to modify the state information displayed via the second display generation component; 27. The method of claim 26, comprising:

28. 28. The method of any one of claims 15 to 27, wherein simultaneously displaying the visual representation of the portion of the user and the one or more graphical elements comprises displaying the visual representation of the portion of the user at a first depth and displaying the one or more graphical elements at a second depth less than the first depth from an external perspective of the state information displayed via the second display generation component.

29. 29. The method of any one of claims 15 to 28, wherein the one or more graphical elements providing a visual indication of content within the computer-generated environment include at least a progress indicator that indicates a progress of the content within the computer-generated environment as displayed via the first display generation component.

30. 30. The method of claim 15, wherein the one or more graphical elements providing a visual indication of content within the computer-generated environment have a first display characteristic, the value of the first display characteristic being based on the value of the first display characteristic of the content within the computer-generated environment.

31. 31. The method of claim 15, wherein the one or more graphical elements that provide a visual indication of content within the computer-generated environment comprise one or more sub-portions of the content of the computer-generated environment.

32. 32. The method of claim 15, wherein the one or more graphical elements that provide a visual indication of content within the computer-generated environment include metadata that identifies the content of the computer-generated environment.

33. Displaying the one or more graphical elements that provide a visual indication of content within the computer-generated environment; detecting a first movement of a second user relative to the second display generation component in a position to view the status information displayed via the second display generation component; in response to detecting the first movement of the second user relative to the second display generation component, updating a display of the one or more graphical elements in accordance with a determination that a distance between the second user and the second display generation component has decreased from above a first threshold distance to below the first threshold distance to increase an information density of the visual indication of content within the computer-generated environment provided by the one or more graphical elements; 33. The method of any one of claims 15 to 32, comprising:

34. while displaying the computer-generated environment via the first display generation component and displaying the state information corresponding to the computing system via the second display generation component, detecting a fifth separate event triggered by a third user in a position to view the state information displayed via the second display generation component; in response to detecting the fifth separate event, in accordance with a determination that the fifth separate event satisfies a fourth criterion requiring that, for the fourth criterion to be satisfied, a preset measure of interaction has increased from below a preset threshold to above the preset threshold as a result of the fifth separate event and that the computer-generated environment is displayed at a third immersion level; changing an immersion level of the computer-generated environment displayed via the first display generation component from the third immersion level to a second immersion level, where the computer-generated environment displayed at the second immersion level includes a representation of an increased amount of the physical environment than the computer-generated environment displayed at the third immersion level; 34. The method of any one of claims 15 to 33, comprising:

35. in response to detecting the fifth separate event, in accordance with a determination that the fifth separate event satisfies the fourth criterion; modifying the state information displayed via the second display generation component, comprising increasing visibility of the visual representation of the portion of the user of the computing system in conjunction with modifying the immersion level of the computer-generated environment displayed via the first display generation component from the third immersion level to the second immersion level; 35. The method of claim 34, comprising:

36. 36. The method of claim 35, wherein detecting the fifth distinct event comprises detecting entry of the third user into a predefined area surrounding the user of the computing system.

37. 37. The method of claim 35 or 36, wherein detecting the fifth distinct event comprises detecting the third user performing a predefined gesture towards the user of the computing system.

38. 1. A computing system including a first display generating component, a second display generating component, and one or more input devices, displaying one or more user interface elements via the second display generation component; detecting that the first display generating component is moved in a predetermined orientation relative to a distinct portion of a user while displaying the one or more user interface elements via the second display generating component; In response to detecting that the first display generation component has been moved in the predetermined orientation relative to the respective portion of the user, pursuant to a determination that the computing system was in a first state corresponding to displaying a first user interface element via the second display generation component when the first display generation component was moved in the predetermined orientation relative to the respective portion of the user, displaying a first user interface element via the first display generation component while the first display generation component is in the predetermined orientation relative to the respective portion of the user; displaying, in accordance with a determination that the computing system was in a second state corresponding to displaying a second user interface element via the second display generation component when the first display generation component was moved in the predetermined orientation relative to the respective portion of the user, a second user interface element different from the first user interface via the first display generation component while the first display generation component is in the predetermined orientation relative to the respective portion of the user; A method comprising:

39. In response to detecting that the first display generation component has been moved in the predetermined orientation relative to the respective portion of the user, displaying, via the first display generation component while the first display generation component is in the predetermined orientation relative to the respective portion of the user, a third user interface element distinct from the first user interface and the second user interface in accordance with a determination that the computing system was in a third state corresponding to displaying a third user interface element via the second display generation component when the first display generation component was moved in the predetermined orientation relative to the respective portion of the user; 39. The method of claim 38, comprising:

40. 40. The method of claim 38 or 39, wherein the first user interface is a first three-dimensional environment including a first virtual content and the second user interface is a second three-dimensional environment including a second virtual content different from the first virtual content.

41. 40. The method of claim 38 or 39, wherein the first user interface is a first three-dimensional environment including a first virtual content without a representation of a physical environment surrounding the first display generating component, and the second user interface is a second three-dimensional environment different from the first three-dimensional environment including a second virtual content without a representation of the physical environment surrounding the first display generating component.

42. 40. The method of claim 38 or 39, wherein the first user interface is a first three-dimensional environment including a first virtual content along with a representation of a physical environment surrounding the first display generating component, and the second user interface is a second three-dimensional environment different from the first three-dimensional environment including a second virtual content along with a representation of the physical environment surrounding the first display generating component.

43. 40. The method of claim 38 or 39, wherein the first user interface is a first three-dimensional environment and the second user interface is a second three-dimensional environment, one of the first three-dimensional environment and the second three-dimensional environment includes a first virtual content along with a representation of a physical environment surrounding the first display generation component, and one of the first three-dimensional environment and the second three-dimensional environment includes a second virtual content without a representation of the physical environment surrounding the first display generation component.

44. displaying the one or more user interface elements includes displaying a calendar element; the first user interface and the second user interface are respective user interfaces corresponding to the calendar; 44. The method of any one of claims 38 to 43.

45. the computing system in the first state detects a first event corresponding to the calendar and displays a first notification corresponding to the first event as part of the one or more user interface elements; the first user interface is a user interface responsive to the first event that includes more details of the first event than the first notification; 45. The method of claim 44.

46. displaying the one or more user interface elements includes displaying a visual representation of a respective media item; the first user interface and the second user interface play content of the media item; 44. The method of any one of claims 38 to 43.

47. the computing system in the first state detects a third event corresponding to availability of a first media item, causing the computing system to display a first visual representation of the first media item as part of the one or more user interface elements; the first user interface is a user interface for playing the first media item; 45. The method of claim 44.

48. displaying the one or more user interface elements includes displaying a notification of a respective incoming communication request; the first user interface and the second user interface are communication interfaces responsive to the respective incoming communication requests; 44. The method of any one of claims 38 to 43.

49. the computing system in the first state detects a fifth event corresponding to a first communication request, and the computing system displays a first visual representation of the first communication request as part of the one or more user interface elements; the first user interface is a user interface that provides a first communication session corresponding to the first communication request; 49. The method of claim 48.

50. Displaying the one or more user interface elements comprises: customizing a display of the one or more user interface elements according to a first set of rules corresponding to the first identity in accordance with a determination that the user has a first identity; customizing a display of the one or more user interface elements according to a second set of rules corresponding to the second identification, the second set of rules being different from the first set of rules, in accordance with a determination that the user has a second identification that is different from the first identification; 50. The method of any one of claims 38 to 49, comprising:

51. displaying, via the second display generation component, selectable options corresponding to a set of two or more modes of the computing system in which a respective one of the first user interface and the second user interface is available for presentation via the first display generation component before detecting that the first display generation component has been moved in the predetermined orientation relative to the respective portion of the user; detecting a user input selecting a respective selectable option corresponding to one of the set of two or more modes of the computing system; In response to detecting the user input, pursuant to determining that the first display generation component has been moved to the predetermined orientation relative to the respective portion of the user, displaying the respective one of the first user interface and the second user interface via the first display generation component in the mode corresponding to the respective selectable option while the first display generation component is in the predetermined orientation relative to the respective portion of the user; 51. The method of any one of claims 38 to 50, comprising:

52. 52. The method of any one of claims 38 to 51, wherein the one or more user interface elements include a preview of a three-dimensional experience available for display via the first display generating component when the first display generating component is moved in the predetermined orientation relative to the respective portion of the user.

53. 1. A computing system including a first display generating component and one or more input devices, comprising: Detecting a first trigger event corresponding to the first display generation component being arranged in a first default configuration with respect to a user; In response to detecting the first trigger event, providing a first computer-generated experience via the first display generation component in accordance with a determination that the computing system including the first display generation component is worn by the user while in the first default configuration for the user; providing, via the first display generation component, a second computer-generated experience distinct from the first computer-generated experience in accordance with a determination that the computing system including the first display generation component is not worn by the user while in the first default configuration for the user; A method comprising:

54. pursuant to a determination that the computing system including the first display generating component is worn by the user while in the first default configuration for the user, the computing system provides at least a first input manner based on movement of the user's hand that is not in contact with the first display generating component or a housing of the first display generating component for interacting with the first computer-generated experience; in accordance with a determination that the computing system including the first display generating component is not worn by the user while in the first default configuration for the user, the computing system does not provide the first input manner based on movement of the user's hand that is not in contact with the first display generating component or the housing of the first display generating component to interact with the second computer-generated experience.

54. The method of claim 53.

55. pursuant to a determination that the computing system including the first display generating component is not worn by the user while in the first default configuration for the user, the computing system provides at least a second input manner based on movement of at least one of the user's hand in contact with the first display generating component or a housing of the first display generating component to interact with the second computer-generated experience.

55. The method of claim 53 or 54.

56. in accordance with a determination that the computing system including the first display generating component is worn by the user while in the first default configuration for the user, the computing system disables the second input method for at least one input device disposed on the first display generating component or the housing of the first display generating component for interacting with the first computer-generated experience.

56. The method of claim 55.

57. the computing system provides first virtual content to both the first computer-generated experience and the second computer-generated experience; the computing system enables a first interaction scheme for the first virtual content when the first virtual content is provided to the first computer-generated experience; the computing system enables a second interaction manner, different from the first interaction manner, for the first virtual content when the first virtual content is provided to the second computer-generated experience.

57. The method of any one of claims 53 to 56.

58. the first computer-generated experience includes a plurality of user interface objects corresponding to different computer-generated experiences, each of the plurality of user interface objects, when activated by user input, causing the computing system to launch a respective computer-generated experience corresponding to the user interface object; the second computer-generated experience is a camera user interface for controlling a camera of the computing system.

58. The method of any one of claims 53 to 57.

59. pursuant to a determination that the computing system including the first display generating component is worn by the user while in the first default configuration for the user, the computing system enables interaction with the first computer-generated experience based on a gesture performed by a hand away from the first display generating component and a housing of the first display generating component; and in response to a determination that the computing system including the first display generating component is not worn by the user while in the first default configuration for the user, the computing system enables interaction with the second computer-generated experience based on a gesture performed by the first display generating component or a hand on the housing of the first display generating component.

59. The method of any one of claims 53 to 58.

60. 1. A computing system including a first display generating component and one or more input devices, comprising: displaying a visual indication that a computer-generated experience corresponding to the physical object is available for display via the first display generation component while displaying a representation of the physical object in a location within the three-dimensional environment that corresponds to a location of the physical object within the physical environment; detecting an interaction with the physical object in the physical environment while displaying the visual indication that a computer-generated experience is available for display via the first display generation component; in response to detecting the interaction with the physical object in the physical environment, displaying, via the first display generation component, the computer-generated experience corresponding to the physical object in accordance with a determination that the interaction with the physical object in the physical environment satisfies a first criterion corresponding to the physical object; and withdrawing from displaying the computer-generated experience corresponding to the physical object in accordance with a determination that the interaction with the physical object in the physical environment does not satisfy the first criterion; and A method comprising:

61. modifying a view of the three-dimensional environment displayed via the first view generation component, including switching from displaying a representation of a first portion of the physical environment to displaying a representation of a second portion of the physical environment; in response to changing the view of the three-dimensional environment, displaying the visual indication at a location within the view of the three dimensional environment that corresponds to the location of the physical object in the physical environment in accordance with a determination that the first portion of the physical environment includes less than the threshold portion of the physical object and the second portion of the physical environment includes more than the threshold portion of the physical object; ceasing to display the visual indication within the view of the three dimensional environment in accordance with a determination that the first portion of the physical environment includes more than the threshold portion of the physical object and the second portion of the physical environment includes less than the threshold portion of the physical object; 61. The method of claim 60, comprising:

62. in response to detecting the interaction with the physical object in the physical environment, displaying, via the first display generation component, visual feedback indicative of availability of the computer-generated experience, the visual feedback being distinct from the visual indication and distinct from the computer-generated experience, in accordance with a determination that the interaction with the physical object in the physical environment satisfies a second criterion corresponding to the physical object that is distinct from the first criterion corresponding to the physical object; 62. The method of claim 60 or 61, comprising:

63. 63. The method of claim 62, wherein the visual feedback provides guidance regarding a type of interaction with the physical object that meets the first criteria.

64. 64. The method of claim 62 or 63, wherein a position of the visual feedback within the three dimensional environment is selected based on a current position of the physical object while the physical object is moving in response to the interaction with the physical object in the physical environment.

65. 65. The method of any one of claims 62 to 64, wherein a value of a first characteristic of the visual feedback is adjusted according to a value of a second characteristic of the interaction with the physical object in the physical environment.

66. a first amount of change in a first characteristic of the interaction with the physical object in the physical environment causes a first amount of change in a value of a second characteristic of the visual feedback; a second amount of change in the first characteristic of the interaction with the physical object in the physical environment causes a second amount of change in the value of the second characteristic of the visual feedback.

66. The method of any one of claims 62 to 65.

67. displaying, via the first display generation component, the computer-generated experience corresponding to the physical object; displaying a first version of the computer-generated experience corresponding to a first predetermined input modality in accordance with determining that the interaction with the physical object corresponds to a first predetermined input modality; displaying a second version of the computer-generated experience corresponding to the second predetermined input modality in accordance with determining that the interaction with the physical object corresponds to a second predetermined input modality; and 67. The method of any one of claims 60 to 66, comprising:

68. 68. The method of any one of claims 60 to 67, wherein the computer-generated experience is a virtual reality experience, and wherein displaying the computer-generated experience comprises ceasing to display the representations of the physical objects in the three-dimensional environment.

69. 68. The method of any one of claims 60 to 67, wherein the computer-generated experience is an augmented reality experience, and displaying the computer-generated experience comprises maintaining a display of the representation of the physical object within the three-dimensional environment.

70. 1. A computing system including a housing, a first display generating component contained within the housing, and one or more input devices, detecting a first hand on the housing containing the first display generating component; in response to detecting the first hand on the housing containing the first display generating component; In conjunction with a second hand being detected on the housing, ceasing to perform a motion associated with the first hand in response to a determination that the first hand has been detected; performing an action associated with the first hand in response to a determination that the first hand is detected on the housing without detecting another hand on the housing; and A method comprising:

71. The computing system has a plurality of motion states and performs the motion associated with the first hand. performing a first operation, in accordance with a determination that the first hand on the housing satisfies a first criterion, to alter an operational state of the computing system in a first manner corresponding to the first criterion; performing a second operation, in accordance with a determination that the first hand on the housing satisfies a second criterion different from the first criterion, to change the operational state of the computing system in a second manner corresponding to the second criterion different from the first manner; 71. The method of claim 70, comprising:

72. the first criterion includes a first position criterion that requires the first hand to be detected on a first predetermined portion of the housing for the first position criterion to be satisfied; the second criteria includes a second position criterion that requires the first hand to be detected on a second predetermined portion of the housing for the second position criterion to be satisfied, the second predetermined portion of the housing being different from the first predetermined portion of the housing.

72. The method of claim 70 or 71.

73. the first criterion includes a first movement criterion that requires the first hand to make a first movement relative to the housing for the first criterion to be met; the second criterion includes a second movement criterion that requires the first hand to make a second movement relative to the housing for the second criterion to be met, the second movement being different from the first movement; 73. The method of any one of claims 70 to 72.

74. the first criterion includes a first posture criterion requiring the first hand to be in a first predetermined posture while on the housing for the first criterion to be met; the second criterion includes a second posture criterion requiring the first hand to be in a second predetermined posture while on the housing for the second criterion to be met, the second predetermined posture being different from the first predetermined posture; 74. The method of any one of claims 70 to 73.

75. in response to detecting the first hand on the housing that houses the first display generating component, and in accordance with a determination that the first hand has been detected on the housing without detecting another hand on the housing, performing the action associated with the first hand. changing an operational state of the computing system from the first operational state to a second operational state different from the first operational state in accordance with a determination that a visual coordination criterion is satisfied, the visual coordination criterion including a respective position criterion and a respective posture criterion to be satisfied in conjunction with one another by the first hand while the computing system is operating in the first operational state such that the visual coordination criterion is satisfied, wherein the first display generation component provides an increased view of a physical environment surrounding the first display generation component while the computing system is in the second operational state compared to when the computing system was in the first operational state.

75. The method of any one of claims 70 to 74.

76. detecting, while the computing system is in the second operating state, that the visual adjustment criterion is no longer satisfied by the first hand; Restoring the operational state of the computing system from the second operational state to the first operational state in response to detecting that the visual adjustment criterion is no longer satisfied by the first hand; 76. The method of claim 75, comprising:

77. in response to detecting the first hand on the housing that houses the first display generating component, and in accordance with a determination that the first hand has been detected on the housing without detecting another hand on the housing, performing the action associated with the first hand. changing an operational state of the computing system from the third operational state to a fourth operational state different from the third operational state in accordance with a determination that an audio adjustment criterion is satisfied, the fourth operational state being different from the third operational state, such that a first audio generating component of the computing system provides an increased amount of pass-through sound from a physical environment surrounding the first audio generating component while the computing system is in the fourth operational state compared to when the computing system was in the third operational state.

77. The method of any one of claims 70 to 76.

78. detecting, while the computing system is in the fourth operating state, that the audio adjustment criteria is no longer satisfied by the first hand; Restoring the operating state of the computing system from the fourth operating state to the third operating state in response to detecting that the audio adjustment criteria is no longer satisfied by the first hand; 78. The method of claim 77, comprising:

79. in response to detecting a first hand on the housing containing the first display generating component and in accordance with a determination that the first hand has been detected on the housing without detecting another hand on the housing, performing the action associated with the first hand.

79. The method of any one of claims 70 to 78, further comprising: changing an operational state of the computing system from a fifth operational state to a sixth operational state different from the fifth operational state in accordance with a determination that a privacy adjustment criterion is satisfied, the sixth operational state being different from the fifth operational state, and a second display generation component of the computing system switching between providing a first amount of state information regarding content displayed via the first display generation component and providing a second amount of state information regarding the content displayed via the first display generation component.

80. 80. The method of claim 79, wherein the state information also includes a representation of the user's eyes facing a content display surface of the first display generation component, and changing the operational state of the computing system from the fifth operational state to the sixth operational state different from the fifth operational state includes switching between displaying the representation of the user's eyes and not displaying the representation of the user's eyes via the first display generation component.

81. detecting, while the computing system is in the sixth operational state, that the privacy adjustment criteria is no longer satisfied by the first hand; maintaining the operational state of the computing system in the sixth operational state in response to detecting that the privacy adjustment criteria is no longer satisfied by the first hand; 81. The method of claim 79 or 80, comprising:

82. detecting that the privacy adjustment criteria is again satisfied by the first hand while the computing system is in the sixth operational state; changing the operational state of the computing system from the sixth operational state to the fifth operational state in response to detecting that the privacy adjustment criteria is again satisfied while the computing system is in the sixth operational state, wherein the second display generation component of the computing system reverses the switching between providing the first amount of state information regarding the content displayed via the first display generation component and providing the second amount of state information regarding the content displayed via the first display generation component in response to the privacy adjustment criteria being last satisfied; 82. The method of any one of claims 79 to 81, comprising:

83. changing the operational state of the computing system from the fifth operational state to the sixth operational state in accordance with the determination that the privacy adjustment criteria is satisfied; In accordance with a determination that the first hand moves in a first direction while satisfying the individual movement criterion and the individual posture criterion of the privacy adjustment criterion, the second display generation component increases an amount of state information related to the content displayed via the first display generation component; In response to a determination that the first hand moves in a second direction different from the first direction, the second display generation component decreases the amount of state information related to the content displayed via the first display generation component; 82. The method of any one of claims 79 to 81, comprising:

84. displaying, via the first display generating component, a respective visual representation corresponding to one or more functions of the computing system in response to detecting the first hand on the housing that houses the first display generating component; 84. The method of any one of claims 70 to 83, comprising:

85. responsive to detecting the first hand approaching the housing enclosing the first display generation component, displaying, via the first display generation component, an animated transition illustrating the respective visual representations corresponding to the one or more functions of the computing system emerging from a peripheral region of the field of view provided by the first display generation component; 85. The method of claim 84, comprising:

86. performing the action associated with the first hand in response to the determination that the first hand is detected in conjunction with not detecting another hand on the housing. activating a first function of the computing system in accordance with a determination that the first hand contacts the housing at a first position corresponding to a first visual representation corresponding to a first function of the computing system and the first hand satisfies a respective control activation criterion associated with the first function; activating a second function of the computing system in accordance with a determination that the first hand contacts the housing at a second position corresponding to a second visual representation corresponding to a second function of the computing system and the first hand satisfies a respective control activation criterion associated with the second function; 86. The method of claim 84 or 85, comprising:

87. A non-transitory computer-readable storage medium storing executable instructions that, when executed by a computing system having one or more processors, a first display generating component, and a second display generating component, cause the computing system to perform the method of any one of claims 1 to 14.

88. one or more processors; A first display generation component; A second display generating component; 15. A computing system comprising: a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing a method according to any one of claims 1 to 14.

89. 1. A computing system having one or more processors, a first display generating system, and a second display generating component, comprising: A computing system comprising means for carrying out the method of any one of claims 1 to 14.

90. 1. An information processing apparatus for use in a computing system having one or more processors, a first display generating component, and a second display generating component, comprising: An information processing device comprising means for carrying out the method according to any one of claims 1 to 14.

91. A non-transitory computer-readable storage medium storing executable instructions that, when executed by a computing system having one or more processors, a first display generating component, and a second display generating component, cause the computing system to perform the method of any one of claims 15 to 37.

92. one or more processors; A first display generation component; A second display generating component; A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing a method according to any one of claims 15 to 37; A computing system comprising:

93. 1. A computing system having one or more processors, a first display generating system, and a second display generating component, comprising: Means for carrying out the method according to any one of claims 15 to 37. A computing system comprising:

94. 1. An information processing apparatus for use in a computing system having one or more processors, a first display generating component, and a second display generating component, comprising:

38. An information processing device comprising means for carrying out the method according to any one of claims 15 to 37.

95. 53. A non-transitory computer-readable storage medium storing executable instructions that, when executed by a computing system having one or more processors, a first display generating component, and a second display generating component, cause the computing system to perform the method of any one of claims 38 to 52.

96. one or more processors; A first display generation component; A second display generating component; A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing a method according to any one of claims 38 to 52; and A computing system comprising:

97. 1. A computing system having one or more processors, a first display generating system, and a second display generating component, comprising:

53. A computing system comprising means for carrying out the method of any one of claims 38 to 52.

98. 1. An information processing apparatus for use in a computing system having one or more processors, a first display generating component, and a second display generating component, comprising:

53. An information processing device comprising means for carrying out a method according to any one of claims 38 to 52.

99. 60. A non-transitory computer-readable storage medium storing executable instructions that, when executed by a computing system having one or more processors, a first display generating component, and a second display generating component, cause the computing system to perform the method of any one of claims 53 to 59.

100. one or more processors; A first display generation component; A second display generating component; A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing a method according to any one of claims 53 to 59; and A computing system comprising:

101. 1. A computing system having one or more processors, a first display generating system, and a second display generating component, comprising:

60. A computing system comprising means for carrying out the method of any one of claims 53 to 59.

102. 1. An information processing apparatus for use in a computing system having one or more processors, a first display generating component, and a second display generating component, comprising:

60. An information processing device comprising means for carrying out a method according to any one of claims 53 to 59.

103. A non-transitory computer-readable storage medium storing executable instructions that, when executed by a computing system having one or more processors and a first display generation component, cause the computing system to perform the method of any one of claims 60 to 69.

104. one or more processors; A first display generation component; A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing a method according to any one of claims 60 to 69; A computing system comprising:

105. 1. A computing system having one or more processors and a first display generating system, comprising:

70. A computing system comprising means for carrying out the method of any one of claims 60 to 69.

106. 1. An information processing apparatus for use in a computing system having one or more processors and a first display generation component, comprising:

70. An information processing apparatus comprising means for carrying out a method according to any one of claims 60 to 69.

107. A non-transitory computer-readable storage medium storing executable instructions that, when executed by a computing system having one or more processors and a first display generation component, cause the computing system to perform the method of any one of claims 70 to 86.

108. one or more processors; A first display generation component; A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing a method according to any one of claims 70 to 86; and A computing system comprising:

109. 1. A computing system having one or more processors and a first display generating system, comprising:

87. A computing system comprising means for carrying out the method of any one of claims 70 to 86.

110. 1. An information processing apparatus for use in a computing system having one or more processors and a first display generation component, comprising:

87. An information processing device comprising means for carrying out a method according to any one of claims 70 to 86.

Images