Device and method for accessing system functions of computer system when displaying three-dimensional environment

Abstract

The present disclosure relates to devices, methods, and graphical user interfaces for accessing system functions of a computer system while displaying a three-dimensional environment. A computer system detects the occurrence of an event. In response, the computer system displays an indication of a notification corresponding to the event. After displaying the indication of the notification: upon determining that the user's attention was pointed to the indication within a threshold amount of time, then moved away from the first indication before a first criterion is met, the computer system maintaining the display of the indication for a first duration and ceasing the display of the indication after expiration of the first duration; and in accordance with a determination that the attention of the user of the computer system has not been directed to the indication within the first threshold amount of time, the computer system maintains display of the indication for a second duration and stops display of the indication after expiration of the second duration.

Description

[0001] This application is a divisional application of Chinese Patent Application No. 202380022417.5, filed on February 16, 2023, entitled "Apparatus, Method and Graphical User Interface for Accessing System Functions of a Computer System When Displaying a Three-Dimensional Environment".

[0002] Related patent applications

[0003] This patent application is a continuation to U.S. Patent Application Serial No. 18 / 110,323, filed February 15, 2023, which claims priority to U.S. Provisional Patent Application Serial No. 63 / 445,698, filed February 14, 2023; U.S. Provisional Patent Application Serial No. 63 / 409,594, filed September 23, 2022; and U.S. Provisional Patent Application Serial No. 63 / 310,970, filed February 16, 2022, each of which is incorporated herein by reference in its entirety. Technical Field

[0004] This disclosure relates in its entirety to computer systems that provide computer-generated experiences in communication with display generation components and optionally with one or more input devices, including but not limited to electronic devices that provide virtual reality and mixed reality experiences via a display. Background Technology

[0005] In recent years, research and development of computer systems for augmented reality has increased significantly. Example augmented reality environments include at least some virtual elements that replace or enhance the physical world. Input devices used in computer systems and other electronic computing devices (such as cameras, controllers, joysticks, touch-sensitive surfaces, and touchscreen displays) are used to interact with the virtual / augmented reality environment. Example virtual elements include virtual objects such as digital images, videos, text, icons, and control elements (such as buttons and other graphics). Summary of the Invention

[0006] Some methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) are cumbersome, inefficient, and limited. For example, systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to achieve a desired result in an augmented reality environment, and systems where the manipulation of virtual objects is complex, tedious, and error-prone, impose a significant cognitive burden on users and detract from the immersive experience of virtual / augmented reality environments. Furthermore, these methods take longer than necessary, thus wasting the energy of the computer system. This latter consideration is particularly important in battery-powered devices.

[0007] Therefore, there is a need for computer systems with improved methods and interfaces that provide users with computer-generated experiences, thereby making user interaction with computer systems more efficient and intuitive for users. Such methods and interfaces optionally complement or replace conventional methods for providing users with extended reality experiences. These methods and interfaces create more efficient human-computer interfaces by helping users understand the relationship between the inputs provided and the device's responses to those inputs, thereby reducing the quantity, extent, and / or nature of user input.

[0008] The disclosed system reduces or eliminates the aforementioned defects and other problems associated with the user interface of a computer system. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is a portable device (e.g., a laptop, tablet, or handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device, such as a watch or head-mounted device). In some embodiments, the computer system has a touchpad. In some embodiments, the computer system has one or more cameras. In some embodiments, the computer system has a touch-sensitive display (also referred to as a "touchscreen" or "touchscreen display"). In some embodiments, the computer system has one or more eye-tracking components. In some embodiments, the computer system has one or more hand-tracking components. In some embodiments, in addition to display generation components, the computer system also has one or more output devices, including one or more haptic output generators and / or one or more audio output devices. In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, and a program or instruction set stored in memory for performing multiple functions. In some implementations, the user interacts with the GUI through contact and gestures of a stylus and / or fingers on a touch-sensitive surface, the user's eyes and hands in space relative to the GUI (and / or computer system) or movement of the user's body (such as captured by a camera and other motion sensors) and / or voice input (such as captured by one or more audio input devices). In some implementations, functions performed through interaction optionally include image editing, drawing, presentations, word processing, spreadsheet creation, playing games, making and receiving phone calls, video conferencing, sending and receiving emails, sending and receiving instant messages, fitness support, digital photography, digital video recording, web browsing, digital music playback, note-taking, and / or digital video playback. Executable instructions for performing these functions are optionally included in transient and / or non-transitory computer-readable storage media or other computer program products configured for execution by one or more processors.

[0009] There is a need for electronic devices with improved methods and interfaces for interacting with a computer system displaying one or more parts of a 3D environment. Such methods and interfaces can complement or replace conventional methods for interacting with a computer system displaying one or more parts of a 3D environment. These methods and interfaces reduce the amount, extent, and / or nature of input from the user, resulting in a more efficient human-computer interface. For battery-powered computing devices, such methods and interfaces save power and increase the time interval between battery charging.

[0010] In some embodiments, a computer system displays a first user interface object via a first display generation component when a first view of a three-dimensional environment is visible, wherein the first user interface object is displayed at a first location in the three-dimensional environment, and wherein the first location in the three-dimensional environment has a first spatial arrangement relative to a corresponding portion of the user. While displaying the first user interface object in the first view of the three-dimensional environment, the computer system detects input via one or more input devices corresponding to a movement of the user's viewpoint in the three-dimensional environment from a first viewpoint to a second viewpoint. In response to detecting the input corresponding to the movement of the user's viewpoint in the three-dimensional environment from the first viewpoint to the second viewpoint, the computer system maintains the display of the first user interface object at the corresponding location in the three-dimensional environment with the first spatial arrangement relative to the corresponding portion of the user in the three-dimensional environment when a second view of the three-dimensional environment is visible. When the second view of the three-dimensional environment is visible and when displaying the first user interface object in the second view of the three-dimensional environment, the computer system detects a first gaze input pointing to the first user interface object via the one or more input devices. In response to detecting a first gaze input directed at the first user interface object, and based on determining that the first gaze input meets an attention criterion with respect to the first user interface object, the computer system displays multiple indications for accessing system functions of the first computer system.

[0011] In some embodiments, a computer system displays a first user interface object via a first display generation component when a first view of a three-dimensional environment is visible. The first user interface object is displayed at a first location in the three-dimensional environment, and the first location in the three-dimensional environment has a first spatial arrangement relative to a corresponding portion of the user. While displaying the first user interface object in the first view of the three-dimensional environment, the computer system detects a first gaze input pointing to the first user interface object via one or more input devices. In response to detecting the first gaze input pointing to the first user interface object: based on determining that a first event for a first notification meets a timing criterion, the computer system displays content associated with the first notification; and based on determining that the first event for the first notification does not meet the timing criterion, the computer system displays a plurality of indications for performing system operations associated with the first computer system without displaying the content associated with the first notification.

[0012] In some embodiments, a computer system displays a first user interface object via a first display generation component when a first view of a three-dimensional environment is visible. The first user interface object is displayed at a first location in the three-dimensional environment, and the first location in the three-dimensional environment has a first spatial arrangement relative to a corresponding portion of the user. While displaying the first user interface object in the first view of the three-dimensional environment, the computer system detects a first gaze input pointing to the first user interface object via one or more input devices. In response to detecting the first gaze input pointing to the first user interface object, based on determining that a request to join a first communication session that meets a timing criterion exists, the computer system displays a first user interface including a capability representation for joining the first communication session; and based on determining that no request to join the communication session meets the timing criterion exists, the computer system displays the plurality of capability representations for performing system operations associated with the first computer system without displaying the capability representation for joining the communication session.

[0013] In some embodiments, a computer system displays a first view of a three-dimensional environment corresponding to a user's first viewpoint via a first display generating component. The first view of the three-dimensional environment includes a first user interface element displayed at a first location within the three-dimensional environment. When the first user interface element is displayed at the first location in the three-dimensional environment, it has a first spatial relationship with the user's first viewpoint, and the first user interface element includes at least a first display representation. While displaying the first view of the three-dimensional environment, including the first user interface element displayed at the first location in the three-dimensional environment, the computer system detects a first movement of the user's viewpoint from the first viewpoint to a second viewpoint. In response to detecting the first movement of the user's viewpoint from the first viewpoint to the second viewpoint, when a second view of the three-dimensional environment corresponding to the user's second viewpoint becomes visible via the one or more display generating components, the computer system displays the first user interface element at the second location in the three-dimensional environment, wherein when the first user interface element is displayed at the second location in the three-dimensional environment, it has the first spatial relationship with the user's second viewpoint. When the first user interface element is displayed in the second view of the three-dimensional environment, the computer system detects a first input corresponding to the activation of the first display representation via one or more input devices. In response to detecting the first input corresponding to the activation of the first display representation, the computer system displays a second user interface element in the second view of the three-dimensional environment, wherein the second user interface element is displayed at a third location in the three-dimensional environment. While displaying the second view of the three-dimensional environment, including displaying the second user interface element at the third location in the three-dimensional environment, the computer system detects a second movement of the user's viewpoint from the second viewpoint to the third viewpoint. In response to detecting the second movement of the user's viewpoint from the second viewpoint to the third viewpoint, when the third view of the three-dimensional environment corresponding to the user's third viewpoint is visible via the one or more display generation components, the computer system displays the second user interface element at a position in the third view of the three-dimensional environment corresponding to the third location in the three-dimensional environment.

[0014] In some implementations, when a first view of a three-dimensional environment is visible through the first display generating component, the computer system detects first user input via one or more input devices, including detecting a first gaze input pointing to a first location in the three-dimensional environment. In response to detecting the first user input, including detecting the first gaze input: based on determining that the first location in the three-dimensional environment has a first spatial relationship with a viewport visible through which the three-dimensional environment is visible, the computer system displays a first user interface object in the first view of the three-dimensional environment, wherein the first user interface object includes one or more display representations for accessing a first set of functions of the first computer system, wherein the first user interface object is displayed at a second location in the three-dimensional environment that has a second spatial relationship different from the first spatial relationship with the viewport visible through which the three-dimensional environment is visible; and based on determining that the first location in the three-dimensional environment does not have the first spatial relationship with the viewport visible through which the three-dimensional environment is visible, the computer system abandons displaying the first user interface object in the first view of the three-dimensional environment.

[0015] In some implementations, when a first view of a three-dimensional environment is visible through the first display generating component, the computer system detects, via one or more input devices, that the user's attention is directed to a first portion of the first view of the three-dimensional environment. In response to detecting that the user's attention is directed to the first portion of the first view of the three-dimensional environment: based on determining that the first portion of the first view of the three-dimensional environment has a first spatial relationship with the viewport through which the three-dimensional environment is visible, and satisfying a first set of one or more contextual conditions when the user's attention is directed to the first portion of the first view of the three-dimensional environment, the computer system displays one or more user interface objects corresponding to the first set of one or more contextual conditions; and based on determining that the first portion of the first view of the three-dimensional environment has the first spatial relationship with the viewport through which the three-dimensional environment is visible, and satisfying a second set of one or more contextual conditions different from the first set of one or more contextual conditions when the user's attention is directed to the first portion of the first view of the three-dimensional environment, the computer system displays one or more user interface objects corresponding to the second set of one or more contextual conditions, the second set of one or more user interface objects being different from the first set of one or more user interface objects.

[0016] In some implementations, when a first view of the environment is visible via one or more display generation components, the computer system detects the occurrence of a first event. In response to detecting the occurrence of the first event, the computer system displays a first indication of a first notification corresponding to the first event via a first display generation component. After displaying the first indication of the first notification corresponding to the first event via the first display generation component: based on determining that the user's attention is directed to the first indication for a first threshold time period and then moves away from the first indication before a first criterion is met, the computer system maintains the display of the first indication for a first duration and stops displaying the first indication after the first duration expires; and based on determining that the user's attention is not directed to the first indication for the first threshold time period, the computer system maintains the display of the first indication for a second duration and stops displaying the first indication after the second duration expires, wherein the second duration is different from the first duration.

[0017] In some implementations, when a corresponding view of a three-dimensional environment including one or more physical objects is visible via a first display generation component, the computer system detects a first input corresponding to a request to display one or more notifications in the corresponding view of the three-dimensional environment. In response to detecting the first input, the computer system displays a corresponding notification at a corresponding location in the three-dimensional environment that corresponds to a previously detected event. Based on a first set of values ​​that determine the representation of the physical environment at the corresponding location where the notification is displayed has a first visual characteristic, the computer system displays the notification with a first visual appearance at least partially based on the first set of values ​​of the first visual characteristic of the representation of the physical environment at the corresponding location; and based on a second set of values ​​that determine the representation of the physical environment at the corresponding location has a different first visual characteristic than the first set of values, the computer system displays the notification with a second visual appearance at least partially based on the second set of values ​​of the second set of values ​​of the first visual characteristic of the representation of the physical environment at the corresponding location, which differs from the first visual appearance.

[0018] In some embodiments, a computer system displays a first user interface object in a first view of a three-dimensional environment corresponding to a user's first viewpoint via a first display generation component. The first user interface object is displayed at a first location in the three-dimensional environment and has a first spatial relationship with the user's first viewpoint. While displaying the first user interface object in the first view of the three-dimensional environment, the computer system detects a first movement of the user's current viewpoint from the first viewpoint to the user's second viewpoint via one or more input devices. In response to detecting a first movement of the user's current viewpoint from the first viewpoint to the second viewpoint, the computer system: replaces the display of the first view of the three-dimensional environment with the display of a second view of the three-dimensional environment corresponding to the user's second viewpoint; based on determining that the first movement of the user's current viewpoint from the first viewpoint to the second viewpoint does not meet a first criterion and that the first user interface object is a first type of user interface object, displays the first user interface object in the second view of the three-dimensional environment, wherein the first user interface object is displayed at a second location in the three-dimensional environment different from the first location and has the first spatial relationship with the user's second viewpoint; and based on determining that the first movement of the user's current viewpoint from the first viewpoint to the second viewpoint does not meet the first criterion and that the first user interface object is a second type of user interface object different from the first type, abandons displaying the first user interface object at the second location in the three-dimensional environment in the second view of the three-dimensional environment. The computer system detects a second movement of the user's current viewpoint from the second viewpoint to the user's third viewpoint via the one or more input devices. In response to detecting a second movement of the user's current viewpoint from the second viewpoint to the third viewpoint, the computer system: replaces the display of the second view of the three-dimensional environment with the display of a third view of the three-dimensional environment corresponding to the user's third viewpoint, and displays a first user interface object in the third view of the three-dimensional environment based on determining that the second movement of the user's current viewpoint from the second viewpoint to the third viewpoint satisfies the first criterion, wherein the first user interface object is displayed at a third location in the three-dimensional environment that is different from the first and second locations and has the first spatial relationship with the user's third viewpoint, regardless of whether the first user interface object is a first type of user interface object or a second type of user interface object.

[0019] It should be noted that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described in this specification are not exhaustive; in particular, many additional features and advantages will be apparent to those skilled in the art from the accompanying drawings, description, and claims. Furthermore, it should be pointed out that the language used in this specification has been chosen in principle for readability and instruction purposes, and such choice may not be necessary to depict or define the subject matter of the invention. Attached Figure Description

[0020] To better understand the various embodiments described, reference should be made to the following detailed description in conjunction with the accompanying drawings, wherein similar reference numerals indicate corresponding parts in all the drawings.

[0021] Figure 1 This is a block diagram illustrating the operating environment of a computer system for providing XR experiences according to some implementation schemes.

[0022] Figure 2 This is a block diagram illustrating a controller of a computer system configured to manage and coordinate a user's XR experience, according to some implementation schemes.

[0023] Figure 3 This is a block diagram illustrating a display generation component of a computer system configured to provide an XR experience to a user, according to some implementation schemes.

[0024] Figure 4 This is a block diagram illustrating a hand tracking unit of a computer system configured to capture user gesture input according to some implementation schemes.

[0025] Figure 5 This is a block diagram illustrating an eye-tracking unit of a computer system configured to capture a user's gaze input according to some implementation schemes.

[0026] Figure 6 This is a flowchart illustrating a flare-assisted gaze tracking pipeline according to some implementation schemes.

[0027] Figures 7A to 7K Example techniques for displaying multiple indications of system functions for accessing a computer system, according to some implementation schemes, are illustrated.

[0028] Figures 7L to 7Q Example techniques for displaying content associated with a first notification, according to some implementation schemes, are illustrated.

[0029] Figures 7R to 7X Example techniques for displaying a first user interface, including a capability representation for joining a communication session, are illustrated according to some implementation schemes.

[0030] Figures 7Y to 7AF Example techniques for displaying a user interface for initiating environment locking from a head-locked user interface, according to some implementation schemes, are illustrated.

[0031] Figures 7AG to 7AM An exemplary area for triggering the display of one or more user interface elements, according to some implementation schemes, is illustrated.

[0032] Figures 7AN to 7AT Exemplary contextual user interfaces displayed when different contextual criteria are met, according to some implementation schemes, are illustrated.

[0033] Figures 7AU to 7AZ Exemplary methods for rescinding instructions on a notification, according to some implementation schemes, are illustrated.

[0034] Figures 7BA to 7BJ Exemplary methods for displaying and interacting with previously received notifications, according to some implementation schemes, are illustrated.

[0035] Figures 7BK to 7BR Example methods for moving, hiding, and redisplaying user interface objects based on the movement of the user's viewpoint, according to some implementation schemes, are illustrated.

[0036] Figure 8 It is a flowchart of a method for displaying multiple indications of system functions for accessing a computer system, according to various implementation schemes.

[0037] Figure 9 It is a flowchart of a method for displaying content associated with the first notification according to various implementation schemes.

[0038] Figure 10 It is a flowchart of a method for displaying a first user interface, including a capability representation for joining a communication session, according to various implementation schemes.

[0039] Figures 11A to 11B This is a flowchart of a method for displaying a user interface that initiates environment locking from a head-locked user interface, according to various implementation schemes.

[0040] Figure 12 This is a flowchart of a method for initiating the display of a user interface in response to detecting that the user's attention is directed to a corresponding area of ​​the three-dimensional environment, according to various implementation schemes.

[0041] Figure 13 This is a flowchart of a method for displaying and interacting with a contextual user interface, according to some implementation schemes.

[0042] Figure 14 It is a flowchart of a method for rescinding instructions on a notification, according to various implementation schemes.

[0043] Figure 15 This is a flowchart of a method for displaying and interacting with previously received notifications in a notification history user interface, according to various implementation schemes.

[0044] Figures 16A to 16B This is a flowchart of a method for moving, hiding, and redisplaying user interface objects according to some implementation schemes. Detailed Implementation

[0045] According to some implementations, this disclosure relates to a user interface for providing extended reality (XR) experiences to users.

[0046] The systems, methods, and GUIs described in this paper improve user interface interaction with virtual / augmented reality environments in a variety of ways.

[0047] In some implementations, in response to a user directing gaze input toward a first user interface object (e.g., a system control indicator) and if the gaze input meets attention criteria regarding the first user interface object (e.g., sustained for a sufficient duration), the computer system displays multiple indications for performing system functions of the computer system (e.g., system settings such as audio volume, centralized notification user interface, search functionality, virtual assistant). Conditionally displaying a set of indications for accessing system functions based on whether the user has sufficiently paid attention to the first user interface object used to trigger the display of multiple indications reduces the amount of input required to access system functions (e.g., providing shortcuts to system functions) without otherwise cluttering the three-dimensional environment with visual controls (e.g., by displaying a single, less intrusive element that provides visual feedback to the user about where to look to elicit multiple indications, without discontinuously displaying multiple indications).

[0048] In some implementations, if the event for the notification meets timing criteria (e.g., recent, occurring within a threshold timeframe), the computer system displays content associated with the notification in response to gaze input pointing to a first user interface object. If the event for the notification does not meet timing criteria, the computer system displays multiple enablements for accessing system functions in response to gaze input pointing to the first user interface object, without displaying content associated with the notification. Conditionally displaying content about the notification that triggered the event based on whether the event is recent enough allows the user to control whether the notification is displayed based on whether the user gazes at the first user interface object, and causes the computer system to automatically and promptly display information and / or controls that are more likely to be relevant to the current context of the computer system.

[0049] In some implementations, if the computer system has received a request to join a communication session and the request remains active, the computer system displays a user interface including a capability representation for joining the communication session in response to gaze input pointing to a first user interface object. If no request to join a communication session is active, the computer system displays multiple capability representations for accessing system functions in response to gaze input pointing to a first user interface object, instead of displaying a user interface with a capability representation for joining the communication session. Conditionally displaying a user interface including a capability representation for joining an actively requested communication session allows the user to control whether to view the request based on whether the user gazes at the first user interface object, and causes the computer system to automatically and promptly display information and / or controls that are likely more relevant to the current context of the computer system (e.g., by allowing the user to respond more quickly to active requests, or by simply displaying system controls).

[0050] Figures 1 to 6 A description of a sample computer system for providing XR experiences to users is provided. Figures 7A to 7K Example techniques are illustrated, according to some embodiments, for displaying multiple indications of system functions for accessing a first computer system in response to detecting a first gaze input pointing to a first user interface object and for determining that the first gaze input meets an attention criterion with respect to the first user interface object. Figures 7L to 7Q An example technique is illustrated, based on some implementation schemes, for displaying content associated with a first notification in response to detecting a first gaze input pointing to a first user interface object, based on determining that a first event for a first notification meets a timing criterion. Figures 7R to 7X Example techniques for displaying a first user interface, including a capability representation for joining a communication session, are illustrated according to some implementation schemes. Figures 7Y to 7AF Example techniques for displaying a user interface for initiating environment locking from a head-locked user interface, according to some implementation schemes, are illustrated. Figures 7AG to 7AM An exemplary area for triggering the display of one or more user interface elements, according to some implementation schemes, is illustrated. Figures 7AN to 7AT Exemplary contextual user interfaces displayed when different contextual criteria are met, according to some implementation schemes, are illustrated. Figures 7AU to 7AZ Exemplary methods for rescinding instructions on a notification, according to some implementation schemes, are illustrated. Figures 7BA to 7BJ Exemplary methods for displaying and interacting with previously received notifications, according to some implementation schemes, are illustrated. Figures 7BK to 7BR Example methods for moving, hiding, and redisplaying user interface objects based on the movement of the user's viewpoint, according to some implementation schemes, are illustrated. Figure 8It is a flowchart of a method for displaying multiple indications of system functions for accessing a first computer system in response to detecting a first gaze input pointing to a first user interface object, according to various implementation schemes, and based on determining that the first gaze input meets attention criteria with respect to the first user interface object. Figure 9 It is a flowchart of a method for displaying content associated with a first notification in response to the detection of a first gaze input pointing to a first user interface object, based on determining various implementation schemes. Figure 10 This is a flowchart of a method for displaying a first user interface, including a capability representation for joining a communication session, in response to detecting a first gaze input pointing to a first user interface object and based on determining that a request for a first computer system to join a first communication session that meets timing criteria, according to various implementation schemes. Figures 11A to 11B This is a flowchart of a method for displaying a user interface that initiates environment locking from a head-locked user interface, according to various implementation schemes. Figure 12 This is a flowchart of a method for initiating the display of a user interface in response to detecting that the user's attention is directed to a corresponding area of ​​the three-dimensional environment, according to various implementation schemes. Figure 13 This is a flowchart of a method for displaying and interacting with a contextual user interface, according to some implementation schemes. Figure 14 It is a flowchart of a method for rescinding instructions on a notification, according to various implementation schemes. Figure 15 This is a flowchart of a method for displaying and interacting with previously received notifications in a notification history user interface, according to various implementation schemes. Figures 16A to 16B This is a flowchart of a method for moving, hiding, and redisplaying user interface objects according to some implementation schemes. Figures 7A to 7BR Used to illustrate, according to some implementation schemes, regarding Figures 8 to 1 Various aspects of the process described in 6.

[0051] The processes described below enhance device operability and make the user-device interface more efficient through various technologies (e.g., by assisting users in providing appropriate input and reducing user errors when operating / interacting with the device). These technologies include: providing users with improved visual feedback; reducing the amount of input required to perform an operation; providing additional control options without cluttering the user interface with additional displayed controls; performing an operation when a set of conditions is met without requiring additional user input; improving privacy and / or security; and / or additional technologies. These technologies also reduce power consumption and extend device battery life by enabling users to use the device faster and more efficiently.

[0052] Furthermore, in methods described herein where one or more steps depend on the satisfaction of one or more conditions, it should be understood that the described method may be repeated in multiple repetitions such that, during the repetitions, all conditions on which the steps in the method depend are satisfied in different repetitions of the method. For example, if a method requires performing a first step (if the condition is satisfied) and a second step (if the condition is not satisfied), those skilled in the art will know that the stated steps are repeated until both the conditions are satisfied and not satisfied (in no particular order). Therefore, a method described as having one or more steps depending on the satisfaction of one or more conditions can be rewritten as a method that repeats until each condition described in the method is satisfied. However, this does not require the system or computer-readable medium to declare that it contains instructions for performing discretionary operations based on the satisfaction of the corresponding one or more conditions, and thus to determine whether possible conditions have been satisfied without explicitly repeating the steps of the method until all conditions on which the steps in the method depend are satisfied. Those skilled in the art will also understand that, similar to methods having discretionary steps, a system or computer-readable storage medium may repeat the steps of the method multiple times as needed to ensure that all discretionary steps have been performed.

[0053] In some implementation schemes, such as Figure 1 As shown, an XR experience is provided to a user via an operating environment 100 including a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), a display generation component 120 (e.g., a head-mounted display (HMD), a monitor, a projector, a touchscreen, etc.), one or more input devices 125 (e.g., an eye-tracking device 130, a hand-tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a haptic output generator 170, and other output devices 180), one or more sensors 190 (e.g., image sensors, light sensors, depth sensors, haptic sensors, orientation sensors, proximity sensors, temperature sensors, position sensors, motion sensors, speed sensors, etc.), and optionally one or more peripheral devices 195 (e.g., home appliances, wearable devices, etc.). In some embodiments, one or more of the input devices 125, output devices 155, sensors 190, and peripheral devices 195 are integrated with the display generation component 120 (e.g., in a head-mounted or handheld device).

[0054] In describing XR experiences, various terms are used to distinguish several related but different environments that a user can sense and / or interact with (e.g., interacting with input detected by the computer system 101 that generates the XR experience, causing the computer system to generate audio, visual, and / or haptic feedback corresponding to various inputs provided to the computer system 101). The following are subgroups of these terms:

[0055] Physical environment: The physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic systems. Physical environments, such as physical parks, include physical objects such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment through senses such as sight, touch, hearing, taste, and smell.

[0056] Extended Reality: In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic systems. In XR, a person's physical motion or a subgroup of its representations is tracked, and in response, one or more features of one or more virtual objects simulated in the XR environment are adjusted in a manner consistent with at least one physical law. For example, an XR system may detect a person's head turning and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), the adjustment of features of virtual objects in the XR environment may be in response to a representation of physical motion (e.g., a voice command). People may use any of their senses to sense and / or interact with XR objects, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with an audio object that creates a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. For example, audio objects can achieve audio transparency, which selectively introduces ambient sounds from the physical environment, with or without computer-generated audio. In some XR environments, a person can sense and / or interact with only audio objects.

[0057] Examples of XR include virtual reality and mixed reality.

[0058] Virtual Reality: A virtual reality (VR) environment is a simulated environment designed to be entirely based on computer-generated sensory input for one or more senses. A VR environment includes multiple virtual objects that a person can sense and / or interact with. Examples of virtual objects include trees, buildings, and computer-generated images representing human avatars. A person can sense and / or interact with virtual objects in a VR environment through the simulation of their presence within the computer-generated environment and / or through the simulation of a subgroup of physical movements of a person within the computer-generated environment.

[0059] Mixed Reality: Compared to VR environments, which are designed to be entirely based on computer-generated sensory input, mixed reality (MR) environments refer to simulated environments designed to incorporate sensory input from the physical environment, or its representations, in addition to computer-generated sensory input (e.g., virtual objects). On the virtual continuum, a mixed reality environment is any state between, but not limited to, a purely physical environment as one end and a virtual reality environment as the other. In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. Additionally, some electronic systems used to present an MR environment can track position and / or orientation relative to the physical environment, enabling virtual objects to interact with real objects (i.e., physical objects or their representations from the physical environment). For example, the system can take motion into account, making virtual trees appear stationary relative to the physical ground.

[0060] Examples of mixed reality include augmented reality and augmented virtual reality.

[0061] Augmented Reality: An augmented reality (AR) environment is a simulated environment in which one or more virtual objects are overlaid on a physical environment or a representation of the physical environment. For example, an electronic system for presenting an AR environment may have a transparent or semi-transparent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on the transparent or semi-transparent display, allowing a person to perceive the virtual objects overlaid on the physical environment using the system. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the images or videos with virtual objects and presents the combination on the opaque display. A person uses the system to indirectly view the physical environment via the images or videos of the physical environment and perceives the virtual objects overlaid on the physical environment. As used herein, video of the physical environment displayed on an opaque display is referred to as “pass-through video,” meaning that the system uses one or more image sensors to capture images of the physical environment and uses these images when presenting the AR environment on the opaque display. Alternatively, the system may have a projection system that projects virtual objects onto the physical environment, such as as a hologram or onto a physical surface, allowing a person to perceive the virtual objects superimposed on the physical environment using the system. Augmented reality environments also refer to simulated environments in which the representation of the physical environment is transformed from computer-generated sensory information. For example, in providing pass-through video, the system may transform one or more sensor images to impose a selected viewpoint (e.g., viewpoint) different from the viewpoint captured by the imaging sensor. As another example, the representation of the physical environment may be transformed by graphically modifying (e.g., magnifying) portions thereof, such that the modified portions are representative but not realistic versions of the originally captured image. Furthermore, the representation of the physical environment may be transformed by graphically removing or blurring portions thereof.

[0062] Augmented Virtual: An augmented virtual (AV) environment is a simulated environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. Sensory inputs can be representations of one or more characteristics of the physical environment. For example, an AV park might have virtual trees and buildings, but the faces of people are realistically reproduced from images taken of physical people. Similarly, virtual objects might adopt the shape or color of physical objects imaged by one or more imaging sensors. Furthermore, virtual objects might use shadows that correspond to the sun's position within the physical environment.

[0063] Viewpoint-locked virtual objects: Virtual objects are also viewpoint-locked when the computer system displays them at the same location and / or position within the user's viewpoint, even if the user's viewpoint shifts (e.g., changes). In embodiments where the computer system is a head-mounted device, the user's viewpoint is locked to the direction forward of the user's head (e.g., when the user is looking straight ahead, the user's viewpoint is at least a portion of the user's field of view); therefore, the user's viewpoint remains fixed even when the user's gaze shifts without moving the user's head. In embodiments where the computer system has a display generating component (e.g., a display screen) that can be repositioned relative to the user's head, the user's viewpoint is the augmented reality view presented to the user on the display generating component of the computer system. For example, a viewpoint-locked virtual object displayed in the upper left corner of the user's viewpoint when the user's viewpoint is in a first orientation (e.g., the user's head is facing north) continues to be displayed in the upper left corner of the user's viewpoint, even when the user's viewpoint changes to a second orientation (e.g., the user's head is facing west). In other words, the position and / or orientation of a viewpoint-locked virtual object displayed in the user's viewpoint is independent of the user's position and / or orientation in the physical environment. In an implementation where the computer system is a head-mounted device, the user's viewpoint is locked to the orientation of the user's head, so the virtual object is also referred to as a "head-locked virtual object".

[0064] Environment-locked visual objects: When a computer system displays a virtual object at a location and / or position within the user's viewpoint, the virtual object is environment-locked (or, "world-locked"), the location and / or position being based on a location and / or object within a three-dimensional environment (e.g., a physical or virtual environment) (e.g., selected and / or anchored to that location and / or object with reference to it). As the user's viewpoint shifts, the location and / or object in the environment relative to the user's viewpoint changes, causing the environment-locked virtual object to be displayed at different locations and / or positions within the user's viewpoint. For example, an environment-locked virtual object locked to a tree immediately in front of the user is displayed at the center of the user's viewpoint. When the user's viewpoint shifts to the right (e.g., the user's head turns to the right) so that the tree is now centered to the left in the user's viewpoint (e.g., the tree's position in the user's viewpoint shifts), the environment-locked virtual object locked to the tree is displayed centered to the left in the user's viewpoint. In other words, the position and / or orientation of an environment-locked virtual object displayed in the user's viewpoint depends on the location to which the virtual object is locked and / or the position and / or orientation of the object in the environment. In some implementations, the computer system uses a stationary reference frame (e.g., a coordinate system anchored to a fixed location and / or object in the physical environment) to determine the orientation of the environment-locked virtual object displayed in the user's viewpoint. An environment-locked virtual object may be locked to a stationary part of the environment (e.g., a floor, wall, table, or other stationary object), or it may be locked to a movable part of the environment (e.g., a vehicle, animal, person, or even a representation of a part of the user's body that moves independently of the user's viewpoint, such as a hand, wrist, arm, or foot), causing the virtual object to move with the viewpoint or that part of the environment to maintain a fixed relationship between the virtual object and that part of the environment.

[0065] In some implementations, environment-locked or viewpoint-locked virtual objects exhibit lazy following behavior, where movement relative to a reference point the virtual object is following is reduced or delayed. In some implementations, when exhibiting lazy following behavior, the computer system intentionally delays the movement of the virtual object when movement of the reference point the virtual object is following (e.g., a portion of the environment, a viewpoint, or a point fixed relative to the viewpoint, such as a point between 5 cm and 300 cm from the viewpoint) is detected. For example, when the reference point (e.g., a portion of the environment or the viewpoint) moves at a first speed, the virtual object is moved by the device to remain locked to the reference point, but moves at a second speed that is slower than the first speed (e.g., until the reference point stops moving or slows down, at which point the virtual object begins to catch up). In some implementations, when the virtual object exhibits lazy following behavior, the device ignores small movements of the reference point (e.g., ignores movements of the reference point below a threshold amount, such as 0 to 5 degrees or 0 to 50 cm). For example, when the reference point (e.g., the part of the environment or viewpoint to which the virtual object is locked) moves by a first amount, the distance between the reference point and the virtual object increases (e.g., because the virtual object is being displayed to maintain a fixed or substantially fixed position relative to a viewpoint or part of the environment different from the reference point to which the virtual object is locked), and when the reference point (e.g., the part of the environment or viewpoint to which the virtual object is locked) moves by a second amount greater than the first amount, the distance between the reference point and the virtual object initially increases (e.g., because the virtual object is being displayed to maintain a fixed or substantially fixed position relative to a viewpoint or part of the environment different from the reference point to which the virtual object is locked), and then decreases as the amount of movement of the reference point increases above a threshold (e.g., a "lazy following" threshold) because the virtual object is moved by the computer system to maintain a fixed or substantially fixed position relative to the reference point. In some implementations, maintaining a substantially fixed position of the virtual object relative to a reference point includes displaying the virtual object within a threshold distance (e.g., 1 cm, 2 cm, 3 cm, 5 cm, 15 cm, 20 cm, 50 cm) of the reference point in one or more dimensions (e.g., up / down, left / right, and / or forward / backward relative to the position of the reference point).

[0066] Hardware: Many different types of electronic systems enable humans to sense and / or interact with a variety of XR environments. Examples include head-mounted systems, projection-based systems, head-up displays, vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays shaped like lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have one or more speakers and an integrated opaque display. Alternatively, head-mounted systems may be configured to receive an external opaque display (e.g., a smartphone). Head-mounted systems may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light representing the image is directed to the person's eyes. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium may be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display may be configured to selectively become opaque. Projection-based systems may employ retinal projection techniques that project graphic images onto a person's retina. Projection systems may also be configured to project virtual objects onto a physical environment, such as as holograms or on a physical surface. In some embodiments, controller 110 is configured to manage and coordinate the user's XR experience. In some embodiments, controller 110 includes a suitable combination of software, firmware, and / or hardware. References below. Figure 2The controller 110 is described in more detail. In some embodiments, the controller 110 is a computing device located locally or remotely relative to scene 105 (e.g., the physical environment). For example, the controller 110 is a local server located within scene 105. Alternatively, the controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside scene 105. In some embodiments, the controller 110 is communicatively coupled to a display generation component 120 (e.g., an HMD, a monitor, a projector, a touchscreen, etc.) via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). For example, controller 110 may be included within the housing (e.g., physical enclosure) of display generation component 120 (e.g., HMD or portable electronic device including display and one or more processors), one or more input devices in input device 125, one or more output devices in output device 155, one or more sensors in sensor 190, and / or one or more peripheral devices in peripheral device 195, or may share the same physical housing or support structure with one or more of the aforementioned devices.

[0067] In some embodiments, the display generation component 120 is configured to provide a user with an XR experience (e.g., at least the visual components of the XR experience). In some embodiments, the display generation component 120 includes a suitable combination of software, firmware, and / or hardware. The following is relative to... Figure 3 The display generation component 120 is described in more detail. In some embodiments, the functionality of the controller 110 is provided by and / or combined with the display generation component 120.

[0068] According to some implementation schemes, when a user is virtually and / or physically present within scene 105, display generation component 120 provides the user with an XR experience.

[0069] In some embodiments, the display generating component is worn on a part of the user's body (e.g., on his / her head, his / her hand, etc.). Thus, the display generating component 120 includes one or more XR displays provided for displaying XR content. For example, in various embodiments, the display generating component 120 surrounds the user's field of view. In some embodiments, the display generating component 120 is a handheld device (such as a smartphone or tablet) configured to present XR content, and the user holds the device having a display facing the user's field of view and a camera facing scene 105. In some embodiments, the handheld device is optionally placed within a housing worn on the user's head. In some embodiments, the handheld device is optionally placed on a support (e.g., a tripod) in front of the user. In some embodiments, the display generating component 120 is an XR chamber, housing, or room configured to present XR content, wherein the user does not wear or hold the display generating component 120. Many user interfaces described with reference to one type of hardware used for displaying XR content (e.g., a handheld device or a tripod-mounted device) can be implemented on another type of hardware used for displaying XR content (e.g., an HMD or other wearable computing device). For example, a user interface illustrating interaction with XR content triggered by an interaction occurring in the space in front of a handheld device or tripod-mounted device can be similarly implemented using an HMD, where the interaction occurs in the space in front of the HMD and the response to the XR content is displayed via the HMD. Similarly, a user interface illustrating interaction with XR content triggered by movement of a handheld device or tripod-mounted device relative to the physical environment (e.g., scene 105 or a part of the user's body (e.g., the user's eyes, head, or hand)) can be similarly implemented using an HMD, where the movement is caused by movement of the HMD relative to the physical environment (e.g., scene 105 or a part of the user's body (e.g., the user's eyes, head, or hand)).

[0070] Despite Figure 1 The relevant features of the operating environment 100 are shown in this disclosure, but those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and to avoid obscuring further relevant aspects of the example embodiments disclosed herein.

[0071] Figure 2This is a block diagram of an example controller 110 according to some implementation schemes. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of simplicity and to avoid obscuring further relevant aspects of the implementation schemes disclosed herein. Therefore, as a non-limiting example, in some embodiments, controller 110 includes one or more processing units 202 (e.g., microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), graphics processing units (GPUs), central processing units (CPUs), processing kernels, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., Universal Serial Bus (USB), IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), Bluetooth, ZigBee, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0072] In some embodiments, one or more communication buses 204 include circuitry for interconnecting and communicating between system components. In some embodiments, one or more I / O devices 206 include at least one of a keyboard, mouse, touchpad, joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0073] Memory 220 includes high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 220 optionally includes one or more storage devices located remotely from one or more processing units 202. Memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, memory 220 or the non-transitory computer-readable storage medium of memory 220 stores programs, modules, and data structures or subgroups thereof, including optional operating system 230 and XR experience module 240.

[0074] Operating system 230 includes instructions for handling various basic system services and for performing hardware-related tasks. In some embodiments, XR experience module 240 is configured to manage and coordinate single or multiple XR experiences for one or more users (e.g., single XR experiences for one or more users, or multiple XR experiences for corresponding groups of one or more users). To this end, in various embodiments, XR experience module 240 includes a data acquisition unit 241, a tracking unit 242, a coordination unit 246, and a data transmission unit 248.

[0075] In some implementations, the data acquisition unit 241 is configured to acquire data from... Figure 1 The data acquisition unit 241 includes at least a display generation component 120 and optionally acquires data (e.g., presentation data, interactive data, sensor data, location data, etc.) from one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, the data acquisition unit 241 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0076] In some implementations, the tracking unit 242 is configured to map scene 105, and the tracking at least shows the generated component 120 relative to... Figure 1 The tracking unit 242 tracks the location / position of scene 105, and optionally the location / position relative to one or more of the tracking input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, the tracking unit 242 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics. In some embodiments, the tracking unit 242 includes a hand tracking unit 244 and / or an eye tracking unit 243. In some embodiments, the hand tracking unit 244 is configured to track the location / position of one or more portions of the user's hand, and / or the location / position of one or more portions of the user's hand relative to... Figure 1 The movement of scene 105 relative to the display generating component 120 and / or relative to a coordinate system (defined relative to the user's hand). The following refers to the movement relative to... Figure 4 The hand tracking unit 244 is described in more detail. In some embodiments, the eye tracking unit 243 is configured to track the user's gaze (or more broadly, the user's eyes, face, or head) relative to scene 105 (e.g., relative to the physical environment and / or relative to the user (e.g., the user's hand)) or relative to XR content displayed via display generation component 120. The following description is relative to... Figure 5 The eye-tracking unit 243 is described in more detail.

[0077] In some implementations, coordination unit 246 is configured to manage and coordinate the XR experience presented to the user by display generation component 120, and optionally by one or more of output device 155 and / or peripheral device 195. To this end, in various implementations, coordination unit 246 includes instructions and / or logic for instructions, as well as heuristics and metadata for heuristics.

[0078] In some embodiments, the data transmission unit 248 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the display generation component 120, and optionally to one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, the data transmission unit 248 includes instructions and / or logic for instructions, as well as heuristics and metadata for heuristics.

[0079] Although the data acquisition unit 241, tracking unit 242 (e.g., including eye tracking unit 243 and hand tracking unit 244), coordination unit 246, and data transmission unit 248 are shown residing on a single device (e.g., controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 241, tracking unit 242 (e.g., including eye tracking unit 243 and hand tracking unit 244), coordination unit 246, and data transmission unit 248 may reside in a separate computing device.

[0080] also, Figure 2 This is used more as a functional description of various features that can exist in a particular specific implementation, and differs from the structural diagrams of the implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separated. For example, Figure 2 Some functional modules shown individually may be implemented in a single module, and the various functions of a single functional block may be implemented in various implementations through one or more functional blocks. The actual number of modules and the division of specific functions and how features are allocated therein will vary depending on the specific implementation, and in some implementations, it depends in part on the specific combination of hardware, software and / or firmware chosen for that particular implementation.

[0081] Figure 3This is a block diagram showing an example of generating component 120 according to some embodiments. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of simplicity and to avoid obscuring further relevant aspects of the embodiments disclosed herein. Therefore, as a non-limiting example, in some embodiments, the display generating component 120 (e.g., HMD) includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, Firewire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BlueTooth, ZigBee, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more XR displays 312, one or more optional internal and / or external image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these components and various other components.

[0082] In some embodiments, one or more communication buses 304 include circuitry for interconnecting and controlling communication between system components. In some embodiments, one or more I / O devices and sensors 306 include inertial measurement units (IMUs), accelerometers, gyroscopes, thermometers, one or more physiological sensors (e.g., blood pressure monitors, heart rate monitors, blood oxygen sensors, blood glucose sensors, etc.), one or more microphones, one or more speakers, haptic engines, and / or one or more depth sensors (e.g., structured light, time-of-flight, etc.).

[0083] In some embodiments, one or more XR displays 312 are configured to provide an XR experience to a user. In some embodiments, the one or more XR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conducting electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical systems (MEMS), and / or similar display types. In some embodiments, the one or more XR displays 312 correspond to waveguide displays such as diffraction, reflection, polarization, and holography. For example, display generation component 120 (e.g., HMD) includes a single XR display. Alternatively, display generation component 120 may include XR displays for each of the user's eyes. In some embodiments, the one or more XR displays 312 are capable of presenting MR and VR content. In some embodiments, the one or more XR displays 312 are capable of presenting either MR or VR content.

[0084] In some embodiments, one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as an eye-tracking camera). In some embodiments, one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's hand and optionally the user's arm (and may be referred to as a hand-tracking camera). In some embodiments, one or more image sensors 314 are configured to face forward in order to acquire image data corresponding to the scene that the user would see in the absence of a display generation component 120 (e.g., an HMD) (and may be referred to as a scene camera). One or more optional image sensors 314 may include one or more RGB cameras (e.g., having a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), one or more infrared (IR) cameras, and / or one or more event-based cameras, etc.

[0085] Memory 320 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 320 optionally includes one or more storage devices located remotely from one or more processing units 302. Memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, memory 320 or the non-transitory computer-readable storage medium of memory 320 stores programs, modules, and data structures or subgroups thereof, including optional operating system 330 and XR rendering module 340.

[0086] Operating system 330 includes instructions for handling various basic system services and for performing hardware-related tasks. In some embodiments, XR rendering module 340 is configured to present XR content to a user via one or more XR displays 312. To this end, in various embodiments, XR rendering module 340 includes a data acquisition unit 342, an XR rendering unit 344, an XR mapping generation unit 346, and a data transmission unit 348.

[0087] In some implementations, the data acquisition unit 342 is configured to acquire data from at least... Figure 1 The controller 110 acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.). To this end, in various embodiments, the data acquisition unit 342 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0088] In some implementations, the XR rendering unit 344 is configured to render XR content via one or more XR displays 312. To this end, in various implementations, the XR rendering unit 344 includes instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0089] In some implementations, the XR mapping generation unit 346 is configured to generate XR maps based on media content data (e.g., 3D maps of mixed reality scenes or maps in which computer-generated objects can be placed to generate extended reality physical environments). To this end, in various implementations, the XR mapping generation unit 346 includes instructions and / or logic for those instructions, as well as heuristics and metadata for those heuristics.

[0090] In some embodiments, the data transmission unit 348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the controller 110, and optionally one or more of the input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, the data transmission unit 348 includes instructions and / or logic for instructions, as well as heuristics and metadata for heuristics.

[0091] Although the data acquisition unit 342, the XR rendering unit 344, the XR mapping generation unit 346, and the data transmission unit 348 are shown residing in a single device (e.g., Figure 1 The display generation unit 120 is located on the display generation unit, but it should be understood that in other embodiments, any combination of the data acquisition unit 342, the XR rendering unit 344, the XR mapping generation unit 346, and the data transmission unit 348 may be located in a separate computing device.

[0092] also, Figure 3 This is used more as a functional description of various features that may exist in a particular implementation, and differs from the structural diagrams of the implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separated. For example, Figure 3 Some functional modules shown individually may be implemented in a single module, and the various functions of a single functional block may be implemented in various implementations through one or more functional blocks. The actual number of modules and the division of specific functions and how features are allocated therein will vary depending on the specific implementation, and in some implementations, it depends in part on the specific combination of hardware, software and / or firmware chosen for that particular implementation.

[0093] Figure 4 This is a schematic illustration of an example embodiment of the hand tracking device 140. In some embodiments, the hand tracking device 140 ( Figure 1 Controlled by hand tracking unit 244 Figure 2 To track the location / position of one or more parts of a user's hand, and / or the location of one or more parts of the user's hand relative to... Figure 1 The movement is defined in scenario 105 (e.g., relative to a portion of the user's surrounding physical environment, relative to display generation component 120, or relative to a portion of the user (e.g., the user's face, eyes, or head) and / or relative to a coordinate system (defined relative to the user's hand)). In some embodiments, the hand tracking device 140 is part of the display generation component 120 (e.g., embedded in or attached to a head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0094] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras, etc.) that captures three-dimensional scene information including at least the human user's hand 406. The image sensor 404 captures images of the hand at sufficient resolution to allow differentiation of the fingers and their corresponding positions. The image sensor 404 typically captures images of other parts of the user's body, or possibly all parts of the body, and may have scaling capabilities or a dedicated sensor with increased magnification to capture images of the hand at the desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with other image sensors to capture the physical environment of scene 105, or serves as the image sensor for capturing the physical environment of scene 105. In some embodiments, the image sensor is positioned relative to the user or the user's environment in a manner that uses the field of view of the image sensor 404 or a portion thereof to define an interaction space in which hand movements captured by the image sensor are considered input to the controller 110.

[0095] In some implementations, image sensor 404 outputs a sequence of frames containing 3D mapping data (and possibly, color image data) to controller 110, which extracts high-level information from the mapping data. This high-level information is typically provided via an application programming interface (API) to an application running on the controller, which in turn drives display generation component 120. For example, a user can interact with software running on controller 110 by moving his hand 406 and changing his hand pose.

[0096] In some embodiments, image sensor 404 projects a speckle pattern onto a scene including hand 406 and captures an image of the projected pattern. In some embodiments, controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) via triangulation based on the lateral displacement of the specks in the pattern. This approach is advantageous because it does not require the user to hold or wear any kind of beacon, sensor, or other marker. This method gives the depth coordinates of points in the scene relative to a predetermined reference plane at a specific distance from image sensor 404. In this disclosure, it is assumed that image sensor 404 defines an orthogonal set of x-axis, y-axis, and z-axis such that the depth coordinates of points in the scene correspond to the z-component measured by the image sensor. Alternatively, image sensor 404 (e.g., a hand-tracking device) may use other 3D mapping methods, such as stereo imaging or time-of-flight measurement, based on a single or multiple cameras or other types of sensors.

[0097] In some implementations, hand tracking device 140 captures and processes time-series depth maps containing the user's hand as the user moves his hand (e.g., the entire hand or one or more fingers). Software running on a processor in image sensor 404 and / or controller 110 processes the 3D mapping data to extract image block descriptors of the hand from these depth maps. The software may match these descriptors with image block descriptors stored in database 408 based on a previous learning process to estimate the pose of the hand in each frame. The pose typically includes the 3D position of the user's hand joints and fingertips.

[0098] The software can also analyze the trajectories of the hand and / or fingers across multiple frames in a sequence to identify pose. The pose estimation function described herein can be interleaved with motion tracking, such that patch-based pose estimation is performed only once every two (or more) frames, while tracking is used to find pose changes occurring in the remaining frames. Pose, motion, and orientation information are provided to an application running on controller 110 via the aforementioned API. This application can, for example, move and modify the image presented on display generation unit 120 in response to the pose and / or orientation information, or perform other functions.

[0099] In some implementations, the posture includes an aerial posture. An aerial posture is a posture detected without the user touching an input element that is part of the device (e.g., computer system 101, one or more input devices 125, and / or hand tracking device 140) (or independent of the input element that is part of the device) and based on the detected movement of a part of the user's body (e.g., head, one or more arms, one or more hands, one or more fingers, and / or one or more legs) through the air (including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movement relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of the user's fingers relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tapping posture including a hand moving in a predetermined position at a predetermined amount and / or speed, or a rocking posture including a part of the user's body at a predetermined rotational speed or amount)).

[0100] In some embodiments, the input gestures used in the various examples and embodiments described herein include aerial gestures performed by the movement of a user's fingers relative to other fingers or portions of the user's hand for interacting with an XR environment (e.g., a virtual or mixed reality environment). In some embodiments, aerial gestures are gestures detected without the user touching an input element that is part of the device (or independently of an input element that is part of the device) and based on detected movements of a part of the user's body through the air (including movements of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movements relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of the user's fingers relative to another finger or portion of the user's hand), and / or absolute movements of a part of the user's body (e.g., a tapping gesture including a hand moving in a predetermined pose at a predetermined amount and / or speed, or a shaking gesture including a predetermined rotational speed or amount of a part of the user's body)).

[0101] In some implementations where the input gesture is an air gesture (e.g., where, in the absence of physical contact with the input device, the input device provides the computer system with information about which user interface element is the target of the user input, such as contact with a user interface element displayed on a touchscreen, or contact with a mouse or touchpad to move the cursor to the user interface element), the gesture takes into account the user's attention (e.g., gaze) to determine the target of the user input (e.g., for direct input, as described below). Therefore, in implementations involving air gestures, the input gesture is, for example, attention (e.g., gaze) detected in conjunction (e.g., concurrently) with the movement of the user's fingers and / or hand performing pinch and / or tap input, as described below.

[0102] In some implementations, input gestures directed towards a user interface object are performed, either directly or indirectly, by referencing the user interface object. For example, user input is performed directly on the user interface object based on the user's hand being positioned at a location corresponding to the user interface object's location in the three-dimensional environment (e.g., determined based on the user's current viewpoint). In some implementations, when user attention to the user interface object (e.g., gazing) is detected, input gestures are performed indirectly on the user interface object based on the user's hand not being positioned at a location corresponding to the user interface object's location in the three-dimensional environment while the user is performing the input gesture. For example, for direct input gestures, the user can guide their input to the user interface object by initiating a gesture at or near a location corresponding to the user interface object's display location (e.g., within 0.5 cm, 1 cm, 5 cm, or a distance between 0 and 5 cm measured from the outer edge or center of the option). For indirect input gestures, the user can guide their input to the user interface object by focusing on it (e.g., by gazing at the user interface object), and while focusing on the option, the user initiates an input gesture (e.g., at any location detectable by the computer system) (e.g., at a location not corresponding to the user interface object's display location).

[0103] In some implementations, the input gestures (e.g., air gestures) used in the various examples and implementations described herein include pinch input and tap input for interacting with a virtual or mixed reality environment. For example, the pinch input and tap input described below are performed as air gestures.

[0104] In some implementations, the pinch input is part of an air gesture that includes one or more of the following: a pinch gesture, a prolonged pinch gesture, a pinch and drag gesture, or a double pinch gesture. For example, a pinch gesture as an air gesture includes the movement of two or more fingers of the hand to contact each other, i.e., optionally followed by an immediate (e.g., within 0 to 1 second) interruption of contact with each other. A prolonged pinch gesture as an air gesture includes the movement of two or more fingers of the hand to contact each other for at least a threshold time (e.g., at least 1 second) before an interruption of contact is detected. For example, a prolonged pinch gesture includes a user holding a pinch gesture (e.g., where two or more fingers are in contact), and the prolonged pinch gesture continues until an interruption of contact between the two or more fingers is detected. In some implementations, a double pinch gesture as an air gesture includes two (e.g., more) pinch inputs (e.g., performed by the same hand) that are detected consecutively with each other immediately (e.g., within a corresponding time period). For example, a user performs a first pinch input (e.g., a pinch input or a long pinch input), releases the first pinch input (e.g., interrupts the contact between two or more fingers), and performs a second pinch input within a corresponding time period after releasing the first pinch input (e.g., within 1 second or within 2 seconds).

[0105] In some embodiments, the pinch and drag gesture as an air gesture includes a pinch gesture (e.g., a pinching gesture or a long pinch gesture) performed in conjunction with (e.g., following) a drag input that changes the position of the user's hand from a first position (e.g., the starting position of the drag) to a second position (e.g., the ending position of the drag). In some embodiments, the user holds the pinch gesture while performing the drag input and releases the pinch gesture (e.g., opening two or more of their fingers) to end the drag gesture (e.g., at the second position). In some embodiments, the pinch input and the drag input are performed by the same hand (e.g., the user pinches two or more fingers together to touch each other and uses the drag gesture to move the same hand to the second position in the air). In some implementations, pinch input is performed by the user's first hand, and drag input is performed by the user's second hand (e.g., while the user continues pinch input with their first hand, their second hand moves in the air from a first position to a second position). In some implementations, input gestures as air gestures include inputs performed using both of the user's hands (e.g., pinch and / or tap inputs). For example, input gestures include two (e.g., more) pinch inputs performed in combination with each other (e.g., concurrently or within a corresponding time period). For example, a first pinch gesture (e.g., pinch input, long pinch input, or pinch and drag input) is performed using the user's first hand, and a second pinch input is performed using the other hand (e.g., the second of the user's two hands). In some implementations, there is movement between the user's two hands (e.g., increasing and / or decreasing the distance or relative orientation between the user's two hands).

[0106] In some embodiments, a tap input performed as an air gesture (e.g., pointing at a user interface element) includes movement of a user's finger toward the user interface element, movement of the user's hand toward the user interface element (optionally, the user's finger extends toward the user interface element), downward movement of the user's finger (e.g., mimicking a mouse click or a tap on a touchscreen), or other movements of the user's hand. In some embodiments, the tap input performed as an air gesture is detected based on the movement characteristics of the finger or hand performing the tap gesture, which is the movement of the finger or hand away from the user's viewpoint and / or toward an object that is the target of the tap input, followed by the end of the movement. In some embodiments, the end of the movement is detected based on changes in the movement characteristics of the finger or hand performing the tap gesture (e.g., the end of movement away from the user's viewpoint and / or toward an object that is the target of the tap input, a reversal of the direction of movement of the finger or hand, and / or a reversal of the acceleration direction of the movement of the finger or hand).

[0107] In some implementations, the user's attention is determined to be directed to a portion of the 3D environment based on the detection of a gaze directed to that portion of the 3D environment (optionally, no other conditions are required). In some implementations, the user's attention is determined to be directed to that portion of the 3D environment based on the detection of a gaze directed to that portion of the 3D environment using one or more additional conditions, such as requiring the gaze to be directed to that portion of the 3D environment for at least a threshold duration (e.g., dwell time) and / or requiring the gaze to be directed to that portion of the 3D environment when the user's viewpoint is within a distance threshold from that portion of the 3D environment, so that the device determines that the user's attention is directed to that portion of the 3D environment, wherein if one of these additional conditions is not met, the device determines that the attention is not directed to the portion of the 3D environment to which the gaze is directed (e.g., until the one or more additional conditions are met).

[0108] In some implementations, the detection of the readiness configuration of a user or a portion of a user is performed by a computer system. The detection of the hand's readiness configuration is used by the computer system as an indication that the user may be preparing to interact with the computer system using one or more aerial gesture inputs performed by the hand (e.g., pinch, tap, pinch and drag, double pinch, long pinch, or other aerial gestures described herein). For example, the readiness of the hand is determined based on whether it has a predetermined hand shape (e.g., a pre-pinch shape with the thumb and one or more fingers extended and spaced apart in preparation for a pinch or grasping gesture, or a pre-tap with one or more fingers extended and the back of the hand facing the user), whether the hand is in a predetermined position relative to the user's viewpoint (e.g., below the user's head and above the user's waist and extending at least 15 cm, 20 cm, 25 cm, 30 cm, or 50 cm from the body), and / or whether the hand has moved in a particular manner (e.g., moving towards an area in front of the user above the user's waist and below the user's head, or moving away from the user's body or legs). In some implementations, the ready state is used to determine whether an interactive element of the user interface responds to attention (e.g., gaze) input.

[0109] In the context of inputting a reference to an aerial posture description, it should be understood that a similar posture can be detected using a hardware input device attached to or held by one or more of the user's hands. The positioning of the hardware input device in space can be tracked using optical tracking, one or more accelerometers, one or more gyroscopes, one or more magnetometers, and / or one or more inertial measurement units. The positioning and / or movement of the hardware input device is used in place of the positioning and / or movement of one or more hands in the corresponding aerial posture. In the context of describing input using air gestures, it should be understood that similar gestures can be detected using hardware input devices attached to or held by one or more of the user's hands. User input can be detected using controls included in the hardware input devices, such as one or more touch-sensitive input elements, one or more pressure-sensitive input elements, one or more buttons, one or more knobs, one or more dials, one or more joysticks, one or more hand or finger covers that can detect the positioning or positional changes of parts and / or fingers of the hand relative to each other, relative to the user's body, and / or relative to the user's physical environment, and / or other hardware input device controls, wherein user input using controls included in the hardware input devices is used in place of hand and / or finger gestures (such as air taps or air pinches corresponding to air gestures). For example, a selection input described as being performed using air tap or air pinch input can alternatively be detected using button presses, taps on touch-sensitive surfaces, presses on pressure-sensitive surfaces, or other hardware inputs. For example, motion input described as being performed using air pinching and dragging can alternatively be detected based on interaction with hardware input controls (such as pressing and holding a button, touching on a touch-sensitive surface, pressing on a pressure-sensitive surface, or other hardware input, followed by movement of the hardware input device (e.g., together with the hand associated with the hardware input device) in space). Similarly, two-handed input involving movement of hands relative to each other can be performed using an air gesture and a hardware input device not in the hand performing the air gesture, two hardware input devices held in different hands, or two air gestures performed by different hands using various combinations of air gestures and / or input detected by one or more of the aforementioned hardware input devices.

[0110] In some embodiments, the software may be downloaded to controller 110 electronically, for example, via a network, or alternatively, may be provided on a tangible, non-transitory medium such as an optical, magnetic, or electronic memory medium. In some embodiments, database 408 is also stored in memory associated with controller 110. Alternatively or additionally, some or all of the functions described for the computer may be implemented in dedicated hardware, such as custom or semi-custom integrated circuits or programmable digital signal processors (DSPs). Although in Figure 4The controller 110 is shown, but for example, as a separate unit from the image sensor 404, some or all of the controller's processing functions may be performed by a suitable microprocessor and software, or by dedicated circuitry within the housing of the image sensor 404 (e.g., a hand-tracking device), or by other devices associated with the image sensor 404. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television, handheld device, or head-mounted device) or with any other suitable computerized device (such as a game console or media player). The sensing function of the image sensor 404 may also be integrated into a computer or other computerized device controlled by the sensor output.

[0111] Figure 4 Also included is a schematic diagram of a depth map 410 captured by image sensor 404 according to some embodiments. As described above, the depth map comprises a matrix of pixels with corresponding depth values. Pixel 412 corresponding to the hand 406 has been segmented from the background and wrist in this map. The brightness of each pixel within the depth map 410 is inversely proportional to its depth value (i.e., the measured z-distance from image sensor 404), where gray shadows become darker as depth increases. Controller 110 processes these depth values ​​to identify and segment components of the image that have human hand features (i.e., groups of adjacent pixels). These features may include, for example, overall size, shape, and frame-to-frame motion from the depth map sequence.

[0112] Figure 4 The controller 110 also schematically illustrates, according to some embodiments, the hand skeleton 414 ultimately extracted from the depth map 410 of the hand 406. Figure 4 In this configuration, the hand skeleton 414 is superimposed on the hand background 416, which has been segmented from the original depth map. In some embodiments, key feature points of the hand, and optionally those on the wrist or arm connected to the hand (e.g., points corresponding to knuckles, fingertips, the center of the palm, the end of the hand connecting to the wrist, etc.), are identified and located on the hand skeleton 414. In some embodiments, the controller 110 uses the position and movement of these key feature points across multiple image frames to determine the pose performed by the hand or the current state of the hand.

[0113] Figure 5 An eye-tracking device 130 is illustrated. Figure 1 Example implementation of ). In some implementations, the eye-tracking device 130 comprises an eye-tracking unit 243 ( Figure 2The eye-tracking device 130 is used to track the positioning and movement of a user's gaze relative to scene 105 or relative to XR content displayed via display generation component 120. In some embodiments, the eye-tracking device 130 is integrated with the display generation component 120. For example, in some embodiments, when the display generation component 120 is a head-mounted device (such as a headset, helmet, goggles, or glasses) or a handheld device placed in a wearable frame, the head-mounted device includes both components for generating XR content for the user to view and components for tracking the user's gaze relative to the XR content. In some embodiments, the eye-tracking device 130 is separate from the display generation component 120. For example, when the display generation component is a handheld device or an XR room, the eye-tracking device 130 is optionally a separate device from the handheld device or XR room. In some embodiments, the eye-tracking device 130 is a head-mounted device or part of a head-mounted device. In some embodiments, the head-mounted eye-tracking device 130 is optionally used in conjunction with a display generation component that is also head-mounted or not head-mounted. In some embodiments, the eye-tracking device 130 is not a head-mounted device and is optionally used in conjunction with head-mounted display generation components. In some embodiments, the eye-tracking device 130 is not a head-mounted device and is optionally part of non-head-mounted display generation components.

[0114] In some embodiments, the display generation unit 120 uses display mechanisms (e.g., a left near-eye display panel and a right near-eye display panel) to display frames including left and right images in front of the user's eyes, thereby providing the user with a 3D virtual view. For example, the head-mounted display generation unit may include left and right optical lenses (referred to herein as eye lenses) located between the display and the user's eyes. In some embodiments, the display generation unit may include or be coupled to one or more external cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generation unit may have a transparent or semi-transparent display on which virtual objects are displayed, allowing the user to view the physical environment directly through the transparent or semi-transparent display. In some embodiments, the display generation unit projects virtual objects onto the physical environment. The virtual objects may, for example, be projected onto a physical surface or as holograms, allowing an individual to observe virtual objects superimposed on the physical environment using the system. In such cases, separate display panels and image frames for the left and right eyes may not be necessary.

[0115] like Figure 5As shown, in some embodiments, eye-tracking device 130 (e.g., gaze tracking device) includes at least one eye-tracking camera (e.g., an infrared (IR) or near-infrared (NIR) camera) and an illumination source (e.g., an array or ring of IR or NIR light sources, such as LEDs) that emits light (e.g., IR or NIR light) toward the user's eyes. The eye-tracking camera may be pointed at the user's eyes to receive IR or NIR light from the light source reflected directly from the eyes, or alternatively, it may be pointed at a "hot" mirror located between the user's eyes and the display panel, which reflects the IR or NIR light from the eyes back to the eye-tracking camera while allowing visible light to pass through. Eye-tracking device 130 optionally captures images of the user's eyes (e.g., as a video stream captured at 60 to 120 frames per second (fps), analyzes the images to generate gaze tracking information, and transmits the gaze tracking information to controller 110. In some embodiments, both of the user's eyes are tracked separately using corresponding eye-tracking cameras and illumination sources. In some embodiments, only one of the user's eyes is tracked using corresponding eye-tracking cameras and illumination sources.

[0116] In some implementations, a device-specific calibration procedure is used to calibrate the eye-tracking device 130 to determine parameters of the eye-tracking device for a specific operating environment 100, such as the 3D geometry and parameters of the LEDs, camera, thermal mirror (if present), eye lenses, and display. The device-specific calibration procedure can be performed at a factory or another facility before the AR / VR equipment is delivered to the end user. The device-specific calibration procedure can be automated or manual. According to some implementations, a user-specific calibration procedure may include estimations of eye parameters for a specific user, such as pupil position, foveal position, optical axis, visual axis, interocular distance, etc. According to some implementations, once the device-specific and user-specific parameters for the eye-tracking device 130 are determined, a flash-assisted method can be used to process the images captured by the eye-tracking camera to determine the user's current visual axis and gaze point relative to the display.

[0117] like Figure 5As shown, the eye-tracking device 130 (e.g., 130A or 130B) includes an eye lens 520 and a gaze tracking system. The gaze tracking system includes at least one eye-tracking camera 540 (e.g., an infrared (IR) or near-infrared (NIR) camera) positioned on the side of the user's face where eye tracking is performed, and an illumination source 530 (e.g., an IR or NIR light source, such as an array or ring of NIR light-emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye 592. The eye-tracking camera 540 may be pointed toward a mirror 550 located between the user's eye 592 and a display 510 (e.g., the left or right display panel of a head-mounted display, or the display of a handheld device, projector, etc.). This mirror reflects the IR or NIR light from the eye 592 while allowing visible light to pass through. Figure 5 (as shown in the top portion), or alternatively, it can be pointed towards the user's eye 592 to receive IR or NIR light reflected from the eye 592 (e.g., as shown in the top portion), Figure 5 (As shown in the bottom part).

[0118] In some implementations, controller 110 renders AR or VR frames 562 (e.g., left and right frames for the left and right display panels) and provides frames 562 to display 510. Controller 110 uses gaze tracking input 542 from eye-tracking camera 540 for various purposes, such as processing frame 562 for display. Controller 110 optionally estimates the user's gaze point on display 510 based on the gaze tracking input 542 obtained from eye-tracking camera 540 using a flash-assisted method or other suitable method. The gaze point estimated based on gaze tracking input 542 is optionally used to determine the direction the user is currently looking.

[0119] The following describes several possible use cases for the user's current gaze direction and is not intended to be limiting. As an example use case, controller 110 may render virtual content in different ways based on the determined user gaze direction. For example, controller 110 may generate virtual content at a higher resolution in the concave region determined according to the user's current gaze direction than in the peripheral region. Alternatively, the controller may position or move virtual content in the view based at least partially on the user's current gaze direction. Alternatively, the controller may display specific virtual content in the view based at least partially on the user's current gaze direction. As another example use case in AR applications, controller 110 may guide an external camera used to capture the physical environment of an XR experience to focus in the determined direction. The external camera's autofocus mechanism may then focus on an object or surface in the environment that the user is currently looking at on display 510. As another example use case, eye lens 520 may be a focusable lens, and the controller uses gaze tracking information to adjust the focus of eye lens 520 so that the virtual object the user is currently looking at has appropriate convergence to match the convergence of the user's eyes 592. The controller 110 can use gaze tracking information to guide the eye lens 520 to adjust its focus so that the nearby object that the user is looking at appears at the correct distance.

[0120] In some embodiments, the eye-tracking device is part of a head-mounted device that includes a display (e.g., display 510), two eye lenses (e.g., eye lens 520), an eye-tracking camera (e.g., eye-tracking camera 540), and a light source (e.g., light source 530 (e.g., IR or NIR LED)) mounted in a wearable housing. The light source emits light (e.g., IR or NIR light) toward the user's eyes 592. In some embodiments, the light source may be arranged in a ring or circle around each lens in the head-mounted device, such as... Figure 5 As shown. In some embodiments, for example, eight light sources 530 (e.g., LEDs) are arranged around each lens 520. However, more or fewer light sources 530 may be used, and other arrangements and positions of the light sources 530 may be used.

[0121] In some embodiments, the display 510 emits light in the visible light range and does not emit light in the IR or NIR range, and therefore does not introduce noise into the gaze tracking system. It should be noted that the positions and angles of the eye-tracking camera 540 are given by way of example and are not intended to be limiting. In some embodiments, a single eye-tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user's face. In some embodiments, cameras 540 with a wider field of view (FOV) and cameras 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, cameras 540 operating at one wavelength (e.g., 850 nm) and cameras 540 operating at different wavelengths (e.g., 940 nm) may be used on each side of the user's face.

[0122] like Figure 5 The illustrated gaze tracking system implementation can be used, for example, in computer-generated reality, virtual reality, and / or mixed reality applications to provide users with computer-generated reality, virtual reality, augmented reality, and / or augmented virtual experiences.

[0123] Figure 6 Examples of flash-assisted gaze tracking pipelines according to some embodiments are illustrated. In some embodiments, the gaze tracking pipeline uses a flash-assisted gaze tracking system (e.g., such as...) Figure 1 and Figure 5 The illustrated eye-tracking device 130) is used for implementation. The flash-assisted gaze tracking system can maintain a tracking state. Initially, the tracking state is off or "no". When in tracking state, the flash-assisted gaze tracking system uses previous information from previous frames when analyzing the current frame to track the pupil outline and flash in the current frame. When not in tracking state, the flash-assisted gaze tracking system attempts to detect the pupil and flash in the current frame, and if successful, initializes the tracking state to "yes" and continues to the next frame in tracking state.

[0124] like Figure 6 As shown, the gaze-tracking camera captures left and right images of the user's left and right eyes. The captured images are then fed into a gaze-tracking pipeline for processing to begin at 610. As indicated by the arrow returning to element 600, the gaze-tracking system can continue capturing images of the user's eyes, for example, at a rate of 60 to 120 frames per second. In some embodiments, each set of captured images may be fed into the pipeline for processing. However, in some embodiments or under certain conditions, not all captured frames are processed by the pipeline.

[0125] At 610, for the currently captured image, if the tracking state is yes, the method proceeds to element 640. At 610, if the tracking state is no, the image is analyzed as indicated at 620 to detect the user's pupil and flash in the image. At 630, if the pupil and flash are successfully detected, the method proceeds to element 640. Otherwise, the method returns to element 610 to process the next image of the user's eye.

[0126] At 640, if proceeding from element 610, the current frame is analyzed to track the pupil and flashes in part based on previous information from the previous frame. At 640, if proceeding from element 630, the tracking state is initialized based on the pupil and flashes detected in the current frame. The processing result at element 640 is checked to verify that the tracking or detection result can be credible. For example, the result can be checked to determine whether a sufficient number of pupils and flashes used for gaze estimation were successfully tracked or detected in the current frame. At 650, if the result cannot be credible, the tracking state is set to no at element 660, and the method returns to element 610 to process the next image of the user's eye. At 650, if the result is credible, the method proceeds to element 670. At 670, the tracking state is set to yes (if it is not already yes), and the pupil and flash information is passed to element 680 to estimate the user's gaze point.

[0127] Figure 6 This is intended as an example of an eye-tracking technology that can be used in a particular specific implementation. As will be appreciated by those skilled in the art, other eye-tracking technologies, currently existing or to be developed in the future, may be used in place of the flash-assisted eye-tracking technology described herein or in combination with the flash-assisted eye-tracking technology, depending on the various implementations.

[0128] In this disclosure, various input methods are described in relation to interaction with a computer system. When an example is provided using one input device or method, and another example is provided using another input device or method, it should be understood that each example is compatible with and optionally utilizes the input device or method described with respect to the other example. Similarly, various output methods are described in relation to interaction with a computer system. When an example is provided using one output device or method, and another example is provided using another output device or method, it should be understood that each example is compatible with and optionally utilizes the output device or method described with respect to the other example. Similarly, various methods are described in relation to interaction with a virtual or mixed reality environment via a computer system. When an example is provided using interaction with a virtual environment, and another example is provided using a mixed reality environment, it should be understood that each example is compatible with and optionally utilizes the methods described with respect to the other example. Therefore, this disclosure discloses embodiments that are combinations of features of a plurality of examples without exhaustively listing all features of the embodiments in the description of each example embodiment.

[0129] User interface and related processes

[0130] Attention will now be focused on implementations of user interfaces (“UI”) and associated processes, which can be implemented on computer systems such as portable multifunction devices or head-mounted devices that communicate with display generation components, one or more input devices, and optionally one or more cameras.

[0131] Figures 7A to 7AFThis includes an example of a three-dimensional environment visible via a display generation component (e.g., display generation component 120) and interactions occurring within the three-dimensional environment caused by user input directed towards the three-dimensional environment and / or input received from other computer systems and / or sensors. In some embodiments, the portion of the three-dimensional environment visible to the user via the display generation component is referred to as a viewport (e.g., a virtual viewport corresponding to the portion of the three-dimensional environment visible to the user of the device when viewing the three-dimensional environment via the display generation component). In some embodiments, input is directed towards a virtual object within the three-dimensional environment by a user gaze detected in an area occupied by a virtual object or by a gesture performed at a location in the physical environment corresponding to the area of ​​the virtual object. In some embodiments, input is directed towards a virtual object within the three-dimensional environment by a gesture performed when the virtual object has input focus (e.g., when the virtual object has been selected by concurrently and / or previously detected gaze input, by concurrently or previously detected pointer input, and / or by concurrently and / or previously detected gesture input) (e.g., optionally, at a location in the physical environment independent of the area of ​​the virtual object in the three-dimensional environment). In some embodiments, input is directed to a virtual object within the 3D environment by an input device that has positioned a focus selector object (e.g., a pointer object or selector object) at the location of the virtual object. In some embodiments, input is directed to a virtual object within the 3D environment via other means (e.g., voice and / or control buttons). In some embodiments, input is directed to a physical object or a representation of a virtual object corresponding to a physical object by user hand movements (e.g., whole hand movement, whole hand movement in a corresponding posture, movement of a portion of the user's hand relative to another portion of the hand, and / or relative movement between the hands) and / or manipulations relative to a physical object (e.g., touch, swipe, tap, open, oriented movement, and / or relative movement). In some embodiments, the computer system displays changes in the 3D environment based on input from sensors (e.g., image sensors, temperature sensors, biometric sensors, motion sensors, and / or proximity sensors) and contextual conditions (e.g., location, time, and / or the presence of other people in the environment) (e.g., displaying additional virtual content, stopping the display of existing virtual content, and / or transitioning between different levels of immersion in the display of visual content). In some implementations, the computer system displays changes in the 3D environment based on input from other computers used by other users sharing the computer-generated environment with the user of the computer system (e.g., in a shared computer-generated experience, in a shared virtual environment, and / or in a shared virtual or augmented reality environment in a communication session). These changes may include displaying additional virtual content, stopping the display of existing virtual content, and / or transitioning between different levels of immersion in the display of visual content.In some implementations, the computer system displays changes in the three-dimensional environment based on input from sensors (e.g., displaying movement, deformation, and / or changes in visual characteristics of user interfaces, virtual surfaces, user interface objects, and / or virtual landscapes), the sensors detecting movement of other people and objects, as well as movement of the user that may not conform to the criteria of identified gestural input that triggers associated operations of the computer system.

[0132] In some embodiments, the 3D environment visible through a viewport provided by the display generation components described herein is a virtual 3D environment without a representation of a physical environment, comprising virtual objects and content at different virtual locations within the 3D environment. In some embodiments, the 3D environment is a mixed reality environment that displays virtual objects at different virtual locations within the 3D environment constrained by one or more physical aspects of the physical environment (e.g., the location and orientation of walls, floors, surfaces, direction of gravity, time of day, and / or spatial relationships between physical objects). In some embodiments, the 3D environment is an augmented reality environment that includes a representation of a physical environment. In some embodiments, the representation of the physical environment includes corresponding representations of physical objects and surfaces at different locations within the 3D environment, such that spatial relationships between different physical objects and surfaces in the physical environment are reflected by spatial relationships between representations of physical objects and surfaces in the 3D environment. In some embodiments, when a virtual object is positioned relative to a representation of physical objects and surfaces in the 3D environment, the virtual object appears to have a corresponding spatial relationship with the physical objects and surfaces in the physical environment. In some implementations, the computer system transitions between different types of environments based on user input and / or contextual conditions (e.g., transitions between computer-generated environments or experiences presenting different levels of immersion, and / or adjusts the relative salience of audio / visual sensory inputs from virtual content and representations from the physical environment).

[0133] In some embodiments, the display generation component includes a transparent portion in which a representation of the physical environment is displayed. In some embodiments, the transparent portion of the display generation component is a transparent or semi-transparent (e.g., see-through) portion of the display generation component that reveals at least a portion of the physical environment surrounding the user or within the user's field of view. For example, the transparent portion is a portion of a head-mounted display or head-up display that is made semi-transparent (e.g., less than 50%, 40%, 30%, 20%, 15%, 10%, or 5% opacity) or transparent, allowing the user to view the real world around them without removing the head-mounted display or moving away from the head-up display. In some embodiments, when displaying a virtual or mixed reality environment, the transparent portion gradually transitions from semi-transparent or transparent to completely opaque. In some embodiments, the transparent portion of the display generation component displays a live feed of images or video of at least a portion of the physical environment captured by one or more cameras (e.g., a rear-facing camera on a mobile device or associated with a head-mounted display, or other cameras that feed image data to a computer system). In some embodiments, the one or more cameras are pointed at a portion of the physical environment that is directly in front of the user's eyes (e.g., behind the user relative to the display generating component). In some embodiments, the one or more cameras are pointed at a portion of the physical environment that is not directly in front of the user's eyes (e.g., in a different physical environment, or to the side or behind the user).

[0134] In some embodiments, when virtual objects are displayed at locations corresponding to the positions of one or more physical objects in a physical environment (e.g., locations in a virtual reality, mixed reality, or augmented reality environment), at least some of the virtual objects are displayed to replace a portion of the camera's live view (e.g., a portion of the physical environment captured in the live view) (e.g., replacing its display). In some embodiments, at least some of the virtual objects and content are projected onto a physical surface or blank space in the physical environment and are visible through a transparent portion of the display generating component (e.g., visible as part of the camera view of the physical environment, or visible through a transparent or semi-transparent portion of the display generating component). In some embodiments, at least some of the virtual objects and virtual content are displayed as part of an overlay display and obscure a view of at least a portion of the physical environment visible through a transparent or semi-transparent portion of the display generating component.

[0135] In some embodiments, the display generation component displays different views of the 3D environment based on user input or movement that alters the virtual positioning of the viewpoint relative to the 3D environment of the currently displayed view. In some embodiments, when the 3D environment is virtual, the viewpoint moves according to navigation or motion requests (e.g., aerial gestures and / or gestures performed by moving one part of the hand relative to another part of the hand), without requiring movement of the user's head, torso, and / or the display generation component in the physical environment. In some embodiments, movement of the user's head and / or torso relative to the physical environment and / or movement of the display generation component or other position sensing elements of the computer system (e.g., due to the user holding the display generation component while wearing or wearing an HMD) causes a corresponding movement of the viewpoint relative to the 3D environment (e.g., with corresponding changes in direction, distance, speed, and / or orientation), thereby causing a corresponding change in the currently displayed view of the 3D environment. In some implementations, when a virtual object has a corresponding spatial relationship with a viewpoint (e.g., anchored or fixed to the viewpoint, and / or maintaining its spatial relationship with the 3D environment through its visible viewport), movement of the viewpoint relative to the 3D environment will cause movement of the virtual object relative to the 3D environment, while maintaining the virtual object's positioning in the field of view and / or viewport (e.g., the virtual object is referred to as head-locked). In some implementations, the virtual object is body-locked to the user and moves relative to the 3D environment as the user moves as a whole in the physical environment (e.g., carrying or wearing display generation components and / or other position sensing components of the computer system), but will not move in the 3D environment in response to individual user head movements (e.g., the display generation components and / or other position sensing components of the computer system rotate around a fixed position of the user in the physical environment). In some implementations, the virtual object is optionally locked to another part of the user, such as the user's hand or wrist, and moves in the 3D environment according to the movement of that part of the user in the physical environment to maintain a spatial relationship between the positioning of the virtual object and the virtual positioning of that part of the user in the 3D environment. In some implementations, the virtual object is locked to a portion of the field of view provided by the display generation component (e.g., having a fixed spatial relationship with the 3D environment through its visible viewport, and / or with the user's viewpoint), and moves in the 3D environment according to the movement of the field of view within the viewport, regardless of the user's movement that does not cause the field of view to change within the viewport.

[0136] In some implementation schemes, such as Figures 7A to 7XAs shown, the view of the 3D environment sometimes does not include a representation of the user's hands, arms, and / or wrists. In some embodiments, a representation of the user's hands, arms, and / or wrists is included in the view of the 3D environment. In some embodiments, a representation of the user's hands, arms, and / or wrists is included in the view of the 3D environment as part of a representation of the physical environment provided via a display generation component. In some embodiments, these representations are not part of the representation of the physical environment and are captured separately (e.g., pointed at the user's hands, arms, and wrists by one or more cameras) and displayed in the 3D environment independently of the currently displayed view of the 3D environment. In some embodiments, these representations include camera images captured by one or more cameras of a computer system or stylized versions of the arms, wrists, and / or hands based on information captured by various sensors. In some embodiments, these representations replace a portion of the display of the representation of the physical environment, overlay that portion of the representation of the physical environment, or obscure that portion of the view of the representation of the physical environment. In some embodiments, when the display generating component does not provide a view of the physical environment and provides a completely virtual environment (e.g., no camera view and no transparent portions), a real-time visual representation (e.g., a stylized representation or segmented camera image) of one or both of the user's arms, wrists, and / or hands may optionally still be displayed in the virtual environment. In some embodiments, if no representation of the user's hands is provided in the view of the three-dimensional environment, the positioning corresponding to the user's hands may optionally be indicated in the three-dimensional environment, for example, by altering the appearance of the virtual content at a positioning in the three-dimensional environment corresponding to the position of the user's hands in the physical environment (e.g., by altering the translucency and / or simulating changes in reflectivity). In some embodiments, the representation of the user's hands or wrists is outside the currently displayed view of the three-dimensional environment, while the virtual positioning in the three-dimensional environment corresponding to the position of the user's hands or wrists is outside the current field of view provided via the display generating component; and in response to the virtual positioning corresponding to the position of the user's hands or wrists moving within the current field of view due to movement of the display generating component, the user's hands or wrists, the user's head, and / or the user as a whole, the representation of the user's hands or wrists becomes visible in the view of the three-dimensional environment.

[0137] Figures 7A to 7K An example is illustrated of displaying multiple indications for accessing system functions of a computer system in response to detecting a first gaze input that meets attention criteria for a first user interface object. Figure 8 This is a flowchart of an exemplary method 8000 for displaying multiple indications of system functions for accessing a computer system in response to detecting a first gaze input to a user interface object that satisfies an attention criterion for a first user interface object. Figures 7A to 7KThe user interface in the document is used to illustrate the processes described below, including... Figure 8 The process in.

[0138] Figure 7A An example is illustrated of a physical environment 7000 including a user 7002 interacting with a computer system 7100. The user 7002 has two hands: hand 7020 and hand 7022. The user's left arm 7028, connected to the user's left hand 7020, is also shown. The physical environment 7000 includes physical objects 7014 and physical walls 7004 and 7006. The physical environment 7000 also includes a physical floor 7008. Figures 7A to 7X As shown in the example, the display generating component of the computer system 7100 is a touchscreen held by the user 7002. In some embodiments, the display generating component of the computer system 7100 is a head-mounted display worn on the head of the user 7002 (e.g., in...). Figures 7A to 7X The content shown is visible via the display generation component of computer system 7100 and corresponds to the field of view of user 7002 when wearing a head-mounted display. In some embodiments, the display generation component is a stand-alone display, projector, or another type of display. In some embodiments, the computer system communicates with one or more input devices, which include cameras or other sensors and input devices that detect movement of the user's hands, movement of the user's whole body, and / or movement of the user's head in the physical environment. In some embodiments, the one or more input devices detect movement of the user's hands, face, and / or whole body and current posture, orientation, and position. In some embodiments, user input is detected via a touch-sensitive surface or touchscreen. In some embodiments, one or more input devices include eye-tracking components that detect the location and movement of the user's gaze. In some embodiments, the display generation component and optionally one or more input devices, along with the computer system, are part of a head-mounted device that moves and rotates with the user's head in the physical environment and changes the user's viewpoint in a three-dimensional environment provided by the display generation component. In some embodiments, the display generating component is a heads-up display that does not move or rotate with the user's head or entire body, but optionally changes the user's viewpoint in the three-dimensional environment based on the movement of the user's head or body relative to the display generating component. In some embodiments, the display generating component is optionally moved and rotated by the user's hand relative to the physical environment or relative to the user's head, and changes the user's viewpoint in the three-dimensional environment based on the movement of the display generating component relative to the user's head or face or relative to the physical environment.

[0139] Figure 7BA first view of a three-dimensional environment (e.g., a virtual three-dimensional environment, an augmented reality environment, a perspective view of a physical environment, or a camera view of a physical environment) visible to user 7002 via computer system 7100 is illustrated, the first view including an indicator 7010 for a system function menu. For example, according to some embodiments, the indicator 7010 is a point, a graphical object, or other type of indication for triggering the display of a system function menu, which includes an indication corresponding to one or more system functions. Depending on user 7002 being located at position 7026-a within physical environment 7000, Figure 7B The view of the illustrated 3D environment is visible to user 7002 via computer system 7100. In some embodiments, the 3D environment is a virtual 3D environment without a representation of the physical environment 7000. In some embodiments, the 3D environment is a mixed reality environment, which is a virtual environment augmented by sensor data corresponding to the physical environment. In some embodiments, the 3D environment is an augmented reality environment, which includes representations of one or more virtual objects and at least a portion of the physical environment (e.g., representations of walls 7004', 7006', a floor 7008', and / or a representation of physical objects 7014'). In some embodiments, the representation of the physical environment includes a camera view of the physical environment. In some embodiments, the representation of the physical environment includes a view of the physical environment through transparent or semi-transparent portions of the computer system or a first display generating component.

[0140] In some embodiments, the system function menu indicator 7010 is displayed in the peripheral area of ​​the viewport provided by the first display generating component (e.g., within a threshold distance from the edge (e.g., 0.5 mm, 1 cm, 2 cm, 5 cm, or any threshold distance between 0 cm and 5 cm)) through which the three-dimensional environment is visible (e.g., relative to the field of view of a user using the first display generating component). Figure 7B As shown, the system function menu indicator 7010 is displayed in the peripheral area of ​​the top edge of the display of the computer system 7100. In embodiments where the display generating component of the computer system 7100 is a head-mounted display, the system function menu indicator 7010 will be displayed in the peripheral area of ​​the user's field of view when viewing the three-dimensional environment via the display generating component. In some embodiments, the system function menu indicator 7010 has a default appearance (e.g., a gray indicator for the system function menu). In some embodiments, the appearance of the system function menu indicator 7010 changes according to one or more sets of criteria (e.g., based on determining that a first event for a first notification meets a timing criterion, the system function menu indicator 7010 has a first appearance (e.g., as referenced below)). Figures 7L to 7Q(described in more detail), and / or based on the determination that a request exists to allow the computer system to join a first communication session that meets timing criteria, the system function menu indicator 7010 has a second appearance (e.g., as referred to below). Figures 7R to 7X (More detailed description)). In some embodiments, the system function menu indicator 7010 is semi-transparent, or otherwise has an appearance that is at least partially based on a three-dimensional environment over which the system function menu indicator 7010 is displayed (e.g., a representation based on the surrounding physical environment (e.g., real-world content) and / or based on computer-generated virtual content). For example, in Figure 7B In this context, the indicator 7010 of the system function menu has an appearance based on the appearance of the wall 7004' (e.g., which is located "below" or "behind" the indicator 7010 of the system function menu in a three-dimensional environment).

[0141] In some implementations, the system function menu indicator 7010 continues to be displayed even when the user interacts with other user interfaces, user interface objects, and / or user interface elements in the three-dimensional environment. For example, the system function menu indicator 7010 continues to be displayed even when the user's attention is directed at a virtual object 7012. The user may interact with or manipulate the virtual object 7012 (e.g., using the user's gaze, aerial gestures, and / or verbal input), and the computer system 7100 will continue to display the system function menu indicator 7010. In some implementations, the system function menu indicator 7010 is visually diminished (e.g., dimmed and / or blurred) when the user interacts with other user interfaces, user interface objects, and / or user interface elements (e.g., to avoid distracting the user when their attention (e.g., gaze) is directed at other user interface elements).

[0142] In some embodiments, the system function menu indicator 7010 is displayed when the user's attention meets a proximity criterion for the system function menu indicator 7010, and the display of the system function menu indicator 7010 is stopped in response to detecting that the user's attention no longer meets the proximity criterion for the system function menu indicator 7010. For example, if the user's attention moves to the center of the display generating component of the computer system 7100, the computer system 7100 stops displaying the system function menu indicator 7010. In some embodiments, after stopping the display of the system function menu indicator 7010, the system function menu indicator is displayed again in response to the user's attention again meeting the proximity criterion (e.g., in response to detecting that the user's attention has returned to a position near the system function menu indicator 7010). Figure 7D As shown), the system function menu indicator 7010 is displayed again.

[0143] exist Figure 7CIn the middle, user 7002 has moved from the first position 7026-a (e.g., ... Figure 7B (As shown) Move to the new location 7026-b in physical environment 7000. (See diagram) Figure 7C As shown, a second view of the three-dimensional environment reflecting the new position of user 7002 is visible via computer system 7100. For example, a virtual object 7012 (e.g., a virtual sphere) is shown as being in a different position (e.g., relative to...). Figure 7B The computer system 7100 in the image is shifted to the left), and the entire object 7014' (e.g., a representation of the physical object 7014) is now visible. In contrast to the virtual objects 7012 and 7014', the system function menu indicator 7010 maintains the same spatial relationship relative to the user's viewpoint (e.g., maintaining the same position on the screen displaying the generated components, maintaining the same spatial relationship with the 3D environment through its visible viewport, and / or maintaining the same position within the user's field of view via the head-mounted display, also referred to as "viewpoint-locked"). In some embodiments, the system function menu indicator 7010 exhibits "lazy following" behavior. For example, Figure 7C Position 7016 in the text reflects the previous location of indicator 7010 in the three-dimensional environment (…). Figure 7B (in the middle), and position 7018 reflects the intermediate position of the system function menu indicator 7010 (for example, because the system function menu indicator 7010 is displayed as moving from position 7016 more slowly than the user's viewpoint when the user 7002 moves from position 7026-a to a new position 7026-b). Figure 7C (Location of indicator 7010 in the system function menu).

[0144] In some implementations, as the system function menu indicator 7010 moves from position 7016 to position 7018 and / or from position 7018 to... Figure 7C As shown in the final positioning, the indicator 7010 of the system function menu changes appearance. In some embodiments, as the indicator 7010 of the system function menu moves from position 7016 to position 7018 and / or from position 7018 to... Figure 7C As indicated, the system function menu indicator 7010 fades out, becomes blurred, or is otherwise visually obscured. In some embodiments, the system function menu indicator 7010 stops displaying during user movement (e.g., as the user's viewpoint changes) and re-displays after the user (e.g., as the user's viewpoint changes) has moved (e.g., after...). Figure 7C (At the location shown).

[0145] Figure 7D An example of user attention 7116 (e.g., sometimes referred to as "user gaze") is shown. Figure 7E In this system, the user's attention 7116 is directed to the indicator 7010 of the system function menu (e.g., the user gazes at the indicator 7010 of the system function menu). If the user's gaze meets the attention criterion, the computer system 7100 displays the system function menu 7024, which includes one or more indications for triggering corresponding system functions. In some embodiments, the user's gaze meets the attention criterion when the user gazes (e.g., the user's attention 7116 is directed) at the position of the indicator 7010 of the system function menu for a threshold amount of time (e.g., at least 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds, 30 seconds, or 60 seconds). In some implementations, a user's gaze meets an attention criterion when the user looks at (e.g., the user's attention 7116 is directed towards) a first region in the view of the 3D environment (e.g., defined as a first region within the visible area surrounding or substantially within the visible area of ​​the system function menu indicator 7010) for a threshold amount of time (e.g., 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds, 30 seconds, or 60 seconds). In some implementations, a user's gaze meets an attention criterion when the user looks at (e.g., the user's attention 7116 is directed towards) the system function menu indicator 7010 and brings their hand into a specified configuration (e.g., a ready state configuration) or performs a specified hand gesture (e.g., an air gesture such as an air tap or air pinch, or another selection input). The system function menu indicator 7010 is sometimes referred to as a "system control indicator" because it indicates a position in the user's field of view that the user can interact with (e.g., gaze at) to trigger the display of the system function menu 7024 (and / or other user interface elements, as described herein). In contrast, in Figure 7D In the process, the user's attention 7116 is not directed to the indicator 7010 of the system function menu (e.g., and has not been directed to the indicator 7010 of the system function menu recently and / or for a long time to meet the attention criterion), therefore the system function menu 7024 is not displayed.

[0146] The system function menu 7024 includes multiple display options for accessing the system functions of the computer system 7100. For example, such as... Figure 7EAs shown, the system function menu 7024 includes a homepage display 7124 (e.g., for accessing one or more applications of the computer system 7100, for initiating a communication session with one or more contacts stored in the memory of the computer system 7100, and / or for initiating the display of a virtual environment), a volume display 7038 for adjusting the volume settings of the computer system 7100, a search display 7042 for searching content stored on the computer system 7100 (e.g., documents, photos, media files, and / or applications) and / or content from the Internet (e.g., Internet search results), a notification display 7044 for displaying multiple notifications (e.g., notifications recently received or generated by the computer system 7100), a control display 7046 for accessing additional settings of the computer system 7100 (e.g., less frequently used settings not displayed in the system function menu 7024), and a virtual assistant display 7048 for invoking and accessing the functions of the virtual assistant. In some embodiments, the system function menu 7024 includes... Figure 7E The power indicators shown are different power indicators (e.g., additional power indicators, fewer power indicators, and / or some power indicators replaced by other power indicators associated with different system functions). In some embodiments, the system function menu 7024 is at least partially customizable, allowing the user to add one or more power indicators to the system function menu 7024 and / or remove power indicators from the system function menu 7024 (e.g., Figure 7E One or more of the power indicators shown in the diagram. User interaction with the power indicators in system function menu 7024 is discussed below. Figure 7J and Figure 7K A more detailed description follows. In some implementations, in addition to multiple capability indicators for accessing the system functions of the computer system 7100, the system function menu 7024 also includes status information about the computer system 7100 (e.g., Wi-Fi connectivity status, cellular connectivity status, current time and / or battery charging status).

[0147] In some embodiments, the system function menu 7024 is displayed within a threshold distance (e.g., within 0.5 cm, 1 cm, 2 cm, or 5 cm) of the system function menu indicator 7010. In some embodiments, the system function menu 7024 is displayed with a corresponding spatial relationship to the system function menu indicator 7010 (e.g., directly below, above, to the right, or to the left of the system function menu indicator, or other desired spatial relationship). In some embodiments, the system function menu 7024 is displayed on top of other visible user interfaces in a corresponding view of the three-dimensional environment. For example, if the application launch user interface and / or application user interface are displayed (e.g., before the system function menu 7024 is displayed, and at least partially overlap with the area above which the system function menu 7024 will be displayed), the system function menu 7024 is displayed on at least a portion of the application launch user interface and / or application user interface when invoked. In some implementations, the system function menu 7024 is displayed such that it appears closer to the user's viewpoint than the application launch user interface and / or the application user interface (e.g., because the system function menu 7024 is displayed above at least a portion of the application launch user interface and / or the application user interface, and / or because the system function menu 7024 appears larger than the application launch user interface and / or the application user interface).

[0148] In some embodiments, displaying the system function menu 7024 includes an animation transition (e.g., the appearance of the system function menu 7024). For example, the animation transition may include an animation of the system function menu 7024 expanding downwards from the indicator 7010 of the system function menu. Alternatively or additionally, the animation transition may include an animation of the system function menu 7024 gradually appearing or fading into the view. In some embodiments, the displayed animation transition depends on the positioning of the indicator 7010 of the system function menu. For example, if the indicator 7010 of the system function menu is instead displayed near the left edge of the display generating component, the animation transition optionally includes an animation of the system function menu 7024 expanding outwards to the right from the indicator 7010 of the system function menu. More generally, the animation transition optionally includes an animation of the system function menu 7024 expanding outwards from the indicator 7010 of the system function menu (e.g., towards the center of the view of the visible three-dimensional environment). In some implementations, when the system function menu 7024 is displayed, the indicator 7010 of the system function menu is visually diminished (e.g., dimmed, faded, grayed out, and / or blurred) or not displayed, as in... Figure 7E The dashed outline of indicator 7010 in the system function menu is indicated in the middle.

[0149] In some implementation schemes, such as Figure 7FAs shown, system function menu 7024 exhibits lazy following behavior. Specifically, Figure 7F An example embodiment is shown in which both system function menu 7024 and system function menu indicator 7010 exhibit lazy follow behavior. In some embodiments, the lazy follow behavior of system function menu indicator 7010 differs from that of system function menu 7024. For example, Figure 7F This illustrates how the system function menu indicator 7010 moves over time from indicator position 7030 to indicator position 7032, and then to... Figure 7F The current position of the system function menu indicator 7010 is determined by the system function menu 7024 (e.g., within the same time period), which moves from system function menu position 7034 to system function menu position 7036, and then to... Figure 7F The current location is shown in system function menu 7024. The distance indicators for locations 7030 and 7032 are also shown. Figure 7F The current location of indicator 7010 in the system function menu is farther than the distance between system function menu locations 7034 and 7036. Figure 7F The current position of the system function menu 7024 is closer, which indicates that the indicator 7010 of the system function menu follows the user's view more closely (e.g., faster) than the system function menu 7024.

[0150] In some implementations, when (e.g., in response to detection) the user's gaze no longer meets the attention criteria (e.g., the user's attention 7116 is no longer directed at the system function menu indicator 7010 or system function menu 7024), the computer system 7100 stops displaying the system function menu 7024. In some implementations, even after the user's gaze no longer meets the attention criteria, the system function menu 7024 may remain displayed for a period of time (e.g., several seconds, optionally taking into account lazy following behavior to allow the system function menu 7024 to settle into a new viewpoint lock position), and if the user's gaze subsequently meets the attention criteria again (e.g., within the threshold amount of time during which the attention criteria were no longer met), the system function menu may remain displayed. For example, in an embodiment where the indicator 7010 of the system function menu exhibits lazy following behavior, the system function menu 7024 may remain displayed even when the user's viewpoint moves (e.g., when the user moves the touchscreen device or turns their head while wearing a head-mounted display), and the user's gaze does not continuously point to the indicator 7010 of the system function menu (e.g., the user's gaze does not precisely track the indicator 7010 of the system function menu).

[0151] Figure 7GAn example is given of user attention 7116 directed to the volume indicator 7038 included in the system function menu 7024. In some implementations, such as Figure 7H As shown, in response to detecting user attention 7116 directed at the volume indicator 7038, the computer system 7100 changes the appearance of the volume indicator 7038 (e.g., highlighting the volume indicator 7038, increasing the thickness of the boundary of the volume indicator 7038, adding a selection outline to the volume indicator 7038, changing the color of the volume indicator 7038, and / or changing the size of the volume indicator 7038) to indicate that the volume indicator 7038 is selected for further user interaction. In some embodiments, the computer system 7100 changes the appearance of the volume indicator 7038 in response to detecting that the user's hand is in a ready configuration. In some embodiments, the computer system 7100 changes the appearance of the volume indicator 7038 in response to detecting both user attention 7116 directed at the volume indicator 7038 and the user's hand being in a ready configuration. In some implementations, the computer system 7100 does not change the appearance of the volume indicator 7038 (e.g., regardless of whether the user's attention 7116 is directed at the volume indicator 7038 and / or whether the user's hand is in a ready configuration).

[0152] In some embodiments, in response to detecting that the user's attention 7116 is directed at the volume indicator 7038, the computer system 7100 outputs additional content associated with the volume indicator 7038 (e.g., a description of the volume setting associated with the volume indicator 7038, instructions for adjusting the volume setting associated with the volume indicator 7038, and / or the current value of the volume setting associated with the volume indicator 7038), which is sometimes referred to as a "tooltip". In some embodiments, in response to detecting that the user's attention 7116 is directed at the volume indicator 7038, the tooltip is displayed after a certain delay (e.g., a delay of 1 second, 5 seconds, or any threshold time between 0 seconds and 10 seconds). In some embodiments, the tooltip is displayed adjacent to the volume indicator 7038 (e.g., immediately above, immediately below, to the left, or to the right of it). In some embodiments, the tooltip is displayed as a partial overlay on at least a portion of the volume indicator 7038 (and / or other indicator in the system function menu 7024).

[0153] Figure 7HIt is also illustrated that, in response to detecting active input pointing towards the volume indicator 7038, the computer system displays system space 7040. In some embodiments, detecting active input includes detecting that the user's attention 7116 is pointing towards the volume indicator 7038. In some embodiments, detecting active input includes detecting input performed by the user 7002 (e.g., such as...) when the volume indicator 7038 is selected (e.g., based on the user 7002's attention 7116 pointing towards the volume indicator 7038). Figure 7H (Indicated by the hand 7020 shown with a moving arrow). Examples of inputs performed by the user 7002 include air gestures (e.g., air pinch or air tap), taps on a touch-sensitive surface, button presses, or other toggle inputs and / or voice commands. Because the activated input points to the volume indicator 7038, the system space 7040 is the system space for adjusting the volume settings of the computer system 7100. In some embodiments, the displayed system space 7040 changes according to the corresponding indicator associated with the system space 7040 activated from the system function menu 7024. For example, if the user's attention 7116 is instead directed to the control indicator 7046, the system space 7040 will be the system space with the control panel user interface, rather than the system space for adjusting the volume settings of the computer system 7100. Examples of different system spaces displayed in response to activation of different indicator in the system function menu 7024 are referenced herein. Figure 7K The following description is provided. In some embodiments, the system space 7040 or additional controls displayed based on interaction with the system function menu are arranged in a corresponding space relative to the system function menu (e.g., to the left, right, above, or below the system function menu) and optionally displayed adjacent to the system function menu (e.g., less than a threshold distance from the system function menu but not overlapping with the system function menu, wherein the threshold distance is less than half, a quarter, or an eighth of the height and / or width of the system function menu).

[0154] In some embodiments, the system space 7040 remains displayed while the user's attention is directed toward the system space 7040, the volume indicator 7038, or the indicator 7010 of the system function menu. In some embodiments, the system space 7040 stops displaying in response to the user's attention 7116 moving away from the volume indicator 7038 or the system space 7040. In some embodiments, the system space 7040 stops displaying after a threshold amount of time (e.g., 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds, 30 seconds, or 60 seconds) following the user's attention having moved away from the volume indicator 7038 or the system space 7040 (e.g., to avoid stopping the display of the system space 7040 when the user does not intend to dismiss it, such as in the case of a brief user distraction, or considering hardware limitations (e.g., the camera of the computer system 7100 temporarily loses focus on the user due to camera movement and / or changes in ambient lighting) for less than the threshold amount of time).

[0155] In some embodiments, if system space 7040 is already displayed (e.g., in response to detecting an active input pointing to volume indicator 7038), and computer system 7100 detects that the user's attention has shifted to another indicator (e.g., control indicator 7046 or any other indicator in system function menu 7024), then computer system 7100 stops displaying system space 7040 and displays a new system space for the other indicator (and optionally replaces the display of system space 7040 with the system space for the other indicator). In some embodiments, the system space for the other indicator is displayed in response to another active input when the user's attention is directed to the other indicator.

[0156] In some embodiments, when system space 7040 is displayed, in response to detecting user attention directed at system function menu 7024, system space 7040 is visually diminished in importance (e.g., darkened, faded, and / or blurred). When system space 7040 is displayed, in response to detecting user attention directed at system space 7040, system function menu 7024 is visually diminished in importance (and optionally, system space 7040 stops being visually diminished in importance). In some embodiments, when user 7002 is looking at or interacting with system function menu 7024 or system space 7040, the indicator 7010 of the system function menu is visually diminished in importance (e.g., darkened, faded, and / or blurred) or is not displayed.

[0157] In some scenarios, computer system 7100 displays an application launch user interface (e.g., a home user interface, which optionally includes one or more indicator lights corresponding to one or more applications that can be launched on computer system 7100). In some embodiments, when the application launch user interface is displayed, an indicator 7010 of the system function menu is displayed, and the application launch user interface remains displayed while the system function menu 7024 is displayed in response to user interaction with the indicator 7010 of the system function menu (e.g., before displaying system space 7040). In some such embodiments, in response to detecting an activation input pointing to the volume indicator 7038, as referenced herein... Figure 7H The computer system 7100 replaces the display of the application launch user interface with the display of the system space 7040 (for example, by stopping the display of the application launch user interface and displaying the system space 7040).

[0158] In some scenarios, the computer system 7100 displays the application user interface before displaying the system space 7040 (e.g., concurrently with the system function menu indicator 7010 and / or system function menu 7024). In response to detecting an activation input pointing to the volume indicator 7038, the computer system 7100 displays the system space 7040 overlaid on at least a portion of the application user interface.

[0159] like Figure 7I As shown, in response to detecting an interactive input (e.g., gaze input, hand gesture, or a combination of gaze and hand gesture) directed toward system space 7040 and specifically toward the slider, computer system 7100 adjusts its volume according to the movement of the interactive input. In some embodiments, the interactive input includes gaze input (e.g., the volume level of computer system 7100 increases as the user's gaze shifts to the right). In some embodiments, the interactive input includes the user's gaze combined with an air gesture (e.g., pinch or pinch and drag). In some embodiments, the volume of computer system 7100 is adjusted based on the movement of the user's hand 7020's hand gesture (e.g., drag or swipe gesture). In some embodiments, a first air gesture (e.g., an air pinch or air tap gesture) activates volume indicator 7038, and a second air gesture (e.g., a second air pinch or air tap gesture combined with a positional change of the user's hand 7020 from a first position to a second position) adjusts the slider of system space 7040. In some implementations, while the user maintains a first air posture with the active volume indicator 7038, the positional change of the user's hand 7020 adjusts the slider of the system space 7040.

[0160] In some embodiments, system space 7040 includes multiple sliders for adjusting the volume of computer system 7100. For example, system space 7040 includes a first slider for adjusting the volume associated with an application (e.g., the volume of content or media within the application's user interface, or the volume of notifications associated with the application), a second slider for adjusting the volume associated with a person / contact (e.g., the volume of audio calls, video calls, and AR / VR communication sessions with other users), and a third slider for adjusting the volume associated with the environment (e.g., the volume of AR or VR experiences). In some embodiments, computer system 7100 determines an activity context (e.g., an application user interface is being displayed and / or the user's attention is directed to the application user interface; an active communication session exists; and / or an AR or VR experience is active) and displays a single slider for adjusting the volume for that activity context. In some embodiments, computer system 7100 determines that no activity context exists (e.g., no application user interface is being displayed, no communication session is active, and no AR / VR experience is active) and displays multiple sliders for adjusting the volume for different contexts (e.g., the three contexts mentioned above).

[0161] Figure 7J Examples of user interactions pointing to different indications in the system function menu 7024 are shown (e.g., the user performs corresponding activation inputs such as by gazing and / or performing gestures). Figure 7K An example system space (e.g., a user interface) is shown that is associated with an indication in the system function menu 7024 and displayed in response to activation of that indication. In some embodiments, a corresponding system space is displayed in response to detecting that the user's attention (e.g., optionally, in combination with aerial gestures and / or verbal input) is directed to a corresponding indication among a plurality of indications in the system function menu 7024.

[0162] Figure 7K (a) shows a system space 7040, which is associated with and displayed in response to the detection of an activation input directed to the volume indicator 7038, as referenced herein. Figures 7H to 7I As stated. Although Figure 7K(a) A single slider (e.g., for adjusting the volume level of computer system 7100) is shown, but in some embodiments, system space 7040 includes additional sliders (e.g., equalizer controls, separate volume controls for two or more speakers of computer system 7100, and / or spatial audio controls) for adjusting related settings of computer system 7100. In some embodiments, system space 7040 includes one or more dials, sliders, and / or buttons (e.g., in addition to) for adjusting settings of computer system 7100. Figure 7K (a) In addition to and / or in place of the single slider shown. In some embodiments, the system space 7054 or additional controls displayed based on interaction with the system function menu are arranged in a corresponding space relative to the system function menu (e.g., to the left, right, above, or below the system function menu) and optionally displayed adjacent to the system function menu (e.g., less than a threshold distance from the system function menu but not overlapping with the system function menu, wherein the threshold distance is less than half, a quarter, or an eighth of the height and / or width of the system function menu).

[0163] Figure 7K (b) illustrates a system space 7050 that is displayed in response to the detection of activation input to a search enable display 7042 and includes text fields for text input. In some embodiments, the text fields of the system space 7050 display the text of a search term or search query entered by the user, such as text with detected verbal input from the user (e.g., such as...). Figure 7K (b) The speech bubble containing the word "weather" is shown as associated (e.g., transcribed from the speech input) text. In some embodiments, as user 7002 speaks, search enablement representation 7042 provides visual feedback (e.g., such as...). Figure 7K (b) shows the dashed circle surrounding the search capability representation 7042. In some embodiments, the visual feedback varies based on at least one characteristic of the verbal input (e.g., volume, speed, and / or length). Figure 7K (b) The dashed circle changes size according to the volume of the detected speech input. In some embodiments, the system space 7050 or additional controls displayed based on interaction with the system function menu are arranged in a corresponding space relative to the system function menu (e.g., to the left, right, above, or below the system function menu) and optionally displayed adjacent to the system function menu (e.g., less than a threshold distance from the system function menu but not overlapping with it, wherein the threshold distance is less than half, a quarter, or an eighth of the height and / or width of the system function menu).

[0164] In some implementations, the text fields of system space 7050 are updated in real time (e.g., when user 7002 speaks). In some implementations, the text fields of system space 7050 display the text of the detected speech input only after the completion of speech input is detected (e.g., in response to the detection of the completion of speech input) (and optionally after a threshold amount of time has elapsed since user 7002 stopped speaking). In some implementations, in response to the detection of speech input (e.g., the completion of speech input), computer system 7100 automatically performs one or more functions associated with the speech input (e.g., internet search, application search, document or file search, or other content search). In some implementations, computer system 7100 performs one or more functions associated with the speech input in response to the detection of a first portion of the speech input, and continues to perform one or more functions as the speech input continues (e.g., continuously updating the search as additional portions of the speech input are detected). For example, computer system 7100 performs a search function in response to the detection of speech input (e.g., presenting results for...). Figure 7K (b) Search results for the word “weather”, and the search results continue to be updated as the speech input continues (e.g., as the speech input continues to add the word “Coppertino”, the computer system 7100 presents updated search results for the term “weather in Cupertino”).

[0165] Figure 7K(c) System space 7052 is shown, which is a notification center displayed in response to the detection of an activation input pointing to notification enable representation 7044. In some embodiments, system space 7052 includes multiple notifications (e.g., recently generated or received notifications). In some embodiments, the notifications appearing in system space 7052 can be configured by the user. For example, the user can configure system space 7052 to display only notifications within a certain time frame (e.g., the past 30 minutes, the past hour, or the past 12 hours), the user can configure system space 7052 to display only notifications from selected applications (e.g., allowing the display of notifications from messaging applications and / or email applications while suppressing the display of notifications from other applications), and / or the user can configure how notifications in system space 7052 are displayed (e.g., grouped by application, grouped by contact, and / or grouped by a specific time window (e.g., received between 9 a.m. and 5 p.m.)). In some implementations, the system space 7052 or additional controls displayed based on interaction with the system function menu are arranged in a corresponding space relative to the system function menu (e.g., to the left, right, above, or below the system function menu) and optionally displayed adjacent to the system function menu (e.g., less than a threshold distance from the system function menu but not overlapping with the system function menu, wherein the threshold distance is less than half, a quarter, or an eighth of the height and / or width of the system function menu).

[0166] Figure 7K(d) System space 7054 is shown, which is a control panel displayed in response to the detection of active input pointing to control indication 7046. In some embodiments, system space 7054 includes one or more indications, such as sliders, buttons, dial pads, dial devices, and / or other controls, for adjusting additional system settings of computer system 7100 (e.g., additional system settings not displayed in the system function menu 7024 itself). In some embodiments, system space 7054 includes indications for transitioning computer system 7100 to airplane mode, indications for enabling or disabling cellular functionality of computer system 7100, indications for enabling or disabling Wi-Fi functionality of computer system 7100, and / or indications for enabling or disabling Bluetooth functionality of computer system 7100. In some embodiments, system space 7054 includes a slider for adjusting the brightness of the display of computer system 7100. In some embodiments, system space 7054 includes one or more controls associated with hardware functions of computer system 7100 (e.g., flashlight function and / or camera function) and / or one or more controls associated with software functions of computer system 7100 (e.g., alarm clock function, timer function, clock function and / or calculator function). In some embodiments, system space 7054 includes one or more indicator displays for controlling system settings of computer system 7100, which can also be accessed using another indicator display in system function menu 7024. For example, system space 7054 optionally includes a slider for adjusting the output volume level of computer system 7100, similar to the slider in system space 7040 that can also be accessed using volume indicator display 7038 in system function menu 7024 (as referenced herein). Figures 7H to 7I and Figure 7K (a) As described. In some embodiments, the system space 7054 or additional controls displayed based on interaction with the system function menu are arranged in a corresponding space relative to the system function menu (e.g., to the left, right, above, or below the system function menu) and optionally displayed adjacent to the system function menu (e.g., less than a threshold distance from the system function menu but not overlapping with the system function menu, wherein the threshold distance is less than half, a quarter, or an eighth of the height and / or width of the system function menu).

[0167] In some implementations, for the corresponding indicator in the system function menu 7024, the system space is not displayed in response to an activation input pointing to the corresponding indicator. For example, as Figure 7KAs shown in (e), the computer system 7100 does not display the system space associated with the virtual assistant capability representation 7048. In some embodiments, the computer system 7100 displays visual feedback that visually emphasizes the virtual assistant capability representation 7048 upon activation, for example, by highlighting the capability representation, displaying a selection outline around the capability representation, zooming in, and / or animing the capability representation, instead of displaying the system space associated with the virtual assistant capability representation 7048. In some embodiments, the computer system 7100 displays visual feedback indicating that verbal input to the virtual assistant is being detected, such as... Figure 7K (e) is indicated by the dashed circle surrounding the virtual assistant enablement representation 7048, thereby indicating the detection of a voice command indicated by a voice bubble with the phrase “Call John Smith”. In some embodiments, in response to the detection of completion of speech input, the virtual assistant associated with the virtual assistant enablement representation 7048 automatically performs one or more functions associated with the speech input (e.g., executes a voice command).

[0168] In some implementations, the system space displayed in response to activation of the power indicator in the system function menu 7024 (e.g., Figure 7K (a) to Figure 7K (d) Any of the system spaces shown is displayed above at least a portion of the system function menu 7024 (e.g., to focus on the system space and increase its visibility). In some embodiments, the computer system 7100 displays the invoked system space (e.g., Figure 7K (a) to Figure 7K (d) Any system space in the system space shown is activated from the system function menu 7024 or more specifically from the system function menu 7024 to invoke the associated power representation of the system space outward (e.g., and downward, according to...). Figure 7J (Example) Extended animation transitions. For example, computer system 7100 displays an animated transition where system space 7040 extends from volume indicator 7038 to cover at least a portion of system function menu 7024.

[0169] related Figures 7A to 7K Additional descriptions are provided below for reference regarding Figures 7A to 7K The method described herein, as well as other figures and methods described herein, are provided.

[0170] Figures 7L to 7Q An example is illustrated of displaying content associated with a first notification in response to detecting a first gaze input pointing to a first user interface object, based on determining that a first event for the first notification meets a timing criterion. Figure 9This is a flowchart of an exemplary method 9000 for displaying content associated with a first notification in response to detecting a first gaze input pointing to a first user interface object, based on determining that a first event for the first notification meets a timing criterion. Figures 7L to 7Q The user interface in the document is used to illustrate the processes described below, including... Figure 9 The process in.

[0171] like Figures 7L to 7Q As shown in the example, content visible via the display generating component of computer system 7100 is displayed on a touchscreen held by user 7002. In some embodiments, the display generating component of computer system 7100 is a head-mounted display worn on the head of user 7002 (e.g., in...). Figures 7L to 7Q The content shown is visible via the display generation component of the computer system 7100 and corresponds to the field of view of the user 7002 when wearing the head-mounted display. Figure 7L An exemplary user interface object 7056 (e.g., a notification indicator 7056) is illustrated when a first event for a first notification that meets a timing criterion is present (e.g., to notify user 7002 about a recently received or generated notification at computer system 7100). For example, user interface object 7056 is displayed when computer system 7100 has recently received a text message from another user (and optionally generates a notification corresponding to the recently received text message). Figure 7L An example is shown with an indicator 7010 that corresponds to the system function menu (for example, see above reference). Figures 7A to 7KThe user interface object 7056 has different appearances (e.g., square shape). When no notification has been recently received (e.g., the first event for the first notification does not meet the timing criteria), an indicator 7010 of the system function menu is displayed (e.g., a first appearance of the system control indicator, optionally the default appearance), and when a notification has been recently received (e.g., the first event for the first notification meets the timing criteria), the user interface object 7056 is displayed (e.g., replacing the system function menu indicator 7010 as a second appearance or alternative appearance of the system control indicator). In some embodiments, the user interface object 7056 has an appearance indicating the application associated with the first notification (e.g., the user interface object 7056 appears as an application icon of a messaging application in which a text message has recently been received). In some such embodiments, the user interface object 7056 optionally includes different indications for different applications (e.g., when the first notification is associated with a first application, the user interface object 7056 has a first appearance indicating the first application, and when the first notification is associated with a second application, the user interface object 7056 has a second appearance indicating the second application (different from the first appearance) (different from the first appearance). In some implementations, the user interface object 7056 is displayed to indicate the appearance of the application associated with the first notification, but after a threshold amount of time (e.g., 5 seconds, 10 seconds, 30 seconds, or 1 minute) has elapsed since the occurrence, generation, or detection of the first event for the first notification, it stops being displayed in that appearance (e.g., and is instead displayed in a default appearance, such as the appearance of the indicator 7010 of the system function menu).

[0172] In some scenarios, the computer system displays other user interfaces, such as user interface 7058, in a three-dimensional environment. In some scenarios, user interface 7058 is an application user interface. In some scenarios, user interface 7058 is an application launch user interface (e.g., a homepage user interface).

[0173] In response to the detection of a first gaze input pointing to the user interface object 7056, such as Figure 7M The user's attention 7116 is shown pointing to the user interface object 7056 (e.g., where the user's attention 7116 is instead pointed to the user interface 7058). Figure 7L(In contrast), computer system 7100 displays notification content 7060 (e.g., message content of a notification for a message received in a messaging application, and / or event information of a notification for a calendar event in a calendar application). In some embodiments, notification content 7060 includes additional information associated with the notification (e.g., a timestamp of a first event for a first notification and / or the application associated with the first notification). In some embodiments, notification content 7060 represents a preview or excerpt of information associated with the received notification (e.g., it shows a portion of a received text message or key event information of a received calendar invitation), and the user can further interact with notification content 7060, as described herein, to view additional (e.g., more complete) information about the received notification and / or additional context of the received notification.

[0174] In some embodiments, notification content 7060 is displayed on top of a portion of user interface 7058 (and / or on top of other displayed user interfaces in a view of a three-dimensional environment). In some embodiments, notification content 7060 is displayed within a threshold distance (e.g., 0.5 cm, 1 cm, 2 cm, or 5 cm) of user interface object 7056. In some embodiments, notification content 7060 is displayed with a corresponding spatial relationship to user interface object 7056 (e.g., directly below, directly above, to the right, or to the left of the user interface object). In some embodiments, notification content 7060 is displayed closer to the user's viewpoint than other user interface objects such as user interface 7058 (e.g., to simulate being "on top" of other user interface elements relative to the user's viewpoint).

[0175] In some implementations, the notification content 7060 is displayed concurrently with the user interface object 7056. In some implementations, such as Figure 7M As indicated by the dashed outline of the user interface object 7056, when (and / or) displaying notification content 7060, the user interface object 7056 is visually diminished in importance or stopped from being displayed. In some embodiments, displaying notification content 7060 includes an animated transition of display (e.g., the appearance of notification content 7060). For example, the animated transition may include an animation of notification content 7060 expanding downwards from user interface object 7056. Alternatively or additionally, the animated transition may include an animation of notification content 7060 gradually increasing in size and / or gradually fading into view. In some embodiments, the displayed animated transition depends on the positioning of user interface object 7056. For example, if user interface object 7056 is instead displayed near the left edge of a display generating component, the animated transition may optionally include an animation of notification content 7060 expanding outwards from user interface object 7056 toward the right edge of a display generating component.

[0176] Figure 7MThe dashed outline of the system function menu 7024 is illustrated below: In some embodiments, in response to the detection of a first gaze input pointing to the user interface object 7056, the system function menu 7024 is displayed concurrently with the notification content 7060 and optionally with the user interface object 7056 (e.g., optionally, the system function menu 7024 is invoked using input pointing to a system control indicator, regardless of whether the system control indicator has the characteristics described in the reference above). Figures 7A to 7K The appearance of the system function menu indicator 7010 or the notification indicator 7056. In some such embodiments, the notification content 7060 is displayed within a threshold distance (e.g., 0.5 cm, 1 cm, 2 cm, or 5 cm) of the system function menu 7024 (e.g., instead of being displayed within a threshold distance of the user interface object 7056, or except for being displayed within a larger threshold distance of the user interface object 7056 (e.g., relative to an embodiment in which the system function menu 7024 is not displayed)). In some such embodiments, the notification content 7060 is optionally displayed in a corresponding spatial relationship with the system function menu 7024 and / or the user interface object 7056 (e.g., directly below, above, to the right, or to the left of the system function menu and / or the user interface object). In other embodiments, the notification content 7060 is displayed without the system function menu 7024 (e.g., as in...). Figure 7N (As shown). In some embodiments, the computer system 7100 initially displays notification content 7060 in response to a first gaze input, but does not display the system function menu 7024. After displaying notification content 7060 in response to the first gaze input, and in response to detecting that the user's gaze continues to point at the user interface object 7056 (e.g., for a threshold amount of time (e.g., 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds, 30 seconds, or 60 seconds), the computer system 7100 also displays the system function menu 7024 (e.g., concurrently with the notification content 7060, in addition to the notification content 7060). In some such embodiments, the notification content 7060 and the user interface object 7056 are displayed in parallel. The threshold distance and / or corresponding spatial relationship changes (e.g., when adding system function menu 7024, the placement of notification content 7060 is updated to accommodate any threshold distance and / or corresponding spatial relationship requirements of system function menu 7024 relative to user interface object 7056 and / or relative to notification content 7060). For example, if notification content 7060 is initially displayed in response to gaze input pointing to user interface object 7056 (e.g., which meets timing criteria for displaying notification content 7060), where notification content 7060 is as follows... Figure 7NThe notification content 7060 is positioned relative to the user interface object 7056, and if the system function menu 7024 is added after a gaze input pointing to the user interface object 7056 satisfies an additional timing criterion, the notification content 7060 is optionally moved to... Figure 7M The positioning is shown relative to the user interface object 7056 and the system function menu 7024 (e.g., to adapt the system function menu 7024 to be displayed closer to the user interface object 7056 than the notification content 7060 is to the user interface object 7056).

[0177] In some implementations, in response to successive gaze inputs directed at user interface object 7056, computer system 7100 switches between displaying notification content 7060 and displaying system function menu 7024. In response to detecting a first gaze input directed at user interface object 7056, computer system 7100 displays notification content 7060. After displaying notification content 7060, the user's gaze moves away from user interface object 7056 (and optionally, the display of notification content 7060 stops). After the gaze moves away from user interface object 7056, the user's gaze returns to user interface object 7056, and in response, computer system 7100 displays system function menu 7024 (e.g., without displaying notification content 7060). If the user repeats this process (e.g., moves their gaze away from user interface object 7056 and then looks back at user interface object 7056), computer system 7100 displays (e.g., redisplays) notification content 7060 (e.g., without displaying system function menu 7024). This allows users to switch between displaying notification content 7060 and displaying system function menu 7024 without cluttering the displayed user interface (e.g., not taking up too much of the user's field of view when displaying notification content 7060 and system function menu 7024 concurrently).

[0178] In some implementations, computer system 7100 switches between displaying notification content 7060 and displaying additional notification content (e.g., additional notification content for a second event of a second notification that also satisfies a timing criterion). In some such implementations, computer 7100 optionally switches between displaying notification content 7024, displaying additional notification content, and displaying system function menus in response to successive gaze input to user interface object 7056 (e.g., cyclically traversing these different user interfaces) (e.g., the displayed user interface may change whenever the user's gaze moves away from user interface object 7056 and then returns to user interface object 7060).

[0179] In some implementations, the user's attention 7116 is directed to the notification content 7060, such as Figure 7NAs shown. In some embodiments, the user performs user input (e.g., air gestures (e.g., air taps or air pinches) or touch input) while looking at the notification content 7060. In some embodiments, the user input includes pinch and drag gestures. In some embodiments, the computer system performs different functions based on the characteristics of the user input (e.g., the direction, speed, and / or amount of movement of the dragged portion of the pinch and drag gesture). For example, in response to a pinch and drag towards the user (e.g., towards the user's viewpoint), the computer system 7100 launches the application associated with the notification and displays the application's user interface 7062, such as... Figure 7O As shown. For example, in response to downward (e.g., relative to the view displayed on the display generating component, oriented towards...) Figure 7N The computer system 7100 stops displaying (e.g., cancels) notification content 7060 when the floor shown in the diagram (represented by 7008') is pinched and dragged. Figure 7N Direct transition to Figure 7P (as shown in the time).

[0180] In some embodiments, as the user 7002's viewpoint moves (e.g., when the user moves the touchscreen device or turns their head while wearing a head-mounted display), the user interface object 7056, notification content 7060, system function menu 7024, and / or application user interface 7062 follow the user 7002's viewpoint (e.g., one or more are viewpoint-locked). In some embodiments, the following behavior of the user interface object 7056, notification content 7060, system function menu 7024, and / or application user interface 7062 is lazy following behavior. In some embodiments, the user interface object 7056 has lazy following behavior. In some embodiments, the following behavior of the user interface object 7056 is consistent with the above-referenced... Figures 7A to 7K The lazy-following behavior of the system function menu indicator 7010 is the same (e.g., regardless of whether the timing criterion is met for the first event of the first notification, and therefore regardless of whether the system control indicator has the appearance of the system function menu indicator 7010 or the appearance of the user interface object 7056, the lazy-following behavior of the system control indicator is the same). In some embodiments, the following behavior of the system function menu indicator 7010 (e.g., the default appearance of the first user interface object), optionally as described herein, is lazy-following behavior, different from the following (e.g., lazy-following) behavior of the user interface object 7056 (e.g., the user interface object 7056 displayed when the timing criterion for the first event of the first notification is met follows the user's viewpoint more closely (or less closely) than the system function menu indicator 7010).

[0181] In some implementations, the user interface object 7056 has a different follow (e.g., lazy follow) behavior than the system function menu 7024 (e.g., similar to how the indicator 7010 of the system function menu may optionally have a different follow behavior than the system function menu 7024, as described above, and as...). Figure 7F (As shown). In some embodiments, the system function menu 7024 has a different follow behavior than the notification content 7060. In some embodiments, the notification content 7060 has a different follow behavior than the application user interface 7062. In some embodiments, the follow behavior of the user interface object 7056 is different from that of the notification content 7060. In some embodiments, the user interface object 7056, the notification content 7060, the system function menu 7024, and the application user interface 7062 each have distinct follow (e.g., lazy follow) behaviors (e.g., relative to each other user interface element that exhibits follow behavior). Optionally, a user interface element (e.g., the application user interface 7062) is world-locked and therefore (e.g., by not exhibiting viewpoint follow behavior) exhibits a different follow behavior than another element that exhibits viewpoint follow behavior.

[0182] In some implementations, the computer system 7100 stops displaying the notification content 7060 when (e.g., in response to detection) the user's attention 7116 is no longer directed to the user interface object 7056 (or notification content 7060). In some implementations, the notification content 7060 may remain displayed for a corresponding period of time (e.g., several seconds, optionally taking into account lazy following behavior) even after the user's attention 7116 has moved away from the user interface object 7056 (or notification content 7060), and the notification content remains displayed if the user's attention 7116 subsequently returns to the user interface object 7056 (or notification content 7060) (e.g., the corresponding period restarts when the user's attention 7116 returns to the user interface object 7056 before the corresponding period expires). In some such implementations, if the computer system 7100 detects that the user's attention 7116 is no longer directed to the user interface object 7056 (or notification content 7060) and has not returned to the user interface object 7056 (or notification content 7060) within the corresponding period, it stops displaying the notification content 7060. In some implementations, after the notification content 7060 is stopped from being displayed, the computer system 7100 redisplays the notification content 7060 in response to the user's attention 7116 subsequently pointing to the user interface object 7056 (e.g., when a timing criterion is met for a first event in response to the first notification).

[0183] For example, in an embodiment where the user interface object 7056 (and / or notification content 7060) exhibits lazy following behavior, the notification content 7060 can remain displayed even if the user's attention 7116 is not continuously focused on the user interface object 7056 (or notification content 7060). This is advantageous, for example, so that the user's attention 7116 does not need to precisely track the user interface object 7056 (or notification content 7060) as the user (and / or the user's viewpoint) moves in order to keep the notification content 7060 displayed, thereby preventing situations where the notification content 7060 stops displaying because the user's attention 7116 has been off for too long and the user must perform another gaze input to the user interface object 7056 to redisplay the notification content 7060.

[0184] After the first event for the first notification no longer meets the timing criteria (e.g., the notification is no longer considered recent enough), the user interface object 7056 reverts to its default appearance (e.g., as shown in the image). Figure 7P and Figure 7Q (As shown in indicator 7010 of the system function menu). Figure 7P This example illustrates the amount of time that has elapsed since the first event in response to the first notification occurred, up to a threshold value. Figure 7L Example transitions are performed. For example... Figure 7Q As shown, if the user's attention 7116 is directed to the system control indicator when the system control indicator is displayed in its default appearance (e.g., the system control indicator appears as indicator 7010 of the system function menu), then the computer system 7100 displays the system function menu 7024 (e.g., the computer system 7100 as described above). Figures 7A to 7K The aforementioned behavior), and the notification content 7060 is not displayed (for example, because the notification corresponding to the notification content 7060 is no longer recent, although in some embodiments, the user can access the content associated with the notification through the notification enablement 7044 of the system function menu 7024, as referenced above). Figure 7K (c) mentioned above.

[0185] In some implementations, after the computer system 7100 displays the notification content 7060, the user interface object 7056 returns to its default appearance (e.g., instead of appearing as...). Figure 7P and Figure 7Q The system function menu indicator 7010 is shown in the figure. In some implementations, after the notification content 7060 stops being displayed (e.g., the user cancels the display of the notification content 7060), the user interface object 7056 returns to its default appearance. Figure 7P It also illustrates the response to user 7002 canceling notification content 7060. Figure 7N Example transition performed.

[0186] related Figures 7L to 7Q Additional descriptions are provided below for reference regarding Figures 7L to 7Q The method described herein, as well as other figures and methods described herein, are provided.

[0187] Figures 7R to 7X An example is shown that displays a first user interface including a capability representation for joining a communication session. Figure 9 This is a flowchart of an exemplary method 9000 for displaying a first user interface including a capability representation for joining a communication session. Figures 7R to 7X The user interface in the document is used to illustrate the processes described below, including... Figure 9 The process in.

[0188] like Figures 7R to 7X As shown in the example, content visible via the display generating component of computer system 7100 is displayed on a touchscreen held by user 7002. In some embodiments, the display generating component of computer system 7100 is a head-mounted display worn on the head of user 7002 (e.g., in...). Figures 7R to 7X The content shown is visible via the display generation component of the computer system 7100 and corresponds to the field of view of the user 7002 when wearing the head-mounted display. Figure 7R An exemplary user interface object 7064 (e.g., a communication session indicator) is illustrated when a request to allow computer system 7100 to join a communication session that meets a timing criterion is present. For example, user interface object 7064 is displayed in response to computer system 7100 receiving a request to allow computer system 7100 to join a communication session (e.g., a telephone call, video call, and / or sharing a virtual experience). In some implementations, a request meets a timing criterion if less than a threshold time amount TH1 (e.g., 5 seconds, 10 seconds, 20 seconds, 30 seconds, 1 minute, 2 minutes, or any threshold time between 0 and 2 minutes, and the threshold time amount may vary depending on the type of communication session requested) has elapsed since the communication session request was first received, while the request remains active (e.g., the initiating user has not cancelled the request). Figure 7R The timer 7066-1 in the system indicates the time t=0 when a communication session request is received. Figure 7R A user interface object 7064 is illustrated, which has a diamond shape to connect the user interface object 7064 with the indicator 7010 of the system function menu (e.g., as referenced above). Figures 7A to 7K Distinguished from the above, and from user interface object 7056 (e.g., as referenced above). Figures 7L to 7QThe user interface object 7064 represents a different appearance (e.g., a third appearance) of the system control indicator from the system function menu indicator 7010 (e.g., a first appearance, which is optionally the default appearance) and the user interface object 7056 (e.g., a second appearance).

[0189] In some implementations, if a request to allow computer system 7100 to join a communication session meets a first timing criterion (e.g., the communication session request has been active for less than a threshold amount of time), then user interface object 7064 is displayed; if a first event for a first notification meets a second timing criterion (e.g., different from the first timing criterion) (e.g., as referenced herein), then... Figures 7L to 7Q If the first timing criterion is met, then user interface object 7056 is displayed; and if neither a request to join a communication session for the computer system 7100 that satisfies the first timing criterion nor a first event for the first notification that satisfies the second timing criterion, then the indicator 7010 of the system function menu is displayed. In some embodiments, if both a request to join a communication session for the computer system 7100 that satisfies the first timing criterion and a first event for the first notification that satisfies the second timing criterion are present, then user interface object 7064 is displayed (e.g., because an incoming communication session request is considered more urgent than an incoming notification). In some embodiments, user interface object 7056 is displayed instead. In some embodiments, the user can configure the computer system 7100 to display user interface object 7064 (e.g., prioritizing alerts for incoming communication session requests) or user interface object 7056 (e.g., prioritizing alerts for notifications) based on user preferences.

[0190] In some embodiments, the user interface object 7064 has an appearance that includes indications to the application associated with the request to have the computer system 7100 join a communication session (e.g., the user interface object 7064 appears as an application icon of the application associated with the communication session). In some such embodiments, the user interface object 7064 optionally includes different indications for different types of communication sessions (e.g., telephone calls, video calls, and / or sharing virtual experiences). In some embodiments, the user interface object 7064 has an appearance that indicates the user associated with the request to have the computer system 7100 join a communication session (e.g., the user who initiated the request) (e.g., as the initiating user's name, initials, username, avatar, photo, etc.).

[0191] In some embodiments, when a timing criterion is met for a request to join a communication session with computer system 7100, the user interface object 7064 has a different color (e.g., green or another color) than when the timing criterion for joining the communication session is no longer met. In some embodiments, in addition to having a different color or instead of having a different color, the user interface object 7064 has a different shape (e.g., compared to the indicator 7010 of the system function menu). In some embodiments, when a timing criterion is met for a request to join a communication session with computer system 7100, the user interface object 7064 displays an animation (e.g., displayed in an animated manner). For example, the user interface object 7064 bounces up and down, the edges of the user interface object 7064 pulsate, the user interface object 7064 changes size, and / or the user interface object 7064 rotates. In some embodiments, the animation is displayed in conjunction with audio output (e.g., beeps, a series of tones (e.g., optionally repeated) and / or ringtones). In some embodiments, the animation is displayed without any corresponding audio output.

[0192] If the request to join the communication session with computer system 7100 meets the timing criteria (e.g., as indicated by timer 7066-2), then... Figure 7S When the timer indicates that the current time is before the time threshold TH1, and the user's attention 7116 is directed to the user interface object 7064, the computer system 7100 displays the incoming call user interface 7068 for joining a communication session, as shown below. Figure 7S As shown. User interface 7068 includes a capability indication for joining a communication session (e.g., as shown by the “Join” capability indication in user interface 7068), and optionally also includes an indication of the user associated with the request to allow computer system 7100 to join the communication session (e.g., the name and avatar of user “John Smith”), and / or additional information about the communication session (e.g., the type of communication session, a list of one or more other users active in the communication session, and / or contact information associated with one or more other users active in the communication session).

[0193] In some embodiments, the user interface 7068 is displayed within a threshold distance (e.g., 0.5 cm, 1 cm, 2 cm, or 5 cm) of the user interface object 7064. In some embodiments, the user interface 7068 is displayed with a corresponding spatial relationship to the user interface object 7064 (e.g., directly below, directly above, to the right, to the left, or other desired spatial relationship). In some embodiments, the user interface 7068 is displayed above at least a portion of other visible user interfaces in a corresponding view of the three-dimensional environment (e.g., as shown in the image). Figure 7SAs shown, user interface 7068 is displayed on top of a portion of user interface 7058.

[0194] Figure 7S The dashed outline of the user interface object 7064 indicates that, in some embodiments, the user interface object 7064 remains displayed while the user interface 7068 is displayed, while in other embodiments, the user interface object 7064 is visually diminished (e.g., faded, grayed out, blurred, and / or shrunk) or stopped from being displayed while the user interface 7068 is displayed. Similarly, Figure 7S The dashed outline of the system function menu 7024 is illustrated in some embodiments: In response to the detection of a first gaze input pointing to the user interface object 7064, the system function menu 7024 is displayed concurrently with the user interface 7068 and optionally with the user interface object 7064 (e.g., optionally, the system function menu 7024 is invoked using input pointing to a system control indicator, regardless of whether the system control indicator has as referenced above). Figures 7A to 7KThe appearance of the system function menu indicator 7010 is the same as the appearance of the user interface object 7064. In some such embodiments, the user interface 7068 is displayed within a threshold distance (e.g., 0.5 cm, 1 cm, 2 cm, or 5 cm) of the system function menu 7024 (e.g., instead of being displayed within a threshold distance of the user interface object 7064, or except for being displayed within a larger threshold distance of the user interface object 7064 (e.g., relative to embodiments in which the system function menu 7024 is not displayed)). In some such embodiments, the user interface 7068 is optionally displayed in a corresponding spatial relationship with the system function menu 7024 and / or the user interface object 7064 (e.g., directly below, directly above, to the right, or to the left of the system function menu and / or the user interface object). In other embodiments, the user interface 7068 is displayed without the system function menu 7024. In some embodiments, in response to the user 7002 gazing at the user interface object 7064, the computer system 7100 initially displays the user interface 7068 without displaying the system function menu 7024. After displaying user interface 7068, and in response to detecting that the user's attention 7116 continues to be directed at user interface object 7064 (e.g., for a threshold amount of time (e.g., 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds, 30 seconds, or 60 seconds)), computer system 7100 also displays system function menu 7024 (e.g., concurrently with user interface 7068, in addition to user interface 7068). In some such embodiments, the threshold distance and / or corresponding spatial relationship between user interface 7068 and user interface object 7064 changes (e.g., when adding system function menu 7024, the placement of user interface 7068 is updated to accommodate any threshold distance and / or spatial relationship requirements of system function menu 7024 relative to user interface object 7064 and / or relative to user interface 7068).

[0195] In some embodiments, displaying the user interface 7068 includes an animated transition of display (e.g., the emergence of the user interface 7068). For example, the animated transition may include an animation of the user interface 7068 expanding downwards from the user interface object 7064. Alternatively or additionally, the animated transition may include an animation of the user interface 7068 gradually enlarging and / or gradually fading into the view. In some embodiments, the animated transition depends on the positioning of the user interface object 7064. For example, if the user interface object 7064 is instead displayed near the left edge of the display generating component, the animated transition optionally includes an animation of the user interface 7068 expanding outwards from the user interface object 7064 toward the right edge of the display generating component.

[0196] In some implementations, as the user's viewpoint moves, the user interface object 7064, system function menu 7024, and / or user interface 7068 follow the user's viewpoint (e.g., one or more are viewpoint-locked). In some implementations, the following behavior of the user interface object 7064, system function menu 7024, and / or user interface 7068 is lazy following behavior. In some implementations, the user interface object 7064, system function menu 7024, and / or user interface 7068 exhibit different following (e.g., lazy following) behaviors (e.g., following the user's viewpoint more closely or less closely than another user interface element).

[0197] In some implementations, after displaying user interface 7068, computer system 7100 responds to user 7002 deselecting user interface 7068 (e.g., by no longer looking at user interface object 7064 or user interface 7068) or stops displaying user interface 7068 after a threshold amount of time has elapsed. After stopping displaying user interface 7068, user interface object 7064 optionally returns to its default appearance (e.g., the appearance of the system function menu indicator 7010), thereby indicating to the user that they have viewed the request to join the communication session (e.g., ...). Figure 7V Examples are given from Figure 7S (Example transitions are shown). In some implementations, if the user interface object 7064 is displayed in an appearance that indicates the user associated with the request to join the communication session with the computer system 7100 (e.g., the user who initiated the request) (before attention 7116 is directed to the user interface object 7064), then when the computer system 7100 stops displaying the user interface 7068, the default appearance of the user interface object 7064 does not indicate the user associated with the request (e.g., restoring the user interface object 7064 to its default appearance includes: the user interface object 7064 ceasing to indicate the user who initiated the communication session request). In some implementations, as long as the request to join the communication session meets the timing criteria, the user can redisplay the user interface 7068 by redirecting the user's attention 7116 back to the user interface object 7064.

[0198] In some implementations, if the user's attention 7116 is not directed to the user interface object 7064 before the request for the computer system 7100 to join the communication session expires (e.g., before the user initiates a request to cancel the communication session) and / or before the request no longer meets a timing criterion (e.g., more than a threshold time TH1 has elapsed since the request was first received), the computer system 7100 abandons displaying the user interface 7068 for joining the communication session (e.g., even if the user's attention 7116 is directed to the user interface object 7064). Figure 7T (i.e. from) Figure 7R As shown in the example transition), in response to the detection of user attention 7116 directed at user interface object 7064 after a threshold time TH1 has elapsed since the first receipt of the communication session request (as indicated by timer 7066-3, which indicates that the current time is after the threshold time TH1) (e.g., and before a second threshold time TH2 has elapsed since the communication session request expired), computer system 7100 displays a missed call user interface 7070 for initiating a communication session (e.g., for calling back the user associated with the initial request to have computer system 7100 join the communication session). In some embodiments, user interface 7070 is displayed based on the determination that: within a threshold time (e.g., 5 seconds, 10 seconds, 20 seconds, 30 seconds, 60 seconds, or any threshold time between 0 and 60 seconds) (e.g., threshold TH2) of the time since the request to have computer system 7100 join the communication session was received, but during which an incoming call can be answered (e.g., this would result in an incoming call user interface 7068 such as...). Figure 7S After different threshold time periods (e.g., threshold TH1) as shown, the user's attention 7116 is directed to the user interface object 7064. In some embodiments, if the user's attention 7116 is not directed to the user interface object 7064 within a threshold time period (e.g., threshold TH2), the user interface 7070 is not displayed, even if the user subsequently directs their attention 7116 to the user interface object 7064. Those skilled in the art will readily recognize that time thresholds can be defined in various ways such that prompting interaction with user interface object 7064 results in the display of incoming call user interface 7068, while delayed interaction with user interface object 7064 results in the display of missed call user interface 7070 (e.g., such that both time thresholds are measured relative to the time corresponding to the reception of a communication session request, where time threshold TH2 occurs after time threshold TH1, or time threshold TH2 is measured relative to the time it takes for the communication session request to become inactive (e.g., due to the time threshold TH1 being reached or due to the requesting user canceling the request)). Optionally, as described in more detail herein, even further delayed interaction with user interface object 7064 after it has been restored to the default appearance of the indicator 7010 of the system function menu results in the display of system function menu 7024, without displaying the incoming, missed, or ongoing call user interface.

[0199] For example, if a communication session request for computer system 7100 lasts for one minute, and the user's attention 7116 is directed to user interface object 7064 during that minute, then computer system 7100 displays user interface 7068 for joining the communication session (e.g., as shown in the image). Figure 7S(As shown). If the user's attention 7116 is not directed to the user interface object 7064 within the 1 minute, the request expires (e.g., automatically times out, not necessarily due to the user initiating the cancellation request). However, if the user's attention 7116 is directed to the user interface object 7064 within 30 seconds of the request's expiration, the computer system 7100 displays the user interface 7070 for initiating a second communication session (e.g., to call back a user whose earlier request to join the communication session was not received). If the user does not direct their attention 7116 to the user interface object 7064 within 30 seconds of the initial request's expiration (e.g., more than 1 minute and 30 seconds have elapsed since the initial request was received), the computer system 7100 does not display the user interface 7070 (e.g., even if the user's attention 7116 is directed to the user interface object 7064), and instead, if the user's attention 7116 is directed to the user interface object 7064, the computer system displays the system function menu 7024 in response (neither the user interface 7068 nor the user interface 7070 is displayed).

[0200] exist Figure 7T In the example shown, user interface 7070 includes an "invitation" indication (e.g., for calling back the user who sent the initial communication session request and optionally inviting one or more other users to join a new communication session) and an indication (e.g., the text "Missed video call") that the request to have computer system 7100 join the communication session has not been viewed in a timely manner (e.g., within a threshold time TH1 from the date the request was received). In some embodiments, user interface 7070 also includes an indication of the user associated with the missed request to have computer system 7100 join the communication session (e.g., the name and avatar of user "John Smith"), and / or additional information about the missed request (e.g., the type of communication session, and / or contact information associated with one or more other users associated with the communication session associated with the missed request). In some embodiments, user interface 7070 is displayed concurrently with system function menu 7024 (e.g., as shown in the image). Figure 7T As shown), and / or displayed concurrently with user interface object 7064. In some embodiments, the display characteristics of user interface 7070 (e.g., its display position, spatial relationship with other user interface elements, displayed animations and / or follow behaviors) are similar to those of user interface 7068 described above.

[0201] Figure 7U Examples are given from Figure 7S Example transitions are performed. For example... Figure 7U As shown, if the user activates the "Join" option in user interface 7068, it indicates that... Figure 7SIf the user activates the "Invite" option in the user interface 7070, the computer system 7100 displays the call control user interface 7072. In some implementations, if the user activates the "Invite" option in the user interface 7070, it indicates that... Figure 7T If a user 7002 initiates a communication session (e.g., even before another user joins the communication session), the computer system 7100 displays the call control user interface 7072. Optionally, the user interface object 7064 has a color associated with an active communication session request that meets timing criteria (e.g., the user interface object 7064 is green), even if the user 7002 initiates the communication session (e.g., instead of receiving a request to join the communication session). In some embodiments, the same call control user interface 7072 is displayed regardless of how the user 7002 joins the communication session (e.g., by joining via a request from another user or by initiating a communication session).

[0202] Figure 7U It also shows examples from Figure 7T Example transitions are performed. In some implementations, if user 7002 of computer system 7100 initiates a communication session (e.g., by activating such...), Figure 7T The user interface 7070 shown has an "Invite" enabled indicator, and another user (e.g., another user who received a request to join the communication session when the "Invite" enabled indicator was activated, such as a user whose call was not answered) Figure 7T Once a communication session is joined, computer system 7100 displays a visual representation of the other user (e.g., an avatar, a profile picture). In some implementations, displaying a visual representation of the other user includes displaying an animated transition (e.g., the visual representation fades into the view, and / or the visual representation appears to expand outward from user interface object 7064, user interface 7070, or another displayed user interface element).

[0203] The call control user interface 7072 includes multiple capability representations for accessing functions of an active communication session. For example, the call control user interface 7072 includes any combination of the following: a message sending / receiving indication (e.g., for launching a separate application (e.g., a text messaging application) for communicating with one or more other users in a communication session and / or viewing previous communications with one or more other users in a communication session (e.g., conversation transcription)); an information indication for displaying additional information about the communication session (e.g., a list of users in the communication session, and / or an indication of the duration of the communication session); a microphone indication (e.g., for muting or unmute user 7002 in the communication session); a session history indication (e.g., for displaying shared content previously shared between one or more users in the communication session); a sharing indication (e.g., for sharing content with other users in the communication session and / or displaying one or more shared content items previously shared (e.g., with other users in the communication session); and / or an exit indication (e.g., marked "Leave" for removing a user from the communication session and / or for ending the communication session for all users).

[0204] In some implementations, the call control user interface 7072 is displayed concurrently with the system function menu 7024 and / or user interface object 7064 (e.g., as shown in the image). Figure 7U (As shown). In some such embodiments, when the user's attention 7116 is directed to the call control user interface 7072, the system function menu 7024 (and / or user interface object 7064) is optionally visually diminished (e.g., blurred or faded), while the call control user interface 7072 is displayed with default visual emphasis. When the user's attention 7116 is directed to the system function menu 7024 (or user interface object 7064), the call control user interface 7072 is optionally visually diminished (e.g., blurred or faded), while the system function menu 7024 (or user interface object 7064) is displayed with default visual emphasis.

[0205] In some implementations, the display characteristics of the call control interface 7072 (e.g., the display location, spatial relationship with other user interface elements, displayed animations and / or follow or lazy follow behavior) are similar to those of the user interfaces 7068 and / or 7070 described above.

[0206] In some embodiments, the call control user interface 7072 includes a power display for switching between different representations of one or more users in a communication session (e.g., a personal picture, a two-dimensional image of the user, the user's name, and / or the user's initials). In some embodiments, the power display is used to switch between displaying a realistic portrait of the user (e.g., which may be two-dimensional or three-dimensional, including, for example, an image of the user, a video or video stream of the user, and / or an avatar that looks like the user and / or is animated based on the user's movement) and an abstract portrait or abstract representation of the user (e.g., the user's name, the user's initials, and / or an image not based on the user's portrait). For example, if an avatar of the corresponding user is displayed when the power display for switching between realistic and abstract portraits is activated, the avatar of the corresponding user is replaced by the abstract portrait. If an abstract portrait is displayed when the power display for switching between realistic and abstract portraits is activated, the abstract portrait is replaced by the avatar of the corresponding user. In some implementations, the call control user interface 7072 includes a power display for switching between different visual representations of one or more users in a communication session (e.g., switching between one or more of an avatar, a 3D character, an icon, a personal picture, the user's name, and the user's initials).

[0207] In some implementations, when (e.g., in response to detection) the user's attention 7116 is no longer directed to the user interface object 7064 (or the call control user interface 7072), the computer system 7100 stops displaying the call control user interface 7072. In some implementations, even after the user's attention 7116 is no longer directed to the user interface object 7064 (or the call control user interface 7072), the call control user interface remains displayed for a corresponding time period (e.g., 1 second, 5 seconds, or a time period that takes into account lazy following behavior), and if the user's attention 7116 subsequently returns to the user interface object 7064 (or the call control user interface 7072), the call control user interface remains displayed (e.g., the corresponding time period restarts when the user's attention 7116 returns to the user interface object 7064 before the corresponding time period expires). In some such implementations, if the computer system 7100 detects that the user's attention 7116 is no longer directed to the user interface object 7064 (or call control user interface 7072) and has not returned to the user interface object 7064 (or user interface 7072) within a corresponding time period, it stops displaying the call control user interface 7072 and keeps displaying the user interface object 7064, or if the user interface object 7064 is not displayed when the computer system 7100 stops displaying the call control user interface 7072, it redisplays the user interface object (e.g., so that the call control user interface 7072 can be redisplayed when the user's attention 7116 subsequently directs to the user interface object 7072). In some implementations, after stopping the display of the call control user interface 7072, if the user's attention 7116 subsequently directs to the user interface object 7064 while the computer system 7100 is still in a communication session, the computer system 7100 redisplays the call control user interface 7072 in response. Although this behavior is described above with reference to the call control user interface 7072, this behavior optionally also involves user interfaces 7068 and / or 7070, which may be displayed concurrently with or in place of the call control user interface 7072.

[0208] In some embodiments, the user interface object 7064, the system function menu 7024, the incoming call user interface 7068, the missed call user interface 7070, and / or the call control user interface 7072 are head-locked virtual objects (e.g., viewpoint-locked virtual objects as described herein). In some embodiments, when the user's viewpoint changes (e.g., the user rotates and / or moves to a new position in the three-dimensional environment, causing a change in the view of the three-dimensional environment), the user interface object 7064 moves cooperatively with the system function menu 7024, the incoming call user interface 7068, the missed call user interface 7070, and / or the call control user interface 7072 (if displayed) (e.g., such that the system function menu 7024, the incoming call user interface 7068, the missed call user interface 7070, and / or the call control user interface (if displayed) maintain the corresponding required spatial relationship with the user interface object 7064, as described herein). In some implementations, the system function menu 7024, the incoming call user interface 7068, the missed call user interface 7070, and / or the call control user interface 7072 move in coordination with each other (e.g., if the system function menu 7024 and the call control user interface 7072 are displayed concurrently, the system function menu 7024 moves in coordination with the call control user interface 7072 such that the call control user interface 7072 maintains a corresponding spatial relationship with the system function menu 7024 (e.g., in a manner similar to maintaining a corresponding spatial relationship with the user interface object 7064)).

[0209] In some implementations, after a request to join a communication session for computer system 7100 no longer meets a timing criterion (e.g., the request to join the communication session expires or times out), the user interface object 7064 returns to its default appearance (e.g., as shown in the image). Figure 7V and Figure 7W (As shown in indicator 7010 of the system function menu). For example... Figure 7V and Figure 7W It shows in Figure 7R After the communication session request expires, it appears as an indicator of the system function menu in the 7010 system control indicator (e.g., Figure 7V Examples are given from Figure 7R Example transition performed). In some implementations, a threshold time amount TH1 has already elapsed (e.g., where interaction with a system control indicator will result in...). Figure 7S The time period displayed for the incoming call from user 7068), followed by the elapsed threshold time TH2 (e.g., where interaction with system control indicators will result in...). Figure 7TAfter the period of time displayed on the missed call user interface 7070 (e.g., the total elapsed time TH1+TH2 since the first receipt of the communication session request), the system control indicator appears as an indicator 7010 for the system function menu. In some embodiments, after the computer system 7100 leaves the communication session (e.g., leaves via...), the system control indicator appears as an indicator 7010 for the system function menu. Figure 7S The "Join" indicator in the incoming call user interface 7068 indicates joining and optionally displays during this period. Figure 7U After the communication session of the call control user interface 7072, the user interface object 7056 returns to the default appearance (e.g., instead of appearing as...). Figure 7V and Figure 7W The system function menu indicator 7010 in the system (e.g., the system function menu indicator 7010). Figure 7V and Figure 7W As shown, if the user's attention 7116 is directed to the indicator 7010 of the system function menu (e.g., after both the time period for invoking the incoming call user interface and the time period for invoking the missed call user interface have expired), the computer system 7100 displays the system function menu 7024 (e.g., the computer system 7100 as described above). Figures 7A to 7K The user interface 7068, user interface 7070, and call control user interface 7072 are not displayed because there are no pending requests to join a communication session and because the computer system 7100 is not in an active communication session.

[0210] In some implementations, (for example) in response to successive gaze inputs pointing to system control indicators, the computer system 7100 displays a user interface 7068, a system function menu 7024, and optionally one or more additional user interface elements (e.g., in cases where a request to join a communication session meets a first timing criterion and a first event for a first notification meets a second timing criterion, as referenced above). Figures 7L to 7Q The display of the notification content 7060 is switched between different states. Those skilled in the art will readily recognize that the following description with reference to the incoming call user interface 7068 similarly applies to the missed call user interface 7070 (e.g., if a user has missed a request to join a communication session but is still within time period TH2, as referred to...). Figure 7T As stated above, the first event of the first notification simultaneously meets the second timing criterion, as referenced herein. Figures 7L to 7Q The aforementioned) and / or call control user interface 7072 (e.g., if a user joins a communication session while a second timing criterion is met for a first event in response to a first notification).

[0211] The request to join the communication session satisfies the first timing criterion (e.g., the request was received before time TH1, as referenced herein). Figure 7SIn an example scenario where the first event of the first notification satisfies the second timing criterion, in response to detecting a first gaze input pointing to a system control indicator, computer system 7100 displays user interface 7068 (e.g., as referenced herein). Figures 7R to 7S (as described above) (for example, without displaying notification content 7060 or system function menu 7024). See references herein. Figure 7R The system control indicator may have an appearance indicating an incoming call or alternatively associated application with the first notification, the appearance being based on the relative priority of the alert and / or based on user preference. After displaying user interface 7068, the user's gaze moves away (e.g., directs the user's attention away) from the system control indicator (and optionally, stops displaying the system control indicator). After the gaze moves away from the system control indicator, the user's gaze returns to the system control indicator, and in response, computer system 7100 displays notification content 7060 (e.g., without displaying user interface 7068 or system function menu 7024). After displaying notification content 7060, the user's gaze moves away from the system control indicator (and optionally, stops displaying notification content 7060). After the gaze moves away from the system control indicator, the user's gaze returns to the system control indicator, and in response, computer system 7100 displays system function menu 7024 (e.g., without displaying notification content 7060 or user interface 7068). This marks the end of a cycle of user interface elements (e.g., which begins with user interface 7068 and ends with system function menu 7024). If the user repeats this process (e.g., looking away from the system control indicator and then back at it), the computer system 7100 begins a new cycle, displaying user interface elements in the same order (e.g., displaying user interface 7068, then notification content 7060, and then system function menu 7024 in response to successive gaze input). This allows the user to switch between displaying user interface 7068, notification content 7060, and system function menu 7024 without cluttering the displayed user interface (e.g., not occupying too much of the user's field of view when multiple user interface elements such as notification content 7060 and system function menu 7024 are displayed concurrently).

[0212] In some implementations, the computer system 7100 switches between displaying the user interface 7068 and displaying another request to join a communication session (e.g., a second request to join a second communication session that also meets timing criteria). In some such implementations, the computer 7100 optionally switches between displaying the user interface 7068 and displaying a second user interface for joining a second communication session (e.g., a similar but different user interface representing a different user who initiated the request for the second communication session) and displaying the system function menu 7024 (e.g., the displayed user interface elements may change whenever the user's gaze moves away from and then returns to the user interface object 7064).

[0213] Figure 7X Example timing of the persistence of a request to join a communication session based on the type of communication session, according to some implementation schemes (e.g., how long the request is active before timeout). Figure 7X The above references are further illustrated. Figures 7A to 7W For specific reference Figures 7R to 7W The concept described. Figure 7X In the diagram, the horizontal axis represents the time axis, with time progressing from left to right.

[0214] For a telephone communication session (e.g., a telephone call), in step 7074, the computer system 7100 (e.g., initially) detects an incoming request to join the telephone communication session. A first time threshold TH is set from the first time the telephone communication session request is received. C1 Previously (e.g., while a request to join a telephone call is ringing), the request to join a telephone communication session met a timing criterion. As shown in step 7076, in response to detecting an incoming request to join a telephone communication session, computer system 7100 displays the request to join the telephone communication session (e.g., incoming call user interface 7068). Figure 7S In some implementations, at the current time in TH C1 Previously, a request to join a telephone communication session was displayed (and optionally remained displayed), regardless of whether the user's gaze was directed at user interface object 7064. Figure 7R (Or more generally, system control indicators) (For example, in Figure 7R The illustrated scenario will also display Figure 7S Incoming call user interface 7068, even in Figure 7R In the middle, user 7002's gaze is not directed at user interface object 7064.

[0215] At time TH C1 Subsequently, the request to join the telephone communication session no longer meets the timing criteria (e.g., the request has timed out). At step 7078, the current time is TH... C1After that, but at the second time threshold TH C2 Previously. If the user is within the time threshold TH C1 and time threshold TH C2 If the user interface object 7064 is viewed between the user interface objects, as in step 7078, then in response, the computer system 7100 displays the missed call user interface (e.g., user interface 7070). Figure 7T The missed call user interface indicates a missed call request to join a telephone communication session and includes an enable signal for initiating a second communication session (e.g., a new communication session), as in step 7080.

[0216] If the user is looking at user interface object 7064, as in step 7082, then because the current time is TH... C1 and TH C2 After both, the request to join the telephone communication session does not meet the timing criteria (e.g., the amount of time since the request was received is not less than TH). C1 ), and within a threshold time period since the request was received (e.g., the time period since the request was received is not less than TH). C2 No gaze input was detected within the system. Therefore, in response to the detection of gaze input in step 7082, and as shown in step 7084, the computer system 7100 does not display either the incoming call user interface or the missed call user interface, but instead displays the system function menu 7024 (e.g., which includes multiple indications for performing system operations associated with the computer system 7100, as referenced herein, for example). Figures 7V to 7W The above).

[0217] For a video communication session (e.g., a video call), in step 7086, the computer system 7100 (e.g., initially) detects an incoming request to join the video communication session. This is done within a first time threshold TH from the first receipt of the video communication session request. V1 Previously (e.g., while a request to join a video call was ringing), the request to join a video communication session met timing criteria. If, as shown in step 7088, the user's gaze is at the time threshold TH... V1 Previously, it pointed to user interface object 7064, so in response to detecting a user's gaze pointing to user interface object 7064 (e.g., and because the current time is TH...), V1 Previously, computer system 7100 displayed an incoming call user interface (e.g., user interface 7068). Figure 7S The incoming call user interface includes a capability indication to join a video communication session, as shown in step 7090.

[0218] At time TH V1Subsequently, the request to join the video communication session no longer meets the timing criteria (e.g., the request has timed out). At step 7092, the current time is TH. V1 After that, but at the second time threshold TH V2 Previously. If the user is within the time threshold TH V1 and time threshold TH V2 If the user interface object 7064 is viewed between the user interfaces, as in step 7092, then in response, the computer system 7100 displays the missed call user interface (e.g., user interface 7070). Figure 7T The missed call user interface indicates a missed call request to join a video communication session and includes an enable signal for initiating a second communication session (e.g., a new communication session), as in step 7094.

[0219] If the user is looking at user interface object 7064, as in step 7096, then because the current time is TH... V1 and TH V2 After both, the request to join the video communication session does not meet the timing criteria (e.g., the amount of time since the request was received is not less than TH). V1 And no gaze input was detected within a threshold time period from when the request was received (e.g., the time period from when the request was received is not less than TH). V2 Therefore, in response to detecting gaze input in step 7096, and as shown in step 7098, computer system 7100 does not display either the incoming call user interface or the missed call user interface, but instead displays the system function menu 7024. As used herein, "video communication session" or "video call" refers to a video telephony protocol or a telephony protocol that supports video but allows the user to enable or disable video (e.g., within the same telephony service, a user can choose between a video call and an audio-only call, or turn off their own and / or another user's video). As used herein, "telephone communication session" or "telephone call" refers to an audio-only call that does not support video (e.g., including some cellular phones and internet phones). Those skilled in the art will readily recognize that an audio-only call made via a video telephony protocol can be considered a "telephone call" rather than a "video call".

[0220] For an extended reality (XR) communication session, in step 7102, the computer system 7100 detects an incoming request to join the XR communication session. A time threshold TH is set from the first received XR communication session request. X Previously (e.g., when requesting the user to keep the invitation to join the XR communication session open), the request to join the XR communication session met a timing criterion. If, as shown in step 7104, the user's gaze is within the time threshold TH... XPreviously pointed to system control indicators (e.g., user interface object 7064). Figure 7R Then, in response to detecting a user's gaze pointing at the user interface object 7064 (e.g., and because the current time is TH...), X Previously, computer system 7100 displayed an incoming call user interface (e.g., user interface 7068). Figure 7S The incoming call user interface includes a sign indicating that the user is joining an XR communication session, as shown in step 7106.

[0221] At time TH X Subsequently, requests to join the XR communication session no longer meet the timing criteria (e.g., the request has timed out). If the user is within the time threshold TH... X Then, focusing on user interface object 7064, as in step 7108, the computer system 7100 displays system function menu 7024, as in step 7110. Figure 7X In the example shown, XR communication session requests are associated with a single time threshold TH. x Relatedly, before the time threshold, the incoming call user interface is displayed, and after the time threshold, the system function menu 7024 is displayed without displaying the missed call user interface. In some embodiments, at the time threshold TH... X Then, and before the second time threshold, there exists a certain time period during which the computer system displays a missed call user interface (e.g., user interface 7070) for XR communication session requests in response to gaze input at user interface object 7064. Figure 7T The method is similar to steps 7078-7080 of the telephone call described above and steps 7092-7094 of the video call described above.

[0222] In some implementations, video communication requests are more persistent than telephone communication requests because incoming video calls ring for a longer time than incoming telephone calls. This is in Figure 7X Zhongyou TH V1 Along the time axis than TH C1 Further (e.g., TH) V1 Visually shown after THC1 (on the right). In some implementations, the request to join the XR communication session remains active as long as the requesting user keeps the invitation open (e.g., the request does not time out automatically but must be manually cancelled by the requesting user). Therefore, Figure 7X This illustrates an example scenario where an XR communication session request is more persistent than a request to join a phone or video communication session because of the time threshold TH. X Farthest along the time axis (e.g., TH) X In TH C1 and TH V1Following both (to its right), this is because the request to keep the user's XR communication session pending exceeds the timeout threshold for both phone and video calls. Those skilled in the art will readily recognize that the time threshold TH... X This can vary based on when the requesting user decides to cancel the XR communication session request (e.g., earlier than a time threshold TH). C1 and / or TH V1 ), and where XR communication session requests are made at time threshold TH C1 and / or time threshold TH V1 Other implementations, such as automatic timeout before or after (e.g., similar to telephone and video calls), are possible. In some implementations, the request to join a telephone communication session is the least persistent (e.g., as...). Figure 7X As instructed, TH C1 It is the lowest / earliest (leftmost) time threshold) but the most prominent one displayed (e.g., displaying a request to join a telephone communication session regardless of whether the user's gaze is on a system control indicator).

[0223] related Figures 7R to 7X Additional descriptions are provided below for reference regarding Figures 7R to 7X The method 10000, as well as other diagrams and methods described herein, are provided.

[0224] Figures 7Y to 7AF An example of the display of a user interface that initiates environment locking from a head-locked user interface is shown. Figures 11A to 11B This is a flowchart of an exemplary method 11000 for displaying a user interface that initiates an environment lock from a user interface with a head-locked interface. Figures 7Y to 7AF The user interface in this document is used to illustrate the procedures described below, including... Figures 11A to 11B The process in [the process]. For ease of discussion, Figures 7Y to 7AF The illustrations illustrate user interaction with a specific display of a particular display that initiates the display of a particular user interface according to some implementation schemes, but the following description generally applies to any suitable display (e.g., which, when selected, initiates the display of the corresponding user interface).

[0225] Figure 7Y An example is shown of a physical environment 7000 including a user 7002 interacting with a computer system 7100. The user 7002 is located at position 7026-c within the physical environment 7000.

[0226] In some implementation schemes (e.g., as referenced above) Figure 7B As described above, the system function menu indicator 7010 is displayed in the outer perimeter area at the top center of the display of the computer system 7100. In some embodiments (e.g., as in...), Figure 7YIn the above-right peripheral area of ​​the display of the computer system 7100 (e.g., at the peripheral area of ​​both the top and right edges of the computer system 7100), the system function menu indicator 7010 is displayed. It should be understood that in some embodiments, the system function menu indicator 7010 is displayed in an alternative peripheral area of ​​the display of the computer system 7100.

[0227] exist Figure 7Z In this system, the user's attention 7116 is directed to the indicator 7010 of the system function menu. In some implementations, if the user's attention 7116 meets an attention criterion (e.g., the user's attention is continuously directed to the indicator 7010 of the system function menu for a threshold amount of time, or the user's attention is directed to the indicator 7010 of the system function menu while the user performs a required gesture (e.g., bringing the hand into a specified configuration, or performing an aerial gesture such as an aerial tap or an aerial pinch)), then the computer system 7100 displays the system function menu 7024. Although some features of the system function menu 7024 are different (e.g., compared to...), the computer system 7100 displays the system function menu 7024. Figures 7H to 7W ,exist Figure 7Z In this context, the system function menu 7024 may have different appearances, a different set of power indicators, and different locations, but other previously described features of the system function menu 7024 (e.g., referenced above) are also present. Figures 7A to 7X The aforementioned features remain applicable to the system function menu 7024.

[0228] In some implementations, the system function menu 7024 includes multiple indications of system functions for accessing the computer system 7100. For example, such as... Figure 7Z As shown, the system function menu 7024 includes a homepage indicator 7124 (e.g., for accessing one or more applications of the computer system 7100, for initiating a communication session with one or more contacts stored in the memory of the computer system 7100, and / or for initiating the display of a virtual environment), together with a search indicator 7042, a volume indicator 7038, a notification indicator 7044, a control indicator 7046, and a virtual assistant indicator 7048 (e.g., these indicators are related to those mentioned above). Figure 7E (Corresponding to the enablement representation of the exemplary system function menu). In some embodiments, the system function menu 7024 includes... Figure 7AC The indicated energy level represents different energy levels (e.g., additional energy levels, fewer energy levels, e.g., as shown in...). Figure 7E (in the system function menu), and / or replace some of the power indicators with other power indicators associated with different functions of the computer system 7100.

[0229] exist Figure 7AA In this context, the user's attention 7116 is directed to the control indication representation 7046. In response to detecting that the user's attention 7116 meets the selection criteria (e.g., which may be the same as or different from the attention criteria described above for displaying the system function menu 7024), the computer system 7100 displays a user interface 7136 (e.g., a control user interface) associated with the control indication representation 7046 (e.g., displayed concurrently with the system function menu 7024). In some implementations, the user interface 7136 includes power indicators 7138, 7140, 7142, and 7144 for interacting with the user interface 7136 and / or for accessing or adjusting settings of the user interface 7136 (e.g., positioning, brightness, or other display characteristics and / or the content of the user interface 7136) and / or settings of the computer system 7100 (e.g., accessing additional system settings of the computer system 7100 not included in the system function menu 7024), and / or for displaying other user interfaces (e.g., additional content associated with the user interface 7136, application user interfaces, and / or user interfaces for adjusting settings of the computer system 7100).

[0230] In some implementations, a user selection of a corresponding feature among the features displayed in the system function menu 7024 (e.g., home feature 7124, search feature 7042, volume feature 7038, notification feature 7044, control feature 7046, virtual assistant feature 7048, and / or any other feature) causes the computer system 101 to display a corresponding user interface for the selected feature. For example, user interface 7136 is a home user interface (e.g., referred to below) displayed in response to detecting a user selection of home feature 7124. Figure 7AD (More details described).

[0231] In some embodiments, the user interface 7136 is a search user interface for performing a search operation (e.g., if the user's attention is directed at the search indicator 7042, rather than the control indicator 7046). In some embodiments, the user interface 7136 includes a search bar that provides visual feedback on the search item. In some embodiments, the user 7002 uses verbal input to enter a search item (e.g., and at least one of the input devices of the computer system 7100 is a microphone configured to detect verbal input). In some embodiments, the user 7002 enters a search item by looking at the search bar displayed in the user interface 7136 (e.g., the computer system 7100 detects the user's attention pointing to the search bar and determines that the user 7002 intends to perform a search function), and / or performs verbal input (e.g., speaking the desired search item aloud). In some embodiments, the search bar is updated in real time as the user performs verbal input (e.g., each word spoken by the user in the search item is displayed in the search bar).

[0232] In some embodiments, the user interface 7136 includes a virtual keyboard (e.g., optionally in addition to a search bar), and the user 7002 uses the virtual keyboard to manually enter search terms in the search bar. In some embodiments, the user 7002 enters search terms via speech input or via a virtual keyboard. In some embodiments, the user 7002 enters search terms using a combination of input devices (e.g., the user begins entering search terms via speech input and edits and / or completes the search terms via a virtual keyboard, or vice versa).

[0233] In some embodiments, the user interface 7136 is a user interface for adjusting the volume settings of the computer system 7100 (e.g., in response to detecting that a user's gaze is directed at the volume indicator 7038, rather than the control indicator 7046). In some embodiments, the user interface 7136 includes one or more controls (e.g., a volume slider, a dial pad, and / or one or more buttons) for adjusting the volume settings of the computer system 7100.

[0234] In some implementations, the user interface 7136 is a notification user interface for displaying the content of one or more notifications received from the computer system 7100 (e.g., in response to detecting that the user's attention is directed to the notification display 7044, rather than the control display 7046).

[0235] In some implementations, user interface 7136 is a system user interface for accessing additional system settings of computer system 7100 that are not included in system function menu 7024. For example, enable indicators 7138, 7140, 7142 and / or 7144 of user interface 7136 allow users to access and / or adjust different settings of computer system 7100 (e.g., which optionally correspond to settings that cannot be accessed or adjusted directly from system function menu 7024).

[0236] exist Figure 7AB In some embodiments, the user's attention 7116 is directed to the indicator 7142 of the user interface 7136. In some implementations, in response to detecting that the user is interacting with a user interface element other than those displayed in the system function menu 7024, and / or the user's attention is no longer directed to the indicator 7010 or the system function menu 7024, the computer system 7100 stops displaying the system function menu 7024. In some implementations, the system function menu 7024 remains displayed, but with reduced salience (e.g., partially blurred, darkened, or otherwise visually diminished) to reflect that the user's attention (e.g., based on gaze) is not directed to the system function menu 7024, and to reduce the risk that the system function menu 7024 will interfere with visibility when the user interacts with other user interface elements displayed outside the system function menu 7024. In some implementations, such as Figure 7AC As illustrated, in response to the detection that the user's attention 7116 has returned to the indicator 7010 of the system function menu, the computer system 7100 redisplays the system function menu 7024.

[0237] exist Figure 7AD In this scenario, user 7002 moves to a new location 7026-d within the physical environment 7000. In some implementations, after moving to the new location, based on user 7002's movement from a third location 7026-c to the new location 7026-d (e.g., causing the user's viewpoint to be updated), different views of the physical environment 7000 (e.g., compared to...) are... Figures 7Z to 7AC The view of the physical environment 7000 visible in the computer system 7100 is visible (e.g., via a viewport provided by the display generation component of the computer system 7100). For example, if the user 7002 has moved further away from the physical object 7014 (e.g., and closer to the physical wall 7006 and further away from the physical wall 7004), the representation 7014' of the physical object 7014 in the view of the physical environment visible via the display generation component of the computer system 7100 will be smaller (e.g., compared to the view of the physical environment 7000). Figure 7AC The size of 7014' is represented in the image and is compared to the user's different relative position (e.g., with the image in the image). Figure 7AC Compared to the view of the visible 3D environment, in Figure 7AD In the view of the visible 3D environment, it is displayed further to the right. For example, as the physical distance between the physical object and the user changes, the representation of the physical object is displayed at a corresponding size that corresponds to the updated physical size of the object as perceived by the user.

[0238] In some implementations, one or more virtual objects (e.g., optionally locked or anchored to a 3D environment) are displayed at a simulated size corresponding to the distance between the virtual object's location in the 3D environment and the user's current location. For example, virtual object 7012 and user interface 7136 are both environment-locked (e.g., also referred to herein as anchored to a 3D environment), and when the user moves to a new location 7026-d, virtual object 7012 and user interface 7136 are... Figure 7AD The visible 3D environment in the view is at a smaller size (e.g., compared to their size in the view). Figure 7AC (corresponding size in) and new position (e.g., with in) Figure 7AC Compared to the view of the visible 3D environment, in Figure 7AD The portion of the currently visible 3D environment is displayed further to the right in the view to reflect the increased simulated distance and positioning changes between the virtual object 7012 and the user 7002.

[0239] In some implementations, the system function menu 7024 is not environment-locked, but instead head-locked, such that the system function menu 7024 has the same size (e.g., compared to...). Figure 7AC The system function menu 7024 is the same size and position regardless of the user's physical location within the physical environment 7000. In some implementations, the system function menu 7024 maintains the same size and position regardless of the user's physical location (e.g., the user can move to any location within the physical environment 7000, and the system function menu 7024 will remain in the same position as shown in the image). Figure 7AC and Figure 7AD (The same size and position shown, or another size and / or position to be displayed). The head-locked system function menu 7024 allows the user 7002 to easily interact with the system function menu 7024 regardless of the user 7002's position, whereas if the system function menu 7024 is instead environment-locked, the user 7002 may move to a location in the physical environment where the system function menu 7024 is no longer visible (e.g., the user 7002 from...). Figure 7AC The orientation shown is rotated 180 degrees) or the system function menu 7024 is so small (e.g., far away) that the user cannot practically interact with the system function menu elements of the system function menu 7024.

[0240] exist Figure 7AD In this context, the user's attention 7116 is directed to the homepage display 7124. In response to detecting that the user's attention 7116 meets the selection criteria (e.g., it optionally aligns with the above reference), Figure 7AA (Whether the selection criteria are the same or different), computer system 7100 displays a user interface 7146 (e.g., optionally a home user interface) corresponding to home display 7124. In some embodiments, user interface 7146 includes display indicators 7148, 7150, 7152, and 7154. In some embodiments, displaying user interface 7146 optionally includes: stopping the display of user interface 7136 (e.g., user interface 7146 replacing user interface 7136), which is determined by... Figures 7AD to 7AF The user interface 7136 is represented by a dashed outline.

[0241] In some embodiments, power representations 7148, 7150, 7152, and 7154 correspond to corresponding applications of computer system 7100 (e.g., power representations are application icons and / or other representations of applications), and a user interacts with the corresponding power representation corresponding to the corresponding application to launch an application user interface for the corresponding application (e.g., by opening the application and / or displaying a new application window for the application). In some embodiments, power representations 7148, 7150, 7152, and 7154 correspond to corresponding contacts of other users stored in the memory of computer system 7100 (e.g., power representations are user avatars, contact information, telephone numbers, user IDs, or entity names), and user 7002 interacts with the corresponding power representation corresponding to the corresponding contact to initiate a communication session with that contact (e.g., another user associated with a different computer system than computer system 7100). In some implementations, the power representations 7148, 7150, 7152 and 7154 correspond to the corresponding virtual environment (e.g., different AR environment, different VR environment, different AR experience and / or different VR experience), and the user 7002 interacts with the corresponding power representation corresponding to the corresponding virtual environment to initiate the display of the corresponding virtual environment in the three-dimensional environment.

[0242] In some embodiments, the user interface 7146 includes different groups of power indicators (e.g., a first group of power indicators for launching the application user interface, a second group of power indicators for initiating a communication session with other users, and / or a third group of power indicators for initiating the display of different virtual environments). In some embodiments, different groups of power indicators are displayed in the user interface 7146 in different contexts (e.g., not every power indicator is always displayed, and / or not every group of power indicators is always displayed). In some embodiments, the user 7002 switches between different groups of power indicators (e.g., by performing air gestures (e.g., air taps or air pinches), by interacting with a specific power indicator used to switch between groups of power indicators, and / or by interacting with a corresponding power indicator used to display the corresponding group of power indicators). In some embodiments, the user interface 7146 includes additional power indicators (e.g., in addition to or as a replacement for power indicators 7148, 7150, 7152, and 7154), and the user 7002 scrolls between displays of these additional power indicators in the user interface 7146 (e.g., by performing air gestures such as air taps or air pinches). In some embodiments, the user 7002 performs a first type of air gesture (e.g., a tap gesture) to switch between different groups of power indicators, and the user 7002 performs a second type of air gesture (e.g., a pinch-drag gesture) to scroll the display of power indicators (e.g., optionally within a group of displayed power indicators).

[0243] In some embodiments, the power indicators of user interface 7146 within a threshold distance (e.g., close to, near, and / or adjacent to) a specific boundary (or multiple boundaries) of user interface 7146 are displayed with different visual characteristics than power indicators outside the threshold distance (e.g., displayed with a certain degree of fading or blurring or other visual de-emphasis). In some embodiments, the specific boundary (or multiple boundaries) of user interface 7146 on which the power indicators are displayed with different visual characteristics are associated with the direction on which the display of the power indicators of user interface 7146 can be scrolled. For example, if user 7002 can scroll the display of the power indicators of user interface 7146 in the up and / or down directions, the power indicators near the lower boundary (and / or upper boundary) of user interface 7146 (e.g., power indicators 7150 and 7154) are displayed with a certain degree of fading or blurring. In some implementations, user 7002 can scroll the display of the power indicators of user interface 7146 to the left (and / or right), and power indicators (e.g., power indicators 7152 and 7154) near the right (and / or left) boundary of user interface 7146 are displayed with a certain degree of fading or blurring. In some implementations, power indicators within a threshold distance from any of the multiple boundaries (e.g., the top and bottom boundaries, left and right boundaries, top and right boundaries, bottom and left boundaries, top and left boundaries, or bottom and right boundaries, or another combination of boundaries) of user interface 7146 are displayed with different visual characteristics (e.g., visual de-emphasis). In some implementations, the boundaries (or multiple boundaries) are not straight edges (e.g., user interface 7146 is circular in shape, and power indicators near the outer boundary of the circle are displayed with different visual characteristics, while power indicators closer to the center of user interface 7146 are not displayed with different visual characteristics). In some implementations, optionally, only a portion (less than all) of the power indications near a specific boundary of the user interface 7146 is displayed with different visual characteristics (e.g., the lower portions of power indications 7150 and 7154 are displayed as faded or blurred, while the upper portions of power indications 7150 and 7154 are displayed without fading or blurring (e.g., displayed with the same or different visual characteristics as power indications 7148 and 7152).

[0244] In some implementations, the visual characteristics of the power representation within a threshold distance from a specific boundary of the user interface 7146 vary depending on the distance from the specific boundary. For example, the lower portions of power representations 7150 and 7154 (e.g., those closest to the lower boundary of the user interface 7146) are displayed with a first (e.g., high) level of blurring or fading, the upper portions of power representations 7150 and 7154 are displayed with a second level of blurring or fading (e.g., no blurring or fading at all), and the portions of power representations 7150 and 7154 between the lower and upper portions are displayed with a third level of blurring or fading (optionally, an intermediate level of blurring or fading between the first and second levels) (e.g., less blurring or fading compared to the lower portions of power representations 7150 and 7154, but more blurring or fading compared to the upper portions of these power representations).

[0245] In some implementations, the energy representation of the user interface 7146 within a threshold distance from the simulated boundary (or multiple boundaries) of the user interface 7146 is displayed with different visual characteristics than the energy representation outside the threshold distance from the simulated boundary (e.g., displayed with a certain degree of fading or blurring). For example, if the user interface 7146 has a circular shape, the simulated boundary may be the right "edge" of the user interface 7146 (even if the user interface 7146 itself does not have a corresponding boundary), and the energy representations near the 2 o'clock, 3 o'clock, and 4 o'clock positions are displayed with different visual characteristics (e.g., and optionally, the energy representation near the 3 o'clock position is displayed with a greater change in visual characteristics (e.g., a higher degree of blurring or fading) than the energy representations near the 2 o'clock and 4 o'clock positions).

[0246] In some implementations, visual characteristics include analog thickness (e.g., an energy representation with a high level of analog thickness appears more three-dimensional or fully three-dimensional, while an energy representation with a low level of analog thickness appears flatter or more two-dimensional).

[0247] Figure 7AE An example is shown of a user's attention 7116 directed at the display indicator 7148 of the user interface 7146. In response to detecting that the user's gaze meets the selection criteria (e.g., which may optionally be the same as the attention criteria described above for displaying the system function menu 7024, and / or the same as the selection criteria for displaying the user interface 7136 or the user interface 7146), the computer system 7100 displays the user interface 7156 associated with the display indicator 7148.

[0248] In some embodiments, user interface 7156 is an application user interface (e.g., a power indicator 7148 is an application icon, and the user launches the application user interface 7156 by selecting the power indicator 7148). In some embodiments, user interface 7156 includes controls for telephone calls or virtual reality sessions, and / or includes a video user interface for video calls or other communication sessions with another user (e.g., the power indicator 7148 is a user avatar, and the user 7002 initiates a communication session with that other user via the power indicator 7148). In some embodiments, user interface 7156 is associated with a virtual environment (e.g., a three-dimensional environment displayed on the monitor of computer system 7100) (e.g., is the virtual environment, or an element of the virtual environment).

[0249] Figure 7AF This illustration depicts the movement of user 7002 to a new location 7026-e in physical environment 7000 (e.g., in some embodiments, it is the same as location 7026-c). Depending on the movement of user 7002 from location 7026-d to location 7026-e, different views of physical environment 7000 (e.g., optionally with...) are shown. Figures 7Y to 7AC The same view in the computer system 7100 is visible (e.g., via a viewport provided by the display generation component of the computer system 7100). Based on the movement of the user 7002, the size of the representation 7014' of the physical object 7014, the virtual object 7012, and the user interface 7136 is adjusted to match the size of the physical object 7014', the virtual object 7012, and the user interface 7136. Figure 7AC The same corresponding size as the visible 3D environment in the view is used to display it (e.g., at their "original" size before any movement by the user 7002).

[0250] For example, similar to the user interface 7136 described above, user interface 7146 (and / or user interface 7156) is also environment-locked. In response to user 7002 moving to position 7026-e, user interface 7146 (and optionally, user interface 7156) has a different size and position in a view of the three-dimensional environment visible through the viewport provided by the display generation component of computer system 7100. Figure 7AF In the middle, based on position 7026-e (relative to position 7026-d), user interfaces 7146 and 7156 have a larger size, and Figure 7AF The user's current view of the visible 3D environment is displayed further to the right (e.g., compared to user interface 7146 and user interface 7156 in...). Figure 7AE Size and positioning in the visible 3D environment view.

[0251] The user's attention 7116 is directed to the indicator 7010 of the system function menu, and in response to the user's attention 7116 satisfying the attention criteria for the indicator 7010 of the system function menu (e.g., as referenced above). Figure 7Z Using the same attention standard, the system function menu 7024 is displayed. As described above, in some embodiments, the system function menu 7024 is head-locked, and therefore the system function menu 7024 remains at the same size and / or in the same position relative to the user's current view (e.g., the same as when the user was previously in the third position 7026-c). Figure 7AC The system function menu 7024 and when the user moves to position 7026-d Figure 7AD The system function menus are displayed in the same size and / or position as the system function menus.

[0252] related Figures 7Y to 7AF Additional descriptions are provided below for reference regarding Figures 7Y to 7AF The method 11000, as well as other figures and methods described herein, are provided.

[0253] Figures 7AG to 7AM An exemplary area is shown for triggering the display of one or more user interface elements. Figure 12 This is a flowchart of an exemplary method 12000 for initiating the display of a user interface in response to detecting that the user's attention is directed to a corresponding area in a three-dimensional environment. Figures 7AG to 7AM The user interface in the document is used to illustrate the processes described below, including... Figure 12 The process in.

[0254] Figure 7AG An example is shown for triggering one or more user interface elements (e.g., indicator 7010 for system function menus, such as...). Figure 7A The two exemplary regions shown are region 7158 and region 7160.

[0255] In some implementations, regions 7158 and 7160 are invisible (e.g., as shown in the image). Figure 7AG (As indicated by the dashed outlines of regions 7158 and 7160 in the text), and Figures 7AG to 7AM The dashed outlines in the diagram are a visual representation of where regions 7158 and 7160 will be on the display of the computer system 7100.

[0256] In some embodiments, region 7158 is larger than region 7160. In some embodiments, region 7158 includes region 7160 (e.g., region 7160 is a sub-region of region 7158). In some embodiments, regions 7158 and 7160 are viewpoint-locked regions and exhibit behavior similar to that of system function menu indicators 7010 and / or system function menu 7024, as referenced above. Figures 7A to 7D and / or Figures 7E to 7G As stated above.

[0257] In some implementations, one or both of region 7158 and region 7160 have the same characteristics as... Figure 7AG The shapes shown are different from the approximate square shape. Other exemplary shapes include polygonal shapes (e.g., triangles, non-square rectangles, or irregular polygons), circular or elliptical shapes, or irregular shapes (e.g., to avoid overlapping with other displayed user interface elements).

[0258] exist Figure 7AG In this scenario, the user's attention 7116 is not directed to area 7158 or area 7160, and no user interface object or element is displayed within area 7158 or area 7160. In some embodiments, if the user's attention 7116 is not directed to area 7158 or area 7160, but rather to the displayed user interface (e.g., application launch user interface, application user interface, or system space) (e.g., combined with air gestures (e.g., air taps or air pinches), input from a hardware controller, and / or verbal input), the computer system performs an operation corresponding to the displayed user interface (e.g., launching an application, performing an application-specific operation, or adjusting system settings of computer system 7100).

[0259] exist Figure 7AH In this system, the user's attention 7116 is directed to a location within area 7158 (but not within area 7160). In response to detecting that the user's attention 7116 is directed to area 7158, the computer system 7100 displays an indicator 7010 for the system function menu. In some embodiments, in response to detecting that the user's attention 7116 has been directed to a location within area 7158 for a threshold time (e.g., 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, or 5 seconds), the computer system 7100 displays the indicator 7010 for the system function menu. In some embodiments, such as... Figure 7AH As shown, the system function menu indicator 7010 is initially in a smaller size (e.g., relative to...). Figure 7AThe system function menu indicator 7010 is displayed at its "normal" or steady-state size. The system function menu indicator 7010 provides visual feedback to the user that the computer system 7100 has detected the user's attention being directed to area 7158. The system function menu indicator 7010 also provides cues (e.g., visual feedback or indications) such as the user's attention 7116 being near a (smaller) area 7160, and / or other user interactions (e.g., accessing the system function menu 7024 via the full-size indicator 7010, such as...). Figure 7E As shown, and / or as referenced below Figure 7AM A more detailed description is available (e.g., via area 7160 and / or indicator 7010 of the system function menu).

[0260] In some implementations, computer system 7100 (e.g., concurrently with system function menu indicators 7010) displays information for (e.g., via reference below) Figure 7AI The system system 7100 extends the system function menu through an indicator 7010 (to one or more inputs as shown in Figure AM) and / or invokes the system function menu 7024 with instructions (e.g., text-based instructions, visual, or picture instructions). In some embodiments, the instructions are non-visual (e.g., auditory) instructions. In some embodiments, the computer system 7100 provides multiple forms of instructions (e.g., visual and audio instructions). In some embodiments, the instructions are persistently displayed until the user's attention is directed to the displayed instruction. In some embodiments, after the user's attention is directed to the displayed instruction, the computer system detects that the user's attention has moved away from the displayed instruction (e.g., for a threshold amount of time, such as 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, or 10 seconds), and in response, the computer system 7100 stops displaying the instruction. In some embodiments, the system system 7100 stops displaying the instruction after the user has successfully displayed the system function menu indicator 7010 (e.g., displayed in full size, as shown in Figure AM). Figure 7A or Figure 7AM Instructions are provided before (in the middle) and / or before calling the system function menu 7024, and no further instructions are provided thereafter (e.g., to avoid unnecessary repetition and distraction when the user is already familiar with how to call the system function menu 7024).

[0261] In some implementations, the system function menu indicator 7010 is not displayed if it would overlap with existing user interface elements. For example, if user interface 7136 ( Figure 7AA In the upper left position of the monitor (for example, this position will be the same as...) Figure 7AHIf the system function menu indicator 7010 overlaps with the position of the system function menu indicator 7010 in the interface, then the system function menu indicator 7010 will not be displayed. Since the user interface 7136 includes power indicators 7138, 7140, 7142, and 7144 as interactive user interface elements, displaying the system function menu indicator 7010 in addition to the power indicators of the user interface 7136 may create situations where the user has difficulty (e.g., via gaze input or other input including gaze components) accurately interacting with user interface elements. This is avoided by abandoning the display of the system function menu indicator 7010 when it overlaps with the system function menu indicator 7010 in another user interface (and / or the system function menu indicator 7010 is displayed on top of another user interface).

[0262] In some implementations, the system function menu indicator 7010 is viewpoint locked (e.g., similar to areas 7158 and 7160) and has, as referenced... Figures 7B to 7D Similar behaviors described and shown.

[0263] In some embodiments, the position of the system function menu indicator 7010 is selected at least in part based on the user's positioning relative to the computer system 7100. In some embodiments, the system function menu indicator 7010 is displayed in a consistent area of ​​the computer system 7100's display (e.g., the upper right corner, or another area consistently selected by the computer system), but the exact position of the system function menu indicator 7010 varies within that area at least in part based on the user's positioning relative to the computer system 7100 (e.g., the exact position of the system function menu indicator 7010 may be a distance from a corresponding default position to a multi-threshold distance). In some embodiments, the computer system 7100 may include one or more cameras (or other sensors) for tracking the user's positioning and for determining the orientation of the computer system 7100 relative to the user (e.g., the amount of tilt and / or rotation).

[0264] For example, in the case where the computer system 7100 is a handheld device, the computer system 7100 detects that its orientation is not precisely aligned with the user (e.g., when the user holds the computer system 7100 only with their right hand on the right side (relative to the user's view), the horizontal axis of the computer system 7100 (which typically runs from the left to the right in the user's view) is slightly rotated due to the weight of the computer system 7100 and the lack of support on the left side of the computer system 7100 (e.g., via the user's left hand) (e.g., from slightly lower left to slightly upper right in the user's view). According to some embodiments, the computer system 7100 displays an indicator 7010 for the system function menu at the adjusted position (e.g., closer to the top edge and / or right edge of the computer system 7100).

[0265] In some implementations, computer system 7100 is a head-mounted display (HMD), and computer system 7100 adapts to how the user wears the device. For example, due to differences in each user's physical characteristics (e.g., different sizes and positions of each user's ears, eyes, and / or nose), asymmetry in the user's physical characteristics (e.g., ears of slightly different heights, eyes of different sizes, and / or imperfectly level eyeliner), or other external factors (e.g., the presence of a headset, glasses, or other accessories worn on or around the user's head), computer system 7100 may be oriented differently for different users. For example, computer system 7100 detects that the head-mounted display is lower on the user's face than another user's (or lower than the default or average height), and computer system 7100 displays an indicator 7010 for the system function menu at the adjusted position (e.g., closer to the top edge of computer system 7100).

[0266] The indicator 7010 for displaying the system function menu at the adjusted position allows the indicator 7010 to be displayed in a consistent position (e.g., relative to the user), even if the computer system 7100 is not "perfectly" aligned with the user (e.g., without tilting or rotating relative to the user), and also maintains the indicator 7010 in a consistent position during minor movements of the computer system 7100 (e.g., due to instability of the user's hand). In some embodiments, the adjusted position of the system function menu indicator 7010 is within a threshold distance (e.g., 0.5 mm, 1 mm, 2 mm, 5 mm, or 1 cm) of the corresponding position (e.g., the center point of the corresponding upper right portion of the display of the computer system 7100). If the adjusted position of the system function menu indicator 7010 would require displaying the system function menu indicator 7010 at a location greater than a threshold distance, then the computer system 7100 will forgo displaying the system function menu indicator 7010 at the adjusted position (e.g., because the adjusted position would require displaying the system function menu indicator 7010 somewhere beyond the physical boundaries of the computer system 7100's display, or because the system function menu indicator 7010 cannot be easily interacted with at the adjusted position (e.g., the system function menu indicator 7010 would be displayed in a location uncomfortable for the user to interact with, and / or the system function menu indicator 7010 would be displayed in a location where other user interface elements are normally displayed (e.g., somewhere near the center of the computer system 7100's display) and / or where other user interface elements are already displayed)). Instead of displaying the system function menu indicator 7010 at an adjusted location that is more than a threshold distance from the corresponding position, the computer system 7100 displays visual instructions to the user (e.g., instructions for correcting the orientation and / or positioning of the computer system 7100 relative to the user, which would allow the system function menu indicator 7010 to be displayed at a position within a threshold distance of the corresponding position).

[0267] In some embodiments, if the position of the system function menu indicator 7010 is adjusted (e.g., the system function menu indicator 7010 is displayed at the adjusted position, as described above), then the system function menu 7024 is also adjusted by the same amount (e.g., in the same direction). In some embodiments, if the position of the system function menu indicator 7010 is adjusted (e.g., the system function menu indicator 7010 is displayed at the adjusted position, as described above), then other user interface elements (e.g., Figure 7L User interface 7058 Figure 7O Application User Interface 7062 Figure 7AA The user interface 7136 and / or Figure 7ADThe user interface 7146 is adjusted by the same amount (e.g., and in the same direction). In some embodiments, if the position of the indicator 7010 of the system function menu is adjusted, the system function menu 7024 is adjusted by the same amount (e.g., and in the same direction), but other user interface elements are not adjusted (e.g., displayed in normal or default positioning, such that only the indicator 7010 and the system function menu 7024 of the system function menu are adjusted).

[0268] According to some implementation plans, in Figure 7AI During this process, the user's attention 7116 remains focused on a location within area 7158 but outside area 7160, and the system function menu indicator 7010 remains (as opposed to) the location within area 7158. Figure 7AH The size of the system function menu indicator 7010 should be the same as that in the middle. Figure 7AH The same size provides visual feedback to the user: the computer system 7100 has detected the user's attention pointing to area 7158, but has not detected any further progress (e.g., toward the full-size indicator 7010 and / or system function menu 7024 displaying the system function menu) (e.g., the user's attention has not yet been directed to a location within area 7160).

[0269] exist Figure 7AJ In this context, the user's attention 7116 is now directed to a location within region 7160, and the computer system 7100 adjusts the size of the input (e.g., in a position such as...). Figure 7AI The size of the system function menu indicator 7010 shown is as follows: Figure 7A (or as described in more detail below) Figures 7AL to 7AM The system function menu indicator 7010 is displayed at an intermediate size between the indicated sizes shown in the diagram. In some embodiments, the computer system 7100 displays the system function menu indicator 7010 in an expanded manner (e.g., from...). Figure 7AI The size of the system function menu indicator 7010 is expanded. Figure 7AJ The animation includes the size of the indicator 7010 of the system function menu. In some embodiments, area 7160 also increases in size (e.g., optionally proportional to the increase in size of the indicator 7010 of the system function menu). In some embodiments, area 7160 increases in size but remains a sub-area of ​​area 7158.

[0270] In some implementations, region 7160 extends in a manner at least partially symmetrical with respect to the system function menu indicator 7010. For example, as Figure 7AJAs shown, in addition to extending downwards relative to the system function menu indicator 7010, region 7160 also extends symmetrically to the left and right of the system function menu indicator 7010. In some embodiments, region 7160 extends in an at least partially asymmetrical manner. For example, in Figure 7AJ In this embodiment, region 7160 does not extend upward relative to the system function menu indicator 7010 (e.g., to avoid overlapping with existing user interface elements, such as the display indicator on the monitor of computer system 7100 located above the system function menu indicator 7010). In some embodiments, region 7160 extends uniformly around the system function menu indicator 7010 (e.g., the system function menu indicator 7010 is centered in region 7160, and region 7160 extends such that the system function menu indicator 7010 remains centered in region 7160).

[0271] exist Figure 7AK In some implementations, the system function menu indicator 7010 shrinks if the user's attention 7116 moves away from area 7160 (and area 7158). In some implementations, the indicator 7010 shrinks if the user's attention 7116 is outside both area 7158 and area 7160. In some implementations, the user's attention 7162 (illustrated as a shaded arrow to distinguish it from the user's attention 7116) is within area 7158 but outside area 7160, and the system function menu indicator 7010 shrinks (e.g., the indicator 7010 shrinks if the user's attention is not directed to a location within area 7160, even if the user's attention is still directed to a location within area 7158).

[0272] In some implementations, the indicator 7010 of the system function menu contracts at a continuous rate when the user's attention 7116 is outside region 7158 (and / or region 7160). In some implementations, the indicator 7010 of the system function menu contracts at a different rate (e.g., faster or slower) than when the user's attention 7116 is within region 7160. In some implementations, the indicator 7010 of the system function menu begins to contract after the user's attention 7116 has been outside region 7158 (and / or region 7160) for a threshold amount of time (e.g., 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5 seconds, 10 seconds, 30 seconds, or 60 seconds), which prevents the indicator 7010 of the system function menu from contracting if the user's attention is temporarily redirected from region 7158 (and / or region 7160) (e.g., if the user is momentarily distracted by something in the real world or virtual / 3D environment). While the user's attention 7116 is outside region 7158 (and / or region 7160) and before a threshold time has elapsed, the indicator 7010 of the system function menu remains at its appropriate size (e.g., the size of the indicator 7010 of the system function menu as the user's attention 7116 moves away from region 7160). In some embodiments, if the user's attention 7116 returns to region 7160 before the threshold time has elapsed, the indicator 7010 of the system function menu resumes its expanded state.

[0273] exist Figure 7AL In this process, the user's attention 7116 returns to a location within area 7160. According to some implementations, as the user's attention 7116 continues to be directed to a location within area 7160, the indicator 7010 of the system function menu continues to expand until it reaches its maximum size (e.g., ...). Figure 7AL As shown, the system function menu indicator 7010 has the same... Figures 7A to 7G The system function menu indicator is the same size as 7010.

[0274] exist Figure 7AM In this context, the user's attention is directed to the (full-size) indicator 7010 of the system function menu, and in response, the computer system 7100 displays the system function menu 7024. In some embodiments, the system function menu indicator 7010 and the system function menu 7024 are as described above. Figures 7A to 7AF The system function menu indicator 7010 and system function menu 7024 are similar and have the same characteristics and behaviors.

[0275] In some implementations, the system function menu indicator 7010 is not displayed (e.g., in...). Figures 7AG to 7AMIn either of these), and the computer system 7100 responds to detecting the location in the region 7160 where the user's attention 7116 is directed (e.g., by...). Figure 7AM The system function menu 7024 is displayed in the location occupied by the system function menu indicator 7010 or any location in area 7160 (e.g., and optionally, area 7158 instead has approximately the same size as the full size of the system function menu indicator 7010). In some embodiments, if the user 7002 has not previously (e.g., when the system function menu 7024 is first displayed) interacted with the system function menu indicator 7010 and / or the system function menu 7024, the system function menu indicator 7010 is displayed (e.g., as referenced above). Figures 7AG to 7AM (as described above), and optionally, the indicator of the system function menu is not displayed again after the system function menu 7024 has been displayed for the first time (e.g., to avoid the user 7002 already knowing how to access the system function menu 7024 (e.g., by directing the user's attention 7116 to area 7160 and / or one of the areas in area 7160)). Figure 7AM (This indicates the location of the system function menu indicator 7010, causing the computer system's display to become confused due to the system function menu indicator 7010).

[0276] In some implementations, without displaying (e.g., or not subsequently displayed after the initial display of the system function menu 7024) the indicator 7010 of the system function menu, when the user's attention 7116 is directed to area 7160 and / or one of the areas in area 7160... Figure 7AM When the position of the indicator 7010 showing the system function menu reaches a threshold time, the computer system 7100 displays (e.g., or redisplays) the system function menu 7024 (e.g., instead of displaying the indicator 7010, which is an expanded version of the system function menu, as shown in the reference). Figures 7AG to 7AM The computer system 7100 detects that the user's attention 7116 is maintained within the pointing area 7160 for a threshold amount of time (e.g., 0.05 seconds, 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, or 5 seconds, optionally related to...). Figures 7AG to 7AM The same amount of time it takes for the indicator 7010 in the system function menu to expand to its full size.

[0277] In some implementations, the location of the syste...

Claims

1. A method for displaying notification information, the method comprising: At a computer system in communication with a first display generating component and one or more input devices: detecting an occurrence of a first event while a first view of an environment is visible via one or more display generating components; in response to detecting said occurrence of said first event, displaying, via said first display generating component, a first indication of a first notification corresponding to said first event; After displaying, via the first display generating component, the first indication of the first notification corresponding to the first event: maintaining display of the first indication for a first duration and ceasing display of the first indication after expiration of the first duration based on determining that attention of a user of the computer system was directed toward the first indication for a first threshold amount of time and then moved away from the first indication before a first criterion was satisfied; as well as Based on determining that the attention of the user of the computer system has not been directed to the first indication within the first threshold amount of time, maintaining display of the first indication for a second duration and ceasing display of the first indication after expiration of the second duration, wherein the second duration is different from the first duration. The method of claim 1 , wherein the second duration is longer than the first duration.

3. The method of claim 1, wherein the occurrence of the first event comprises at least one of receiving an incoming communication request, an occurrence of a state change of an application, or an occurrence of a state change of the computer system.

4. The method according to claim 1, comprising: After displaying, via the first display generating component, the first indication of the first notification corresponding to the first event: Based on determining that the attention of the user of the computer system was directed to the first indication and the first criterion is satisfied, the first notification including first notification content is displayed.

5. A method according to claim 4, wherein the first notification including the first notification content is displayed concurrently with one or more user interface objects corresponding to a first set of one or more contextual conditions that were satisfied when the user's attention was directed to the first indication.

6. The method according to claim 4, comprising: while displaying the first notification including the first notification content, detecting a first user input that satisfies a second criterion with respect to the first notification; In response to detecting the first user input that satisfies the second criterion, the first notification is expanded to display the first notification content and at least some of additional content corresponding to the first notification that was not included in the first notification content.

7. The method of claim 6, wherein the second criterion comprises the first criterion, the first criterion requiring the user's attention to be maintained on the first notification for at least a second threshold amount of time in order to satisfy the first criterion.

8. The method of claim 6, wherein the second criterion comprises a second criterion requiring that the user input directed to the first notification be a user input of a first input type in order to satisfy the second criterion.

9. The method of any one of claims 1 to 8, wherein displaying the first indication of the first notification in response to detecting the occurrence of the first event comprises: displaying the first indication of the first notification at a first location in the environment, wherein the first location has a first spatial relationship to the first view of the environment; and The method comprises: prior to detecting said occurrence of said first event, detecting that said attention of said user is directed toward said first location in a first view of said environment; as well as In response to detecting that the attention of the user is directed to the first location in the first view of the environment, a first user interface object is displayed at the first location in the first view of the environment.

10. The method of any one of claims 1 to 8, wherein displaying the first indication of the first notification in response to detecting the occurrence of the first event comprises: displaying the first indication of the first notification at a first location in the first view of the environment; and The method comprises: After stopping display of the first indication, detecting that the attention of the user is directed to the first position; as well as In response to detecting that the user's attention is directed toward the first position, A representation of the first notification is displayed.

11. The method according to claim 10, comprising: In response to detecting that the attention of the user is directed to the first location, a first set of one or more user interface objects corresponding to a first set of one or more contextual conditions are displayed concurrently with the representation of the first notification.

12. The method of any one of claims 1 to 8, wherein the environment comprises a three-dimensional environment, and displaying the first indication of the first notification in response to detecting the occurrence of the first event comprises: displaying the first indication of the first notification at a first location in the three-dimensional environment; and The method comprises: detecting that the attention of the user is directed to the first location after ceasing display of the first indication of the first notification after expiration of the first duration or the second duration; and In response to detecting that the user's attention is directed toward the first position: displaying a representation of the first notification based on determining that the attention of the user was directed to the first location within a third threshold amount of time of ceasing display of the first indication of the first notification; Based on determining that the user's attention has not been directed to the first location within the third threshold amount of time of ceasing display of the first indication of the first notification, displaying a second set of one or more user interface objects without displaying the representation of the first notification.

13. A method according to claim 12, wherein the second group of user interface objects includes one or more user interface objects corresponding to a corresponding set of one or more contextual conditions that were satisfied when the user's attention was directed to the first location.

14. The method of claim 12, wherein the second set of user interface objects includes one or more affordances for accessing system functions of the computer system.

15. The method according to any one of claims 1 to 8, comprising: detecting a first request to display a notification user interface after ceasing display of the first indication of the first notification after expiration of the first duration or the second duration; as well as In response to detecting the first request to display the notification user interface, the notification user interface is displayed, wherein the notification user interface includes a plurality of previously received notifications and the first notification.

16. The method of claim 15, wherein detecting the first request to display the notification user interface comprises: Detecting a user interaction with a system user interface that has been displayed in response to detecting that the attention of the user is directed toward a corresponding area of ​​the first view of the environment.

17. The method of claim 16, wherein the corresponding area of ​​the first view of the environment is located at a first location in the environment having a first spatial relationship with the first view of the environment; and The method comprises: detecting a change in a current viewpoint of the user from a first viewpoint corresponding to the first view of the environment to a second viewpoint corresponding to a second view of the environment; as well as in response to detecting the change in the current viewpoint of the user from the first viewpoint to the second viewpoint, updating a currently displayed view of the environment from the first view of the environment to the second view; While displaying the second view of the environment: detecting that the attention of the user is directed toward a corresponding area of ​​the second view of the environment, wherein the corresponding area of ​​the second view of the environment has the first spatial relationship with the second view of the environment; as well as In response to detecting that the attention of the user is directed to a corresponding area of ​​the second view of the environment, the notification user interface is displayed.

18. The method according to any one of claims 1 to 8, wherein: The first event is an event of a first type; the first indication of the first notification corresponding to the first event being displayed at a first location in the environment; The method comprises: detecting an occurrence of a second event of a second type different from the first type; as well as In response to detecting the occurrence of the second event, displaying a second indication of a first alert corresponding to the second event at a second location in the environment that is different from the first location in the environment, wherein the first location is within an area for displaying notifications and the second location is within an area for displaying alerts.

19. The method according to claim 18, comprising: ceasing to display the second indication of the first alert corresponding to the second event; as well as detecting a second request to display a notification user interface after ceasing display of the second indication of the first alert corresponding to the second event; as well as In response to detecting the second request to display the notification user interface, displaying the notification user interface, wherein the notification user interface includes one or more notifications corresponding to the first type of event, and wherein the notification user interface does not include an alert corresponding to the second type of event.

20. The method according to claim 18, comprising: while displaying the second indication of the first alert, detecting that the attention of the user is directed to a corresponding area of ​​the first view of the environment that includes a plurality of affordances for accessing system functions of the computer system; as well as In response to detecting that the user's attention is directed toward the corresponding area including the plurality of affordances for accessing system functions of the computer system, ceasing to display the second indication for the first alert.

21. The method of claim 18, wherein the second indication of the first alert includes one or more selectable options for performing an operation corresponding to the first alert or the second event.

22. A computer system, comprising: a first display generating component; one or more input devices; one or more processors; and A memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the following operations: detecting an occurrence of a first event while a first view of an environment is visible via one or more display generating components; in response to detecting said occurrence of said first event, displaying, via said first display generating component, a first indication of a first notification corresponding to said first event; After displaying, via the first display generating component, the first indication of the first notification corresponding to the first event: maintaining display of the first indication for a first duration and ceasing display of the first indication after expiration of the first duration based on determining that attention of a user of the computer system was directed toward the first indication for a first threshold amount of time and then moved away from the first indication before a first criterion was satisfied; as well as Based on determining that the attention of the user of the computer system has not been directed to the first indication within the first threshold amount of time, maintaining display of the first indication for a second duration and ceasing display of the first indication after expiration of the second duration, wherein the second duration is different from the first duration.

23. The computer system of claim 22, wherein the one or more programs include instructions for performing any of the methods of claims 2 to 21.

24. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that, when executed by a computer system comprising and / or in communication with a first display generating component and one or more input devices, cause the computer system to: detecting an occurrence of a first event while a first view of an environment is visible via one or more display generating components; in response to detecting said occurrence of said first event, displaying, via said first display generating component, a first indication of a first notification corresponding to said first event; After displaying, via the first display generating component, the first indication of the first notification corresponding to the first event: maintaining display of the first indication for a first duration and ceasing display of the first indication after expiration of the first duration based on determining that attention of a user of the computer system was directed toward the first indication for a first threshold amount of time and then moved away from the first indication before a first criterion was satisfied; as well as Based on determining that the attention of the user of the computer system has not been directed to the first indication within the first threshold amount of time, maintaining display of the first indication for a second duration and ceasing display of the first indication after expiration of the second duration, wherein the second duration is different from the first duration.

25. The computer-readable storage medium of claim 24, wherein the one or more programs include instructions that, when executed by the computer system, cause the computer system to perform any of the methods of claims 2 to 21.

Images