Devices, methods, and graphical user interfaces for interacting with virtual objects using hand gestures

Abstract

The present disclosure generally relates to interacting with virtual objects using hand gestures. In some embodiments, methods and user interfaces for navigating content using hand gestures are described. In some embodiments, methods and user interfaces for performing various operations using hand gestures are described. In some embodiments, methods and user interfaces for activating virtual objects are described. In some embodiments, methods and user interfaces for displaying information are described. In some embodiments, methods and user interfaces for manipulating the display of virtual objects are described.

Description

[0001] This application is a divisional application of the invention patent application with application number 202380020934.9, application date February 3, 2023, and name “Device, method and graphical user interface for interacting with virtual objects using hand gestures”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Patent Application No. 18 / 102,036, filed on January 26, 2023, entitled “DEVICES, METHODS, AND GRAPHICALUSER INTERFACES FOR INTERACTING WITH VIRTUAL OBJECTS USING HAND GESTURES” and U.S. Provisional Patent Application No. 63 / 308,012, filed on February 8, 2022, entitled “DEVICES, METHODS, AND GRAPHICAL USER INTERFACES FOR INTERACTING WITH VIRTUAL OBJECTS USING HAND GESTURES”. The entire contents of each of these applications are hereby incorporated by reference herein in their entirety. Technical Field

[0004] The present disclosure generally relates to a computer system in communication with a display generation component and optionally with a wearable device that provides a computer-generated experience, including but not limited to an electronic device that provides virtual reality and mixed reality experiences via a display. Background Art

[0005] In recent years, the development of computer systems for augmented reality has increased significantly. Example augmented reality environments include at least some virtual elements that replace or augment the physical world. Input devices (such as cameras, controllers, joysticks, touch-sensitive surfaces, and touch screen displays) for computer systems and other electronic computing devices are used to interact with virtual / augmented reality environments. Example virtual elements include virtual objects such as digital images, videos, text, icons, and control elements (such as buttons and other graphics). Summary of the invention

[0006] Some methods and interfaces for using hand gestures to display and interact with virtual objects are cumbersome, inefficient, and limited. For example, systems that provide insufficient control over virtual objects, systems that require a series of complex inputs to achieve desired results in an augmented reality environment, and systems where virtual object manipulation is complex, cumbersome, and error-prone can impose a significant cognitive burden on users and detract from the experience of the virtual / augmented reality environment. In addition, these methods take longer than necessary, wasting energy on the computer system. This latter consideration is particularly important in battery-powered devices.

[0007] Therefore, there is a need for a computer system with an improved method and interface for displaying and interacting with virtual objects using hand gestures, making the interaction more efficient and intuitive for the user. Such methods and interfaces optionally supplement or replace conventional methods for displaying and interacting with virtual objects using hand gestures. Such methods and interfaces reduce the amount, degree and / or nature of inputs from the user by helping the user understand the connection between the inputs provided and the device's response to those inputs, thereby forming a more effective human-computer interface.

[0008] The above-mentioned defects and other problems associated with the user interface of the computer system are reduced or eliminated by the disclosed system. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is a portable device (e.g., a notebook computer, a tablet computer, or a handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device, such as a watch or a head-mounted device). In some embodiments, the computer system has a touch pad. In some embodiments, the computer system has one or more cameras. In some embodiments, the computer system has a touch-sensitive display (also referred to as a "touch screen" or "touch screen display"). In some embodiments, the computer system has one or more eye tracking components. In some embodiments, the computer system has one or more hand tracking components. In some embodiments, in addition to the display generation component, the computer system also has one or more output devices, which include one or more tactile output generators and / or one or more audio output devices. In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, a memory, and one or more modules, a program or instruction set stored in the memory for performing multiple functions. In some embodiments, the user interacts with the GUI through contact and gestures of a stylus and / or finger on a touch-sensitive surface, movement of the user's eyes and hands in space relative to the GUI (and / or computer system) or the user's body (as captured by a camera and other motion sensors), and / or voice input (as captured by one or more audio input devices). In some embodiments, the functions performed by interaction optionally include image editing, drawing, presentations, word processing, spreadsheet creation, playing games, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, testing support, digital photography, digital video recording, web browsing, digital music playback, note-taking, and / or digital video playback. Executable instructions for performing these functions are optionally included in a transient and / or non-transient computer-readable storage medium or other computer program product configured for execution by one or more processors.

[0009] There is a need for electronic devices with improved methods and interfaces for displaying and interacting with virtual objects using hand gestures. Such methods and interfaces can supplement or replace traditional methods for displaying and interacting with virtual objects using hand gestures. Such methods and interfaces reduce the amount, degree, and / or nature of input from the user and produce a more efficient human-computer interface. For battery-powered computing devices, such methods and interfaces save power and increase the time interval between battery charges.

[0010] According to some embodiments, a method performed at a computer system that communicates with a display generation component is described. The method includes: when a hand-worn device is being worn by a user, displaying a corresponding user interface including a first portion of content and a second portion of the content that is different from the first portion of the content via the display generation component; when displaying the corresponding user interface including the first portion of the content and the second portion of the content, receiving an indication that the hand-worn device detects a hand input including a rotation of the hand; and in response to receiving an indication that the hand-worn device detects a hand input including a rotation of the hand: navigating between the first portion of the content and the second portion of the content based on determining that the hand-worn device detects that the hand is clenched when performing the hand input; and abandoning navigating between the first portion of the content and the second portion of the content based on determining that the hand-worn device does not detect that the hand is clenched when performing the hand input.

[0011] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system that communicates with a display generation component, and the one or more programs include instructions for performing the following operations: when a hand-worn device is being worn by a user, display a corresponding user interface via the display generation component, the corresponding user interface including a first part of the content and a second part of the content that is different from the first part of the content; when displaying the corresponding user interface including the first part of the content and the second part of the content, receive an indication that the hand-worn device detects a hand input including a rotation of the hand; and in response to receiving an indication that the hand-worn device detects a hand input including a rotation of the hand: based on determining that the hand-worn device detects that the hand is clenched when performing the hand input, navigate between the first part of the content and the second part of the content; and based on determining that the hand-worn device does not detect that the hand is clenched when performing the hand input, abandon navigation between the first part of the content and the second part of the content.

[0012] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system that communicates with a display generation component, and the one or more programs include instructions for performing the following operations: when a hand-worn device is being worn by a user, displaying a corresponding user interface via the display generation component, the corresponding user interface including a first part of the content and a second part of the content that is different from the first part of the content; when displaying the corresponding user interface including the first part of the content and the second part of the content, receiving an indication that the hand-worn device detects a hand input including a rotation of the hand; and in response to receiving an indication that the hand-worn device detects a hand input including a rotation of the hand: navigating between the first part of the content and the second part of the content based on determining that the hand-worn device detects that the hand is clenched when performing the hand input; and abandoning navigating between the first part of the content and the second part of the content based on determining that the hand-worn device does not detect that the hand is clenched when performing the hand input.

[0013] According to some embodiments, a computer system configured to communicate with a display generation component. The computer system includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: when a hand-worn device is being worn by a user, displaying a corresponding user interface via the display generation component, the corresponding user interface including a first part of content and a second part of the content that is different from the first part of the content; when displaying the corresponding user interface including the first part of the content and the second part of the content, receiving an indication that the hand-worn device detects a hand input including a rotation of the hand; and in response to receiving an indication that the hand-worn device detects a hand input including a rotation of the hand: navigating between the first part of the content and the second part of the content based on determining that the hand-worn device detects that the hand is clenched when performing the hand input; and abandoning navigating between the first part of the content and the second part of the content based on determining that the hand-worn device does not detect that the hand is clenched when performing the hand input.

[0014] According to some embodiments, a computer system is configured to communicate with a display generation component. The computer system includes: a component for displaying a corresponding user interface including a first part of content and a second part of the content different from the first part of the content via the display generation component when the hand-worn device is being worn by a user; a component for receiving an indication that the hand-worn device detects a hand input including a rotation of the hand when displaying the corresponding user interface including the first part of the content and the second part of the content; and a component for performing the following operations in response to receiving an indication that the hand-worn device detects a hand input including a rotation of the hand: navigating between the first part of the content and the second part of the content based on determining that the hand-worn device detects that the hand is clenched when performing the hand input; and abandoning navigation between the first part of the content and the second part of the content based on determining that the hand-worn device does not detect that the hand is clenched when performing the hand input.

[0015] According to some embodiments, a computer program product is described. The computer program product includes one or more programs configured to be executed by one or more processors of a computer system that communicates with a display generation component. The one or more programs include instructions for performing the following operations: when a hand-worn device is being worn by a user, a corresponding user interface including a first part of the content and a second part of the content that is different from the first part of the content is displayed via the display generation component; when displaying the corresponding user interface including the first part of the content and the second part of the content, an indication that the hand-worn device detects a hand input including the rotation of the hand is received; and in response to receiving an indication that the hand-worn device detects a hand input including the rotation of the hand: based on determining that the hand-worn device detects that the hand is clenched when performing the hand input, navigate between the first part of the content and the second part of the content; and based on determining that the hand-worn device does not detect that the hand is clenched when performing the hand input, abandon navigation between the first part of the content and the second part of the content.

[0016] According to some embodiments, a method performed at a computer system that communicates with a display generation component is described. The method includes: displaying a corresponding user interface via a display generation component, wherein displaying the corresponding user interface includes simultaneously displaying: a first control virtual object, which causes the computer system to perform a first operation when activated with a first type of input; a second control virtual object, which causes the computer system to perform a second operation different from the first operation when activated with a first type of input; and a first virtual object, which indicates that the first control virtual object can be activated in response to executing a second type of input, wherein the second type of input does not point to a position in the corresponding user interface; when displaying the first control virtual object, receiving an indication that the corresponding input has been executed; and in response to receiving an indication that the corresponding input has been executed: initiating a process for performing the first operation based on determining that the corresponding input is a first type of input pointing to a position corresponding to the first control virtual object; initiating a process for performing the second operation based on determining that the corresponding input is a first type of input pointing to a position corresponding to the second control virtual object; and initiating a process for performing the first operation based on determining that the corresponding input is a second type of input.

[0017] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system that communicates with a display generation component, the one or more programs including instructions for performing the following operations: displaying a corresponding user interface via the display generation component, wherein displaying the corresponding user interface includes simultaneously displaying: a first control virtual object, which, when activated with a first type of input, causes the computer system to perform a first operation; a second control virtual object, which, when activated with the first type of input, causes the computer system to perform a second operation different from the first operation; and a first virtual object, the first virtual object indicating that the first control virtual object can be activated in response to executing a second type of input, wherein the second type of input does not point to a location in the corresponding user interface; when displaying the first control virtual object, receiving an indication that the corresponding input has been executed; and in response to receiving the indication that the corresponding input has been executed: initiating a process for performing the first operation based on determining that the corresponding input is a first type of input pointing to a location corresponding to the first control virtual object; initiating a process for performing the second operation based on determining that the corresponding input is a first type of input pointing to a location corresponding to the second control virtual object; and initiating a process for performing the first operation based on determining that the corresponding input is a second type of input.

[0018] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component, the one or more programs including instructions for performing the following operations: displaying a corresponding user interface via a display generation component, wherein displaying the corresponding user interface includes simultaneously displaying: a first control virtual object, which, when activated with a first type of input, causes the computer system to perform a first operation; a second control virtual object, which, when activated with a first type of input, causes the computer system to perform a second operation different from the first operation; and a first virtual object, the first virtual object indicating that the first control virtual object can be activated in response to executing a second type of input, wherein the second type of input does not point to a location in the corresponding user interface; when displaying the first control virtual object, receiving an indication that the corresponding input has been executed; and in response to receiving an indication that the corresponding input has been executed: initiating a process for performing the first operation based on determining that the corresponding input is a first type of input pointing to a location corresponding to the first control virtual object; initiating a process for performing the second operation based on determining that the corresponding input is a first type of input pointing to a location corresponding to the second control virtual object; and initiating a process for performing the first operation based on determining that the corresponding input is a second type of input.

[0019] According to some embodiments, a computer system configured to communicate with a display generation component is described. The computer system includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: displaying a corresponding user interface via a display generation component, wherein displaying the corresponding user interface includes simultaneously displaying: a first control virtual object, which, when activated with a first type of input, causes the computer system to perform a first operation; a second control virtual object, which, when activated with the first type of input, causes the computer system to perform a second operation different from the first operation; and a first virtual object, which indicates that the first control virtual object can be activated in response to executing a second type of input, wherein the second type of input does not point to a location in the corresponding user interface; when displaying the first control virtual object, receiving an indication that the corresponding input has been executed; and in response to receiving the indication that the corresponding input has been executed: initiating a process for performing the first operation based on determining that the corresponding input is a first type of input pointing to a location corresponding to the first control virtual object; initiating a process for performing the second operation based on determining that the corresponding input is a first type of input pointing to a location corresponding to the second control virtual object; and initiating a process for performing the first operation based on determining that the corresponding input is a second type of input.

[0020] According to some embodiments, a computer system configured to communicate with a display generation component is described. The computer system includes: a component for displaying a corresponding user interface via a display generation component, wherein displaying the corresponding user interface includes simultaneously displaying: a first control virtual object, which, when activated with a first type of input, causes the computer system to perform a first operation; a second control virtual object, which, when activated with a first type of input, causes the computer system to perform a second operation different from the first operation; and a first virtual object, which indicates that the first control virtual object can be activated in response to executing a second type of input, wherein the second type of input does not point to a position in the corresponding user interface; a component for receiving an indication that the corresponding input has been executed when displaying the first control virtual object; and a component for performing the following operations in response to receiving an indication that the corresponding input has been executed: initiating a process for performing the first operation based on determining that the corresponding input is a first type of input pointing to a position corresponding to the first control virtual object; initiating a process for performing the second operation based on determining that the corresponding input is a first type of input pointing to a position corresponding to the second control virtual object; and initiating a process for performing the first operation based on determining that the corresponding input is a second type of input.

[0021] According to some embodiments, a computer program product is described. The computer program product includes one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component. The one or more programs include instructions for performing the following operations: displaying a corresponding user interface via a display generation component, wherein displaying the corresponding user interface includes simultaneously displaying: a first control virtual object, which causes the computer system to perform a first operation when activated with a first type of input; a second control virtual object, which causes the computer system to perform a second operation different from the first operation when activated with a first type of input; and a first virtual object, which indicates that the first control virtual object can be activated in response to executing a second type of input, wherein the second type of input does not point to a position in the corresponding user interface; when displaying the first control virtual object, receiving an indication that the corresponding input has been executed; and in response to receiving an indication that the corresponding input has been executed: based on determining that the corresponding input is a first type of input pointing to a position corresponding to the first control virtual object, initiating a process for performing the first operation; based on determining that the corresponding input is a first type of input pointing to a position corresponding to the second control virtual object, initiating a process for performing the second operation; and based on determining that the corresponding input is a second type of input, initiating a process for performing the first operation.

[0022] According to some embodiments, a method performed at a computer system in communication with a display generation component is described. The method includes: while displaying an extended reality environment including a virtual object via the display generation component, receiving an indication of a first hand input performed by a hand on which a wearable device is being worn, the virtual object occluding at least a first portion of a physical environment including the wearable device, wherein the first hand input includes movement of one or more fingers of the hand relative to a portion of the hand; and in response to receiving an indication that a hand input has been performed by the hand on which the wearable device is being worn, displaying information about the wearable device via the display generation component.

[0023] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system communicating with a display generation component, and the one or more programs include instructions for performing the following operations: when displaying an extended reality environment including a virtual object via the display generation component, receiving an indication of a first hand input performed by a hand on which a wearable device is being worn, the virtual object occluding at least a first portion of the physical environment including the wearable device, wherein the first hand input includes movement of one or more fingers of the hand relative to a portion of the hand; and in response to receiving an indication that a hand input has been performed by the hand on which the wearable device is being worn, displaying information about the wearable device via the display generation component.

[0024] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system that communicates with a display generation component, the one or more programs including instructions for performing the following operations: when displaying an extended reality environment including a virtual object via the display generation component, receiving an indication of a first hand input performed by a hand on which a wearable device is being worn, the virtual object occluding at least a first portion of a physical environment including the wearable device, wherein the first hand input includes movement of one or more fingers of the hand relative to a portion of the hand; and in response to receiving an indication that a hand input has been performed by the hand on which the wearable device is being worn, displaying information about the wearable device via the display generation component.

[0025] According to some embodiments, a computer system configured to communicate with a display generation component is described. The computer system includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: when displaying an extended reality environment including a virtual object via the display generation component, receiving an indication of a first hand input performed by a hand on which a wearable device is being worn, the virtual object occluding at least a first portion of the physical environment including the wearable device, wherein the first hand input includes movement of one or more fingers of the hand relative to a portion of the hand; and in response to receiving an indication that a hand input has been performed by the hand on which the wearable device is being worn, displaying information about the wearable device via the display generation component.

[0026] According to some embodiments, a computer system configured to communicate with a display generation component is described. The computer system includes: a component for receiving an indication of a first hand input performed by a hand on which a wearable device is being worn when an extended reality environment including a virtual object is displayed via the display generation component, the virtual object occluding at least a first portion of a physical environment including the wearable device, wherein the first hand input includes movement of one or more fingers of the hand relative to a portion of the hand; and a component for displaying information about the wearable device via the display generation component in response to receiving an indication that a hand input has been performed by the hand on which the wearable device is being worn.

[0027] According to some embodiments, a computer program product is described. The computer program product includes one or more programs configured to be executed by one or more processors of a computer system that communicates with a display generation component. The one or more programs include instructions for performing the following operations: when an extended reality environment including a virtual object is displayed via the display generation component, an indication of a first hand input performed by a hand on which a wearable device is being worn is received, and the virtual object obscures at least a first portion of the physical environment including the wearable device, wherein the first hand input includes movement of one or more fingers of the hand relative to a portion of the hand; and in response to receiving an indication that a hand input has been performed by a hand on which the wearable device is being worn, information about the wearable device is displayed via the display generation component.

[0028] According to some embodiments, a method performed at a computer system in communication with a display generation component and a wearable device is described. The method includes: when displaying an augmented reality environment user interface via the display generation component, receiving an indication that a first hand input is performed by a first hand of a user; and in response to receiving an indication that the first hand input is performed by the first hand of the user: performing a first operation based on determining that the first hand input is performed when a second hand input is performed by a second hand of the user, wherein the second hand of the user is different from the first hand; and abandoning the execution of the first operation based on determining that the first hand input is performed when the second hand input is not performed by the second hand of the user.

[0029] According to some embodiments, a non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system that communicates with a display generation component and a wearable device, and the one or more programs include instructions for performing the following operations: when displaying an augmented reality environment user interface via the display generation component, receiving an indication that a first hand input is performed by a first hand of a user; and in response to receiving an indication that the first hand input is performed by the first hand of the user: performing a first operation based on determining that the first hand input is performed when a second hand input is performed by a second hand of the user, wherein the second hand of the user is different from the first hand; and abandoning the execution of the first operation based on determining that the first hand input is performed when the second hand input is not performed by the second hand of the user.

[0030] According to some embodiments, a transient computer-readable storage medium is described. The transient computer-readable storage medium stores one or more programs configured to be executed by one or more processors of a computer system that communicates with a display generation component and a wearable device, and the one or more programs include instructions for performing the following operations: when displaying an augmented reality environment user interface via the display generation component, receiving an indication that a first hand input is performed by a first hand of a user; and in response to receiving an indication that the first hand input is performed by the first hand of the user: performing a first operation based on determining that the first hand input is performed when a second hand input is performed by a second hand of the user, wherein the second hand of the user is different from the first hand; and abandoning the execution of the first operation based on determining that the first hand input is performed when the second hand input is not performed by the second hand of the user.

[0031] According to some embodiments, a computer system configured to communicate with a display generation component and a wearable device is described. The computer system includes: one or more processors; and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: when displaying an augmented reality environment user interface via the display generation component, receiving an indication that a first hand input is performed by a first hand of a user; and in response to receiving an indication that a first hand input is performed by a first hand of a user: performing a first operation based on determining that the first hand input is performed when a second hand input is performed by a second hand of the user, wherein the second hand of the user is different from the first hand; and abandoning the execution of the first operation based on determining that the first hand input is performed when the second hand input is not performed by the second hand of the user.

[0032] According to some embodiments, a computer system configured to communicate with a display generation component and a wearable device is described. The computer system includes: a component for receiving an indication of a first hand input performed by a user's first hand when an augmented reality environment user interface is displayed via the display generation component; and a component for performing the following operations in response to receiving an indication of a first hand input performed by the user's first hand: performing a first operation based on determining that the first hand input is performed when a second hand input is performed by a second hand of the user, wherein the second hand of the user is different from the first hand; and abandoning the execution of the first operation based on determining that the first hand input is performed when the second hand input is not performed by the second hand of the user.

[0033] According to some embodiments, a computer program product is described. The computer program product includes one or more programs configured to be executed by one or more processors of a computer system that communicates with a display generation component and a wearable device. The one or more programs include instructions for performing the following operations: when displaying an augmented reality environment user interface via a display generation component, receiving an indication that a first hand input is performed by a user's first hand; and in response to receiving an indication that a first hand input is performed by a user's first hand: performing a first operation based on determining that the first hand input is performed when a second hand input is performed by a user's second hand, wherein the user's second hand is different from the first hand; and performing the first operation based on determining that the first hand input is performed when the user's second hand does not perform the second hand input, abandoning the execution of the first operation.

[0034] It should be noted that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described in this specification are not comprehensive, and in particular, many additional features and advantages will be apparent to those of ordinary skill in the art based on the drawings, the specification, and the claims. In addition, it should be noted that the language used in this specification is selected in principle for readability and instructional purposes, and may not be selected to describe or define the subject matter of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, wherein like reference numerals designate corresponding parts throughout the several views.

[0036] Figure 1 is a block diagram illustrating an operating environment of a computer system for providing an XR experience according to some embodiments.

[0037] Figure 2 is a block diagram illustrating a controller of a computer system configured to manage and coordinate a user's XR experience according to some embodiments.

[0038] Figure 3 is a block diagram illustrating display generation components of a computer system configured to provide a visual component of an XR experience to a user according to some embodiments.

[0039] Figure 4 is a block diagram illustrating a hand tracking unit of a computer system configured to capture gesture input from a user according to some embodiments.

[0040] Figure 5 is a block diagram illustrating an eye tracking unit of a computer system configured to capture gaze input of a user according to some embodiments.

[0041] Figure 6 is a flow chart illustrating a flash-assisted gaze tracking pipeline according to some embodiments.

[0042] 7A to 7F Example techniques for navigating content using hand gestures according to some embodiments are shown.

[0043] FIG. 8A to FIG. 8H Example techniques for using hand gestures to perform various operations according to some embodiments are shown.

[0044] Fig. 9 is a flow chart of a method of navigating content using hand gestures according to various embodiments.

[0045] FIG. 10A to FIG. 10IExample techniques for activating virtual objects according to some embodiments are shown.

[0046] FIG. 11A to FIG. 11B is a flow chart of a method of activating a virtual object according to various embodiments.

[0047] FIG. 12A to FIG. 12E Example techniques for displaying information according to some embodiments are shown.

[0048] Fig.13 is a flow chart of a method of displaying an object according to various embodiments.

[0049] FIG. 14A to FIG. 14G Example techniques for manipulating the display of virtual objects according to some embodiments are shown.

[0050] Fig.15 is a flowchart of a method for manipulating the display of a virtual object according to some embodiments. DETAILED DESCRIPTION

[0051] According to some embodiments, the present disclosure relates to a user interface for providing an extended reality (XR) experience to a user.

[0052] The systems, methods, and GUIs described herein improve user interface interactions with virtual / augmented reality environments in a number of ways.

[0053] Figures 1 to 6 A description of an example computer system for providing an XR experience to a user is provided. 7A to 7F Example techniques for navigating content using hand gestures according to some embodiments are shown. FIG. 8A to FIG. 8H Example techniques for using hand gestures to perform various operations are shown. Fig. 9 is a flow chart of a method of navigating content using hand gestures according to various embodiments. 7A to 7F and FIG. 8A to FIG. 8H The user interface in Fig. 9 process. FIG. 10A to FIG. 10I Example techniques for activating virtual objects according to some embodiments are shown. FIG. 11A to FIG. 11B is a flow chart of a method of activating a virtual object according to various embodiments. FIG. 10A to FIG. 10I The user interface in FIG. 11A to FIG. 11B process. FIG. 12A to FIG. 12E Example techniques for displaying information according to some embodiments are shown. Fig.13 is a flow chart of a method of displaying information according to various embodiments. FIG. 12A to FIG. 12E The user interface in Fig.13 process. FIG. 14A to FIG. 14GExample techniques for manipulating the display of virtual objects according to some embodiments are shown. Fig.15 is a flow chart of a method of manipulating the display of a virtual object. FIG. 14A to FIG. 14G The user interface in Fig.15 process.

[0054] The processes described below enhance the operability of the device and make the user-device interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the device) through various techniques, including by providing improved visual feedback to the user, reducing the number of inputs required to perform an operation, providing additional control options without cluttering the user interface with additional display controls, performing an operation without further user input when a set of conditions have been met, improving privacy and / or security, and / or additional techniques. These techniques also reduce power usage and extend the battery life of the device by enabling the user to use the device more quickly and efficiently.

[0055] In addition, in the method described herein where one or more steps depend on one or more conditions being met, it should be understood that the method can be repeated in multiple repetitions so that in the process of repetition, all conditions for determining the steps in the method have been met in different repetitions of the method. For example, if the method needs to perform the first step (if the condition is met), and perform the second step (if the condition is not met), then the ordinary technician will know that the steps stated are repeated until both the condition is met and the condition is not met (in no particular order). Therefore, the method described as having one or more steps depending on one or more conditions being met can be rewritten as a method of repeating until each condition described in the method is met. However, this does not require the system or computer-readable medium to declare that the system or computer-readable medium contains instructions for performing a contingent operation based on the satisfaction of the corresponding one or more conditions, and is therefore able to determine whether the possible situation has been met without explicitly repeating the steps of the method until all conditions for determining the steps in the method have been met. It will also be understood by ordinary technicians in the art that, similar to the method with the contingent step, the system or computer-readable storage medium can repeat the steps of the method as many times as needed to ensure that all the contingent steps have been performed.

[0056] In some embodiments, such as Figure 1As shown in , an XR experience is provided to a user via an operating environment 100 including a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), a display generation component 120 (e.g., a head mounted device (HMD), a display, a projector, a touch screen, etc.), one or more input devices 125 (e.g., an eye tracking device 130, a hand tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a tactile output generator 170, and other output devices 180), one or more sensors 190 (e.g., an image sensor, a light sensor, a depth sensor, a tactile sensor, an orientation sensor, a proximity sensor, a temperature sensor, a position sensor, a motion sensor, a speed sensor, etc.), and optionally one or more peripheral devices 195 (e.g., a household appliance, a wearable device, etc.). In some embodiments, one or more of the input device 125, the output device 155, the sensor 190, and the peripheral device 195 are integrated with the display generation component 120 (e.g., in a head mounted device or a handheld device).

[0057] In describing an XR experience, various terms are used to distinguishably refer to several related but distinct environments that a user can sense and / or with which the user can interact (e.g., using inputs detected by the computer system 101 generating the XR experience, which inputs cause the computer system generating the XR experience to generate audio, visual, and / or tactile feedback corresponding to the various inputs provided to the computer system 101). The following is a subset of these terms:

[0058] Physical environment: The physical environment refers to the physical world that people can sense and / or interact with without the help of electronic systems. A physical environment, such as a physical park, includes physical objects, such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0059] Extended Reality: In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via an electronic system. In XR, a subset of a person's physical movements or a representation thereof is tracked, and in response, one or more features of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one physical law. For example, an XR system may detect a person's head turn, and in response, adjust the graphical content and sound field presented to the person in a manner similar to the way such views and sounds change in a physical environment. In some cases (e.g., for accessibility reasons), adjustments to the features of virtual objects in the XR environment may be made in response to a representation of physical movement (e.g., a voice command). People can sense and / or interact with XR objects using any of their senses, including vision, hearing, touch, taste, and smell. For example, people can sense and / or interact with audio objects, which create a 3D or spatial audio environment that provides the perception of a point audio source in 3D space. As another example, an audio object can enable audio transparency that selectively introduces ambient sounds from the physical environment with or without computer-generated audio. In some XR environments, a person can sense and / or interact only with an audio object.

[0060] Examples of XR include virtual reality and mixed reality.

[0061] Virtual Reality: A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input to one or more senses. A VR environment includes a plurality of virtual objects that a person can sense and / or interact with. For example, trees, buildings, and computer-generated images representing avatars of people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence within the computer-generated environment and / or through a simulation of a subset of the person's physical movement within the computer-generated environment.

[0062] Mixed Reality: In contrast to VR environments, which are designed to be based entirely on computer-generated sensory input, a mixed reality (MR) environment refers to a simulated environment that is designed to include sensory input from the physical environment, or representations thereof, in addition to computer-generated sensory input (e.g., virtual objects). On the virtuality continuum, a mixed reality environment is anything between, but not including, a fully physical environment at one end and a virtual reality environment at the other end. In some MR environments, computer-generated sensory input can respond to changes in sensory input from the physical environment. In addition, some electronic systems used to render MR environments can track position and / or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical items from the physical environment, or representations thereof). For example, the system can cause motion so that virtual trees appear stationary relative to the physical ground.

[0063] Examples of mixed reality include augmented reality and augmented virtuality.

[0064] Augmented reality: An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation of a physical environment. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on a transparent or translucent display so that a person uses the system to perceive virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. A person uses the system to indirectly view the physical environment via an image or video of the physical environment and perceives virtual objects superimposed on the physical environment. As used herein, a video of a physical environment displayed on an opaque display is referred to as a "transparent video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images when presenting the AR environment on an opaque display. Further alternatively, the system may have a projection system that projects virtual objects into a physical environment, such as as a hologram or on a physical surface, so that a person using the system perceives virtual objects superimposed on the physical environment. An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a pass-through video, the system may transform one or more sensor images to apply a selected perspective (e.g., a viewpoint) that is different from the perspective captured by the imaging sensor. For another example, the representation of the physical environment may be transformed by graphically modifying (e.g., enlarging) a portion thereof so that the modified portion may be a representative but not real version of the original captured image. For another example, the representation of the physical environment may be transformed by graphically eliminating a portion thereof or blurring a portion thereof.

[0065] Augmented Virtual: An augmented virtual (AV) environment refers to a simulated environment in which a virtual environment or a computer-generated environment incorporates one or more sensory inputs from the physical environment. The sensory input may be a representation of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the faces of people are realistically reproduced from images taken of physical people. For another example, a virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, a virtual object may take a shadow that conforms to the positioning of the sun in the physical environment.

[0066] Viewpoint-locked virtual objects: When a computer system displays a virtual object at the same position and / or location in a user's viewpoint, the virtual object is viewpoint-locked even if the user's viewpoint shifts (e.g., changes). In embodiments where the computer system is a head-mounted device, the user's viewpoint is locked to the forward direction of the user's head (e.g., when the user is looking straight ahead, the user's viewpoint is at least a portion of the user's field of view); thus, without moving the user's head, the user's viewpoint remains fixed even when the user's gaze shifts. In embodiments where the computer system has a display generation component (e.g., a display screen) that is repositionable relative to the user's head, the user's viewpoint is an augmented reality view presented to the user on the display generation component of the computer system. For example, a viewpoint-locked virtual object that is displayed in the upper left corner of the user's viewpoint when the user's viewpoint is in a first orientation (e.g., the user's head is facing north) continues to be displayed in the upper left corner of the user's viewpoint even when the user's viewpoint changes to a second orientation (e.g., the user's head is facing west). In other words, the position and / or location of the viewpoint-locked virtual object displayed in the user's viewpoint is independent of the user's position and / or orientation in the physical environment. In embodiments where the computer system is a head-mounted device, the user's viewpoint is locked to the orientation of the user's head, such that the virtual object is also referred to as a "head-locked virtual object."

[0067] Environment-locked visual objects: A virtual object is environment-locked (alternatively, "world-locked") when a computer system displays a virtual object at a position and / or location in a user's viewpoint that is based on (e.g., selected with reference to and / or anchored to) a location and / or object in a three-dimensional environment (e.g., a physical environment or a virtual environment). As the user's viewpoint moves, the location and / or objects in the environment change relative to the user's viewpoint, which causes the environment-locked virtual object to be displayed at a different location and / or location in the user's viewpoint. For example, an environment-locked virtual object locked to a tree immediately in front of the user is displayed at the center of the user's viewpoint. When the user's viewpoint shifts to the right (e.g., the user's head turns to the right) such that the tree is now to the left of center in the user's viewpoint (e.g., the tree location in the user's viewpoint shifts), the environment-locked virtual object locked to the tree is displayed to the left of center in the user's viewpoint. In other words, the position and / or orientation of the virtual object that is locked to the environment in the user's viewpoint depends on the position and / or orientation of the object and / or the position to which the virtual object is locked in the environment. In some embodiments, the computer system uses a stationary reference system (e.g., a coordinate system anchored to a fixed position and / or object in the physical environment) to determine the positioning of the virtual object that is locked to the user's viewpoint. The virtual object that is locked to the environment can be locked to a stationary part of the environment (e.g., a floor, wall, table, or other stationary object), or can be locked to a movable part of the environment (e.g., a vehicle, animal, person, or even a representation of a part of the user's body that moves independently of the user's viewpoint, such as a hand, wrist, arm, or foot of the user) so that the virtual object moves as the viewpoint or the part of the environment moves to maintain a fixed relationship between the virtual object and the part of the environment.

[0068] In some embodiments, an environment-locked or viewpoint-locked virtual object exhibits an inertial following behavior that reduces or delays the movement of the environment-locked or viewpoint-locked virtual object relative to the movement of the reference point followed by the virtual object. In some embodiments, when exhibiting inertial following behavior, when detecting the movement of a reference point (e.g., a portion of the environment, a viewpoint, or a point fixed relative to the viewpoint, such as a point between 5cm and 300cm from the viewpoint) that the virtual object is following, the computer system intentionally delays the movement of the virtual object. For example, when the reference point (e.g., the portion of the environment or the viewpoint) moves at a first speed, the virtual object is moved by the device to remain locked to the reference point, but moves at a second speed that is slower than the first speed (e.g., until the reference point stops moving or slows down, at which point the virtual object begins to catch up with the reference point). In some embodiments, when the virtual object exhibits inertial following behavior, the device ignores small amounts of movement of the reference point (e.g., ignoring movements of the reference point below a threshold movement amount, such as moving 0 degrees to 5 degrees or moving 0cm to 50cm). For example, when a reference point (e.g., the portion of the environment or viewpoint to which the virtual object is locked) moves a first amount, the distance between the reference point and the virtual object increases (e.g., because the virtual object is being displayed so as to maintain a fixed or substantially fixed position relative to a viewpoint or portion of the environment that is different from the reference point to which the virtual object is locked), and when the reference point (e.g., the portion of the environment or viewpoint to which the virtual object is locked) moves a second amount greater than the first amount, the distance between the reference point and the virtual object initially increases (e.g., because the virtual object is being displayed so as to maintain a fixed or substantially fixed position relative to a viewpoint or portion of the environment that is different from the reference point to which the virtual object is locked), and then decreases as the amount of movement of the reference point increases above a threshold (e.g., a “lazy follow” threshold) because the virtual object is moved by the computer system to maintain a fixed or substantially fixed position relative to the reference point. In some embodiments, maintaining a substantially fixed position of the virtual object relative to the reference point includes displaying the virtual object within a threshold distance (e.g., 1 cm, 2 cm, 3 cm, 5 cm, 15 cm, 20 cm, 50 cm) of the reference point in one or more dimensions (e.g., up / down, left / right, and / or forward / backward relative to the position of the reference point).

[0069] Hardware: There are many different types of electronic systems that enable a person to sense various XR environments and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablet devices, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system can incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. Instead of an opaque display, a head-mounted system can have a transparent or translucent display. A transparent or translucent display can have a medium through which light representing the image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, a transparent or translucent display can be configured to become selectively opaque. The projection-based system can employ retinal projection technology that projects graphic images onto a person's retina. The projection system can also be configured to project virtual objects into a physical environment, such as as a hologram or on a physical surface. In some embodiments, the controller 110 is configured to manage and coordinate the user's XR experience. In some embodiments, the controller 110 includes a suitable combination of software, firmware, and / or hardware. The following description with respect to Figure 2Controller 110 is described in more detail. In some embodiments, controller 110 is a computing device that is located locally or remotely relative to scene 105 (e.g., physical environment). For example, controller 110 is a local server located within scene 105. As another example, controller 110 is a remote server (e.g., cloud server, central server, etc.) located outside scene 105. In some embodiments, controller 110 is communicatively coupled to display generation component 120 (e.g., HMD, display, projector, touch screen, etc.) via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is included within a housing (e.g., a physical housing) of the display generating component 120 (e.g., an HMD or a portable electronic device including a display and one or more processors, etc.), one or more input devices of the input devices 125, one or more output devices of the output devices 155, one or more sensors of the sensors 190, and / or one or more peripheral devices of the peripheral devices 195, or shares the same physical housing or support structure with one or more of the above devices.

[0070] In some embodiments, the display generation component 120 is configured to provide an XR experience (e.g., at least the visual component of the XR experience) to the user. In some embodiments, the display generation component 120 includes a suitable combination of software, firmware, and / or hardware. Figure 3 Display generation component 120 is described in further detail. In some embodiments, the functionality of controller 110 is provided by and / or combined with display generation component 120.

[0071] According to some embodiments, display generation component 120 provides an XR experience to the user when the user is virtually and / or physically present within scene 105.

[0072] In some embodiments, the display generation component is worn on a part of the user's body (e.g., on his / her head, on his / her hand, etc.). In this way, the display generation component 120 includes one or more XR displays provided for displaying XR content. For example, in various embodiments, the display generation component 120 surrounds the user's field of view. In some embodiments, the display generation component 120 is a handheld device (such as a smart phone or tablet device) configured to present XR content, and the user holds a device with a display facing the user's field of view and a camera facing the scene 105. In some embodiments, the handheld device is optionally placed in a shell worn on the user's head. In some embodiments, the handheld device is optionally placed on a support (e.g., a tripod) in front of the user. In some embodiments, the display generation component 120 is an XR room, shell, or room configured to present XR content, wherein the user does not wear or hold the display generation component 120. Many user interfaces described with reference to one type of hardware for displaying XR content (e.g., a handheld device or a device on a tripod) may be implemented on another type of hardware for displaying XR content (e.g., an HMD or other wearable computing device). For example, a user interface showing interactions with XR content triggered based on interactions occurring in the space in front of a handheld device or a tripod-mounted device may similarly be implemented with an HMD, where the interactions occur in the space in front of the HMD and responses to the XR content are displayed via the HMD. Similarly, a user interface showing interactions with XR content triggered based on movement of a handheld device or a tripod-mounted device relative to a physical environment (e.g., scene 105 or a part of a user's body (e.g., the user's eyes, head, or hands)) may similarly be implemented with an HMD, where the movement is caused by movement of the HMD relative to the physical environment (e.g., scene 105 or a part of a user's body (e.g., the user's eyes, head, or hands)).

[0073] Despite Figure 1 Relevant features of operating environment 100 are shown, but those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the example embodiments disclosed herein.

[0074] Figure 2is a block diagram of an example of a controller 110 according to some embodiments. While some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the embodiments disclosed herein. To this end, as a non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., a universal serial bus (USB), IEEE802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), infrared (IR), Bluetooth, ZIGBEE and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, a memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0075] In some embodiments, the one or more communication buses 204 include circuits that interconnect and control communications between system components. In some embodiments, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a touch pad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, and the like.

[0076] The memory 220 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some embodiments, the memory 220 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 220 optionally includes one or more storage devices located away from the one or more processing units 202. The memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 220 or the non-transitory computer-readable storage medium of the memory 220 stores the following programs, modules, and data structures or subsets thereof, including an optional operating system 230 and an XR experience module 240.

[0077] The operating system 230 includes instructions for handling various basic system services and for performing hardware-related tasks. In some embodiments, the XR experience module 240 is configured to manage and coordinate single or multiple XR experiences of one or more users (e.g., a single XR experience of one or more users, or multiple XR experiences of corresponding groups of one or more users). To this end, in various embodiments, the XR experience module 240 includes a data acquisition unit 241, a tracking unit 242, a coordination unit 246, and a data transmission unit 248.

[0078] In some embodiments, the data acquisition unit 241 is configured to obtain Figure 1 120, and optionally acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.) from one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, data acquisition unit 241 includes instructions and / or logic for instructions as well as heuristics and metadata for the heuristics.

[0079] In some embodiments, tracking unit 242 is configured to map scene 105 and track at least display generation component 120 relative to Figure 1 105, and optionally tracks the position of one or more of the input device 125, the output device 155, the sensor 190, and / or the peripheral device 195. To this end, in various embodiments, the tracking unit 242 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics. In some embodiments, the tracking unit 242 includes a hand tracking unit 244 and / or an eye tracking unit 243. In some embodiments, the hand tracking unit 244 is configured to track the position / location of one or more parts of the user's hand and / or the position of one or more parts of the user's hand relative to the user's hand. Figure 1 The movement of the scene 105 relative to the display generation component 120 and / or relative to a coordinate system (the coordinate system is defined relative to the user's hand). Figure 4 The hand tracking unit 244 is described in more detail. In some embodiments, the eye tracking unit 243 is configured to track the position or movement of the user's gaze (or more generally, the user's eyes, face, or head) relative to the scene 105 (e.g., relative to the physical environment and / or relative to the user (e.g., the user's hands)) or relative to the XR content displayed via the display generation component 120. Figure 5 The eye tracking unit 243 is described in more detail.

[0080] In some embodiments, coordination unit 246 is configured to manage and coordinate the XR experience presented to the user by display generation component 120, and optionally by one or more of output device 155 and / or peripherals 195. To this end, in various embodiments, coordination unit 246 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0081] In some embodiments, data transfer unit 248 is configured to transfer data (e.g., presentation data, position data, etc.) to at least display generation component 120, and optionally to one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To this end, in various embodiments, data transfer unit 248 includes instructions and / or logic for instructions as well as heuristics and metadata for the heuristics.

[0082] Although the data acquisition unit 241, the tracking unit 242 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the coordination unit 246, and the data transmission unit 248 are illustrated as residing on a single device (e.g., the controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 241, the tracking unit 242 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the coordination unit 246, and the data transmission unit 248 may be located in separate computing devices.

[0083] also, Figure 2 It serves more as a functional description of various features that may be present in a particular implementation, as opposed to a block diagram of the embodiments described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 2 Some functional modules shown separately in the figure can be implemented in a single module, and the various functions of a single functional block can be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some embodiments, depends in part on the specific combination of hardware, software and / or firmware selected for a specific implementation.

[0084] Figure 31 is a block diagram of an example of a display generation component 120 according to some embodiments. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the embodiments disclosed herein. For this purpose, as a non-limiting example, in some embodiments, the display generation component 120 (e.g., an HMD) includes one or more processing units 302 (e.g., a microprocessor, an ASIC, an FPGA, a GPU, a CPU, a processing core, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZIGBEE and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more XR displays 312, one or more optional internal-facing and / or external-facing image sensors 314, a memory 320, and one or more communication buses 304 for interconnecting these components and various other components.

[0085] In some embodiments, one or more communication buses 304 include circuits for interconnecting and controlling communications between various system components. In some embodiments, one or more I / O devices and sensors 306 include an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, and / or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0086] In some embodiments, one or more XR displays 312 are configured to provide an XR experience to the user. In some embodiments, one or more XR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emission display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS) and / or similar display types. In some embodiments, one or more XR displays 312 correspond to diffraction, reflection, polarization, holographic and other waveguide displays. For example, the display generation component 120 (e.g., HMD) includes a single XR display. In another example, the display generation component 120 includes an XR display for each eye of the user. In some embodiments, one or more XR displays 312 can present MR and VR content. In some embodiments, one or more XR displays 312 can present MR or VR content.

[0087] In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as an eye tracking camera). In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to the user's hands and, optionally, at least a portion of the user's arms (and may be referred to as a hand tracking camera). In some embodiments, the one or more image sensors 314 are configured to face forward so as to acquire image data corresponding to a scene that the user would see in the absence of the display generating component 120 (e.g., an HMD) (and may be referred to as a scene camera). The one or more optional image sensors 314 may include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor), one or more infrared (IR) cameras, and / or one or more event-based cameras, etc.

[0088] The memory 320 includes a high-speed random access memory, such as a DRAM, SRAM, DDR RAM, or other random access solid-state memory device. In some embodiments, the memory 320 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 320 optionally includes one or more storage devices located away from the one or more processing units 302. The memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, the memory 320 or the non-transitory computer-readable storage medium of the memory 320 stores the following programs, modules, and data structures or a subset thereof, including an optional operating system 330 and an XR rendering module 340.

[0089] The operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks. In some embodiments, the XR rendering module 340 is configured to present XR content to a user via one or more XR displays 312. To this end, in various embodiments, the XR rendering module 340 includes a data acquisition unit 342, an XR rendering unit 344, an XR map generation unit 346, and a data transmission unit 348.

[0090] In some embodiments, the data acquisition unit 342 is configured to at least Figure 1 The controller 110 acquires data (e.g., presentation data, interaction data, sensor data, location data, etc.). For the purposes described, in various embodiments, the data acquisition unit 342 includes instructions and / or logic for instructions and heuristics and metadata for the heuristics.

[0091] In some embodiments, the XR rendering unit 344 is configured to render XR content via one or more XR displays 312. For such purposes, in various embodiments, the XR rendering unit 344 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0092] In some embodiments, the XR map generation unit 346 is configured to generate an XR map (e.g., a 3D map of a mixed reality scene or a map of a physical environment in which a computer-generated object can be placed to generate an extended reality) based on the media content data. For such purposes, in various embodiments, the XR map generation unit 346 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0093] In some embodiments, the data transmission unit 348 is configured to transmit data (e.g., presentation data, position data, etc.) to at least the controller 110, and optionally one or more of the input device 125, the output device 155, the sensor 190, and / or the peripheral device 195. For such purposes, in various embodiments, the data transmission unit 348 includes instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0094] Although the data acquisition unit 342, the XR rendering unit 344, the XR map generation unit 346, and the data transfer unit 348 are shown as residing on a single device (e.g., Figure 1 , but it should be understood that in other embodiments, any combination of the data acquisition unit 342, the XR rendering unit 344, the XR mapping generation unit 346, and the data transmission unit 348 may be located in a separate computing device.

[0095] also, Figure 3 More as a functional description of various features that may be present in a particular embodiment, rather than a schematic diagram of the structures of the embodiments described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 3 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some embodiments, depends in part on the specific combination of hardware, software and / or firmware selected for a specific implementation.

[0096] Figure 4 is a schematic illustration of an example implementation of the hand tracking device 140. In some implementations, the hand tracking device 140 ( Figure 1 ) is controlled by the hand tracking unit 244 ( Figure 2 ) to track the position / location of one or more parts of a user's hand and / or one or more parts of a user's hand relative to Figure 1The hand tracking device 140 can be used to track movement of the scene 105 (e.g., relative to a portion of the physical environment surrounding the user, relative to the display generation component 120, or relative to a portion of the user (e.g., the user's face, eyes, or head), and / or relative to a coordinate system (which is defined relative to the user's hands)). In some embodiments, the hand tracking device 140 is part of the display generation component 120 (e.g., embedded in or attached to the head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0097] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras, etc.) that captures three-dimensional scene information including at least a human user's hand 406. The image sensor 404 captures hand images with sufficient resolution so that fingers and their corresponding positions can be distinguished. The image sensor 404 typically captures images of other parts of the user's body, and also or possibly all parts of the body, and may have zoom capabilities or a dedicated sensor with increased magnification to capture images of the hand with a desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with other image sensors to capture the physical environment of the scene 105, or is used as an image sensor to capture the physical environment of the scene 105. In some embodiments, the image sensor is positioned relative to the user or the user's environment in a manner that the field of view of the image sensor 404 or a portion thereof is used to define an interaction space, in which the hand movements captured by the image sensor are considered as input to the controller 110.

[0098] In some embodiments, the image sensor 404 outputs a sequence of frames containing 3D mapping data (and in addition, possible color image data) to the controller 110, which extracts high-level information from the mapping data. This high-level information is typically provided to an application running on the controller via an application program interface (API), which drives the display generation component 120 accordingly. For example, a user can interact with the software running on the controller 110 by moving his hand 406 and changing his hand posture.

[0099] In some embodiments, the image sensor 404 projects a speckled pattern onto a scene containing the hand 406 and captures an image of the projected pattern. In some embodiments, the controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on the lateral offset of the spots in the pattern. This method is advantageous because it does not require the user to hold or wear any kind of beacon, sensor, or other marker. The method gives the depth coordinates of points in the scene at a specific distance from the image sensor 404 relative to a predetermined reference plane. In the present disclosure, it is assumed that the image sensor 404 defines an orthogonal set of x-axis, y-axis, and z-axis, so that the depth coordinates of points in the scene correspond to the z component measured by the image sensor. Alternatively, the image sensor 404 (e.g., a hand tracking device) may use other 3D mapping methods based on a single or multiple cameras or other types of sensors, such as stereo imaging or time-of-flight measurement.

[0100] In some embodiments, the hand tracking device 140 captures and processes a time series of depth maps containing the user's hand as the user moves his hand (e.g., the entire hand or one or more fingers). Software running on the image sensor 404 and / or the processor in the controller 110 processes the 3D mapping data to extract image patch descriptors of the hand in these depth maps. The software can match these descriptors with image patch descriptors stored in the database 408 based on a previous learning process to estimate the pose of the hand in each frame. The pose typically includes the 3D position of the user's hand joints and finger tips.

[0101] The software can also analyze the trajectory of the hand and / or finger over multiple frames in the sequence to recognize gestures. The pose estimation function described herein can be alternated with the motion tracking function so that the image block-based pose estimation is performed only once every two (or more) frames, and the tracking is used to find the changes in pose that occur on the remaining frames. The pose, motion, and gesture information is provided to the application running on the controller 110 via the above-mentioned API. The program can, for example, move and modify the image presented on the display generation component 120 in response to the pose and / or gesture information, or perform other functions.

[0102] In some embodiments, gestures include air gestures. An air gesture is a gesture detected without the user touching an input element that is part of a device (e.g., computer system 101, one or more input devices 125, and / or hand tracking device 140) (or independent of an input element that is part of a device) and based on detected movement of a part of the user's body (e.g., head, one or more arms, one or more hands, one or more fingers, and / or one or more legs) through the air (including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movement relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of a user's finger relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tap gesture in which the hand moves a predetermined amount and / or speed in a predetermined posture, or a shake gesture including a predetermined speed or amount of rotation of a part of the user's body)).

[0103] In some embodiments, according to some embodiments, the input gestures used in the various examples and embodiments described herein include air gestures for interacting with an XR environment (e.g., a virtual or mixed reality environment) performed by movement of a user's fingers relative to other fingers or parts of the user's hand. In some embodiments, an air gesture is a gesture detected without the user touching an input element that is part of the device (or independent of an input element that is part of the device) and based on detected movement of a part of the user's body through the air (including movement of the user's body relative to an absolute reference (e.g., the angle of the user's arm relative to the ground or the distance of the user's hand relative to the ground), movement relative to another part of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one of the user's hands relative to the user's other hand, and / or movement of the user's fingers relative to another finger or part of the user's hand), and / or absolute movement of a part of the user's body (e.g., a tap gesture in which the hand moves a predetermined amount and / or speed in a predetermined posture, or a shake gesture including a predetermined speed or rotation amount of a part of the user's body)).

[0104] In some embodiments where the input gesture is an in-air gesture (e.g., in the absence of physical contact with an input device that provides information to the computer system about which user interface element is the target of the user input, such as contact with a user interface element displayed on a touch screen, or contact with a mouse or trackpad to move a cursor to a user interface element), the gesture takes into account the user's attention (e.g., gaze) to determine the target of the user input (e.g., for direct input, as described below). Thus, in embodiments involving in-air gestures, for example, the input gesture is combined (e.g., simultaneously) with movement of the user's fingers and / or hand to detect attention (e.g., gaze) toward a user interface element to perform a pinch and / or tap input, as described below.

[0105] In some embodiments, an input gesture pointing to a user interface object is performed directly or indirectly with reference to the user interface object. For example, the user input is performed directly on the user interface object according to performing input with the user's hand at a location corresponding to the location of the user interface object in the three-dimensional environment (e.g., as determined based on the user's current viewpoint). In some embodiments, when the user's attention (e.g., gaze) to the user interface object is detected, the input gesture is performed indirectly on the user interface object according to the location of the user's hand not being at the location corresponding to the location of the user interface object in the three-dimensional environment while the user performs the input gesture. For example, for a direct input gesture, the user can guide the user's input to the user interface object by initiating a gesture at or near a location corresponding to the display location of the user interface object (e.g., within 0.5 cm, 1 cm, 5 cm, or a distance between 0 and 5 cm measured from the outer edge of the option or the center portion of the option). For an indirect input gesture, the user can guide the user's input to the user interface object by focusing on the user interface object (e.g., by gazing at the user interface object), and while focusing on the option, the user initiates an input gesture (e.g., at any location detectable by the computer system) (e.g., at a location that does not correspond to the display location of the user interface object).

[0106] In some embodiments, according to some embodiments, the input gestures (e.g., air gestures) used in various examples and embodiments described herein include pinch input and tap input for interacting with a virtual or mixed reality environment. For example, the pinch input and tap input described below are performed as air gestures.

[0107] In some embodiments, the pinch input is part of an air gesture that includes one or more of the following: a pinch gesture, a long pinch gesture, a pinch and drag gesture, or a double pinch gesture. For example, a pinch gesture as an air gesture includes movement of two or more fingers of a hand to contact each other, that is, optionally followed by an immediate (e.g., within 0 to 1 second) interruption of contact with each other. A long pinch gesture as an air gesture includes movement of two or more fingers of a hand in contact with each other for at least a threshold amount of time (e.g., at least 1 second) before an interruption of contact with each other is detected. For example, a long pinch gesture includes the user maintaining a pinch gesture (e.g., in which two or more fingers are in contact), and the long pinch gesture continues until an interruption of contact between the two or more fingers is detected. In some embodiments, a double pinch gesture as an air gesture includes two (e.g., or more) pinch inputs (e.g., performed by the same hand) that are detected consecutively immediately (e.g., within a predefined time period) with each other. For example, the user performs a first pinch input (e.g., a pinch input or a long pinch input), releases the first pinch input (e.g., interrupts contact between two or more fingers), and performs a second pinch input within a predefined time period after releasing the first pinch input (e.g., within 1 second or within 2 seconds).

[0108] In some embodiments, the pinch and drag gesture as an air gesture includes a pinch gesture (e.g., a pinch gesture or a long pinch gesture) performed in conjunction with (e.g., following) a drag input that changes the position of the user's hand from a first position (e.g., the start position of the drag) to a second position (e.g., the end position of the drag). In some embodiments, the user maintains the pinch gesture while performing the drag input, and releases the pinch gesture (e.g., opens their two or more fingers) to end the drag gesture (e.g., at the second position). In some embodiments, the pinch input and the drag input are performed by the same hand (e.g., the user pinches two or more fingers to contact each other and moves the same hand to a second position in the air using a drag gesture). In some embodiments, the pinch input is performed by the user's first hand, and the drag input is performed by the user's second hand (e.g., while the user continues the pinch input with the user's first hand, the user's second hand moves from the first position to the second position in the air). In some embodiments, the input gesture as an air gesture includes an input performed using both of the user's hands (e.g., a pinch and / or tap input). For example, the input gesture includes two (e.g., or more) pinch inputs that are performed in conjunction with each other (e.g., simultaneously or within a predefined time period). For example, a first pinch gesture (e.g., a pinch input, a long pinch input, or a pinch and drag input) is performed using a first hand of the user, and a second pinch input is performed using another hand (e.g., a second hand of the user's two hands) in conjunction with the pinch input performed using the first hand. In some embodiments, movement between the user's two hands (e.g., increasing and / or decreasing the distance or relative orientation between the user's two hands).

[0109] In some embodiments, a tap input performed as an air gesture (e.g., pointing to a user interface element) includes movement of a user's finger toward the user interface element, movement of a user's hand toward the user interface element (optionally, extension of the user's finger toward the user interface element), downward movement of a user's finger (e.g., mimicking a mouse click motion or a tap on a touch screen), or other predefined movement of a user's hand. In some embodiments, a tap input performed as an air gesture is detected based on movement characteristics of the finger or hand performing the tap gesture movement of the finger or hand, which is a movement of the finger or hand away from the user's viewpoint and / or toward an object that is the target of the tap input, followed by an end of the movement. In some embodiments, the end of the movement is detected based on a change in the movement characteristics of the finger or hand performing the tap gesture (e.g., the end of the movement away from the user's viewpoint and / or toward an object that is the target of the tap input, a reversal of the direction of movement of the finger or hand, and / or a reversal of the acceleration direction of the movement of the finger or hand).

[0110] In some embodiments, it is determined that the user's attention is directed to a portion of the three-dimensional environment based on detection of a gaze directed to that portion of the three-dimensional environment (optionally, no other conditions are required). In some embodiments, it is determined that the user's attention is directed to a portion of the three-dimensional environment based on detection of a gaze directed to that portion of the three-dimensional environment using one or more additional conditions, such as requiring the gaze to be directed to the portion of the three-dimensional environment for at least a threshold duration (e.g., a dwell duration) and / or requiring the gaze to be directed to the portion of the three-dimensional environment when the user's viewpoint is within a distance threshold from the portion of the three-dimensional environment, so that the device determines that the user's attention is directed to the portion of the three-dimensional environment, wherein if one of these additional conditions is not met, the device determines that the attention is not directed to the portion of the three-dimensional environment to which the gaze is directed (e.g., until the one or more additional conditions are met).

[0111] In some embodiments, detection of a ready state configuration of a user or a portion of a user is detected by a computer system. Detection of a ready state configuration of a hand is used by the computer system as an indication that the user may be preparing to interact with the computer system using one or more air gesture inputs (e.g., pinch, tap, pinch and drag, double pinch, long pinch, or other air gestures described herein) performed by the hand. For example, the ready state of the hand is determined based on whether the hand has a predetermined hand shape (e.g., a pre-pinch shape with the thumb and one or more fingers extended and spaced apart to prepare for a pinch or grab gesture, or a pre-tap with one or more fingers extended and the palm facing away from the user), based on whether the hand is in a predetermined position relative to the user's viewpoint (e.g., below the user's head and above the user's waist and extending at least 15 cm, 20 cm, 25 cm, 30 cm, or 50 cm from the body), and / or based on whether the hand has moved in a particular manner (e.g., toward an area in front of the user above the user's waist and below the user's head, or away from the user's body or legs). In some embodiments, the ready state is used to determine whether an interactive element of a user interface responds to attention (e.g., gaze) input.

[0112] In some embodiments, the software may be downloaded to the controller 110 in electronic form, for example, over a network, or may alternatively be provided on a tangible, non-transitory medium such as an optical, magnetic, or electronic memory medium. In some embodiments, the database 408 is also stored in a memory associated with the controller 110. Alternatively or in addition, some or all of the described functions of the computer may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although in Figure 4Controller 110 is shown in FIG. 1 , but some or all of the processing functions of the controller may be performed by a suitable microprocessor and software or by dedicated circuitry within the housing of the image sensor 404 (e.g., a hand tracking device) or other device associated with the image sensor 404, for example, as a separate unit from the image sensor 404. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television receiver, handheld device, or head-mounted device) or with any other suitable computerized device (such as a game console or media player). The sensing functions of the image sensor 404 may also be integrated into a computer or other computerized device to be controlled by the sensor output.

[0113] Figure 4 Also included is a schematic diagram of a depth map 410 captured by the image sensor 404 according to some embodiments. As described above, the depth map includes a matrix of pixels with corresponding depth values. Pixels 412 corresponding to the hand 406 have been segmented from the background and wrist in the map. The brightness of each pixel within the depth map 410 is inversely proportional to its depth value (i.e., the measured z distance from the image sensor 404), where gray shades become darker as depth increases. The controller 110 processes these depth values ​​in order to identify and segment components of the image that have human hand features (i.e., a group of adjacent pixels). These features may include, for example, overall size, shape, and motion from frame to frame in the depth map sequence.

[0114] Figure 4 Also schematically shown is a hand skeleton 414 that the controller 110 ultimately extracts from the depth map 410 of the hand 406 according to some embodiments. Figure 4 In the hand skeleton 414, a hand background 416 that has been segmented from the original depth map is superimposed. In some embodiments, key feature points of the hand and optionally on the wrist or arm connected to the hand (e.g., points corresponding to knuckles, finger tips, palm center, end of the hand connected to the wrist, etc.) are identified and located on the hand skeleton 414. In some embodiments, the controller 110 uses the position and movement of these key feature points over multiple image frames to determine the gesture performed by the hand or the current state of the hand according to some embodiments.

[0115] Figure 5 The eye tracking device 130 ( Figure 1 ). In some embodiments, the eye tracking device 130 is composed of an eye tracking unit 243 ( Figure 2) controls to track the position and movement of the user's gaze relative to the scene 105 or relative to the XR content displayed via the display generation component 120. In some embodiments, the eye tracking device 130 is integrated with the display generation component 120. For example, in some embodiments, when the display generation component 120 is a head-mounted device (such as headphones, helmets, goggles, or glasses) or a handheld device placed in a wearable frame, the head-mounted device includes both components for generating XR content for viewing by the user and components for tracking the user's gaze relative to the XR content. In some embodiments, the eye tracking device 130 is separate from the display generation component 120. For example, when the display generation component is a handheld device or an XR room, the eye tracking device 130 is optionally a device separate from the handheld device or the XR room. In some embodiments, the eye tracking device 130 is a head-mounted device or a part of a head-mounted device. In some embodiments, the head-mounted eye tracking device 130 is optionally used in conjunction with a display generation component that is also head-mounted or a display generation component that is not head-mounted. In some embodiments, the eye tracking device 130 is not a head mounted device, and is optionally used in conjunction with a head mounted display generation component. In some embodiments, the eye tracking device 130 is not a head mounted device, and is optionally part of a non-head mounted display generation component.

[0116] In some embodiments, the display generation component 120 uses a display mechanism (e.g., a left near-eye display panel and a right near-eye display panel) to display a frame including a left image and a right image in front of the user's eyes, thereby providing a 3D virtual view to the user. For example, the head-mounted display generation component may include a left optical lens and a right optical lens (referred to herein as eye lenses) located between the display and the user's eyes. In some embodiments, the display generation component may include or be coupled to one or more external cameras that capture a video of the user's environment for display. In some embodiments, the head-mounted display generation component may have a transparent or translucent display, and display a virtual object on the transparent or translucent display, and the user can directly view the physical environment through the transparent or translucent display. In some embodiments, the display generation component projects the virtual object into the physical environment. The virtual object may, for example, be projected on a physical surface or projected as a hologram so that an individual using the system observes the virtual object superimposed on the physical environment. In this case, separate display panels and image frames for the left and right eyes may not be required.

[0117] like Figure 5As shown in , in some embodiments, the eye tracking device 130 (e.g., gaze tracking device) includes at least one eye tracking camera (e.g., an infrared (IR) or near infrared (NIR) camera), and an illumination source (e.g., an IR or NIR light source, such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eyes. The eye tracking camera can be pointed at the user's eyes to receive the IR or NIR light reflected directly from the eyes by the light source, or alternatively can be pointed at "hot" mirrors located between the user's eyes and the display panel, which reflect the IR or NIR light from the eyes to the eye tracking camera while allowing visible light to pass through. The eye tracking device 130 optionally captures images of the user's eyes (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes the images to generate gaze tracking information, and transmits the gaze tracking information to the controller 110. In some embodiments, the user's two eyes are tracked separately by corresponding eye tracking cameras and illumination sources. In some embodiments, only one eye of the user is tracked by the corresponding eye tracking camera and illumination source.

[0118] In some embodiments, the eye tracking device 130 is calibrated using a device-specific calibration process to determine the parameters of the eye tracking device for a specific operating environment 100, such as the 3D geometric relationships and parameters of the LED, camera, thermal mirror (if present), eye lens, and display screen. The device-specific calibration process can be performed at a factory or another facility before the AR / VR equipment is delivered to the end user. The device-specific calibration process can be an automatic calibration process or a manual calibration process. According to some embodiments, the user-specific calibration process may include an estimate of the eye parameters of a specific user, such as pupil position, foveal position, optical axis, visual axis, eye spacing, etc. According to some embodiments, once the device-specific parameters and user-specific parameters are determined for the eye tracking device 130, a flash-assisted method can be used to process the images captured by the eye tracking camera to determine the current visual axis and the user's gaze point relative to the display.

[0119] like Figure 5As shown in , the eye tracking device 130 (e.g., 130A or 130B) includes an eye lens 520 and a gaze tracking system, which includes at least one eye tracking camera 540 (e.g., an infrared (IR) or near infrared (NIR) camera) positioned on the side of the user's face on which eye tracking is performed, and an illumination source 530 (e.g., an IR or NIR light source, such as an array or ring of NIR light emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye 592. The eye tracking camera 540 can be directed toward a mirror 550 (which reflects IR or NIR light from the eye 592 while allowing visible light to pass) located between the user's eye 592 and a display 510 (e.g., a left display panel or a right display panel of a head-mounted display, or a display of a handheld device, a projector, etc.) Figure 5 ), or alternatively may be directed toward the user's eye 592 to receive reflected IR or NIR light from the eye 592 (e.g., as shown in the top portion of Figure 5 ).

[0120] In some embodiments, the controller 110 renders AR or VR frames 562 (e.g., left and right frames for left and right display panels) and provides the frames 562 to the display 510. The controller 110 uses the gaze tracking input 542 from the eye tracking camera 540 for various purposes, such as for processing the frames 562 for display. The controller 110 optionally estimates the user's gaze point on the display 510 based on the gaze tracking input 542 obtained from the eye tracking camera 540 using a flash-assisted method or other suitable method. The gaze point estimated from the gaze tracking input 542 is optionally used to determine the direction in which the user is currently looking.

[0121] Several possible use cases for the user's current gaze direction are described below and are not intended to be limiting. As an example use case, the controller 110 may render virtual content differently based on the determined direction of the user's gaze. For example, the controller 110 may generate virtual content at a higher resolution in the foveal area determined according to the user's current gaze direction than in the peripheral area. As another example, the controller may position or move virtual content in a view based at least in part on the user's current gaze direction. As another example, the controller may display specific virtual content in a view based at least in part on the user's current gaze direction. As another example use case in an AR application, the controller 110 may guide an external camera for capturing the physical environment of the XR experience to focus in the determined direction. The autofocus mechanism of the external camera may then focus on an object or surface in the environment that the user is currently looking at on the display 510. As another example use case, the eye lens 520 may be a focusable lens, and the controller uses gaze tracking information to adjust the focus of the eye lens 520 so that the virtual object that the user is currently looking at has an appropriate degree of convergence to match the convergence of the user's eye 592. The controller 110 can use the gaze tracking information to guide the eye lens 520 to adjust the focus so that nearby objects that the user is looking at appear at the correct distance.

[0122] In some embodiments, the eye tracking device is part of a head mounted device that includes a display (e.g., display 510), two eye lenses (e.g., eye lenses 520), an eye tracking camera (e.g., eye tracking camera 540), and a light source (e.g., light source 530 (e.g., IR or NIR LED)) mounted in a wearable housing. The light source emits light (e.g., IR or NIR light) toward the user's eye 592. In some embodiments, the light sources may be arranged in a ring or circle around each of the lenses, such as Figure 5 In some embodiments, for example, eight light sources 530 (eg, LEDs) are arranged around each lens 520. However, more or fewer light sources 530 may be used, and other arrangements and locations of the light sources 530 may be used.

[0123] In some embodiments, the display 510 emits light in the visible range and does not emit light in the IR or NIR range, and therefore does not introduce noise into the gaze tracking system. Note that the positions and angles of the eye tracking cameras 540 are given by way of example and are not intended to be limiting. In some embodiments, a single eye tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user's face. In some embodiments, a camera 540 with a wider field of view (FOV) and a camera 540 with a narrower FOV may be used on each side of the user's face. In some embodiments, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

[0124] like Figure 5 Embodiments of the gaze tracking system shown may be used, for example, in computer-generated reality, virtual reality, and / or mixed reality applications to provide a user with a computer-generated reality, virtual reality, augmented reality, and / or augmented virtual experience.

[0125] Figure 6 FIG. 1 shows a flash-assisted gaze tracking pipeline according to some embodiments. In some embodiments, the gaze tracking pipeline is implemented by a flash-assisted gaze tracking system (e.g., Figure 1 and Figure 5 The flash-assisted gaze tracking system can maintain a tracking state. Initially, the tracking state is off or "no". When in the tracking state, the flash-assisted gaze tracking system uses previous information from previous frames when analyzing the current frame to track the pupil outline and the flash in the current frame. When not in the tracking state, the flash-assisted gaze tracking system attempts to detect the pupil and the flash in the current frame, and if successful, initializes the tracking state to "yes" and continues with the next frame in the tracking state.

[0126] like Figure 6 As shown in , the gaze tracking camera can capture left and right images of the user's left and right eyes. The captured images are then input to the gaze tracking pipeline for processing starting at 610. As indicated by the arrow returning to element 600, the gaze tracking system can continue to capture images of the user's eyes at a rate of, for example, 60 to 120 frames per second. In some embodiments, each group of captured images can be input to the pipeline for processing. However, in some embodiments or under some conditions, not all captured frames are processed by the pipeline.

[0127] At 610, for the currently captured image, if the tracking status is yes, the method proceeds to element 640. At 610, if the tracking status is no, the image is analyzed to detect the user's pupil and glint in the image, as indicated at 620. At 630, if the pupil and glint are successfully detected, the method proceeds to element 640. Otherwise, the method returns to element 610 to process the next image of the user's eye.

[0128] At 640, if advancing from element 610, the current frame is analyzed to track pupils and glints based in part on previous information from previous frames. At 640, if advancing from element 630, the tracking state is initialized based on the pupils and glints detected in the current frame. The processing results at element 640 are checked to verify that the results of the tracking or detection can be credible. For example, the results can be checked to determine whether the pupil and a sufficient number of glints for performing gaze estimation are successfully tracked or detected in the current frame. At 650, if the results are not likely to be credible, at element 660, the tracking state is set to no, and the method returns to element 610 to process the next image of the user's eye. At 650, if the results are credible, the method advances to element 670. At 670, the tracking state is set to yes (if not already yes), and the pupil and glint information is passed to element 680 to estimate the user's gaze point.

[0129] Figure 6 It is intended to be used as an example of an eye tracking technology that can be used for a particular implementation. As recognized by one of ordinary skill in the art, according to various embodiments, other eye tracking technologies currently existing or developed in the future can be used in place of or in combination with the flash-assisted eye tracking technology described herein in a computer system 101 for providing an XR experience to a user.

[0130] In the present disclosure, various input methods are described with respect to interaction with a computer system. When an input device or input method is used to provide an example, and another input device or input method is used to provide another example, it should be understood that each example is compatible with the input device or input method described with respect to another example and optionally utilizes the input device or input method. Similarly, various output methods are described with respect to interaction with a computer system. When an output device or output method is used to provide an example, and another output device or output method is used to provide another example, it should be understood that each example is compatible with the output device or output method described with respect to another example and optionally utilizes the output device or output method. Similarly, various methods are described with respect to interaction with a virtual environment or a mixed reality environment by a computer system. When an example is provided using interaction with a virtual environment, and another example is provided using a mixed reality environment, it should be understood that each example is compatible with the method described with respect to another example and optionally utilizes these methods. Therefore, the present disclosure discloses an embodiment as a combination of features of multiple examples, without exhaustively listing all features of the embodiment in the description of each example embodiment.

[0131] User interface and associated processes

[0132] Attention is now directed to embodiments of a user interface ("UI") and associated processes that may be implemented on a computer system, such as a portable multifunction device or a head mounted device, in communication with display generating components and (optionally) a wearable device.

[0133] 7A to 7F An example of using hand gestures to navigate content is shown. Fig. 9 is a flow chart of an exemplary method 900 for navigating content using hand gestures. 7A to 7F The user interface in the following sections is used to illustrate the process described below, including Fig. 9 process.

[0134] Fig. 7A A user 702 is shown wearing a wearable device 706. The wearable device 706 is on the right hand of the user 702. The wearable device 706 includes one or more sensors (e.g., one or more heart rate sensors, accelerometers, and / or gyroscopes) that detect movement (e.g., rotation and / or lateral movement), orientation, gestures, and positioning of the right hand of the user 702. Fig. 7A As shown in FIG. 7 , the right hand of the user 702 is in a neutral position (eg, the right hand of the user 702 is not rotated) and is released. The wearable device 706 is Fig. 7AHowever, in some embodiments, wearable device 706 is another device that can be worn and track hand movements, such as a camera. In some embodiments, computer system 704 includes one or more components of computer system 101.

[0135] like Fig. 7A , user 702 is standing in front of computer system 700. Computer system 700 includes display 718 and communicates with computer system 704 (e.g., wired communication and / or wireless communication). Computer system 704 transmits instructions to computer system 700 to display user interface 710. In response to computer system 700 receiving instructions from computer system 704, computer system 700 displays user interface 710. In some embodiments, computer system 700 does not communicate with computer system 704. In some embodiments, computer system 700 is an independent system and displays user interface 710 via a processor integrated into computer system 700. In some embodiments, user interface 710 is a media user interface (e.g., a user interface for displaying video data, audio data (e.g., music) and / or photo data) (e.g., a user interface for controlling the playback of video data and / or audio data). In some embodiments, user interface 710 is an augmented reality user interface displayed by computer system 700 as part of an XR environment. In some embodiments, computer system 700 displays user interface 710 from an immersive perspective. In some embodiments, the computer system 700 presents the content included in the user interface 710 from multiple perspectives in response to the detected change in the orientation / position of the computer system 700. In some embodiments, the computer system 700 updates the content of the user interface 710 based on the positioning and / or orientation of the computer system 700 (e.g., the computer system 700 displays additional content within the user interface 710 and / or removes content from the user interface in response to the computer system 700 moving in a particular direction). In some embodiments, the computer system 700 displays the user interface 710 in response to receiving (e.g., via the computer system 704) an indication that the wearable device 706 has detected that the right hand of the user 702 is clenched. In some embodiments, the computer system 706 includes one or more components of the computer system 101. In some embodiments, the computer system 700 includes one or more components of the computer system 101. In some embodiments, the hand gesture is an air gesture (e.g., as discussed above with respect to the description of the input gesture). In some embodiments, the hand gesture is a hand input (e.g., an air tap, an air swipe, an air pinch, and / or an air open).

[0136] The wearable device 706 communicates with the computer system 704 (e.g., wired communication and / or wireless communication). The wearable device 706 transmits positioning data and / or movement data about the right hand of the user 702 to the computer system 704. The computer system 704 transmits display instructions to the computer system 700 based on the received positioning data and / or movement data of the right hand of the user 702. Therefore, the computer system 700 displays changes when the wearable device 706 detects changes in the positioning and / or movement of the right hand of the user 702. The above description of how the computer system 702, the computer system 704, and the wearable device 706 communicate is optional, and other communication methods between the computer system 702, the computer system 704, and the wearable device 706 can be envisioned. In some embodiments, the wearable device 706 communicates with the computer system 700 (e.g., direct communication and / or indirect communication) (e.g., wireless communication and / or wired communication) (e.g., Bluetooth communication, ultra-wideband communication and / or near field communication) (e.g., the computer system 704 does not act as an intermediary between the wearable device 706 and the computer system 700). In some embodiments, the wearable device 706 transmits display instructions to the computer system 700 based on the position and / or movement of the right hand of the user 702. In some embodiments, the wearable device 706 transmits the position data and / or movement data about the right hand of the user 702 to the computer system 700. Fig. 7A As shown in , the computer system 700 is a television or large display device. In some embodiments, the computer system 700 is one or more other devices, such as a handheld device (smart phone) and / or an HMD. In the embodiment where the computer system 700 is an HMD device, the user interface 710 is displayed in the XR environment.

[0137] like Fig. 7A As shown in FIG. 7 , the user interface 710 includes virtual objects 714a to 714g, a selection indicator virtual object 720, and a hand indicator virtual object 736. Fig. 7A As shown in FIG. 7 , virtual objects 714a to 714g are avatars, where each avatar represents a different individual. Virtual objects 714a to 714g are selectable (e.g., selectable via input performed by user 702). Fig. 7AAs shown in , the computer system 700 displays a selection indicator virtual object 720 around the virtual object 714d, which indicates that the virtual object 714d is in focus. It is worth noting that the virtual objects 714a to 714c and 714e to 714g are not in focus (for example, no selection indicator virtual object is displayed around the virtual objects 714a to 714c and 714e to 714g). Because the virtual object 714d is in focus, the computer system 700 is configured to perform operations with respect to the virtual object 714d. However, the computer system 700 is not configured to perform operations with respect to the virtual objects 714a to 714c and 714e to 714g because the virtual objects 714a to 714c and 714e to 714g are not in focus. As shown in FIG. Fig. 7A As shown in , the computer system 700 displays a representation of an individual represented by a virtual object 714d within the central display area 722. The hand indicator virtual object 736 indicates the state of the hand of the user 702 detected by the wearable device 706. The state of the hand of the user 702 includes one or more of the position (e.g., whether the user's hand is clenched and / or loosened) and / or displacement (e.g., rotation and / or lateral displacement) of the hand of the user 702. Fig. 7A In FIG. 7 , the wearable device 706 detects that the right hand of the user 702 is released. In response to receiving the indication that the hand of the user 702 is released, the computer system 700 displays the hand indication virtual object 736 as representing the released hand. Fig. 7A , the wearable device 706 detects that the right hand of the user 702 is rotating in a clockwise direction when in the loose position.

[0138] exist Figure 7B In response to receiving (e.g., via computer system 704) an indication that the hand of user 702 is rotating (and / or has rotated) in a clockwise direction when the right hand of user 702 is released, computer system 700 maintains the display of selection indicating virtual object 720 around virtual object 714d (e.g., virtual object 714d remains in focus). Because wearable device 706 did not detect that the right hand of user 702 was clenched when wearable device 706 detected the rotation of the right hand of user 702, computer system 700 does not navigate the display of virtual objects 714a through 714g. Figure 7C, the right hand of user 702 is in a neutral position (e.g., the right hand of user 702 does not rotate) and is in a clenched position. In response to receiving an indication that the right hand of user 702 is clenched (e.g., via computer system 704), the computer system 700 displays the hand indication identification virtual object 736 as representing a clenched hand. In some embodiments, the computer system 700 cyclically displays the hand indication identification virtual object 736 in an animated manner (e.g., such as how to display some graphics interchange formats (e.g., GIF)). In some embodiments, the animation of the hand indication identification virtual object 736 includes an animation that is displayed in a cycle between a clenched hand position and a loosened hand position. In some embodiments, the cyclic animation is displayed in a first visual appearance (e.g., a graphical representation of a clenched hand) within a predetermined amount of time (e.g., 0.1 seconds, 0.3 seconds, 0.5 seconds, 1 second, 3 seconds), and after the predetermined amount of time has passed, the cyclic animation is displayed in a second visual appearance (e.g., a graphical representation of a loosened hand) for a predetermined amount of time, and then displayed again in the first visual appearance. In some embodiments, the hand indicates that the animation of the virtual object 736 includes showing the hand (or another object) rotating (e.g., in a clockwise or counterclockwise direction) and / or moving. In some embodiments, the hand (or other object) rotates and / or moves based on the amount of rotation and / or movement of the wearable device 706. Figure 7C At , the wearable device 706 detects that the right hand of the user 702 is clenched in a neutral position and begins to rotate in a clockwise direction.

[0139] exist Fig.7D In response to receiving (e.g., via computer system 704) an indication that user 702's hands are clenched and rotated in a clockwise direction, computer system 700 moves virtual objects 714a to 714g in a counterclockwise direction and navigates from virtual object 714d to virtual object 714e. By navigating from virtual object 714d to virtual object 714e, computer system 700 displays a selection indicator virtual object 720 around virtual object 714e instead of virtual object 714d to indicate that virtual object 714e is in focus and virtual object 714d is out of focus. Fig.7D As shown in FIG. 1 , when the computer system 700 navigates from the virtual object 714d to the virtual object 714e, the computer system 700 moves the virtual objects 714b to 714g to the virtual objects 714d to 714e. Figure 7C The positions of the virtual objects 714b to 714g are displayed counterclockwise. Fig.7D At this point, the computer system 700 stops displaying Figure 7CIn some embodiments, the computer system 700 moves the virtual object in the first direction as long as the wearable device 706 detects that the user's 702 right hand is clenched and rotated in the second direction (and / or the first direction). In some embodiments, in response to receiving an indication that the wearable device 706 detects that the user's 702 hand has been rotated in a counterclockwise direction, the computer system 700 displays navigation between the displayed virtual objects in a clockwise direction (e.g., the computer system 700 displays navigation from virtual object 714d to virtual object 714c). In some embodiments, in response to receiving (e.g., via computer system 704) an indication that wearable device 706 detected that user 702's right hand was rotating in a clockwise direction when the wearable device detected that user 702's right hand was clenched, computer system 700 moves the display of selection indicator identifying virtual object 720 from a corresponding virtual object to a different corresponding virtual object based on the amount of rotation detected by wearable device 706 (e.g., the greater the amount of rotation detected by wearable device 706, the greater the distance computer system 700 moves the selection indicator identifying virtual object 720) (e.g., and computer system 700 does not move the display of any of virtual object 714a, virtual object 714b, virtual object 714c, virtual object 714d, virtual object 714e, virtual object 714f and / or virtual object 714g). In some embodiments, while the computer system 700 moves the display of one or more of the virtual objects, the computer system 700 displays an indication of the release gesture (e.g., a graphical representation of a hand performing a release gesture) in response to receiving an indication that the wearable device 706 has detected the release gesture to indicate that the computer system 700 is configured to perform a corresponding operation corresponding to the virtual object in focus. In some embodiments, the indication of the release gesture is a graphical representation of a hand that is animated (e.g., cycles) between a clenched hand position and an unclosed hand position. In some embodiments, the computer system 700 scrolls in a first direction between virtual objects in response to receiving (e.g., via the computer system 704) an indication that the wearable device 706 has detected that the hand of the user 702 is clenched and rotated along one or more Cartesian axes in the first direction (e.g., roll, pitch, and / or yaw of the user 702's hand), and the computer system 700 scrolls in a second direction (e.g., opposite to the first direction) in response to receiving an indication that the wearable device 706 has detected that the hand of the user 702 is clenched and rotated along one or more Cartesian axes in a second direction opposite to the first direction.

[0140] like Fig.7DAs shown in , in response to receiving an indication that the wearable device 706 has detected that the hand of the user 702 is clenched and has been rotated in a clockwise direction, the computer system 700 displays a rotation indicator virtual object 724. The rotation indicator virtual object 724 indicates the amount of rotation that the wearable device 706 has detected. The rotation indicator virtual object 724 includes radial lines and a circle. The positioning of the radial lines within the circle indicates the amount of rotation that the wearable device 706 has detected. As the amount of rotation detected by the wearable device 706 increases, the computer system 700 further displays radial lines starting from the twelve o'clock position of the circle (e.g., where the twelve o'clock position of the circle is the top of the circle). In addition, the position of the radial lines included in the rotation indicator virtual object 724 indicates the direction of the rotation detected by the wearable device 706. That is, when the wearable device 706 detects that the right hand of the user 702 rotates in a clockwise direction, the radial line will be to the right of the twelve o'clock position of the circle (e.g., the top of the circle), and when the wearable device 706 detects that the right hand of the user 702 rotates in a counterclockwise direction, the radial line will be to the left of the twelve o'clock position of the circle (e.g., the top of the circle). In some embodiments, the computer system 700 displays the rotation indicator virtual object 724 and the hand indicator virtual object 736 simultaneously.

[0141] like Fig.7D As shown in , the computer system 700 displays a selection indicator identifying a virtual object 720 around the virtual object 714e, the selection indicator identifying the virtual object indicating that the computer system 700 is configured to perform an operation corresponding to the virtual object 714e (e.g., in response to receiving an indication that one or more inputs have been performed). Fig.7D As shown in , the computer system 700 displays a representation of the individual represented by the virtual object 714e within the central display area 722 because the virtual object 714e is in focus. Fig.7D At , the wearable device 706 detects that the right hand of the user 702 is clenched and further rotated in the clockwise direction.

[0142] exist Fig. 7E In response to receiving (via computer system 704) an indication that wearable device 706 has detected that user 702's right hand is clenched and further rotated in a clockwise direction, computer system 700 moves the virtual object in a counterclockwise direction and displays navigation from virtual object 714e to virtual object 714g. Fig. 7E At , the wearable device 706 detects that the right hand of the user 702 is at a position greater than the hand of the user 702. Fig.7D In response to receiving that the hand of user 702 is rotating at a rotation angle greater than the rotation angle of the rotation in the middle. Fig.7DIn response to the large rotation angle of the hand of user 702 at the position indicated by the rotation of the user 702, the computer system 700 navigates between virtual objects at a faster rate (e.g., the computer system 700 navigates to a virtual object (e.g., virtual object 714g) that is one virtual object removed from virtual object 714e, rather than navigating to a virtual object (e.g., virtual object 714f) that is directly adjacent to virtual object 714e). Because the computer system 700 navigates from virtual object 714e to virtual object 714g, the computer system 700 displays virtual objects 714d to 714g to the computer system 700 at the position indicated by the rotation of the user 702 at the position indicated by the rotation of the user 702. Fig.7D 714d, 714e, 714f and 714g are displayed at the left side of the position. In some embodiments, the computer system 700 communicates with an external device (e.g., a computer monitor, a smart phone and / or a smart watch) that displays the virtual object. In some embodiments, the computer system 700 transmits instructions to an external device to adjust the display of the virtual object at least based on the rotation amount detected by the wearable device 706 (e.g., when the wearable device 706 detects the larger rotation amount of the right hand of the user 702, the external device increases the size of the virtual object). In some embodiments, the computer system 700 communicates with one or more sensors that can detect hand motion (e.g., wireless communication and / or wired communication). In some embodiments, when the right hand of the user 702 is clenched and rotated, the right hand of the user 702 is outside the detection capability range of one or more sensors. In some embodiments, the computer system 700 displays a user interface for controlling the playback of the media item. In some embodiments, user 702 can perform discrete gestures (eg, pinch and scroll) to modify the playback state of a media item (eg, pause playback of a media item, initiate playback of a media item).

[0143] As described above, as the amount of rotation detected by the wearable device 706 increases, the additional computer system 700 displays radial lines of the rotation indicator virtual object 724 starting from the twelve o'clock position of the circle (e.g., where the twelve o'clock position of the circle is at the top of the circle of the rotation indicator virtual object 724). Fig. 7E Because the wearable device 706 detects that the right hand of the user 702 is rotated at a larger rotation angle, the computer system 700 displays the radial line included in the rotation indicator virtual object 724 as being aligned with the radial line of the rotation indicator virtual object 724. Fig.7D The position of the radial line in is further away from the 12 o'clock position (eg, the top of the circle) of the circle included in the rotation indicator virtual object 724. Fig. 7E At , the wearable device 706 detects that the right hand of the user 702 performs a release gesture.

[0144] exist Figure 7FIn response to receiving (e.g., via computer system 704) an indication that wearable device 706 has detected that the right hand of user 702 has performed an unclasp gesture, computer system 700 performs operations associated with virtual object 714g, including stopping navigating between displayed virtual objects, stopping displaying other virtual objects, and showing virtual object 714g as being selected. Figure 7F At , the computer system 700 displays the virtual object 714g as being selected because the virtual object 714g was in focus when the computer system 700 received the indication that the user 702's hand performed the release gesture. Figure 7F As shown in , in response to receiving an indication that the user 702's hand is performing a release gesture, the computer system 700 displays a hand indication identification virtual object 736 representing the released hand. In some embodiments, the computer system 700 uses the above-mentioned 7A to 7F In some embodiments, the computer system 700 displays a communication user interface. In some embodiments, in response to the wearable device 706 (and / or the computer system 700 or the computer system 704) detecting a request to connect to a third-party device, the computer system 700 displays an indication that the connection has been accepted (e.g., the phone call has been answered) and / or that the wearable device 706 has answered the phone call. In some embodiments, the request to connect to the third-party device is detected based on a release gesture detected by the wearable device 706. In addition, the wearable device 706 has been described as being able to connect to the third-party device. 7A to 7F The various descriptions of FIG. 700 may be used to state that the computer system 700 receives an indication from the wearable device 706. It should be understood that (for example, as described above in Fig. 7A ), the computer system 704 may receive an indication from the wearable device 706 and cause the computer system 700 to perform one or more actions in response to receiving one or more indications from the wearable device 706 (e.g., one or more actions performed by the computer system 700, as described above with respect to 7A to 7F described).

[0145] about 7A to 7F See below for additional description of 7A to 7F Method 900 is described.

[0146] FIG. 8A to FIG. 8H An exemplary user interface is shown for performing various operations using hand gestures in some embodiments. FIG. 8A to FIG. 8H The user interface in is used to illustrate the process described below, including Fig. 9 and FIG. 11A to FIG. 11B process.

[0147] FIG. 8A to FIG. 8CAn exemplary scenario of performing one or more operations after hand input (e.g., and / or an air gesture) is detected by the wearable device 706 is shown. Fig. 8A As shown in FIG. 8 , the computer system 700 displays an alarm notification 804. The alarm notification 804 includes a snooze control virtual object 806 and a stop control virtual object 808. Fig. 8A At , the wearable device 706 detects that the right hand of the user 702 is clenched.

[0148] exist Figure 8B In response to receiving (e.g., via computer system 704) an indication that wearable device 706 has detected that user 702's right hand is clenched, computer system 700 moves display of snooze control virtual object 806 to the right and displays a left-facing arrow 822 to the left of snooze control virtual object 806. The left-facing arrow 822 indicates that snooze control virtual object 806 will be activated by sliding snooze control virtual object 806 to the left. In addition, in response to receiving an indication that wearable device 706 has detected that user 702's right hand is clenched, computer system 700 moves display of stop control virtual object 808 to the left, and computer system 700 displays a right-facing arrow 824 to the right of stop control virtual object 808. The right-facing arrow 824 indicates that stop control virtual object 808 will be activated by sliding stop control virtual object 808 to the right. Figure 8B At , wearable device 706 detects that the right hand of user 702 is tilted downward.

[0149] exist Figure 8CAt, in response to receiving the indication that the wearable device 706 has detected the right hand of the user 702 tilted downward, the computer system 700 slides the snooze control virtual object 806 to the left. By sliding the snooze control virtual object 806 to the left, the computer system 700 indicates that the snooze control virtual object 806 is being activated. In some embodiments, the wearable device 706 activates the control and / or performs the function associated with the snooze control virtual object 806. In some embodiments, the computer system 700 and / or the computer system 704 perform the function associated with the snooze control virtual object 806. In some embodiments, the function performed is to initiate the snooze function to temporarily pause the "wake up" alarm. In some embodiments, the wearable device 706 detects that the hand of the user 702 tilts to the right, and in response to (e.g., via the computer system 704) receiving the indication that the hand of the user 702 tilts to the right, the computer system 700 stops the control virtual object 708 from sliding to the right. In some embodiments, in response to receiving (e.g., via computer system 704) an indication that the user 702's hand is tilted to the right, one or more of computer system 700, computer system 704, and / or wearable device 706 executes a function of stopping the "wake up" alarm.

[0150] In some embodiments, when the wearable device 706 displays a notification corresponding to the alarm notification 804, the wearable device 706 moves the user interface object corresponding to the snooze control virtual object 806 or the user interface object corresponding to the stop control virtual object 808 based on whether the direction of the hand of the user 702 is tilted in the direction of gravity or tilted in the direction away from gravity. In some embodiments, if the hand of the user 702 is detected to be tilted in the same direction as gravity, the wearable device 706 (and / or the computer system 700 and / or the computer system 704) slides the user interface object corresponding to the snooze control virtual object 806 to the left. In some embodiments, if the hand of the user 702 is detected to be tilted in the direction opposite to gravity, the wearable device 706 slides the user interface object corresponding to the stop control virtual object 808 to the right.

[0151] FIG. 8D to FIG. 8F An exemplary scenario of selecting a virtual object after hand input (e.g., and / or an air gesture) is detected by the wearable device 706 is shown. Fig.8D At , the wearable device 706 detects that the user 702 is performing indoor walking exercise. Fig.8DAs shown in , in response to the computer system 700 (e.g., from the computer system 704) receiving an indication that the wearable device 706 detects that the user 702 is performing an indoor walking exercise, the computer system 700 displays a physical activity notification 818. The physical activity notification 818 includes a tracking function initiating virtual object 820a and a dismissing virtual object 820b. In response to the tracking function initiating virtual object 820a being activated, the wearable device 706 initiates a tracking function that tracks metrics (e.g., heart rate, calories burned, and / or time spent exercising) during the indoor walking exercise of the user 702. The wearable device 706 abandons initiating a tracking function corresponding to the indoor walking exercise of the user 702 in response to the dismissing virtual object being activated. In some embodiments, the wearable device 706 displays the physical activity notification 818 (e.g., instead of the computer system 700). In some embodiments, the wearable device 706 and the computer system 700 simultaneously display user interface objects and / or virtual objects corresponding to the physical activity notification 818. In Fig.8D At , the wearable device 706 detects that the right hand of the user 702 is clenched.

[0152] like Fig. 8E As shown in , in response to receiving an indication that the wearable device 706 has detected that the right hand of the user 702 is clenched, the computer system 700 displays the tracking function initiating virtual object 820a as visually prominent (e.g., the computer system 700 displays the tracking function initiating virtual object 820a as larger than the releasing virtual object 820b). If the wearable device 706 detects that the user 702 performs a release gesture when the computer system 700 displays the tracking function initiating virtual object 820a as visually prominent, the wearable device 706 activates the tracking function initiating virtual object 820a. In some embodiments, the computer system 700 visually highlights the tracking function initiating virtual object 820a by thickening the border of the tracking function initiating virtual object 820a. In some embodiments, the computer system 700 visually highlights the tracking function initiating virtual object 820a by displaying the tracking function initiating virtual object 820a as a different color from the releasing virtual object 820b. In some embodiments, the computer system 700 visually highlights the tracking function initiating virtual object 820a by reducing the size of the dismissed virtual object 820b (e.g., and maintaining the displayed size of the tracking function initiating virtual object 820a). Fig. 8E At this point, the wearable device continues to detect that the user's 702 right hand is clenched.

[0153] exist Fig.8FAt , in response to receiving an indication that the wearable device 706 continues to detect that the right hand of the user 702 is clenched, the computer system 700 stops displaying the tracking function initiating virtual object 820a as visually prominent, and the computer system 700 displays the releasing virtual object 820b as visually prominent. As long as the wearable device 706 detects the right hand of the user 702, the computer system 700 alternates between displaying the tracking function initiating virtual object 820a and displaying the releasing virtual object 820b as visually prominent. If the wearable device 706 detects that the user 702 performs a release gesture while the computer system 700 displays the releasing virtual object 820b as visually prominent, the wearable device 706 activates the releasing virtual object 820b. In some embodiments, the computer system 700 alternates between displaying the tracking function initiating virtual object 820a and displaying the virtual object 820b as visually prominent based on time (e.g., the computer system 700 displays the tracking function initiating virtual object 820a as visually prominent for a predetermined amount of time (e.g., 1 second, 3 seconds, or 5 seconds), where after the predetermined amount of time has passed, the computer system 700 stops displaying the tracking function initiating virtual object 820a as visually prominent and displays the releasing virtual object 820b as visually prominent for the predetermined amount of time). Fig.8F At , the wearable device 706 detects that the right hand of the user 702 has performed a release gesture.

[0154] exist Figure 8G In response to the wearable device 706 detecting that the right hand of the user 702 has performed a release gesture, the wearable device 706 activates the release virtual object 820b. In response to receiving (e.g., via the computer system 704) an indication that the wearable device 706 activates the release virtual object 820b, the computer system 700 stops displaying the physical activity notification 818. Figure 8G As shown in , computer system 700 displays user interface 820. Figure 8G At , the wearable device 706 detects that the user 702 has executed a request for the computer system 700 to display the control center user interface 830 (e.g., the user 702 performs an air tap, hand rotation, air swipe, air pinch, and / or air expand that is detected by the wearable device 706).

[0155] like Figure 8H As shown in , in response to receiving an indication that the wearable device 706 has detected that the user 702 has executed a request to display the control center user interface 830, the computer system 700 displays the control center user interface 830. Figure 8HAs shown in , the control center user interface 830 includes a plurality of virtual objects 836. The wearable device 706 performs a corresponding operation (e.g., entering a do not disturb mode, adjusting the volume of a pair of external speakers, spraying water within the wearable device 706, and / or activating a light source on the wearable device 706) in response to detecting that the corresponding virtual object is selected. The plurality of virtual objects 836 include a hand gesture detection virtual object 836a. Figure 8H , activates (e.g., switches on) the hand gesture detection virtual object 836a. When the hand gesture detection virtual object 836a is activated, the computer system 700 (e.g., from the computer system 704) receives instructions to perform various operations (e.g., as discussed above) in response to the wearable device 706 detecting an air gesture by the right hand of the user 702. In some embodiments, the wearable device 706 detects a request to deactivate the hand gesture detection virtual object 836a (e.g., the user 702 performs an air tap, hand rotation, air swipe, air pinch, and / or air open detected by the wearable device 706). In an embodiment in which the hand gesture detection virtual object 836a is deactivated, the computer system 700 will not receive instructions to perform various operations in response to the wearable device 706 detecting that the right hand of the user 702 has performed an air gesture. In addition, it has been described FIG. 8A to FIG. 8H The various descriptions of the present invention may be used to state that the computer system 700 receives an indication from the wearable device 706. It should be understood that (for example, as described above in Fig. 7A ), the computer system 704 may receive an indication from the wearable device 706 and cause the computer system 700 to perform one or more actions in response to receiving one or more indications from the wearable device 706 (e.g., one or more actions performed by the computer system 700, as described above with respect to FIG. 8A to FIG. 8H described).

[0156] The following references are relative to FIG. 8A to FIG. 8H The methods 900 and 1100 described provide for FIG. 8A to FIG. 8H Additional description of .

[0157] Fig. 9 is a flow chart of an exemplary method 900 for navigating content using hand gestures according to some embodiments. In some embodiments, the method 900 includes a display generation component (e.g., Figure 1 , Figure 3 and Figure 4 ) and a computer system that optionally includes one or more sensors (e.g., a display generating component 120 in the display) (e.g., a head-up display, a display, a touch screen, a projector, etc.) Figure 1In some embodiments, the computer system communicates with one or more input devices (e.g., a touch-sensitive surface). In some embodiments, method 900 is performed by storing in a non-transitory (or transient) computer-readable storage medium and executed by one or more processors of a computer system (such as one or more processors 202 of computer system 101) (e.g., Figure 1 Some operations in method 900 are optionally combined, and / or the order of some operations is optionally changed.

[0158] When the hand-worn device (e.g., 706) is being worn by a user (e.g., 702), a computer system (e.g., 700) displays (902) via a display generation component a corresponding user interface (e.g., a virtual user interface or an AR user interface) including a first portion (e.g., one or more virtual objects and / or virtual content) of content (e.g., 714a to 714g) and a second portion (e.g., one or more virtual objects and / or virtual content) of the content (e.g., 714a to 714g) that is different from the first portion of the content. In some embodiments, the first portion of the content and the second portion of the content are selectable. In some embodiments, the first portion of the content has a visual appearance that is different from the visual appearance (e.g., size, color, shape) of the second portion of the content. In some embodiments, the first portion of the content and the second portion of the content are displayed in an array, in a vertical / horizontal line, in a circle, a semicircle, and / or any other suitable configuration. In some embodiments, the first portion of the content and the second portion of the content are displayed simultaneously. In some embodiments, the first portion of the content or the second portion of the content is displayed with a visual prominence indicating that the corresponding portion of the content is currently selectable. In some embodiments, the first portion of the content and the second portion of the content are displayed in an augmented reality environment.

[0159] While displaying a corresponding user interface including a first portion of content and a second portion of content, the computer system receives (904) a signal from the hand-worn device (e.g., 706) that the hand (e.g., Figure 7B and Fig.7DIn some embodiments, the hand input is detected by one or more sensors (e.g., gyroscopes, accelerometers, heart rate sensors, etc.) of an external device (e.g., a smart watch) worn by the user. In some embodiments, the external device communicates with the computer system (e.g., wirelessly). In some embodiments, one or more cameras integrated into the computer system detect the hand input.

[0160] In response to receiving (906) an indication that the hand-worn device (e.g., 706) detected a hand input including a rotation of the hand and determining according to (908) that the hand input (e.g., at least a portion of the hand input) was performed (e.g., Fig.7D When the hand-worn device detects that the hand (e.g., the right hand of the user 702) (e.g., the hand of the user) (e.g., the hand being rotated) is clenched, the computer system navigates between the first part of the content and the second part of the content (e.g., as described above with respect to Fig.7D and Fig. 7E The computer system (e.g., 700) abandons navigating between the first portion of the content and the second portion of the content (e.g., as discussed above with respect to the rotation direction of performing the hand input) and determining according to (910) that the hand-worn device (e.g., 702) did not detect that the hand (e.g., the right hand of the user 702) was clenched when performing the hand input (e.g., at least a portion of the hand input), then Fig. 7AAs discussed) (e.g., in the direction of rotation in which the hand input is performed). In some embodiments, navigating between the first portion of the content and the second portion of the content includes performing a scrolling operation (e.g., horizontal scrolling, vertical scrolling, and / or diagonal scrolling) on ​​a user interface (e.g., updating / changing which portion of the content is displayed as visually prominent (e.g., being differently enlarged, changing color, bolding, and / or animated when compared to other portions of the content)) (e.g., scrolling the first portion of the content (and / or the second portion of the content) to display the second portion of the content and / or an additional portion of the content that is different from the second portion of the content and / or the first portion of the content). In some embodiments, navigating between the first portion of the content and the second portion of the content includes moving a selection indicator between the first portion of the content and the second portion of the content (e.g., moving from the first portion of the content to the second portion of the content and / or vice versa). In some embodiments, navigating between the first portion of the content and the second portion of the content includes alternating between which portion of the content is visually prominent. In some embodiments, navigating between the first portion of the content and the second portion of the content includes changing the size of the first portion of the content or the second portion of the content. In some embodiments, navigating between the first part of the content and the second part of the content includes changing the size of both the first part of the content and the second part of the content (e.g., making the first part of the content smaller and making the second part of the content larger). In some embodiments, navigating between the first part of the content and the second part of the content includes stopping displaying the first part of the content or the second part of the content. In some embodiments, navigating between the first part of the content and the second part of the content includes displaying the first part of the content in an animated manner and / or stopping displaying one of the second part of the content in an animated manner, or vice versa. In some embodiments, navigating between the first part of the content and the second part of the content includes outputting an audio signal corresponding to the first part of the content and / or the second part of the content via one or more speakers communicating with the computer system. In some embodiments, navigating between the first part of the content and the second part of the content includes displaying the first part of the content in the background of the corresponding user interface and displaying the second part of the content in the foreground of the corresponding user interface, or vice versa. In some embodiments, navigating between the first part of the content and the second part of the content includes visually blurring the first part of the content or the second part of the content. In some embodiments, the determination that the hand is clenched when the hand input is made is performed at an external device (e.g., a smart watch) (e.g., in communication (e.g., wireless communication) with the computer system), at the computer system, and / or at another computer system (e.g., a server) that is different from the computer system and / or the external device. In some embodiments, the determination that the hand is clenched when the hand input is made is performed by the computer system based on data received from an external device (e.g., a smart watch) worn by the user.In some embodiments, a hand is determined to be clenched when a hand input is performed by one or more cameras in communication with the computer system. In response to receiving an indication that a hand-worn device detected that a hand input including a rotation of the hand was performed and navigating between a first portion of content and a second portion of content based on determining that the hand-worn device detected that the hand was clenched when the hand input was performed, this provides the user with greater control over the computer system by allowing the user to navigate between content displayed by the computer system without the user touching the computer system and without the computer system displaying additional controls (which provides additional control options without cluttering the user interface). Providing additional control over the system without cluttering the UI with additional displayed controls enhances the operability of the system and makes the user-system interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the system), which in turn reduces power usage and extends the battery life of the system by enabling the user to use the system more quickly and efficiently.

[0161] In some embodiments, a computer system (e.g., 700) communicates (e.g., wired communication, wireless communication) with one or more sensors (e.g., one or more accelerometers, gyroscopes, and / or heart rate sensors) (e.g., for detecting hand movements) (e.g., one or more cameras), the one or more sensors having a detection capability range (e.g., a distance range (e.g., 0 inches to 5 inches or 10 inches to 24 inches) within which the user's corresponding body part movement must be relative to the one or more sensors so that the one or more sensors can detect movement of the corresponding body part), and wherein when the hand-worn device detects hand input, the hand is not within the detection capability range of the one or more sensors. When the user's hand is outside the range of the one or more sensors, detecting hand input with the hand-worn device when the hand is not within the detection capability range of the one or more sensors improves the usability of the hand-worn device for certain users.

[0162] In some embodiments, navigating between a first portion of content (e.g., 714a to 714g) and a second portion of content (e.g., 714a to 714g) includes visually highlighting the second portion of content relative to the first portion of content (e.g., as described above with respect to FIG. 7C to FIG. 7EIn some embodiments, the second portion of the content is larger than the first portion of the content, the second portion of the content is surrounded by a selection indicating virtual object and the first portion of the content is not surrounded by a selection indicating virtual object, the second portion of the content is displayed as a first color and the first portion of the content is displayed as a second color different from the first color, and / or the first portion of the content is visually obscured while the second portion of the content is not visually obscured. In some embodiments, while visually highlighting the second portion of the content, the computer system receives an indication (e.g., generated by the computer system and / or generated by the hand-worn device) that the hand-worn device detects that the hand is released (e.g., as described above in Fig. 7E and Figure 7F In some embodiments, in response to receiving an indication that the hand-worn device detects that the hand is released, the computer system performs operations corresponding to the second portion of the content (e.g., as described above with respect to Figure 7F The present invention also provides a method of controlling the display of the second portion of the content by the computer system and / or performing a control operation corresponding to the second portion of the content. The method further provides a method of controlling the display of the second portion of the content by the computer system and / or performing a control operation corresponding to the second portion of the content. The method further provides a method of controlling the display of the second portion of the content by the computer system and / or performing a control operation corresponding to the second portion of the content. The method further provides a method of controlling the display of the second portion of the content by the computer system and / or performing a control operation corresponding to the second portion of the content. The method further provides a method of controlling the display of the second portion of the content by the computer system and / or performing a control operation corresponding to the second portion of the content. The method further provides a method of controlling the display of the second portion of the content by the computer system and / or performing a control operation corresponding to the second portion of the content. The method further provides a method of controlling the display of the second portion of the content by the computer system and / or performing a control operation corresponding to the second portion of the content.

[0163] In some embodiments, navigation between a first portion of content (e.g., 714a to 714g) and a second portion of content (e.g., 714a to 714g) is based on a duration (e.g., a time (e.g., milliseconds, seconds, or minutes) (e.g., 1 second, 3 seconds, or 5 seconds) that the hand-worn device detects a portion of the hand input (e.g., a clenched portion of the hand input) (e.g., as described above with respect to FIG. 7D to FIG. 7EAs described above) (e.g., navigating between the first portion of the content and the second portion of the content as long as the hand input is detected). In some embodiments, the navigation between the first portion of the content and the second portion of the content is based on a combination of parameters of the hand input (e.g., duration of the hand input, magnitude of the hand input). In some embodiments, based on determining that a portion of the hand input is detected within a first time period, the navigation between the first portion of the content and the second portion of the content occurs within a second time period (e.g., which has a predefined relationship with the first time period (e.g., the second time period is 0.5 times, 2 times, or 4 times the first time period)). In some embodiments, based on determining that the portion of the hand input is detected within a third time period that is longer than the first time period, the navigation between the first portion of the content and the second portion of the content occurs within a fourth time period that is longer than the second time period. Navigating between the first portion of the content and the second portion of the content within the duration of the manual input provides the user with visual feedback about the duration that the hand input is detected by the hand-worn device, which provides improved visual feedback and can result in more effective control of the user interface for some users.

[0164] In some embodiments, navigation between a first portion of content (e.g., 714a to 714g) and a second portion of content (e.g., 714a to 714g) is based on a manually entered rotation angle (e.g., 5°, 10°, 15°, or 25°) (e.g., as described above with respect to FIG. 7D to FIG. 7E As described) (e.g., when the hand input is gripped) (e.g., the larger the rotation angle, the faster the computer system navigates between the first portion of the content and the second portion of the content). In some embodiments, based on determining that the rotation angle is positive, navigation between the first portion of the content and the second portion of the content occurs in a first direction. In some embodiments, based on determining that the rotation angle is negative, navigation between the first portion of the content and the second portion of the content occurs in a second direction opposite to the first direction. In some embodiments, navigation between the first portion of the content and the second portion of the content is based on a combination of parameters (e.g., duration of the hand input, magnitude of the hand input). Navigating between the first portion of the content and the second portion of the content based on the rotation angle of the hand input provides the user with visual feedback about the rotation angle at which the hand input is being performed, which provides improved visual feedback and can result in more effective control of the user interface for some users.

[0165] In some embodiments, navigation between a first portion of content (e.g., 714a to 714g) and a second portion of content (e.g., 714a to 714g) is based on the rotation direction (e.g., counterclockwise and / or clockwise) of the hand input (e.g., as described above with respect to FIG. 7D to FIG. 7Edescribed) (e.g., when the hand input is clenched) (e.g., when the rotation direction of the hand input is clockwise, navigation between the first part of the content and the second part of the content occurs in a first direction, and when the rotation of the hand input is counterclockwise, navigation between the first part of the content and the second part of the content occurs in a second direction opposite to the first direction). In some embodiments, navigation between the first part of the content and the second part of the content is based on a combination of parameters of the hand input (e.g., duration of the hand input, magnitude of the hand input). Navigating between the first part of the content and the second part of the content based on the rotation direction of the hand input provides the user with visual feedback about the direction in which the hand input was performed, which provides improved visual feedback and can result in more effective control of the user interface for some users.

[0166] In some embodiments, in response to receiving an indication that the hand-worn device (e.g., 706) detected a hand input that included rotation of the hand and based on determining that the hand-worn device did not detect that the hand was clenched when performing the hand input, the computer system displays via a display generation component (e.g., simultaneously with the display of the first portion of content and the second portion of content) a first hand input virtual object (e.g., a graphical indication that the first hand input virtual object is a first type of hand input) representing a clenched hand. Figure 7C 736 at ). In some embodiments, the first hand input virtual object represents a clenched hand that is rotated. In some embodiments, the first hand input virtual object is displayed in response to receiving an indication that the hand is clenched. Displaying the first hand input virtual object representing a clenched hand in response to receiving an indication that the hand-worn device detects that a hand input including rotation of the hand is performed provides the user with visual feedback about whether the hand-worn device detects that the hand is clenched when performing the hand input, which improves the visual feedback and can result in more effective control of the user interface for some users.

[0167] In some embodiments, displaying the first hand input virtual object (e.g., 736) includes displaying an animation of the first hand input virtual object changing from displaying a virtual object representing a clenched hand position to displaying a virtual object representing a loosened hand position (e.g., as described above with respect to Figure 7CAs described). In some embodiments, displaying the first hand input virtual object includes displaying a cyclic animation (e.g., a cyclic mode). In some embodiments, the cyclic animation cycles between a clenched hand position and an unclosed hand position (e.g., the first hand input virtual object has a first visual appearance (e.g., a clenched hand) for a first amount of time, and has a second visual appearance (e.g., an unclosed hand) for a second amount of time (e.g., the second amount of time is the same amount of time as the first amount of time)). Displaying the first hand input virtual object as changing between a virtual object representing a clenched hand position and a virtual object representing an unclosed hand position provides the user with visual feedback about the hand gesture that needs to be detected in order for the computer system to navigate between a first portion of content (e.g., 714a to 714g) and a second portion of content (e.g., 714a to 714g), which improves visual feedback and can result in more efficient control of the user interface for some users.

[0168] In some embodiments, before receiving an indication that the hand-worn device detected a hand input including a rotation of the hand, the computer system receives an indication that the hand-worn device detected that the hand was clenched (e.g., as described above with respect to Figure 7C In some embodiments, in response to receiving an indication that the hand-worn device detects that the hand is clenched (for example, when the hand-worn device detects that the hand is clenched), the computer system displays a second hand input virtual object representing a rotated clenched hand (for example, as described above with respect to Figure 7C 736) (e.g., a graphical indication of a rotated clenched hand). In some embodiments, in response to displaying the second hand input virtual object, display of the first hand input virtual object ceases. In some embodiments, in response to receiving an indication that the hand-worn device detects that a hand input including rotation of the hand is performed, the second hand input virtual object ceases to be displayed. Displaying a second hand input virtual object representing a rotated clenched hand in response to receiving an indication that the hand-worn device detects that the hand is clenched provides the user with visual feedback about the necessary type of input (e.g., a clenched, rotated hand) required to navigate between the first portion of content and the second portion of content, which provides improved visual feedback and can result in more effective control of the user interface for some users.

[0169] In some embodiments, displaying the second hand input virtual object (e.g., 736) includes animating the second hand input virtual object to change between a first rotation amount (e.g., 5°, 10°, 15°, or 25°) and a second rotation amount different from the first rotation amount (e.g., as described above with respect to Figure 7C . described). In some embodiments, the animation (e.g., a looping pattern) cycles between a first rotation amount (e.g., 5°, 10°, 15°, or 25°) and a second rotation amount that is different from the first rotation amount. In some embodiments, the second hand input virtual object is displayed as cycling between rotation in a first direction (e.g., clockwise) and rotation in a second direction (e.g., counterclockwise). In some embodiments, the first rotation amount and the second rotation amount are in the same direction (e.g., clockwise and / or counterclockwise). Displaying the second hand input virtual object as changing between the first rotation amount and the second rotation amount provides visual feedback to the user about the type of hand gesture that needs to be performed to cause the computer system to navigate between the first portion of the content (e.g., 714a to 714g) and the second portion of the content (e.g., 714a to 714g), which provides improved visual feedback and can result in more effective control of the user interface for some users.

[0170] In some embodiments, before receiving an indication that the hand-worn device detects hand input including rotation of the hand, the computer system displays a selection indication identifying a virtual object (e.g., Fig. 7A In some embodiments, in response to receiving an indication that the hand-worn device detected a hand input that includes rotation and based on determining that the hand-worn device detected a clenching when performing the hand input, the computer system moves the display of the selection indicator virtual object from the first position to the second position on the corresponding user interface (e.g., as described above with respect to Fig.7Ddescribed) (e.g., different from the first position) (e.g., the second position is positioned to the right, left, above, and / or below the first position), wherein the display of the selection indicator virtual object moves based on the rotation angle of the hand (e.g., the distance between the first position and the second position is based on the rotation angle of the hand (e.g., the greater the rotation angle of the hand, the farther the second position is from the first position)) (e.g., the greater the rotation angle of the hand input, the farther the second position is from the first position). In some embodiments, the selection indicator virtual object is displayed around the first portion of the content and / or the second portion of the content. In some embodiments, the selection indicator virtual object moves in a direction based on the rotation of the hand (e.g., when the hand rotates in a clockwise direction, the selection indicator virtual object moves to the right, and when the hand rotates in a counterclockwise direction, the selection indicator virtual object moves to the left). Moving the display of the selection indicator virtual object from the first position to the second position in response to receiving an indication that the hand-worn device detects a hand input that includes a rotation provides the user with the ability to control the computer system to display the location of the selection indicator virtual object without displaying additional controls, which provides additional control options without cluttering the user interface.

[0171] In some embodiments, navigation between a first portion of content (e.g., 714a to 714g) and a second portion of content (e.g., 714a to 714g) includes scrolling (e.g., horizontal scrolling, vertical scrolling, and / or diagonal scrolling) between the first portion of content and the second portion of content. In some embodiments, the scrolling speed between the first portion of content and the second portion of content is based on the rotation angle of the hand (e.g., the greater the rotation angle of the hand, the faster the scrolling operation). In some embodiments, the direction of scrolling between the first portion of content and the second portion of content is based on the direction of rotation of the hand (e.g., counterclockwise and / or clockwise) (e.g., when the rotation of the hand is in a clockwise direction, scrolling includes scrolling to the right, and when the rotation of the hand is in a counterclockwise direction, scrolling includes scrolling to the left).

[0172] In some embodiments, the first portion of the content (e.g., 714a to 714g) (at least a subset thereof) and the second portion of the content (e.g., 714a to 714g) (at least a subset thereof) are selectable (e.g., as described above with respect to Fig. 7AIn some embodiments, selection of a first portion of the content causes the computer system to perform a first operation (e.g., displaying a media item, playing back an audio file, changing a mode of the computer system), and selection of a second portion of the content causes the computer system to perform a second operation different from the first operation. In some embodiments, only the first portion of the content or the second portion of the content is selectable at any point in time. Navigating between a selectable first portion of the content and a selectable second portion of the content based on determining that the hand-worn device detects that the hand is clenched when performing hand input provides the user with the ability to browse between multiple selectable content options and decide to select one or more of the selectable content options without displaying additional controls, which provides additional control options without cluttering the user interface.

[0173] In some embodiments, when navigating between a first portion of content and a second portion of content (e.g., and while the hand is clenched), the computer system receives an indication (e.g., generated by the computer system) (e.g., generated by the hand-worn device) that the hand is released (e.g., as described above with respect to Fig. 7E In some embodiments, in response to receiving an indication that the hand is released, the computer system (e.g., 700) stops navigating (e.g., scrolling) between the first portion of the content and the second portion of the content (e.g., as described above with respect to Figure 7F In some embodiments, in response to stopping navigating between the first portion of the content and the second portion of the content, the second portion of the content becomes selectable. In some embodiments, in response to stopping navigating between the first portion of the content and the second portion of the content, the computer system selects the second portion of the content (e.g., the computer system performs an operation corresponding to the second portion of the content). Stopping navigating between the first portion of the content and the second portion of the content in response to receiving an indication that the hand is released allows the user to control the navigation operation performed by the computer system without displaying additional controls, which provides additional control options without cluttering the user interface.

[0174] In some embodiments, when navigating between the first portion of the content and the second portion of the content, the computer system displays, via the display generation component, a third hand input virtual object (e.g., the third hand input virtual object depicts the released hand) that represents the released hand. Fig. 7A736 at ) (e.g., a graphical indication). In some embodiments, a third hand input virtual object is displayed in response to receiving an indication that the hand is released. In some embodiments, the third hand input virtual object is displayed as cycling between a released hand gesture and a clenched hand gesture. In some embodiments, the first rotation amount is in the same direction as the second rotation amount (e.g., clockwise and / or counterclockwise). Displaying the third hand input virtual object while the first portion of content and the second portion of content are being navigated provides the user with visual feedback about the state of the computer system (e.g., the computer system has detected hand input including rotation while the hand is clenched), which provides improved visual feedback and can result in more effective control of the user interface for some users.

[0175] In some embodiments, displaying the third hand input virtual object (e.g., 736) includes displaying an animation of the third hand input changing from a virtual object representing a clenched hand position to a virtual object representing a loosened hand position (e.g., a looping animation that cycles through a pattern) (as described above with respect to Fig.7D As described herein) (e.g., a third hand input virtual object has a first visual appearance (e.g., a clenched hand) for a first amount of time, and has a second visual appearance (e.g., an unclosed hand) for a second amount of time (e.g., the second amount of time is the same as the first amount of time). Displaying the third hand input virtual object as an animation that changes from a virtual object representing a clenched hand position to a virtual object representing an unclosed hand position provides the user with visual feedback about the hand gesture that needs to be detected in order for the computer system to perform a desired operation (e.g., selection of a virtual object or stopping navigation between a first portion of content and a second portion of content), which provides improved visual feedback and can result in more effective control of the user interface for some users.

[0176] In some embodiments, the computer system communicates (e.g., wired communication and / or wireless communication) with an external device (e.g., a television, computer monitor, smart phone, and / or smart watch separate from the computer system), and wherein the external device causes a virtual object (e.g., 720) (e.g., a selectable virtual object) to be displayed in a first visual appearance (e.g., a first size, a first color). In some embodiments, after receiving an indication that a hand-worn device (e.g., 706) detected a hand input comprising a rotation of a hand, the computer system receives a second indication (e.g., generated by the computer system) (e.g., generated by the hand-worn device) that the hand-worn device detected a fourth hand input (e.g., an air swipe and / or air drag) comprising a second rotation of the hand (e.g., a clockwise, counterclockwise, forward, and / or backward rotation of the user's hand) when the hand is clenched, wherein the fourth hand input comprises a first rotation amount (e.g., 5°, 10°, 15°, or 25°). In some embodiments, in response to receiving a second indication that the hand-worn device detected a fourth hand input comprising a second rotation of the hand, the computer system transmits instructions to the external device to cause the virtual object to be displayed in a second visual appearance different from the first visual appearance (e.g., increasing the size of the virtual object, displaying the virtual object in a different color, displaying the virtual object in a different shade and / or tint of the same color) (e.g., as described above with respect to Fig. 7E In some embodiments, while displaying the virtual object in the second visual appearance, the computer system receives a third indication (e.g., generated by the computer system) that the hand-worn device detects that the fourth hand input includes a second rotation amount (e.g., greater than the first rotation amount). In some embodiments, in response to receiving a third indication that the hand-worn device detects that the fourth hand input includes a second rotation amount (e.g., in the same direction as the first rotation amount), the computer system transmits instructions to an external device to cause the virtual object to be displayed in a third visual appearance that is different from the second visual appearance (e.g., as well as the first visual appearance) (e.g., as described above with respect to Fig. 7EIn some embodiments, after receiving the third indication, the computer system transmits an instruction to the external device to display the virtual object in the second visual appearance in response to the hand-worn device detecting that the second hand input includes the first rotation amount. In some embodiments, after receiving the second indication, the computer system transmits an instruction to the external device to display the virtual object in the first visual appearance in response to the hand-worn device detecting that the second hand input does not include rotation. In some embodiments, the second hand input is detected by one or more cameras integrated into the computer system. Changing the visual appearance of the virtual object in response to receiving an indication that the rotation amount of the hand input has changed allows the user to change the visual appearance of the virtual object without displaying additional controls, which provides additional control options without cluttering the user interface. Changing the visual appearance of the virtual object provides the user with visual feedback about the state of the hand-worn device (e.g., the hand-worn device has detected a change in rotation), which provides improved visual feedback and can result in more effective control of the user interface for some users.

[0177] In some embodiments, the corresponding user interface (e.g., 710) is a media user interface (e.g., as described above with respect to Fig. 7E Described) (e.g., a media user interface associated with a television application) (e.g., a user interface that includes one or more representations of media (e.g., video, photo, and / or audio media)).

[0178] In some embodiments, the corresponding user interface (e.g., 710) is an augmented reality user interface (e.g., as described above in Fig. 7A described in ).

[0179] In some embodiments, after receiving a hand indication that the hand-worn device (e.g., 706) detects a hand input including a rotation of the hand, the computer receives a hand indication that the hand-worn device (e.g., using one or more sensors integrated into the hand-worn device) detects a corresponding hand input (e.g., as described above with respect to Fig. 7E In some embodiments, in response to receiving an indication that the hand-worn device detected the corresponding hand input, the computer system performs a second operation (e.g., as described above with respect to Fig. 7EIn some embodiments, the second operation is performed while the computer system is navigating between a first portion of content and a second portion of content. In some embodiments, performing the second operation causes the computer system to stop navigating between the first portion of content and the second portion of content. In some embodiments, performing the second operation causes the computer system to stop displaying the corresponding user interface. Performing an operation in response to receiving an indication that the hand-worn device detected a corresponding hand input allows a user to control when the computer system performs various operations without displaying additional controls, which provides additional control options without cluttering the user interface.

[0180] In some embodiments, displaying a corresponding user interface (e.g., 710) includes displaying playback of a video (e.g., a video media item stored on a computer system) (e.g., a movie and / or television program), wherein the corresponding hand input includes a clenching and rolling (e.g., a gesture including a user clenching a hand and rotating their clenched hand in one or more directions (e.g., clockwise, counterclockwise, upward, and / or downward)) input, and wherein performing an operation includes pausing playback of the video media item (e.g., as described above with respect to 7E). In some embodiments, when playback of the video media item is paused, and in response to receiving an indication that the hand-worn device detected the corresponding hand input, playback of the video media item is resumed. In some embodiments, pausing playback of the video media item causes the computer system to display multiple playback controls. Pausing playback of the video media in response to receiving an indication that the hand-worn device detected the corresponding hand gesture allows the user to modify the playback state of the video media item without displaying additional controls, which provides additional control options without cluttering the user interface.

[0181] In some embodiments, the corresponding hand input includes a clenching and scrolling input (e.g., a hand gesture including a user clenching their hands together and rotating their clenched hands in one or more directions (e.g., clockwise, counterclockwise, upward, and / or downward), and wherein performing the operation includes scrolling between a first portion of content and a second portion of content (e.g., as described above with respect to Fig.7DAs described above) (e.g., the direction of the squeeze and scroll gesture determines the direction of scrolling). In some embodiments, scrolling between the first portion of content and the second portion of content occurs in a combination of directions (e.g., scrolling between the first portion of content and the second portion of content includes scrolling in two or more directions (e.g., vertical, horizontal, and / or diagonal directions)). In some embodiments, scrolling is performed based on parameters of the squeeze and scroll gesture (e.g., scrolling direction, rotation angle, and / or duration of the squeeze gesture). Scrolling between the first portion of content and the second portion of content in response to receiving an indication that a squeeze and scroll gesture has been detected by the hand-worn device allows a user to control display operations of the computer system without displaying additional controls, which provides additional control options without cluttering the user interface.

[0182] In some embodiments, aspects / operations of methods 1100, 1300, 1500 may be interchanged, replaced, and / or added between these methods. For example, navigating between displayed virtual objects using hand gestures as described in method 900 is optionally used to navigate displayed virtual objects in method 1100. For the sake of brevity, these details are not repeated here.

[0183] FIG. 10A to FIG. 10I An example of activating a virtual object is shown. FIG. 11A to FIG. 11B is a flow chart of an exemplary method 1100 for activating a virtual object. FIG. 10A to FIG. 10I The user interface in is used to show the following description including FIG. 11A to FIG. 11B The process of the process.

[0184] like Fig. 10A As shown in FIG. , the wearable device 706 (e.g., as described above with respect to Fig. 7A The complex function block 1002 is displayed on the wearable device 706. The complex function block 1002 includes a first set of information, which includes date information 1004a, time information 1004b, and selectable icons 1004c. The wearable device 706 starts the corresponding application in response to detecting that the corresponding icon in the selectable icons 1004c is selected. It should be understood that displaying the complex function block 1002 on the wearable device 706 is only exemplary. Although the wearable device 706 is Fig. 10A Complex function block 1002 is displayed at the wearable device 706, but in some embodiments, a device external to the wearable device 706 (e.g., a television, a computer monitor, a smart phone, and / or a display generating component of an HMD) displays complex function block 1002. In embodiments where an external device displays complex function block 1002, the wearable device 706 transmits display instructions to the external device in response to the wearable device 706 detecting one or more hand gestures (e.g., one or more air gestures or inputs on the hardware or touch-sensitive input elements of the HMD). It should also be understood that, as Fig. 10A The configuration of the first set of information shown is exemplary. In some embodiments, the first set of information includes Fig. 10A additional information not shown in the first set of information (e.g., weather information, fitness information, and / or news information), or the first set of information includes less than Fig. 10A The information shown in Fig. 10A , the wearable device 706 detects that the hand 1018 is clenched.

[0185] exist Fig. 10B In response to detecting that hand 1018 is clenched, wearable device 706 updates the display of complex function block 1002 to include a second set of information. The second set of information includes date information 1004a, time information 1004b, selectable icons 1004c, and event information 1004d. The second set of information includes more information than the above about Fig. 10A In some embodiments, the wearable device 706 displays the second set of information as long as the wearable device 706 detects that the hand 1018 is clenched. In some embodiments, in response to detecting that the hand 1018 is no longer clenched, the wearable device 706 stops displaying the second set of information and redisplays the first set of information. In some embodiments, displaying the second set of information does not include one or more information portions and / or any information included in the first set of information. In some embodiments, displaying the second set of information includes displaying a subset of the information included in the first set of information.

[0186] like Fig. 10C As shown in FIG. 8 , the wearable device 706 displays an alarm notification 804 including a snooze control virtual object 1020 and a stop control virtual object 1024 (e.g., as described above with respect to FIG. 8 ). Fig. 8A ). The alarm notification 804 is generated by a timer / clock application installed on the wearable device 706, and the wearable device 706 displays the alarm notification 804 because a determination has been made that the time criteria has been met (e.g., the current time of day coincides with the time at which the wearable device 706 has been programmed to generate the alarm notification 804). Fig. 10C , the snooze control virtual object 1020 includes the word “snooze” within the boundary of the snooze control virtual object 1020, and the stop control virtual object 1024 includes the word “stop” within the boundary of the stop control virtual object 1024. In some embodiments, the wearable device 706 performs an operation in response to detecting a tap gesture on the snooze control virtual object 1020. In some embodiments, the wearable device 706 performs an operation in response to detecting a tap gesture on the stop control virtual object 1024.

[0187] like Fig. 10DAs shown in , the wearable device 706 displays the hand gesture virtual object 1016 within the snooze control virtual object 1020. Fig. 10D At , the hand gesture virtual object 1016 corresponds to a clenching gesture. Fig. 10D As shown in , the wearable device 706 displays the hand gesture virtual object 1016 in a first appearance (e.g., an appearance representing a loosened hand). The display of the hand gesture virtual object 1016 within the snooze control virtual object 1020 indicates that in response to the wearable device 706 detecting the clenching gesture, the wearable device 706 will perform an operation corresponding to the snooze control virtual object 1020 (e.g., placing the alarm associated with the alarm notification 804 in snooze mode). In some embodiments, when the wearable device 706 displays the hand gesture virtual object 1016 within the snooze control virtual object, the wearable device 706 displays the snooze control virtual object 1020 as visually prominent (e.g., the snooze control virtual object 1020 is larger than the stop control virtual object 1024, and the snooze control virtual object 1020 is displayed as a different color from the stop control virtual object 1024 (e.g., highlighted)).

[0188] When the wearable device 706 displays the hand gesture virtual object 1016 within the snooze control virtual object 1020, the wearable device 706 stops displaying the word "snooze". In some embodiments, the wearable device 706 displays both the word "snooze" and the hand gesture virtual object 1016 within the boundaries of the snooze control virtual object 1020 (e.g., simultaneously). In some embodiments, the wearable device 706 displays the hand gesture virtual object 1016 adjacent to the snooze control virtual object 1020 (e.g., outside it). In some embodiments, the hand gesture virtual object 1016 represents different types of gestures (e.g., different types of air gestures) that can be detected by the wearable device 706 (e.g., a pinch gesture, a release gesture, an open gesture, and / or an air tap). In some embodiments, the wearable device 706 displays a hand gesture virtual object 1016 next to a control (e.g., a snooze control virtual object 1020 and / or a stop control virtual object 1024) that is in focus (e.g., a control that is a preferred control and / or a primary option), and the wearable device 706 does not display a hand gesture virtual object next to a control that is not in focus.

[0189] like Fig.10E As shown in FIG. , the wearable device 706 has a different appearance than the first appearance (e.g., as described above with respect to Fig. 10DThe hand gesture virtual object 1016 is displayed with a second visual appearance (e.g., an appearance representing a clenched hand) as described above. The wearable device 706 displays the hand gesture virtual object 1016 in an animated manner (e.g., as a looping animation) that animates the type of gesture that the wearable device 706 must detect in order to activate the snooze control virtual object 1020. In some embodiments, the looping animation is displayed with a first visual appearance for a predetermined amount of time (e.g., 0.1 seconds, 0.3 seconds, 0.5 seconds, 1 second, or 3 seconds), and after the predetermined amount of time has passed, the looping animation is displayed with a second visual appearance for a predetermined amount of time (e.g., such as how some graphics interchange formats (e.g., GIFs) are displayed). Fig.10FAs shown in , the wearable device 706 displays the hand gesture virtual object 1016 in the first visual appearance within the stop control virtual object 1024. The wearable device 706 alternates between displaying the hand gesture virtual object 1016 within the snooze control virtual object 1020 and displaying the hand gesture virtual object 1016 within the stop control virtual object 1024. The wearable device 706 alternates between displaying the hand gesture virtual object 1016 within the snooze control virtual object 1020 and displaying the hand gesture virtual object 1016 within the stop control virtual object 1024 based on one or more factors or a combination of factors. In some embodiments, the wearable device 706 alternates between displaying the hand gesture virtual object 1016 within the snooze control virtual object 1020 and displaying the hand gesture virtual object 1016 within the stop control virtual object 1024 based on time. In some embodiments, the wearable device 706 displays the hand gesture virtual object 1016 within the snooze control virtual object 1020 for a predetermined amount of time (e.g., 0.3 seconds, 0.5 seconds, 1 second, 5 seconds, 15 seconds, or 20 seconds). In some embodiments, after the predetermined amount of time has passed, the wearable device 706 displays the hand gesture virtual object 1016 within the stop control virtual object 1024 for a predetermined amount of time. In some embodiments, after displaying the hand gesture virtual object 1016 within the stop control virtual object 1024 for a predetermined amount of time, the wearable device 706 repeats the process (e.g., displaying the hand gesture virtual object 1016 in the snooze control virtual object 1020 for a predetermined period of time, and after the predetermined period of time has passed, displaying the hand gesture virtual object 1016 within the stop control virtual object 1024 for a predetermined amount of time). In some embodiments, the wearable device 706 alternately displays the hand gesture virtual object 1016 within the snooze control virtual object 1020 and displays the hand gesture virtual object 1016 within the stop control virtual object 1024 based on the wearable device 706 detecting the hand gesture. In some embodiments, the wearable device 706 alternately displays the hand gesture virtual object 1016 within the snooze control virtual object 1024 or displays the hand gesture virtual object 1016 within the stop control virtual object 1016 in response to the wearable device 706 detecting a hand gesture (e.g., a hand gesture represented by the hand gesture virtual object 1016 and / or a hand gesture different from the hand gesture represented by the hand gesture virtual object 1020).In some embodiments, based on the wearable device 706 animating the hand gesture virtual object 1016 to complete a rotation between the hand gesture virtual object 1016 having the first appearance object 1016 and the hand gesture virtual object 1016 having the second appearance, the wearable device 706 alternately displays the hand gesture virtual object 1016 within the snooze control virtual object 1020 and displays the hand gesture virtual object 1016 within the stop control virtual object 1024. In some embodiments, the wearable device 706 alternates between displaying the hand gesture virtual object 1016 and displaying the second hand gesture virtual object. In some embodiments, the wearable device 706 displays only one of the hand gesture virtual object 1016 or the second hand gesture virtual object at any given time. In some embodiments, the hand gesture virtual object 1016 represents a first type of hand gesture (e.g., a clenching gesture, a pinching gesture, and / or a rotation gesture) and corresponds to a first control (e.g., the stop control virtual object 1024 and / or the snooze control virtual object 1020). In some embodiments, the second hand gesture virtual object represents a second type of hand gesture different from the first type of hand gesture, and corresponds to a second control different from the first control. In some embodiments, the wearable device 706 alternates between displaying the hand gesture virtual object 1016 and displaying the second hand gesture virtual object based on time (e.g., the wearable device 706 displays the hand gesture virtual object 1016 for a predetermined time period, and after the predetermined time period has passed, the wearable device 706 displays the second hand gesture virtual object for a predetermined amount of time). In some embodiments, the wearable device 706 alternates between displaying the hand gesture virtual object and displaying the second hand gesture virtual object in response to the wearable device 706 detecting a hand gesture. In some embodiments, the hand gesture is an air gesture (e.g., as described above with respect to the description of the input gesture).

[0190] exist Fig.10F At , when the wearable device 706 displays the hand gesture virtual object 1016 in the stop control virtual object 1024, the wearable device 706 stops displaying the word “Stop” in the stop control virtual object 1024. When the wearable device 706 displays the hand gesture virtual object 1016 in the stop control virtual object 1024, the wearable device 706 redisplays the word “Snooze” in the snooze control virtual object 1020. Fig.10FAt the point where the hand gesture virtual object 1016 indicates that, when the wearable device 706 displays the hand gesture virtual object 1016 within the stop control virtual object 1024, in response to the wearable device 706 detecting that the hand 1018 has performed a clenching gesture, the wearable device 706 will perform an operation corresponding to the stop control virtual object 1024 (e.g., stopping the alarm).

[0191] like Figure 10G As shown in , the wearable device 706 displays the hand gesture virtual object 1016 in a second visual appearance (e.g., the hand is in a clenched position) within the stop control virtual object 1024. As described above, the wearable device 706 displays the hand gesture virtual object 1016 in a cycling mode between the first visual appearance (e.g., the hand is in an unfastened position) and the second visual appearance (e.g., the hand is in a clenched position). Figure 10G At , when the wearable device 706 displays the hand gesture virtual object 1016 within the stop control virtual object 1024, the wearable device 706 detects that the hand 1018 has performed a clenching gesture.

[0192] exist Fig. 10H In response to the wearable device 706 detecting that the hand 1018 has performed a clenching gesture when the wearable device 706 displays the hand gesture virtual object 1016 within the stop control virtual object 1024, the wearable device 706 stops displaying the alarm notification 804 and stops the alarm operation, and the wearable device 706 displays the complex function block 1002.

[0193] like Fig.10I As shown in FIG. 1 , the wearable device 706 displays a meditation notification 1014 associated with a meditation application installed on the wearable device 706. The meditation notification 1014 includes a start control virtual object 1022 and a release control virtual object 1026. Fig.10I As shown in , the wearable device 706 displays the hand gesture virtual object 1010 within the start control virtual object 1022. The hand gesture virtual object 1010 includes the same features and characteristics as the hand gesture virtual object 1016 described above. However, the hand gesture virtual object 1010 represents a pinch gesture. That is, in response to the wearable device 706 detecting a pinch gesture when the wearable device 706 displays the hand gesture virtual object 1010 within the start control virtual object 1022, the wearable device 706 performs an operation corresponding to the start control virtual object 1022 (e.g., initiating a meditation exercise). Fig.10I As shown in , the wearable device 706 displays the start control virtual object 1022 in a first color scheme (e.g., horizontal shading). Fig.10IAt , when the wearable device 706 initially displays the meditation notification 1014, the start control virtual object 1022 is a default control (e.g., a control that the user can select (e.g., via the wearable device 706 detecting that the user has performed a pinch gesture)). The wearable device 706 displays the hand gesture virtual object 1010 within the start control virtual object (e.g., and not the dismiss control virtual object 1026) because the start control virtual object 1022 is the default control for the meditation notification 1014. The wearable device 706 displays the default controls in a color scheme based on which application is associated with the notification. Because the meditation notification 1014 is associated with the meditation application, the wearable device 706 displays the start control virtual object 1022 (e.g., the default control for the media notification 1014) in a first color scheme. That is, the wearable device 706 will display the default controls included in the notification associated with the meditation application in the first color scheme. Wearable device 706 will display default controls included in notifications associated with other applications (e.g., an email application, a text messaging application, or a weather application) in a second color scheme that is different from the first color scheme.

[0194] about FIG. 10A to FIG. 10I See below for additional description of FIG. 10A to FIG. 10I Method 1100 is described.

[0195] FIG. 11A to FIG. 11B is a flow chart of an exemplary method 1000 for activating a virtual object according to some embodiments. In some embodiments, in a process including a display generation component (e.g., Figure 1 , Figure 3 and Figure 4 A computer system (e.g., 706) (e.g., a smart watch, a smart phone, a desktop computer, a laptop computer, a tablet device, a head-up display unit, a head-mounted display unit, an optical head-mounted display unit, and / or a wearable device) (e.g., Figure 1 In some embodiments, method 1000 is performed by storing in a non-transitory (or transient) computer-readable storage medium and executed by one or more processors of a computer system (such as one or more processors 202 of computer system 101) (e.g., Figure 1 Some operations in method 1000 are optionally combined, and / or the order of some operations is optionally changed.

[0196] The computer system (e.g., 706) displays (1102) a corresponding user interface (e.g., 804) via a display generation component, wherein displaying the corresponding user interface includes simultaneously displaying: a first control virtual object (1104) (e.g., 1020), which, when activated with a first type of input, causes the computer system (e.g., 706) to perform a first operation (e.g., stop playback of an audio tone, initiate playback of an audio tone, answer a phone call, reject a phone call, view a notification, and / or initiate an application); a second control virtual object (e.g., 1106) (e.g., 1024 ), which, when activated with the first type of input, causes the computer system to perform a second operation different from the first operation (e.g., stopping playback of an audio tone, initiating playback of an audio tone, answering a phone call, rejecting a phone call, viewing a notification, and / or initiating an application); and a first virtual object (1108) (e.g., 1016 and / or 1010) (e.g., a visual indication (e.g., a graphic) (e.g., a looping animation) (e.g., a visual indication displayed as part of the first control virtual object), which first virtual object indication may be responsive to executing (e.g., detecting) the second type of input. In some embodiments, the first control virtual object is activated by an input of a second type, wherein the second type of input does not point to a location in the corresponding user interface (e.g., when the first input is detected and / or performed) (e.g., air pinching, hand clenching, hand swiping, finger (e.g., one or more fingers and / or thumb) movement in the air) (e.g., the virtual object includes an animation of the gesture being performed). In some embodiments, a second virtual object indicating that the second control can be activated in response to detecting the second input is displayed simultaneously with the virtual object. In some embodiments, in response to detecting the first input, the virtual object stops being displayed. In some embodiments, the display of the virtual object replaces the description of the first control virtual object (e.g., a description of the operation associated with the first control virtual object). In some embodiments, the first control virtual object and the virtual object are displayed in response to a change in the orientation of the computer system (e.g., wrist raising). In some embodiments, the first control virtual object and the virtual object are displayed in response to detecting a tactile input. In some embodiments, the first control virtual object and the virtual object are displayed in response to receiving a notification from an external device. In some embodiments, the description of the virtual object and the first control virtual object are displayed simultaneously.

[0197] While displaying the first control virtual object, the computer system (e.g., 706) receives (1110) an indication (e.g., generated by the computer system) that a corresponding input (e.g., an air pinch, a hand clench, a hand swipe, movement of a finger (e.g., one or more fingers and / or a thumb) , an air tap, a tap, and / or a mouse click) has been performed (e.g., and / or detected).

[0198] In response to (1112) receiving an indication that a corresponding input has been performed (e.g., and / or detected) and in accordance with (1114) determining that the corresponding input is a first type of input directed to a location corresponding to a first control virtual object (e.g., 1020) (and / or in accordance with determining that a second input is a direct input (e.g., an input directed to a location corresponding to a virtual object)) (e.g., when the second input (e.g., a portion of the second input) is performed), the computer system initiates a process for performing a first operation (e.g., as described above with respect to Fig. 10C As described above) and according to (1116) determining that the corresponding input is a first type of input pointing to a location corresponding to the second control virtual object (e.g., 1024) (and / or according to determining that the corresponding input is a direct input (e.g., an input pointing to a location corresponding to the virtual object) (e.g., when the corresponding input (e.g., a portion of the corresponding input) is executed, the computer system initiates a process for performing a second operation (e.g., as described above with respect to Fig. 10C As described above) and in accordance with (1118) determining that the corresponding input is an input of the second type (and / or in accordance with determining that the second input is a specific indirect input (e.g., an input that does not point to a location corresponding to the virtual object)) (e.g., when the second input (e.g., a portion of the second input) is performed), the computer system initiates a process for performing the first operation (e.g., as described above with respect to Fig. 10DIn some embodiments, in response to receiving an indication that a second input has been detected and according to determining that the second input is not the first input, the computer system does not initiate (e.g., abandons) initiating a process for performing the first operation. In some embodiments, the second control virtual object and / or virtual object are stopped from being displayed in response to initiating a process for performing the second operation. In some embodiments, the first control virtual object and / or virtual object are stopped from being displayed in response to initiating a process for performing the first operation. In some embodiments, in response to receiving an indication that a second input has been detected and according to determining that the second input is a third input (e.g., an input that does not point to a position corresponding to the first virtual object and an input that is different from the first input), the computer system does not initiate (e.g., abandons initiating) a process for performing the first operation. Initiating a process for performing the first operation in response to receiving an indication that a corresponding input (wherein the corresponding input is an input of the second type) provides the user with greater control over the computer system by allowing the user to control when the computer system performs the first operation without the user touching the computer system and without the computer system displaying additional controls (which provides additional control options without cluttering the user interface). Providing additional control over the system without cluttering the UI with additional displayed controls enhances the operability of the system and makes the user-system interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the system), which in turn reduces power usage and extends the battery life of the system by enabling the user to use the system more quickly and efficiently. Displaying a virtual object indicating that the first control virtual object may be activated in response to a second type of input being performed provides visual feedback to the user regarding the type of gesture that needs to be performed to cause the first control virtual object to be activated, which provides improved visual feedback and may result in more efficient control of the user interface for some users.

[0199] In some embodiments, when displaying a corresponding user interface (e.g., 804), the computer system (e.g., 706) displays an animation that alternates (e.g., transitions (e.g., transitions back and forth)) between displaying a first virtual object (e.g., 1016) and displaying a second virtual object (e.g., 1010) (e.g., displaying the first virtual object on the corresponding user interface or displaying the second virtual object on the corresponding user interface at any given point in time), wherein the second virtual object indicates that the second control virtual object can be activated in response to a third type of input (e.g., a pinch gesture, an open gesture, an air tap, and / or an air swipe), and wherein the second type of input is different from (e.g., corresponds to a different gesture) the third type of input (e.g., as described above with respect to Fig.10FIn some embodiments, the first virtual object and the second virtual object are displayed at different locations on the respective user interfaces. In some embodiments, the first virtual object and the second virtual object are displayed at the same location on the respective user interfaces. In some embodiments, the second type of input is the same as the third type of input. Alternating between the display of the first virtual object and the second virtual object provides visual feedback to the user about the type of input required to activate various controls that the computer system can perform, which provides improved visual feedback and can result in more effective control of the user interface for some users.

[0200] In some embodiments, the display of the first virtual object (e.g., 1016) and the display of the second virtual object (e.g., 1010) are alternated based on time (e.g., as described above with respect to Fig.10F As described) (e.g., milliseconds, seconds, or minutes) (e.g., 0.1 seconds, 0.3 seconds, 0.5 seconds, 1 second, 3 seconds, or 10 seconds) to perform (e.g., display of the first virtual object alternates with display of the second virtual object after a predetermined amount of time has passed, and vice versa (e.g., displaying the first virtual object for a predetermined amount of time when the second virtual object is not displayed, and after the predetermined amount of time has passed, stopping displaying the first virtual object and displaying the second virtual object for a predetermined amount of time)). Alternating between display of the first virtual object and display of the second virtual object based on time provides visual feedback to the user about the type of input required to activate the first control virtual object and the type of input required to activate the second control virtual object, which provides improved visual feedback and can result in more effective control of the user interface for some users.

[0201] In some embodiments, alternating between display of a first virtual object and display of a second virtual object (e.g., display of the first virtual object alternates with display of the second virtual object in response to detecting a gesture) is performed based on detecting (e.g., via one or more sensors in communication (e.g., wireless communication and / or wired communication) with a computer system) a first input (e.g., a hand input) (e.g., a gesture corresponding to a second type of input and / or a third type of input) (e.g., as described above with respect to Fig.10F Alternating between display of a first virtual object and display of a second virtual object based on detecting a first input provides visual feedback to the user regarding a state of the computer system (e.g., whether the computer system has detected a gesture), which provides improved visual feedback and can result in more effective control of the user interface for some users.

[0202] In some embodiments, displaying the first virtual object (e.g., 1016) includes displaying a second animation (e.g., as described above with respect to the first virtual object) alternating (e.g., based on time) between a first position corresponding to the first control virtual object (e.g., 1020) (e.g., a position within the boundaries of the first virtual control object) and a second position corresponding to the second control virtual object (e.g., 1024) (e.g., a position within the boundaries of the second virtual control object) (e.g., the second position is different from the first position). Fig.10F Described). In some embodiments, when the computer system receives an indication to perform a second type of input when the first virtual object is displayed at the second position, the computer system initiates a process for performing the second operation. In some embodiments, when the computer system receives an indication to perform a second type of input when the first virtual object is displayed at the second position, the computer system abandons initiating a process for performing the first operation. Alternating the display of the first virtual object between the first position and the second position provides the user with the ability to activate at least two different controls without displaying additional controls, which provides additional control options without cluttering the user interface. Alternating the display of the first virtual object between the first position corresponding to the first control virtual object and the second position corresponding to the second control virtual object provides the user with more control over the operations performed by the computer system without having to touch the computer system, which can result in more effective control of the computer system for some users.

[0203] In some embodiments, while displaying the corresponding user interface, the computer system (e.g., 706) displays a third virtual object (e.g., 1010) via a display generation component (e.g., simultaneously with the first control virtual object, the second control virtual object, and / or the first virtual object), the third virtual object indicating that the second operation can be activated in response to performing a fourth type of input (e.g., an air tap gesture, an air pinch gesture, an air swipe gesture, and / or an air open gesture), wherein the fourth type of input does not point to a location in the corresponding user interface (e.g., as described above with respect to Fig.10F. described). In some embodiments, the third virtual object is displayed simultaneously with the first virtual object. In some embodiments, the computer system alternately displays the first virtual object and the third virtual object. In some embodiments, the fourth type of input is the same as the second type of input. In some embodiments, the fourth type of input is different from the second type of input. Displaying a third virtual object indicating that the second operation can be activated in response to the fourth type of input (e.g., when the first virtual object is displayed) provides the user with visual feedback on the input type that needs to be detected to activate both the first control virtual object and the second control virtual object, which provides improved visual feedback and can result in more effective control of the user interface for some users. Simultaneously displaying the first virtual object and the third virtual object provides the user with greater control of the computer system by allowing the user to cause the computer system to perform the first operation or the second operation without the user contacting the computer system, which can result in more effective control of the computer system for some users.

[0204] In some embodiments, the first control virtual object (e.g., 1016) corresponds to a first default control option (e.g., as described above with respect to Fig.10I Described) (e.g., a control that is pre-selected as a default control relative to other displayed controls (e.g., by the computer system or by the manufacturer of the computer system) (e.g., a user may initially select only the default control without selecting other controls included in the corresponding user interface). In some embodiments, the second control virtual object corresponds to the default control option. Displaying the first virtual object indicating that the default control object can be activated in response to performing the second type of input provides the user with visual feedback about which control virtual object is the default control option, which provides improved visual feedback and can result in more effective control of the user interface for some users.

[0205] In some embodiments, displaying the first virtual object (e.g., 1016) includes visually highlighting the first control virtual object (e.g., 1020) (e.g., as described above with respect to Fig. 10D In some embodiments, when the first virtual object indicates that the second control virtual object can be activated in response to the second type of input, the second control virtual object is visually highlighted. Displaying the first control virtual object as visually highlighted provides the user with visual feedback about which control will be activated in response to the second type of input being performed, which provides improved visual feedback and can result in more effective control of the user interface for some users.

[0206] In some embodiments, the corresponding user interface (e.g., 1014) corresponds to a first application (e.g., an email application, a weather application, and / or a map application) (e.g., a third-party application previously installed on the computer system) (e.g., a default application (e.g., an application installed by the manufacturer of the computer system)), and wherein the first control virtual object (e.g., 1022) is a second default virtual object (e.g., a pre-selected control virtual object that provides the user with an option to select when the corresponding user interface is initially displayed) and is displayed with a first color scheme (e.g., Fig.10I 1022 at (e.g., the computer system displays the first control virtual object as filled with one or more colors) (e.g., the computer system displays the first control virtual object with a shade of the first color). In some embodiments, after receiving an indication that a corresponding input has been performed, the computer system receives a request to display a second corresponding user interface (e.g., one or more inputs (e.g., air tap, air swipe, air tap, swipe and / or tap)). In some embodiments, in response to receiving the request to display the second corresponding user interface, the computer system displays the second corresponding user interface via a display generation component, wherein the second corresponding user interface corresponds to a second application (e.g., an email application, a weather application, and / or a map application) different from the first application (e.g., an application different from the first application) (e.g., stored on the computer system and / or an external device), and wherein displaying the second corresponding user interface includes displaying a third control virtual object, wherein the third control virtual object is a second default virtual object and is displayed with a second color scheme different from the first color scheme (e.g., the second color scheme includes colors different from the first color scheme) (e.g., as described above with respect to Fig.10I In some embodiments, displaying the second corresponding user interface causes the computer system to stop displaying the first corresponding user interface. In some embodiments, the second corresponding user interface is displayed simultaneously with the first corresponding user interface.

[0207] In some embodiments, displaying the first virtual object (e.g., 1016) includes displaying a graphical (e.g., indicative) representation of the second type of input (e.g., as described above with respect to Fig. 10D and Fig.10E Displaying the first virtual object with a graphical representation of the second type of input provides visual feedback to the user about the gesture that needs to be detected to activate the first control virtual object, which provides improved visual feedback and can result in more effective control of the user interface for some users.

[0208] In some embodiments, displaying the first virtual object (e.g., 1016) includes displaying an animation representing the second type of input (e.g., displaying the first virtual object in a looping pattern that animates the second type of input) (e.g., as described above with respect to Fig. 10D and Fig.10E Displaying an animation representing the second type of input provides visual feedback to the user about the type of gesture that needs to be detected in order to activate the first control virtual object, which provides improved visual feedback and can result in more effective control of the user interface for some users.

[0209] In some embodiments, the first virtual object (e.g., 1016) is displayed adjacent to (e.g., next to, on top of, within, and / or in contact with) the first control virtual object (e.g., 1020) (e.g., as described above with respect to Fig. 10D As described above) (e.g., displaying the first control virtual object includes displaying a border (e.g., a continuous line and / or a dashed line) around the first control virtual object, and wherein the first virtual object is displayed within a threshold distance of the border (e.g., the first virtual object is displayed adjacent to the first virtual control object and / or the first virtual object is displayed within the border surrounding the first control virtual object)). In some embodiments, the first control virtual object includes static first information (e.g., text (e.g., text indicating a first operation)), and the first control virtual object includes dynamic second information (e.g., the first virtual object) (e.g., the first virtual object is displayed with a looping animation). Displaying the first virtual object adjacent to the first control virtual object provides visual feedback to the user about which operation will be activated if a second type of input is detected, which provides improved visual feedback and can result in more effective control of the user interface for some users.

[0210] In some embodiments, the first type of input is a tap input (e.g., as described above with respect to Fig. 10C Described) (e.g., an air tap or an air pinch) (e.g., performed by a user) (e.g., a tap input directed to a display of the first control virtual object and / or the second control virtual object). In some embodiments, the first type of input is a non-tap input, such as a mouse click, a swipe input, and / or a press and hold input.

[0211] In some embodiments, displaying a corresponding user interface includes displaying a first set of information (e.g., Fig. 10AIn some embodiments, after receiving an indication that a corresponding input has been executed, the computer system (e.g., 706) receives an indication (e.g., an indication generated by the computer system) that a second corresponding input (e.g., a clenching gesture, a hand rotation, and / or an unclamping gesture) has been executed (e.g., by a user). In some embodiments, in response to receiving an indication that the corresponding second input has been executed (e.g., while the corresponding second input is being executed), the computer system updates the display of the first complex function block to include a second set of information that is different from the first set of information (e.g., Fig. 10B 1004a to 1004d at (e.g., the complex function block includes both the first set of information and the second set of information) (e.g., weather information, event information, time information, date information and / or exercise information). In some embodiments, updating the display of the first complex function block to include the second set of information includes moving the display position of the first set of information. In some embodiments, after the computer receives an indication that the corresponding input is no longer to be performed, the second set of information is stopped from being displayed. In some embodiments, the display of the complex function block is updated when the first operation and / or the second operation is performed. Updating the display of the first complex function block to include the second set of information in response to receiving an indication that the second corresponding input has been performed provides the user with control over the amount of information displayed by the computer system without displaying additional controls, which provides additional control options without cluttering the user interface.

[0212] In some embodiments, based on determining that the first control virtual object (e.g., 1020) is in focus (the first control virtual object is displayed as visually prominent (e.g., displayed as larger than the second control virtual object, displayed in a different color than the second control virtual object, for example, displayed as having a border)) (e.g., the first control virtual object is a preferred control and / or a primary control), displaying the first virtual object includes displaying the first virtual object (e.g., 1016) adjacent to the first control virtual object (e.g., as described above with respect to Fig. 10D In some embodiments, based on determining that the second control virtual object (e.g., 1024) is in focus (e.g., the second control virtual object is a preferred control and / or a primary control), displaying the first virtual object includes displaying the first virtual object adjacent to the second control virtual object (e.g., the first virtual object is not displayed next to the first control virtual object) (e.g., as described above with respect to Fig. 10DDescribed). In some embodiments, based on determining that the first control virtual object is in focus, the first virtual object is displayed within the boundary of the first control virtual object. In some embodiments, when neither the first virtual control object nor the second virtual control object is in focus, the first virtual object is displayed as not adjacent to both the first virtual control object and the second virtual control object. In some embodiments, when neither the first virtual control object nor the second virtual control object is in focus, the first virtual object is displayed as adjacent to both the first virtual control object and the second virtual control object. When a set of specified conditions are met, displaying the first virtual object next to the first control virtual object or the second control virtual object allows the computer system to automatically indicate to the user which control virtual object is in focus, and the control virtual object performs an operation when a set of conditions are met without further user input. Displaying the first virtual object next to the control virtual object in focus provides the user with visual feedback about which control virtual object is the default / primary option, which provides improved visual feedback and can result in more effective control of the user interface for some users.

[0213] In some embodiments, in response to receiving a request to connect to an external device (e.g., a phone and / or an external computer system separate from the computer system), a corresponding user interface is displayed (e.g., in response to receiving a phone call, a corresponding user interface is displayed). In some embodiments, based on determining that the corresponding input is an input of the second type (e.g., a pinch gesture, a pinch gesture, and / or a rotation gesture) and based on determining that the corresponding input is being performed, initiating a process for performing the first operation includes connecting to an external device (e.g., as described above with respect to Figure 7F In some embodiments, performing the first operation also includes displaying an animation of the first control virtual object expanding from a first size to a second size different from the first size (e.g., activating the first control virtual object). Connecting to an external device in response to an indication that a corresponding input has been performed provides the user with the ability to answer a phone call without displaying additional controls, which provides additional control options without cluttering the user interface.

[0214] In some embodiments, the first control virtual object (e.g., Fig. 8A 806 at ) is displayed at a third position on the corresponding user interface, and the second control virtual object (eg, Fig. 8A808 at ) is displayed at a fourth position (e.g., different from the first position) on the corresponding user interface. In some embodiments, when the first control virtual object is displayed at a third position on the corresponding user interface and the second control virtual object is displayed at a fourth position on the corresponding user interface, the computer system (e.g., 700) receives an indication (e.g., generated by the computer system) (e.g., generated by the wearable device) that a clenching gesture has been performed (e.g., by a user) (e.g., by a hand on which the wearable device is being worn). In some embodiments, in response to receiving an indication that a clenching gesture has been performed, the computer system displays the first control virtual object (e.g., generated by the computer system) at a fifth position (e.g., different from the third position) (e.g., horizontally shifted from the third position) on the corresponding user interface. Figure 8B 806 at the location), and the computer system displays the second control virtual object (e.g., different from the fourth location) at a sixth location (e.g., different from the fourth location) (e.g., horizontally shifted from the third location) on the corresponding user interface. Figure 8B In some embodiments, when displaying the first control virtual object at the fifth position on the corresponding user interface and displaying the second control virtual object at the sixth position on the corresponding user interface, the computer system receives an indication (e.g., generated by the computer system) (e.g., generated by the wearable device) that the computer system has tilted (e.g., tilted relative to the ground) (e.g., tilted upward away from the ground and / or tilted downward toward the ground). In some embodiments, in response to receiving the indication that the computer system has tilted and based on determining that the computer system has tilted in the first direction, the computer system displays the first control object (e.g., tilted relative to the ground) at the third position on the corresponding user interface. Figure 8C 806 at the corresponding user interface) (e.g., displaying the first control virtual object in an animated manner as sliding from the fifth position to the third position) (e.g., maintaining display of the second control object at the sixth position on the corresponding user interface) and based on determining that the computer system has been tilted in a second direction different from the first direction, the computer system displays the second control object at the second position on the corresponding user interface (e.g., displaying the second control object in an animated manner as sliding from the sixth position to the fourth position) (e.g., as described above with respect to FIG. 8A to FIG. 8C Moving the display of the first control object in response to receiving an indication that the computer system has been tilted allows a user to modify the display operation of the computer system without displaying additional controls, which provides additional control options without cluttering the user interface. Moving the display of the first control object in response to receiving an indication that the computer system has been tilted helps the user navigate the user interface without touching the computer system, which can result in more efficient control of the user interface for some users.

[0215] In some embodiments, aspects / operations of methods 900, 1300, and 1500 may be interchanged, replaced, and / or added between these methods. For example, a hand gesture used to activate a virtual object in method 1100 may be used to activate a virtual object displayed as part of method 900. For the sake of brevity, these details are not repeated here.

[0216] FIG. 12A to FIG. 12E An example of display information is shown. Fig.13 is a flow chart of an exemplary method 1300 for displaying information. FIG. 12A to FIG. 12E The user interface in is used to show the following description including Fig.13 The process of the process.

[0217] Fig. 12A The user 702 is shown sitting at a table with the computer system 1200 placed on the table. The user 702 and the computer system 1200 are positioned in a physical environment (e.g., a room in a home). The physical environment includes a sofa 1208a, a picture 1208b, a first individual 1208c1, a second individual 1208c2, a television 1208d, and a table 1208e. The computer system 1200 communicates with one or more cameras (e.g., wired communication and / or wireless communication). The one or more cameras communicating with the computer system 1200 have a field of view such that the sofa 1208a, the picture 1208b, the user 702, the first individual 1208c1, and the second individual 1208c2 are within the field of view of the one or more cameras communicating with the computer system 1200 (e.g., the one or more cameras communicating with the computer system 1200 are integrated into the front of the computer system 1200 (e.g., facing the user 702) and / or integrated into the back of the computer system 1200 (e.g., facing the physical environment)). Fig.14A, computer system 1200 is a tablet device. However, it should be understood that depicting computer system 1200 as a tablet device is merely exemplary. In some embodiments, computer system 1200 is a display generation component of one or more other devices, such as a handheld device (smart phone) and / or an HMD. In an embodiment where computer system 1200 is a display generation component of an HMD, wearable device 706 transmits display instructions to the HMD in response to wearable device 706 detecting one or more hand gestures (e.g., one or more air gestures or inputs on the hardware or touch-sensitive input elements of the HMD). In some embodiments, computer system 1200 includes a transparent display, in which a representation of physical environment 1230 is visible because light can "pass through" the transparent display (e.g., like a pair of glasses). In some embodiments, the sofa 1208a, the drawing 1208b, the user 702, the first individual 1208c1, and the second individual 1208c2 are within the user's viewpoint, such that each of these objects can be visible through the display of the computer system 1200 when the computer system 1200 does not display one or more virtual objects and / or one or more portions of the XR user interface on these objects. For ease of discussion, the following description will be described with respect to the computer system 1200 displaying a representation of the physical environment 1230 based on receiving visual content being captured by one or more cameras in communication with the computer system 1200. FIG. 12C to FIG. 12E .

[0218] exist Fig. 12A At the right hand of user 702 (for example, as shown below Fig. 12B The hand 1212 indicated in FIG. 1 is positioned on the back of the computer system 1200 and within the field of view of one or more rear-facing cameras in communication with the computer system 1200. The right hand of the user 702 includes a wearable device 706 (e.g., as described above with respect to FIG. 1 ). 7A to 7F The wearable device 706 communicates with the computer system 1200 (eg, wireless communication (Bluetooth communication, ultra-wideband communication, near field communication) and / or wired communication). Fig. 12A As shown in , computer system 1200 is a tablet device. In some embodiments, computer system 1200 is one or more other devices, such as a handheld device (e.g., a smart phone), a laptop computer, and / or a head-mounted device. In some embodiments, computer system 1200 includes one or more components of computer system 101.

[0219] like Fig. 12B As shown in FIG. 1 , computer system 1200 displays user interface 1210, which is an XR user interface overlaid on a representation of a physical environment. Fig. 12B, the user interface 1210 includes a tent virtual object 1222a, a fire virtual object 1222b, and a chair virtual object 1222c. In some embodiments, the user interface 1210 is displayed from an immersive perspective (e.g., in response to a detected change in the orientation / position of the computer system 1200, the content included in the user interface 1210 is presented from multiple perspectives). In some embodiments, the computer system 1200 updates the content of the user interface 1210 based on the positioning and / or orientation of the computer system 1200 (e.g., the computer system 1200 displays additional content and / or removes content in response to the computer system 1200 moving in a particular direction). In some embodiments, the user interface 1210 is not overlaid on a representation of the physical environment because the computer system 1200 does not display a representation of the physical environment. In some embodiments, the user interface 1210 includes a transparent display, and the user interface 1210 is overlaid on a portion of the physical environment that passes through the transparent display of the computer system 1200.

[0220] FIG. 12B to FIG. 12E Includes an enlarged representation of hand 1212 and wearable device 706 (e.g., FIG. 12B to FIG. 12E ). An enlarged representation of hand 1212 is shown in FIG. 12B to FIG. 12E 12 is provided as a visual aid to illustrate the position and orientation of the hand 1212 (e.g., the right hand of the user 702) at a particular moment. Fig. 12B As shown in FIG. , the wearable device 706 displays a complex function block 1002 (e.g., as described above with respect to Fig. 10A described).

[0221] like Fig. 12C , the wearable device 706 displays a user interface 1224 indicating that the user 702 is receiving a phone call. In some embodiments, the wearable device 706 communicates with an external device (e.g., a smart phone). In some embodiments, in response to receiving an instruction to display the user interface 1224 from the external device, the wearable device 706 displays the user interface 1224. In some embodiments, the wearable device 706 displays the user interface 1224 in response to receiving an instruction to display the user interface 1224 from the computer system 1200.

[0222] exist Fig. 12CAt , wearable device 706 detects that hand 1212 has performed a clenching gesture (e.g., as indicated by hand 1212). In response to receiving an indication (e.g., from wearable device 706) that hand 1212 has performed a clenching gesture, computer system 1200 ceases displaying a portion of user interface 1210 and displays a representation of physical environment 1230 in place of the portion of user interface 1210 (e.g., the representation of physical environment 1230 "penetrates" the display of user interface 1210). The representation of physical environment 1230 includes image data captured by one or more cameras in communication with computer system 1200. Fig. 12C At this point, the representation of the physical environment 1230 is an actual representation of the physical environment being captured by one or more cameras of the computer system 1200. That is, the representation of the physical environment 1230 is a real-time feed (e.g., and / or a delayed feed) from one or more cameras of the computer system 1200. Fig. 12C As shown in , the representation of the physical environment includes a wearable device 706 in the field of view of one or more cameras of the computer system 1200. In some embodiments, the computer system 1200 displays a corresponding virtual object corresponding to a portion of the physical environment included in the representation of the physical environment 1230. In some embodiments, the corresponding virtual object is a virtual representation included in the representation of the physical environment 1230 (e.g., not an actual representation of the portion of the physical environment). In some embodiments, the corresponding virtual object is one or more graphical objects in the representation of the physical environment 1230 (e.g., an avatar, an avatar of the wearable device 706, an avatar wearing the wearable device 706, text indicating one or more subjects (e.g., objects, people, animals, and / or items) (e.g., the time displayed on the wearable device 706 and / or information about an incoming call)). In some embodiments, the corresponding virtual object is dynamic (e.g., changes as one or more representative objects and / or the physical environment changes). In some embodiments, the corresponding virtual object is static (e.g., does not change as one or more representative objects and / or the physical environment changes). As described above with respect to Fig. 12A As discussed, in some embodiments, computer system 1200 includes a transparent display in which the representation of physical environment 1230 is visible because light can "pass through" the transparent display (e.g., like a pair of glasses). Thus, in some of these embodiments, computer system 1200 does not display a representation of physical environment 1230; instead, the representation of physical environment 1230 is visible through the display of computer system 1200. In some embodiments, computer system 1200 allows the physical environment 1230 to be included in the representation of physical environment 1230 (e.g., as Fig. 12C) is visible in response to receiving an indication that hand 1212 has performed a clenching gesture. Thus, in some embodiments, portions of the physical environment that are permitted to be visible (e.g., in response to receiving an indication that hand 1212 has performed a clenching gesture) are visible through the XR user interface (e.g., user interface 1210) and / or a portion of the XR user interface.

[0223] like Fig. 12C, the computer system 1200 displays a representation of the physical environment 1230 such that the representation of the physical environment 1230 obscures a portion of the chair virtual object 1222c and the fire virtual object 1222b. However, while the representation of the physical environment 1230 is displayed, other portions of the user interface 1210 (e.g., 1220a) are visible on the display of the computer system 1200. Thus, in some embodiments, the computer system 1200 displays the representation of the physical environment 1230 simultaneously with one or more portions of the XR user interface displayed prior to displaying the representation of the physical environment 1230. In some embodiments, the representation of the wearable device 706 included in the representation of the physical environment 1230 includes corresponding information that was most recently displayed by the wearable device 706 (e.g., and is not currently displayed by the wearable device 706). In some embodiments, the corresponding information is displayed within a control user interface portion that is overlaid on the display of the computer system 1200. In some embodiments, the control user interface portion includes one or more selectable virtual objects. In some embodiments, in response to detecting a selection of one or more selectable virtual objects, the computer system 1200 performs one or more corresponding operations (e.g., adjusting the volume of an audio output, launching a camera application, and / or modifying (e.g., pausing or playing) the playback state of a media item). In some embodiments, the computer system 1200 displays a representation of the physical environment 1230 in response to receiving an indication that the wearable device 706 detects that the user 702 is double-gripping (e.g., and / or has performed a double-gripping gesture). In some embodiments, the computer system 1200 displays a representation of the physical environment 1230 in response to the computer system 1200 (e.g., through one or more cameras in communication with the computer system 122) detecting a hand gesture (e.g., a single grip gesture, a double grip gesture, an open gesture, and / or an open gesture) and / or an air gesture. In some embodiments, the representation of the physical environment 1230 includes a virtual representation (e.g., virtual objects) of the wearable device 706 (e.g., and not an actual representation of the wearable device 706), a virtual representation of the hand 1212, and a virtual representation of information displayed by the wearable device 706. In some embodiments, the computer system 1200 stops displaying the virtual representation of the wearable device 706 and maintains display of the virtual representation of the hand 1212 in response to receiving an indication that the wearable device 706 detects that the hand 1212 has performed an air gesture. In some embodiments, the display of the representation of the physical environment 1230 obscures the display of the tent virtual object 1222a, the fire virtual object 1222b, and the chair virtual object 1222c. In some embodiments, the representation of the physical environment 1230 includes a virtual representation of the wearable device 706 on a virtual representation of the wrist of the user 702.

[0224] The location of the representation of the physical environment 1230 on the computer system 1200 is based at least on the location of the wearable device 706 (e.g., as tracked by one or more cameras in communication with the computer system 1200). Fig. 12C At , computer system 1200 (e.g., through one or more cameras in communication with computer system 1200) detects that wearable device 706 is positioned in front of the lower right corner of computer system 1200. Because computer system 1200 detects that wearable device 706 is positioned in front of the lower right corner of computer system 1200, computer system 1200 displays a representation of physical environment 1230 in the lower right corner of the display of computer system 1200. Fig. 12C At , computer system 1200 detects that wearable device 706 has moved so that wearable device 706 is positioned in front of the lower left corner of computer system 1200. In some embodiments, computer system 1200 detects that the wearable device has moved via one or more cameras in communication with computer system 1200. In some embodiments, computer system 1200 detects that the wearable device has moved based on receiving information (e.g., location information) from the wearable device.

[0225] like Fig.12D In response to detecting that the wearable device 706 has moved (e.g., positioned in front of the lower left corner of the computer system 1200), the computer system 1200 moves the representation of the physical environment 1230 to the lower left corner of the display of the computer system 1200 (e.g., from its position in the lower left corner of the display). Fig. 12C 1200). As described above, the location (e.g., position) of the representation of the physical environment 1230 is based at least on the position of the wearable device 706 detected in the physical environment (e.g., by one or more cameras in communication with the computer system 1200). In some embodiments, the computer system 1200 moves the representation of the physical environment 1230 in response to detecting that the computer system 1200 has moved. In some embodiments, the computer system 1200 moves the representation of the physical environment 1230 in response to detecting that the computer system 1200 and the wearable device 706 have moved. In some embodiments, the computer system 1200 does not move the representation of the physical environment 1230 in response to detecting that the computer system 1200 and the wearable device 706 have moved.

[0226] exist Fig.12D At, the representation of the physical environment 1230 includes Fig. 12C The portion of the physical environment included in the representation of the physical environment 1230 is different from the portion of the physical environment. Fig.12DBecause computer system 1200 detects that wearable device 706 is positioned in front of the lower left corner of computer system 1200, the representation of physical environment 1230 includes a portion of the left side of the physical environment (e.g., at Fig. 12A 1200) rather than the right side of the physical environment. In some embodiments, the representation of the physical environment 1230 is a representation of a virtual environment (e.g., such as a home environment inside a virtual world) rather than a representation of the physical environment visible from the viewpoint of the computer system 1200 and / or being captured by one or more cameras in communication with the computer system 1200. In some embodiments, the representation of the virtual environment is displayed on a portion of the user interface 1210 (e.g., using one or more similar techniques as described herein with respect to the representation of the physical environment 1230). In some embodiments, the representation of the virtual environment is an actual representation of the virtual environment. In some embodiments, the representation of the virtual environment is not an actual representation of a virtual embodiment, but includes one or more characteristics and / or information about the virtual environment.

[0227] exist Fig.12D At , computer system 1200 increases the size of the representation of physical environment 1230 so that the representation of physical environment 1230 is larger than Fig. 12C In some embodiments, the computer system 1200 increases (e.g., gradually increases) the size of the representation of the physical environment 1230 over a predetermined time period (e.g., 1 second to 30 seconds). Fig.12D At , the wearable device 706 detects that the hand 1212 has performed a release gesture (e.g., returned to a neutral position). Fig.12D , in response to receiving an indication that hand 1212 has performed an unwind gesture, computer system 1200 continues to display a representation of physical environment 1230. In some embodiments, computer system 1200 stops displaying a representation of physical environment 1230 in response to receiving an indication that hand 1212 has performed an unwind gesture.

[0228] exist Fig.12E At , the wearable device 706 detects that the hand 1212 has performed a clenching gesture. Fig.12E As shown in FIG, in response to receiving an indication that hand 1212 has performed a clenching gesture, computer system 1200 stops displaying a representation of physical environment 1230. It is noteworthy that the user interface 1210 is Fig.12D The portion of the physical environment 1230 in FIG. 1230 (e.g., the chair virtual object 1222c) that represents the obstruction is shown in FIG. Fig.12E is visible in the , because the representation of the physical environment 1230 is no longer displayed in Fig.12EIn some embodiments, computer system 1200 displays a representation of physical environment 1230 and, in response to receiving an indication that wearable device 706 has detected a gesture of the same type (or a different type in other embodiments) (e.g., a clenching gesture), stops displaying the representation of physical environment 1230. In some embodiments, in response to receiving an indication that hand 1212 has performed a clenching gesture, computer system 1200 gradually stops displaying the representation of physical environment 1230. In some embodiments, computer system 700 gradually reduces the size of the representation of physical environment 1230 before stopping displaying the representation of physical environment 1230.

[0229] about FIG. 12A to FIG. 12E See below for additional description of FIG. 12A to FIG. 12E Method 1300 is described.

[0230] Fig.13 is a flow chart of an exemplary method 1300 for displaying information according to some embodiments. In some embodiments, in a process including a display generation component (e.g., Figure 1 , Figure 3 and Figure 4 A computer system (e.g., 1200) (e.g., a smart watch, a smart phone, a desktop computer, a laptop computer, a tablet device, a head-up display unit, a head-mounted display unit, an optical head-mounted display unit, and / or a wearable device) (e.g., a display generating component 120 in the embodiment of the present invention) (e.g., a display controller, a touch-sensitive display system; a display (e.g., integrated and / or connected), a 3D display, a transparent display, a projector, a head-up display, and / or a head-mounted display) (e.g., a smart watch, a smart phone, a desktop computer, a laptop computer, a tablet device, a head-up display unit, a head-mounted display unit, a optical head-mounted display unit, and / or a wearable device) (e.g., Figure 1 In some embodiments, the computer system communicates with an external device (e.g., an external device (e.g., a device separate from the computer system) (e.g., a smart watch), (e.g., a wearable device and / or another device being worn by a user of the computer system (e.g., on the user's body)). In some embodiments, the computer system communicates with one or more input devices (e.g., a touch-sensitive surface). In some embodiments, method 1300 is performed by storing in a non-transitory (or transient) computer-readable storage medium and by one or more processors of a computer system (such as one or more processors 202 of computer system 101) (e.g., Figure 1 Some operations in method 1300 are optionally combined, and / or the order of some operations is optionally changed.

[0231] While displaying, via a display generation component, an extended reality environment (e.g., 1210) including a virtual object (e.g., 1222a to 1222c) that occludes at least a first portion of a physical environment including a wearable device (e.g., 706) (e.g., an external device (e.g., a device separate from the computer system) (e.g., a smartwatch) (e.g., a wearable device and / or another device being worn by a user of the computer system (e.g., on the user's body))), the computer system (e.g., 1200) receives (1302) an indication of a first hand input performed by a hand on which the wearable device is being worn (e.g., via a set of one or more sensors integrated into the wearable device (e.g., a gyroscope, an accelerometer, a heart rate sensor, etc.)) (e.g., as described above with respect to Fig. 12C described), wherein the first hand input includes movement of one or more digits of the hand (e.g., fingers and / or thumb) relative to a part of the hand (e.g., palm and / or wrist) (e.g., clenching the user's hand, snapping one or more of the user's fingers, and / or pinching the user's fingers).

[0232] In response to receiving an indication that a hand input has been performed by a hand on which the wearable device is being worn, the computer system displays (1304) (e.g., displays within an augmented reality environment user interface) information about the wearable device (e.g., 706) via a display generation component (e.g., Fig. 12C and Fig.12D 1230) (e.g., information related to the current state of the external device) (e.g., information related to data received by the external device from other external devices (e.g., as described above with respect to Fig. 12C and Fig.12DIn some embodiments, displaying information includes displaying a depiction of the physical environment in which the computer system is currently located. In some embodiments, in response to displaying information about the wearable device, a portion of the augmented reality environment user interface is stopped from being displayed. In some embodiments, the information is overlaid on top of the augmented reality environment user interface. In some embodiments, displaying information includes displaying a representation of a body part of the user (e.g., the user's wrist). In some embodiments, in response to not receiving an indication that a hand input has been performed, a virtual object including information about the external device is abandoned. In some embodiments, the information about the external device is overlaid on a representation of a portion of the environment. In response to receiving an indication that a hand input has been performed by the hand on which the wearable device is being worn, information about the wearable device is displayed, providing the user with greater control over the computer system by allowing the user to control what information the computer system displays without the user touching the computer system and without the computer system displaying additional controls (which provides additional control options without cluttering the user interface). Providing additional control over the system without cluttering the UI with additional displayed controls enhances the operability of the system and makes the user-system interface more efficient (e.g., by helping the user provide appropriate input and reducing user errors when operating / interacting with the system), which in turn reduces power usage and extends the battery life of the system by enabling the user to use the system more quickly and efficiently. Displaying information about the wearable device provides visual feedback to the user about the state of the computer system (e.g., the computer system has received an indication that hand input has been performed by the hand on which the wearable device is being worn), which provides improved visual feedback and can result in more efficient control of the user interface for some users.

[0233] In some embodiments, displaying information about the wearable device (e.g., 1230) includes ceasing to display at least a portion of a virtual object (e.g., 1222a, 1222b, and / or 1222c) (e.g., displaying information about the wearable device includes "punching through" (e.g., ceasing to display a portion of the extended reality environment and displaying information about the wearable device in place of the portion of the extended reality environment that is no longer visible) the portion of the extended reality environment). Ceasing to display at least a portion of the virtual object while displaying information about the wearable device provides visual feedback to the user by allowing the user to simultaneously view the information about the wearable device while viewing at least a portion of the virtual object, which provides improved visual feedback and can result in more effective control of the user interface for some users.

[0234] In some embodiments, stopping displaying a virtual object (e.g., Fig. 12C 1222a to 1222c) at the first portion of the physical environment is visible (eg, Fig. 12C 1230 at ) (e.g., the portion of the physical environment is no longer obscured by the virtual user interface object) (e.g., displaying information about the wearable device includes displaying a real-time representation of at least a portion of the physical environment). In some embodiments, in response to a change in the position of the wearable device in the physical environment (e.g., the visible portion of the physical environment depends on the position of the wearable device), a first portion of the physical environment that is visible after ceasing to display the portion of the virtual item is updated. Making the first portion of the physical environment visible provides the user with visual feedback about the physical environment surrounding the user, which allows the user to safely navigate within the physical environment, which provides improved visual feedback and can result in more effective control of the user interface for some users. Making the first portion of the physical environment visible in response to receiving an indication that a hand input has been performed allows the user to control when various portions of the physical environment are visible and when various portions of the physical environment are not visible without displaying additional controls, which provides additional control options without cluttering the user interface.

[0235] In some embodiments, before ceasing to display the portion of the virtual object, a second portion of the physical environment that is different from the first portion of the physical environment (e.g., the second portion of the physical environment represents a different area of ​​the physical environment) is not visible, and wherein after ceasing to display the portion of the virtual object, the second portion of the physical environment is not visible (e.g., as described above with respect to FIG. 12C to FIG. 12D In some embodiments, content within the second portion of the physical environment overlaps content within the first portion of the physical environment.

[0236] In some embodiments, while displaying information about a wearable device (e.g., 1230), a computer system (e.g., 1200) receives an indication (e.g., an indication generated by the computer system) of the position of a hand (e.g., 1212) on which the wearable device (e.g., 706) is being worn (e.g., received from an external computer system) (e.g., an indication of the position of the hand on which the wearable device is being worn detected via one or more cameras integrated into the computer system, one or more cameras external to the computer system, and / or one or more sensors integrated into the wearable device) (e.g., the position of the hand on which the wearable device is being worn corresponds to the position of the wearable device). In some embodiments, in response to receiving an indication of the position of a hand (e.g., 1212) on which the wearable device is being worn and based on determining that the position of the hand on which the wearable device is being worn is at a first position (e.g., a first position in a physical environment), a computer system (e.g., 1200) displays information about an external device including displaying information at a second position in the extended reality environment corresponding to the first position (e.g., the first position and the second position have a spatial relationship (e.g., when the first position changes within the physical environment, the second position changes accordingly)), and based on determining that the position of the hand on which the wearable device is being worn is at a third position (e.g., a position in the physical environment different from the first position), the computer system displays information about the external device including displaying information at a fourth position in the extended reality environment corresponding to the third position (e.g., different from the second position) (e.g., as described above with reference to FIG. 1 ). Fig. 12C and Fig.12D As described). (For example, the display of information about the wearable device depends on the position of the wearable device (e.g., the user's hand) in the physical environment). In some embodiments, while the first hand input is being performed, an indication of the position of the hand on which the wearable device is being worn is received. When a set of specified conditions are met, displaying information about the external device at a specific location in the extended reality environment corresponding to the position of the user's hand in the physical environment allows the computer system to automatically display the information at a location of interest to the user in the extended reality environment, and allows the computer system to automatically display the information at a convenient location so that the user can easily view and analyze the information, which performs operations without further user input when a set of conditions are met.

[0237] In some embodiments, based on determining that the position of the hand on which the wearable device is being worn is at a first position (e.g., a first position in the physical world), displaying information (e.g., 1230) includes displaying the information at a second position in the extended reality environment (e.g., the position of the second position in the extended reality environment corresponds to the position of the first position) within a first amount of time (e.g., 1 second, 3 seconds, or 5 seconds). Fig. 12C 1230) (e.g., 0.25 inch diameter, 0.5 inch diameter, or 1 inch diameter) to a second size (e.g., Fig.12D In some embodiments, based on determining that the position of the hand on which the wearable device is being worn is at a third position (e.g., a third position in the physical world), displaying information includes displaying information at a fourth position in the extended reality environment (e.g., the position of the fourth position in the extended reality environment corresponds to the position of the third position) from the first size (e.g., a second size) within a first amount of time. Fig. 12C 1230 at the address) is changed to a second size (for example, Fig.12D In some embodiments, in response to the hand on which the wearable device is being worn moving within the physical environment, information is displayed as changing from a first size to a second size at multiple locations in the extended reality environment. Displaying information as changing from a first size to a second size at locations in the extended reality environment within a certain amount of time based on the location of the wearable device allows the computer system to automatically display information at locations of interest to the user in the extended reality environment, so that the user can easily view and analyze the information, which performs operations when a set of conditions are met without further user input. Displaying information as changing size from a first size to a second size provides visual feedback to the user about the state of the computer system (e.g., the computer system has received an indication that hand input has been performed by the hand on which the wearable device is being worn), which provides improved visual feedback and can result in more effective control of the user interface for some users.

[0238] In some embodiments, the information about the wearable device is a virtual representation of the wearable device (e.g., as described above with respect to Fig. 12C as discussed) (e.g., the information includes an avatar of the user's wrist and a digital representation of the wearable device).

[0239] In some embodiments, the information includes one or more notifications corresponding to the wearable device (e.g., Fig. 12C1224) (e.g., a notification received by the wearable device from an external device) (a notification generated by a third-party application installed on the wearable device, and / or a notification generated by a non-third-party application installed on the wearable device (e.g., an application initially installed on the wearable device by the manufacturer)) (e.g., as described above with respect to Fig. 12C The information may include information about the wearable device's recent notifications (e.g., the information includes notifications recently received by the wearable device) (e.g., the information mimics one or more notifications displayed by the wearable device in a physical environment). Displaying one or more notifications corresponding to the wearable device provides the user with visual feedback about the operating state of the wearable device (e.g., whether the wearable device has recently received a notification), which provides improved visual feedback and may result in more effective control of the user interface for some users.

[0240] In some embodiments, the information (e.g., Fig. 12C and Fig.12D 1230) is displayed on a representation (e.g., a digital representation of the wrist and / or an optical representation of the wrist) of a wrist (e.g., a user's hand) of a hand (e.g., 1212) on which a wearable device (e.g., 706) is being worn. In some embodiments, the information is displayed on a digital representation of the wearable device, and the representation of the wearable device is displayed on the representation of the wrist of the hand. Displaying the information on the representation of the wrist of the hand on which the wearable device is being worn provides the user with visual feedback about the location of the information relative to the location of the user's hand, which provides improved visual feedback and can result in more effective control of the user interface for some users.

[0241] In some embodiments, in response to receiving an indication that a hand input is performed by a hand (e.g., 1212) on which a wearable device (e.g., 706) is worn, a computer system (e.g., 1200) displays a control user interface (e.g., a user interface including a plurality of selectable virtual objects (e.g., selection of a corresponding selectable virtual object in the plurality of selectable virtual objects causes the computer system to perform a corresponding operation (e.g., adjusting the volume of an audio output, launching a camera application, and / or modifying (e.g., pausing or playing) a playback state of a media item), wherein information is displayed within the control user interface (e.g., as described above with respect to Fig. 12CAs described) (e.g., the information is displayed as part of the control user interface) (e.g., the information is displayed simultaneously with multiple selectable virtual objects). In some embodiments, the information is displayed within the boundaries of the control user interface. In some embodiments, the display of the information is dependent on the display of the control user interface (e.g., changing the location of the display of the control user interface will cause the location of the display of the information to change). Displaying the control user interface in response to receiving an indication to perform hand input allows the user to control when the computer system displays the control user interface without displaying additional controls, which provides additional control options without cluttering the user interface. Displaying the information within the control user interface provides the user with additional information about the state of the computer system (e.g., information included in the control user interface that is not displayed as part of the information about the wearable device), which provides improved visual feedback and can result in more effective control of the user interface for some users.

[0242] In some embodiments, the extended reality environment includes a virtual representation of a hand (e.g., 1212) on which the wearable device is being worn (e.g., a virtual avatar representation of the hand of a user on which the wearable device is being worn), and wherein displaying information (e.g., 1230) includes displaying the information on the virtual representation of the hand (e.g., the information is displayed on the virtual representation of the wearable device (e.g., a digital representation of the wearable device), and the virtual representation of the wearable device is displayed on the virtual representation of the hand). In some embodiments, while displaying the information on the virtual representation of the hand, the computer receives (e.g., 1200) an indication (e.g., receives the indication from the wearable device) that a second hand input (e.g., a clenching gesture and / or an unclamping gesture) has been performed (e.g., the second hand input is the same as the first hand input) (e.g., an indication generated by the computer system). In some embodiments, in response to receiving the indication that the second hand input has been performed, the computer system stops displaying information on the virtual representation of the hand on which the wearable device is being worn, and the computer system maintains display of the virtual representation of the hand on which the wearable device is being worn (e.g., as described above with respect to Fig. 12C In some embodiments, while the information is being displayed, the virtual object is not visible, and ceasing to display the information makes the virtual object visible. In some embodiments, maintaining display of the virtual representation of the hand on which the wearable device is being worn includes maintaining display of the virtual representation of the wearable device on the virtual representation of the hand. Ceasing to display information on the virtual representation of the hand on which the wearable device is being worn in response to receiving an indication that a second hand input has been performed provides the user with the ability to control what information the computer system is displaying without displaying additional controls, which provides additional control options without cluttering the user interface.

[0243] In some embodiments, the first hand input includes a double clenched gesture (e.g., as described above with respect to Fig. 12C Described) (e.g., continuous clenching gestures detected (e.g., by the wearable device) (e.g., the first hand input includes two hand clenching gestures in rapid succession to each other (e.g., the second clenching gesture is detected within a predetermined amount of time of the first clenching gesture)). Displaying information about the wearable device in response to receiving an indication that a double clenching gesture has been performed allows a user to control when information about the wearable device is displayed without displaying additional controls, which provides additional control options without cluttering the user interface. Displaying information about the wearable device in response to receiving an indication that a double clenching gesture has been performed reduces the cognitive burden imposed on the user when directing a computer system to display information about the wearable device.

[0244] In some embodiments, while displaying information about the wearable device, the computer system receives an indication that the wearable device has detected a third hand input (e.g., a third hand input performed by the hand on which the wearable device is being worn) (e.g., the wearable device detects the third hand input using one or more sensors integrated into the wearable device) (e.g., the third hand input is detected by one or more cameras in communication with the computer system) (e.g., the third hand input is a hand gesture of the same type as the first hand input) (e.g., the third hand input is a hand gesture of a different type than the first hand input). In some embodiments, in response to receiving an indication that a third hand input is performed by the hand on which the wearable device is being worn, the computer system stops displaying information about the wearable device (e.g., as described above regarding Fig.12E In some embodiments, the virtual object is not visible when the information is displayed, and ceasing to display the information makes the virtual object visible. Ceasing to display information about the wearable device in response to receiving an indication of a third hand input performed by the hand on which the wearable device is being worn provides the user with the ability to control what information the computer system is displaying without displaying additional controls, which provides additional control options without cluttering the user interface.

[0245] In some embodiments, the first hand input is a gesture of the first type (e.g., a clenching gesture), and wherein the third hand input is a gesture of the first type (e.g., as described above with respect to Fig.12EThe present invention also provides a method for displaying information in a computer system that is not described herein (e.g., the first hand input includes the same gesture as the third hand input). Stopping displaying information in response to receiving an indication that the same gesture that caused the information to be initially displayed was performed provides feedback to the user about the state of the computer system (e.g., the computer system detected the same type of input that caused the information to be initially displayed), which provides improved visual feedback and can result in more effective control of the user interface for some users. Stopping displaying information in response to receiving an indication that the user performed the gesture that caused the information to be initially displayed reduces the cognitive burden imposed on the user when the user wishes to stop displaying the information.

[0246] In some embodiments, the first hand input is a second type of gesture (e.g., a clenching gesture), and wherein the third hand input is a third type of gesture that is different from the second type of gesture (e.g., rotation of the wrist of the hand on which the wearable device is being worn) (e.g., as described above with respect to Fig.12E Ceasing to display information in response to receiving an indication that a gesture different from the gesture that caused the information to be initially displayed has been received provides feedback to the user about the state of the computer system (e.g., the computer system detected the same type of input that caused the information to be initially displayed), which provides improved visual feedback and can result in more effective control of the user interface for some users.

[0247] In some embodiments, aspects / operations of methods 900, 1100, and 1500 may be interchanged, replaced, and / or added between these methods. For example, a hand gesture performed in method 1300 to display a representation of a physical environment is optionally used to display a representation of a physical environment in method 1500. For the sake of brevity, these details are not repeated here.

[0248] FIG. 14A to FIG. 14G Examples of manipulating the display of a virtual object according to some embodiments are shown. Fig.15 is a flowchart of an exemplary method 1500 for manipulating the display of a virtual object according to some embodiments. FIG. 14A to FIG. 14G The user interface in is used to illustrate the process described below, including Fig.15 process.

[0249] like Fig.14A As shown in FIG. 1 , the computer system 1200 displays a user interface 1210 (eg, as described above with respect to FIG. 1 ). Fig. 12B The user interface 1210 includes a tent virtual object 1222a, a fire virtual object 1222b, a chair virtual object 1222c, and a hand position virtual object 1436 (eg, using the above description of the virtual object 1436). Fig. 7AThe hand position virtual object 1436 may be displayed using one or more of the techniques discussed in Displaying the Hand Position Virtual Object 736 in FIG. Fig.14A As shown in FIG. 7 , user 702 is wearing wearable device 706 on the right hand of user 702 (e.g., as described above with respect to FIG. 7 ). Fig. 7A As described above Fig. 12A As discussed, the computer system 1200 is in communication with one or more cameras. The one or more cameras are front-facing (e.g., facing toward the user 702) and / or rear-facing (e.g., facing away from the user 702). The one or more cameras in communication with the computer system 1200 can track the position of the hands (e.g., both the right and left hands) of the user 702. Fig.14A , computer system 1200 is a tablet device. However, it should be understood that depicting computer system 1200 as a tablet device is merely exemplary. In some embodiments, computer system 1200 is one or more other devices, such as a handheld device (smart phone) and / or a display generation component of an HMD. In an embodiment where computer system 1200 is a display generation component of an HMD, wearable device 706 transmits display instructions to the HMD in response to wearable device 706 detecting one or more hand gestures (e.g., one or more air gestures or inputs on the hardware or touch-sensitive input element of the HMD). In some embodiments, one or more cameras communicating with computer system 1200 only detect the positioning of the dominant hand of user 702 (e.g., the hand of user 702 not wearing wearable device 706). In some embodiments, one or more cameras communicating with computer system 1200 are located on the front of computer system 1200 or the back of computer system 1200.

[0250] exist Fig. 14B At , the right hand of user 702 is detected as being released, and the left hand of user 702 is detected as performing an air gesture, such as an air tap pointing to a point in space corresponding to the display of one of virtual objects 1222a to 1222c (e.g., as described above). In some embodiments, the wearable device 706 detects (e.g., via one or more sensors integrated into the wearable device 706) that the right hand of user 702 is released. In some embodiments, the computer system 700 detects (e.g., via one or more cameras in communication with the computer system 700) that the right hand of user 702 is released. In some embodiments, the computer system 700 detects (e.g., from the wearable device 706) that the right hand of user 702 is released, and at the same time detects that the left hand of user 702 is detected as performing an air gesture.

[0251] like Fig. 14BAs shown in , in response to receiving an indication that the right hand of user 702 is detected as being released and the left hand of user 702 is detected as performing an air gesture (e.g., generated by computer system 1200 and / or generated by wearable device 706), computer system 1200 displays fire virtual object 1222b as being selected (e.g., computer system 1200 displays a box around fire virtual object 1222b). Computer system 1200 performs an operation (e.g., selects a virtual object) in response to receiving an indication that the right hand of user 702 is detected as being released and the left hand of user 702 is detected as performing an air gesture. In some embodiments, when computer system 1200 displays fire virtual object 1222b as being selected, computer system 1200 stops displaying tent virtual object 1222a and chair virtual object 1222c. In some embodiments, computer system 1200 displays multiple virtual objects being selected. In some embodiments, when the computer system 1200 displays the fire virtual object 1222b as being selected, the computer system 1200 manipulates the display of the fire virtual object 1222b (e.g., increasing the size of the fire virtual object 1222b, reducing the size of the fire virtual object 1222b, rotating the display of the fire virtual object 1222b (e.g., clockwise and / or counterclockwise)) in response to receiving an indication that the wearable device 706 has detected one or more hand gestures and / or air gestures. In some embodiments, when the computer system 1200 displays the fire virtual object 1222b as being selected, the computer system 1200 cannot manipulate the display of the tent virtual object 1222a and the chair virtual object 1222c. In some embodiments, the computer system 1200 displays the fire virtual object 1222b as being selected in response to receiving an indication that the user's 702 gaze is directed toward the display of the fire virtual object 1222b and the user's 702 left hand has performed an air gesture (e.g., as detected by one or more cameras in communication with the computer system 1200 and / or by the wearable device 706).

[0252] like Fig. 14B As shown in FIG. 1 , the computer system 1200 displays a hand position virtual object 1436 (e.g., as described above with respect to FIG. 1 ). Fig. 7A Because the computer system 1200 receives an indication that the user's right hand is released, the computer system 1200 displays the hand position virtual object 1436 as representing the released hand. Fig. 14CAt , the right hand of user 702 is detected as being released, and the left hand of user 702 is detected as having moved to the left (e.g., so that the left hand of user 702 is pointing to the area on the computer system 1200 to the left of the chair virtual object 1222c). In some embodiments, the wearable device 706 (e.g., via one or more sensors integrated into the wearable device 706) detects that the right hand of user 702 is released. In some embodiments, the computer system 700 (e.g., via one or more cameras in communicatio...

Claims

1. A method, comprising: At a computer system in communication with a display generating component: Displaying a corresponding user interface via the display generation component, wherein displaying the corresponding user interface includes simultaneously displaying: a first control virtual object that, when activated with a first type of input, causes the computer system to perform a first operation; a second control virtual object that, when activated with the first type of input, causes the computer system to perform a second operation different from the first operation; as well as a first virtual object, the first virtual object indicating that the first control virtual object can be activated in response to performing a second type of input, wherein the second type of input is not directed to a location in the corresponding user interface; While displaying the first control virtual object, receiving an indication that a corresponding input has been performed; as well as In response to receiving an indication that the corresponding input has been executed: Initiating a process for performing the first operation based on determining that the corresponding input is the first type of input directed to a location corresponding to the first control virtual object; Initiating a process for performing the second operation based on determining that the corresponding input is the first type of input directed to a position corresponding to the second control virtual object; as well as Based on determining that the corresponding input is the second type of input, a process for performing the first operation is initiated.

2. The method according to claim 1, further comprising: While displaying the corresponding user interface, an animation is displayed that alternates between displaying the first virtual object and displaying a second virtual object, wherein the second virtual object indicates that the second control virtual object can be activated in response to a third type of input, and wherein the second type of input is different from the third type of input. 3 . The method of claim 2 , wherein alternating between display of the first virtual object and display of the second virtual object is performed based on time. 4 . The method of claim 2 , wherein alternating between display of the first virtual object and display of the second virtual object is performed based on detecting a first input.

5. The method according to any one of claims 1 to 4, wherein displaying the first virtual object includes displaying a second animation, and the second animation alternates between a first position corresponding to the first control virtual object and a second position corresponding to the second control virtual object.

6. The method according to any one of claims 1 to 4, further comprising: While displaying the corresponding user interface, a third virtual object is displayed via the display generation component, the third virtual object indicating that the second operation can be activated in response to a fourth type of input being performed, wherein the fourth type of input does not point to a location in the corresponding user interface.

7. The method according to any one of claims 1 to 4, wherein the first control virtual object corresponds to a first default control option.

8. The method according to any one of claims 1 to 4, wherein displaying the first virtual object includes visually highlighting the first control virtual object.

9. The method according to any one of claims 1 to 4, wherein the corresponding user interface corresponds to a first application, and wherein the first control virtual object is a second default virtual object and is displayed in a first color scheme, the method further comprising: receiving, after receiving an indication that the corresponding input has been executed, a request to display a second corresponding user interface; as well as In response to receiving the request to display the second corresponding user interface, the second corresponding user interface is displayed via the display generating component, wherein the second corresponding user interface corresponds to a second application different from the first application, and wherein displaying the second corresponding user interface includes displaying a third control virtual object, wherein the third control virtual object is the second default virtual object and is displayed in a second color scheme different from the first color scheme.

10. The method of any one of claims 1 to 4, wherein displaying the first virtual object comprises displaying a graphical representation of the second type of input.

11. The method of any one of claims 1 to 4, wherein displaying the first virtual object comprises displaying an animation representing the second type of input.

12. The method according to any one of claims 1 to 4, wherein the first virtual object is displayed adjacent to the first control virtual object.

13. The method of any one of claims 1 to 4, wherein the first type of input is a tap.

14. The method according to any one of claims 1 to 4, wherein displaying the corresponding user interface comprises displaying a first complex function block having a first set of information, the method further comprising: After receiving the indication that the corresponding input has been performed, receiving an indication that a second corresponding input has been performed; as well as In response to receiving an indication that the second corresponding input has been performed, the display of the first complex function block is updated to include a second set of information different from the first set of information.

15. The method according to any one of claims 1 to 4, wherein: Based on determining that the first control virtual object is in focus, displaying the first virtual object includes displaying the first virtual object adjacent to the first control virtual object; and Based on determining that the second control virtual object is in focus, displaying the first virtual object includes displaying the first virtual object adjacent to the second control virtual object.

16. The method according to any one of claims 1 to 4, wherein the corresponding user interface is displayed in response to receiving a request to connect to an external device, wherein: Based on determining that the corresponding input is the second type of input and based on determining that the corresponding input is being performed, initiating a process for performing the first operation includes connecting to the external device.

17. The method according to any one of claims 1 to 4, wherein the first control virtual object is displayed at a third position on the corresponding user interface, and the second control virtual object is displayed at a fourth position on the corresponding user interface, the method further comprising: receiving an indication that a squeeze gesture has been performed while displaying the first control virtual object at the third position on the corresponding user interface and displaying the second control virtual object at the fourth position on the corresponding user interface; In response to receiving an indication that the squeeze gesture has been performed, displaying the first control virtual object at a fifth position on the corresponding user interface and displaying the second control virtual object at a sixth position on the corresponding user interface; as well as While displaying the first control virtual object at the fifth position on the corresponding user interface and while displaying the second control virtual object at the sixth position on the corresponding user interface, receiving an indication that the computer system has been tilted: displaying the first control object at the third position on the corresponding user interface based on determining that the computer system has been tilted in the first direction; as well as Based on determining that the computer system has been tilted in a second direction different from the first direction, the second control object is displayed at the second position on the corresponding user interface.

18. A computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a computer system in communication with a display generator, the one or more programs comprising instructions for performing the method according to any one of claims 1 to 17.

19. A computer system configured to communicate with a display generation component, the computer system comprising: one or more processors; as well as A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to any one of claims 1 to 17.

20. A computer system configured to communicate with a display generation component, the computer system comprising: A component for carrying out the method according to any one of claims 1 to 17.

21. A computer program product comprising one or more programs configured to be executed by one or more processors of a computer system in communication with a display generating component, the one or more programs comprising instructions for performing the method according to any one of claims 1 to 17.

Images