DEVICE, METHOD AND GRAPHICAL USER INTERFACE FOR INTERACTING WITH A THREE-DIMENSIONAL ENVIRONMENT - Patent application

Abstract

To present an improved method and a computer system with an interface for presenting, to a user, a computer-generated experience which makes interaction with a computer system more efficient and more intuitive for a user.SOLUTION: While a view of a three-dimensional environment is displayed, a computer system detects movement of a user's thumb over the user's index finger by using a camera. In accordance with determination that the movement is a swipe operation of the thumb on the index finger in a first direction, the computer system executes a first operation. In accordance with determination that the movement is a tapping operation of the thumb on the index finger at a first position of the index finger, the computer system carries out a second operation different from the first operation.SELECTED DRAWING: Figure 7B

Description

[Technical field]

[0001] (Related Applications) This application claims priority to U.S. Provisional Application No. 62 / 907,480, filed September 27, 2019, and U.S. Application No. 17 / 030,200, filed September 23, 2020, and is a continuation of U.S. Application No. 17 / 030,200, filed September 23, 2020.

[0002] The present disclosure generally relates to computer systems having a display generation component, including but not limited to electronic devices that provide virtual reality and mixed reality experiences via a display, and one or more input devices that provide a computer-generated experience. [Background technology]

[0003] The development of computer systems for augmented reality has progressed significantly in recent years. Exemplary augmented reality environments include at least some virtual elements that replace or augment the physical world. Input devices such as cameras, controllers, joysticks, touch-sensitive surfaces, and touch screen displays for computer systems and other electronic computing devices are used to interact with the virtual / augmented reality environment. Exemplary virtual elements include digital images, videos, text, icons, and virtual objects such as buttons and other graphics.

[0004] However, methods and interfaces for interacting with environments (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) that include at least some virtual elements are cumbersome, inefficient, and limited. For example, systems that provide insufficient feedback to perform actions associated with virtual objects, systems that require a series of inputs to achieve a desired result in an augmented reality environment, and systems in which manipulation of virtual objects is complex and error-prone cause significant cognitive burden to users and detract from the experience of the virtual / augmented reality environment. In addition, the methods are unnecessarily time-consuming, thereby wasting energy. The latter problem is particularly acute in battery-operated devices. Summary of the Invention

[0005] Thus, there is a need for a computer system having improved methods and interfaces for providing a user with a computer-generated experience that makes interaction with the computer system more efficient and intuitive for the user. Such methods and interfaces can complement or replace conventional methods of providing a user with a computer-generated reality experience. Such methods and interfaces reduce the number, extent and / or type of inputs from the user by helping the user understand the connection between the inputs provided and the device response to that input, thereby creating a more efficient human-machine interface.

[0006] The above deficiencies and other problems associated with user interfaces for computer systems having a display generating component and one or more input devices are reduced or eliminated by the disclosed system. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is a portable device (e.g., a notebook computer, a tablet computer, or a handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device such as a watch or a head-mounted device). In some embodiments, the computer system has a touchpad. In some embodiments, the computer system has one or more cameras. In some embodiments, the computer system has a touch-sensitive display (also known as a "touch screen" or "touch screen display"). In some embodiments, the computer system has one or more eye tracking components. In some embodiments, the computer system has one or more hand tracking components. In some embodiments, the computer system has one or more output devices in addition to the display generating component, the output devices including one or more tactile output generators and one or more audio output devices. In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or sets of instructions stored in the memory for performing a number of functions. In some embodiments, a user interacts with the GUI through stylus and / or finger contacts and gestures on a touch-sensitive surface, movements of the user's eyes and hands in space relative to the GUI or the user's body as captured by cameras and other movement sensors, and voice input as captured by one or more audio input devices.In some embodiments, the functions performed through the interactions optionally include image editing, drawing, presenting, word processing, creating spreadsheets, playing games, making phone calls, video conferencing, emailing, instant messaging, training support, digital photography, digital videography, web browsing, playing digital music, taking notes, and / or playing digital videos. Executable instructions to perform those functions are optionally contained on a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0007] There is a need for electronic devices with improved methods and interfaces for interacting with a three-dimensional environment. Such methods and interfaces can complement or replace conventional methods for interacting with a three-dimensional environment. Such methods and interfaces reduce the number, extent, and / or type of input from a user, creating a more efficient human-machine interface.

[0008] In some embodiments, the method is performed in a computer system including a display generation component and one or more cameras, the method comprising: The method includes displaying a view of a three-dimensional environment; detecting, using one or more cameras, movement of a user's thumb on a user's index finger of a first hand of the user while displaying the view of the three-dimensional environment; and in response to detecting, using the one or more cameras, movement of the user's thumb on the user's index finger, performing a first action in accordance with a determination that the movement is a swipe of the thumb on the index finger of the first hand in a first direction; and performing a second action, different from the first action, in accordance with a determination that the movement is a tap of the thumb on the index finger at a first location on the index finger of the first hand.

[0009] In some embodiments, the method is performed in a computing system including a display generation component and one or more input devices, the method comprising: The method includes displaying a view of a three-dimensional environment; detecting a hand at a first position corresponding to a portion of the three-dimensional environment while displaying the three-dimensional environment; and in response to detecting the hand at the first position corresponding to the portion of the three-dimensional environment, displaying a visual indication of a first operational context for gesture input using a hand gesture within the three-dimensional environment in accordance with a determination that the hand is maintained in a first predetermined configuration; and not displaying the visual indication of the first operational context for gesture input using a hand gesture within the three-dimensional environment in accordance with a determination that the hand is not maintained in the first predetermined configuration.

[0010] In some embodiments, the method is performed in a computer system including a display generation component and one or more input devices, the method comprising: The method includes displaying a three-dimensional environment including displaying a representation of the physical environment; detecting a gesture while displaying the representation of the physical environment; and in response to detecting the gesture, displaying a system user interface in the three-dimensional environment in accordance with a determination that the user's gaze is directed toward a position corresponding to a predetermined physical location in the physical environment; and performing an operation in a current context of the three-dimensional environment without displaying the system user interface in accordance with a determination that the user's gaze is not directed toward a position corresponding to the predetermined physical location in the physical environment.

[0011] In some embodiments, the method is performed in an electronic device that includes a display generation component and one or more input devices, the method comprising: The method includes displaying a three-dimensional environment including one or more virtual objects; detecting a gaze directed toward a first object in the three-dimensional environment, where the gaze meets a first criterion and the first object responds to at least one gesture input; and in response to detecting the gaze directed toward the first object that meets the first criterion and responds to the at least one gesture input, displaying an indication of one or more interaction options available for the first object in the three-dimensional environment in accordance with a determination that the hand is in a predetermined ready state for providing gesture input; and not displaying an indication of the one or more interaction options available for the first object in accordance with a determination that the hand is not in a predetermined ready state for providing gesture input.

[0012] There is a need for electronic devices with improved methods and interfaces for facilitating the use of electronic devices by users interacting with three-dimensional environments. Such methods and interfaces can complement or replace conventional methods for facilitating the use of electronic devices by users interacting with three-dimensional environments. Such methods and interfaces can create a more efficient human-machine interface, allowing users more control over the device, and allowing users to use the device in a safer, less cognitively intensive, and improved user experience.

[0013] In some embodiments, a method is executed on a computer system including a display generating component and one or more input devices and includes detecting placement of the display generating component at a predetermined position relative to a user of the electronic device; in response to detecting placement of the display generating component at a predetermined position relative to a user of the computer system, displaying, through the display generating component, a first view of a three-dimensional environment including a pass-through portion, the pass-through portion including a representation of at least a portion of the real world surrounding the user; detecting a change in hand grip on a housing physically coupled to the display generating component while displaying the first view of the three-dimensional environment including the pass-through portion; in response to detecting the change in hand grip on the housing physically coupled to the display generating component, in accordance with a determination that the change in hand grip on the housing physically coupled to the display generating component satisfies a first criterion, replacing the first view of the three-dimensional environment with a second view of the three-dimensional environment, the second view replacing at least a portion of the pass-through portion with virtual content.

[0014] In some embodiments, a method is executed on a computer system including a display generation component and one or more input devices, comprising: displaying, via the display generation component, a view of the virtual environment; detecting a first movement of the user within the physical environment while displaying the view of the virtual environment, and while the view of the virtual environment does not include a visual representation of a first portion of a first physical object present in the physical environment in which the user is located; and in response to detecting the first movement of the user within the physical environment, detecting a range that may be visible to the user based on the user's field of view with respect to the virtual environment, and detecting a range that may be visible to the user based on the user's field of view with respect to the first physical object. In accordance with a determination that the user is within a threshold distance of the first portion, altering the appearance of the view of the virtual environment in a first manner that indicates a physical characteristic of the first portion of the first physical object, without altering the appearance of the view of the virtual environment to indicate a second portion of the first physical object that is part of a range of the first physical object that may be visible to the user based on the user's field of view of the virtual environment; and in accordance with a determination that the user is not within a threshold distance of a first physical object present in the physical environment surrounding the user, not altering the appearance of the view of the virtual environment in the first manner that indicates a physical characteristic of the first portion of the first physical object.

[0015] According to some embodiments, a computer system includes a display generation component (e.g., a display, projector, head mounted display, etc.), one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors for detecting intensity of contact with the touch-sensitive surface), optionally one or more haptic output generators, one or more processors, and a memory storing one or more programs, the one or more programs configured to be executed by the one or more processors, the one or more programs including instructions to perform or cause to be performed any of the operations of the methods described herein. According to some embodiments, a non-transitory computer readable storage medium has instructions stored therein that, when executed by a computer system having a display generation component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors for detecting intensity of contact with the touch-sensitive surface), and optionally one or more tactile output generators, cause the device to perform or cause to be performed any of the operations of the methods described herein. According to some embodiments, a graphical user interface of a computer system having a display generation component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), optionally one or more tactile output generators, a memory, and one or more processors executing one or more programs stored in the memory includes one or more of the elements displayed in any of the methods described herein, which elements are updated in response to input as described in any of the methods described herein. According to some embodiments, a computer system includes a display generation component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), optionally one or more tactile output generators, and means for performing or causing to be performed the operations of any of the methods described herein.According to some embodiments, an information processing apparatus for use in a computer system having a display generation component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors that detect the intensity of contact with the touch-sensitive surface), and optionally one or more tactile output generators, includes means for performing or causing to be performed any of the operations of the methods described herein.

[0016] Thus, computer systems having display generating components are provided with improved methods and interfaces for interacting with three-dimensional environments and facilitating a user's use of the computer systems when interacting with the three-dimensional environments, thereby increasing the effectiveness, efficiency, and safety and satisfaction of such computer systems. Such methods and interfaces may complement or replace conventional methods for interacting with three-dimensional environments and facilitating a user's use of the computer systems when interacting with the three-dimensional environments.

[0017] It should be noted that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described herein are not exhaustive, and many additional features and advantages will be apparent to those skilled in the art, especially in view of the drawings, the specification, and the claims. Furthermore, it should be noted that the language used in this specification has been selected solely for the purposes of readability and description, and not to define or limit the subject matter of the present invention. [Brief description of the drawings]

[0018] For a better understanding of the various described embodiments, reference should be made to the following Detailed Description in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout:

[0019] [Figure 1]FIG. 1 is a block diagram illustrating a computer system operating environment for providing a CGR experience, according to some embodiments.

[0020] [Diagram 2] FIG. 2 is a block diagram illustrating a controller of a computer system configured to manage and coordinate a user's CGR experience, according to some embodiments.

[0021] [Diagram 3] FIG. 1 is a block diagram illustrating display generation components of a computer system configured to provide a visual component of a CGR experience to a user, according to some embodiments.

[0022] [Figure 4] FIG. 1 is a block diagram illustrating a hand tracking unit of a computer system configured to capture a user's gesture input, according to some embodiments.

[0023] [Diagram 5] FIG. 1 is a block diagram illustrating an eye-tracking unit of a computer system configured to capture a user's gaze input, according to some embodiments.

[0024] [Figure 6] 1 is a flowchart illustrating a glint-assisted gaze tracking pipeline, according to some embodiments.

[0025] [Figure 7A] 1 is a block diagram illustrating user interaction with a three-dimensional environment, according to some embodiments. [Figure 7B] 1 is a block diagram illustrating user interaction with a three-dimensional environment, according to some embodiments. [Figure 7C] 1 is a block diagram illustrating user interaction with a three-dimensional environment, according to some embodiments. [Figure 7D]1 is a block diagram illustrating user interaction with a three-dimensional environment, according to some embodiments. [Figure 7E] 1 is a block diagram illustrating user interaction with a three-dimensional environment, according to some embodiments. [Figure 7F] 1 is a block diagram illustrating user interaction with a three-dimensional environment, according to some embodiments. [Figure 7G] 1 is a block diagram illustrating user interaction with a three-dimensional environment, according to some embodiments. [Figure 7H] 1 is a block diagram illustrating user interaction with a three-dimensional environment, according to some embodiments. [Figure 7I] 1 is a block diagram illustrating user interaction with a three-dimensional environment, according to some embodiments. [Figure 7J] 1 is a block diagram illustrating user interaction with a three-dimensional environment, according to some embodiments.

[0026] [Figure 7K] FIG. 1 is a block diagram illustrating a method for facilitating a user's use of a device in a physical environment to interact with a computer-generated three-dimensional environment, according to some embodiments. [Figure 7L] FIG. 1 is a block diagram illustrating a method for facilitating a user's use of a device in a physical environment to interact with a computer-generated three-dimensional environment, according to some embodiments. [Figure 7M] FIG. 1 is a block diagram illustrating a method for facilitating a user's use of a device in a physical environment to interact with a computer-generated three-dimensional environment, according to some embodiments. [Figure 7N] FIG. 1 is a block diagram illustrating a method for facilitating a user's use of a device in a physical environment to interact with a computer-generated three-dimensional environment, according to some embodiments. [Figure 7O] FIG. 1 is a block diagram illustrating a method for facilitating a user's use of a device in a physical environment to interact with a computer-generated three-dimensional environment, according to some embodiments. [Figure 7P]1 is a block diagram illustrating a method for facilitating a user's use of a device in a physical environment to interact with a computer-generated three-dimensional environment, according to some embodiments.

[0027] [Figure 8] 1 is a flowchart of a method for interacting with a three-dimensional environment, according to some embodiments.

[0028] [Figure 9] 1 is a flowchart of a method for interacting with a three-dimensional environment, according to some embodiments.

[0029] [Figure 10] 1 is a flowchart of a method for interacting with a three-dimensional environment, according to some embodiments.

[0030] [Figure 11] 1 is a flowchart of a method for interacting with a three-dimensional environment, according to some embodiments.

[0031] [Figure 12] 1 is a flowchart of a method for facilitating a user's transition within a three-dimensional environment, according to some embodiments.

[0032] [Figure 13] 1 is a flowchart of a method for facilitating a user's transition within a three-dimensional environment, according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] The present disclosure relates to a user interface that provides a computer-generated reality (CGR) experience to a user, according to some embodiments.

[0034] The systems, methods, and GUIs described herein improve user interface interaction with virtual / augmented reality environments in several ways.

[0035] In some embodiments, the computer system allows a user to interact with a three-dimensional environment (e.g., a virtual or mixed reality environment) using microgestures performed with small movements of fingers relative to other fingers or parts of the same hand. The microgestures are detected using a camera (e.g., integrated with a head-mounted device or installed remotely from the user (e.g., in a CGR room)) as opposed to, for example, a touch-sensitive surface or other physical controller. Different movements and positions of the microgestures and various movement parameters are used to determine actions to be performed in the three-dimensional environment. Using a camera to capture microgestures and interact with the three-dimensional environment allows the user to move freely around the physical environment without being hindered by physical input devices, thereby allowing the user to explore the three-dimensional environment more naturally and efficiently. Furthermore, microgestures are discrete, non-obtrusive, and suitable for interactions that can be made in public and / or require politeness.

[0036] In some embodiments, a hand ready state configuration is defined. The additional requirement that the hands are detected at a position corresponding to a portion of the displayed three-dimensional environment ensures that the hand ready state configuration is not erroneously recognized by the computer system. The hand ready state configuration is used by the computer system as an indication that the user intends to interact with the computer system in a predefined operational context that is different from the currently displayed operational context. For example, the predefined operational context is one or more interactions with the device outside the currently displayed application (e.g., a game, a communication session, a media playback session, navigation, etc.). The predefined operational context is optionally a system interaction, such as, for example, displaying a home or start user interface from which other experiences and / or applications can be launched, a multitasking user interface from which recently displayed experiences and / or applications can be selected and resumed, a control user interface for adjusting one or more device parameters of the computer system (e.g., display brightness, audio volume, network connection, etc.). By using a special hand gesture to trigger the display of a visual indication of a predefined action context for a gesture input that is different from the currently displayed action context, a user can easily access a predefined action context without cluttering the three-dimensional environment with visual controls and without inadvertently triggering an interaction in the predefined action context.

[0037] In some embodiments, a physical object or portion thereof (e.g., a user's hand or a hardware device) is selected by a user or a computer system to be associated with a system user interface (e.g., a control user interface for a device) that is not currently displayed in the three-dimensional environment (e.g., a mixed reality environment). When the user's gaze is directed to a location in the three-dimensional environment other than a location corresponding to the given physical object or portion thereof, a gesture performed by the user's hand causes an operation to be performed in the currently displayed context without causing the system user interface to be displayed, and when the user's gaze is directed to a location in the three-dimensional environment corresponding to the given physical object or an option thereof, a gesture performed by the user's hand causes the system user interface to be displayed. By selectively performing an action in a currently displayed operational context or displaying a system user interface in response to an input gesture based on whether the user's gaze is directed at a given physical object (e.g., the user's hand performing a gesture, or a physical object the user is attempting to control using the gesture), a user can efficiently interact with a three-dimensional environment in two or more contexts without visually cluttering the three-dimensional environment with multiple controls, improving the interaction efficiency of the user interface (e.g., reducing the number of inputs required to achieve a desired result).

[0038] In some embodiments, a user's gaze directed at a virtual object in a three-dimensional environment responsive to a gesture input causes a visual indication of one or more available interaction options for the virtual object to be displayed only if the user's hands are found to be in a predetermined ready state to provide the gesture input. If the user's hands are not found in a ready state to provide the gesture input, the user's gaze directed at the virtual object does not trigger the display of a visual indication. Using a combination of the user's gaze and the ready state of the user's hands to determine whether to display a visual indication of whether the virtual object has an associated interaction option for the gesture input provides the user with useful feedback as he explores the three-dimensional environment with his eyes, reducing the user's confusion when exploring the three-dimensional environment, without unnecessarily exposing the user to constant changes as the user shifts his gaze around the three-dimensional environment.

[0039] In some embodiments, when a user places the display generating component of the computer system in a predetermined position relative to the user (e.g., placing a display in front of their eyes or a head-mounted device on their head), the user's view of the real world is occluded by the display generating component and the content presented by the display generating component dominates the user's view. Sometimes users benefit from a more gradual and controlled process for transitioning from the real world to a computer-generated experience. Thus, when displaying content to the user through the display generating component, the computer system displays a pass-through portion including a representation of at least a portion of the real world surrounding the user, and replaces and displays at least a portion of the pass-through portion with virtual content only in response to detecting a change in the grip of the user's hand on the housing of the display generating component. The change in the grip of the user's hand is used as an indication that the user is ready to transition to a more immersive experience than is currently presented through the display generating component. A gradual transition of the immersive environment as controlled by a change in the grip of the user's hand on the housing of the display generating component is intuitive and natural to the user, improving the user's experience and comfort when using the computer system for computer-generated immersive experiences.

[0040] In some embodiments, when the computer system displays the virtual three-dimensional environment, the computer system applies visual changes to portions of the virtual environment at locations that correspond to the locations of portions of the physical objects that fall within the user's threshold distance and that may be within the user's field of view for the virtual environment (e.g., the portions of the physical objects are visible to the user, but the presence of the display generating component blocks the user's view of the real world around the user). Furthermore, instead of simply presenting all portions of the physical objects that may be within the field of view, the portions of the physical objects that are not within the user's threshold distance are not visually presented to the user (e.g., by changing the appearance of the portions of the virtual environment at locations that correspond to those portions of the physical objects that are not within the user's threshold distance). In some embodiments, the visual changes applied to the portions of the virtual environment cause one or more physical characteristics of the portions of the physical objects that are within the user's threshold distance to be presented in the virtual environment without ceasing to display those portions of the virtual environment entirely or ceasing to provide an immersive virtual experience to the user entirely. This technique allows the user to be alerted to physical obstacles approaching the user as the user moves about the physical environment while exploring the immersive virtual environment, without unduly intruding on and disrupting the user's immersive virtual experience. Therefore, a safer and smoother immersive virtual experience can be provided to the user.

[0041] 1-6 illustrate an exemplary computer system for providing a CGR experience to a user. FIGs. 7A-7G illustrate an exemplary interaction with a three-dimensional environment using gesture input and / or gaze input, according to some embodiments. FIGs. 7K-7M illustrate an exemplary user interface displayed when a user transitions between interacting with a three-dimensional environment, according to some embodiments. FIGs. 7N-7P illustrate an exemplary user interface displayed when a user navigates a physical environment while interacting with a virtual environment, according to some embodiments. FIGs. 8-11 are flowcharts of a method of interacting with a three-dimensional environment, according to various embodiments. The user interfaces of FIGs. 7A-7G are used to explain the process of FIGs. 8-11. FIG. 12 is a flowchart of a method of facilitating a user's use of a computer system for interacting with a three-dimensional environment, according to various embodiments. The user interfaces of FIGs. 7K-7M are used to explain the process of FIGs. 12-13.

[0042] 1, a CGR experience is provided to a user via an operating environment 100 that includes a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), a display generation component 120 (e.g., a head mounted device (HMD), a display, a projector, a touch screen, etc.), one or more input devices 125 (e.g., an eye tracking device 130, a hand tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a tactile output generator 170, and other output devices 180), one or more sensors 190 (e.g., image sensors, light sensors, depth sensors, tactile sensors, orientation sensors, proximity sensors, temperature sensors, position sensors, motion sensors, speed sensors, etc.), and optionally one or more peripheral devices 195 (e.g., consumer electronics, wearable devices, etc.). In some embodiments, one or more of the input device 125, the output device 155, the sensors 190, and the peripheral devices 195 are integrated with the display generation component 120 (eg, within a head-mounted or handheld device).

[0043] When describing a CGR experience, various terms are used to individually refer to several related, but distinct, environments that a user senses and / or that the user can interact with (e.g., using inputs detected by computer system 101 that cause the computer system generating the CGR experience to generate audio, visual, and / or haptic feedback corresponding to various inputs provided to computer system 101 generating the CGR experience). The following are a subset of these terms:

[0044] Physical Environment: The physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic systems. A physical environment, such as a physical park, includes physical objects such as physical trees, physical buildings, and physical people. People can sense and / or interact with the physical environment directly, such as through sight, touch, hearing, taste, and smell.

[0045] Computer-Generated Reality: In contrast, a Computer-Generated Reality (CGR) environment refers to a wholly or partially mimicked environment that people sense and / or interact with via electronic systems. In a CGR, a subset of a person's body movements or representations thereof are tracked, and in response, one or more properties of one or more virtual objects simulated within the CGR environment are adjusted to behave with at least one law of physics. For example, a CGR system may detect a person's head rotation and in response adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. In some circumstances (e.g., for accessibility reasons), adjustments to a property(s) of a virtual object(s) in a CGR environment may be made in response to a representation of a body movement (e.g., a voice command). A person may sense and / or interact with a CGR object using any one of these senses, including vision, hearing, touch, taste, and smell. For example, a person may sense and / or interact with audio objects that create an audio environment with 3D or spatial breadth that provides the perception of a point sound source in 3D space. In another example, audio objects may enable audio transparency that selectively incorporates ambient sounds from the physical environment, with or without computer-generated audio. In some CGR environments, a person may sense and / or interact with only audio objects.

[0046] Examples of CGR include virtual reality and mixed reality.

[0047] Virtual Reality: A virtual reality (VR) environment refers to an emulated environment designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes a number of virtual objects that a person can sense and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence in the computer-generated environment and / or through a simulation of a subset of the person's physical movements in the computer-generated environment.

[0048] Mixed reality: A mixed reality (MR) environment refers to a mimetic environment designed to incorporate sensory input from a physical environment or a representation thereof in addition to including computer-generated sensory input (e.g., virtual objects) as opposed to a VR environment designed to be based entirely on computer-generated sensory input. On the virtual continuum, a mixed reality environment is anywhere between but not including a complete physical environment at one end and a virtual reality environment at the other end. In some MR environments, the computer-generated sensory input may respond to changes in sensory input from the physical environment. Some electronic systems for presenting MR environments may also track position and / or orientation relative to the physical environment to allow virtual objects to interact with real objects (i.e., physical items from the physical environment or their representations). For example, the system may account for movement so that a virtual tree appears stationary relative to the physical ground.

[0049] Examples of mixed reality include augmented reality and augmented virtuality.

[0050] Augmented reality: An augmented reality (AR) environment refers to a mimicked environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or semi-transparent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on the transparent or semi-transparent display, whereby a person uses the system to perceive the virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system composites the images or videos with the virtual objects and presents the composite on the opaque display. A person uses the system to indirectly view the physical environment through the images or videos of the physical environment and perceive the virtual objects superimposed on the physical environment. As used herein, the video of the physical environment shown on the opaque display is referred to as "pass-through video," meaning that the system uses one or more image sensors to capture images of the physical environment and uses those images in presenting the AR environment on the opaque display. Alternatively, the system may have a projection system that projects virtual objects, for example as holograms, into the physical environment or onto a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment. An augmented reality environment also refers to a simulated environment in which a representation of the physical environment is transformed by computer-generated sensory information. For example, in providing pass-through video, the system may distort one or more sensor images to impose a selected perspective (e.g., viewpoint) other than the perspective captured by the image sensor. As another example, the representation of the physical environment may be distorted by graphically altering (e.g., enlarging) a portion thereof, such that the altered portion is a modified version that represents the original captured image but is non-photorealistic. As a further example, the representation of the physical environment may be distorted by graphically removing or obscuring a portion thereof.

[0051] Augmented Virtual: An augmented virtual (AV) environment refers to a mimicking environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. The sensory inputs may be representations of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, while people with faces are realistically recreated from images taken of physical people. As another example, virtual objects may adopt the shape or color of physical items imaged by one or more imaging sensors. As a further example, virtual objects may adopt shadows that match the position of the sun in the physical environment.

[0052] Hardware: There are many different types of electronic systems that allow a person to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed over a person's eyes (e.g., similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have one or more speaker(s) and an integrated opaque display. Alternatively, the head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). The head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. The head-mounted system may have a transparent or translucent display rather than an opaque display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eye. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanned light source, or any combination of these technologies. The medium may be a light guide, a holographic medium, an optical combiner, an optical reflector, or any combination of these. In an embodiment, a transparent or translucent display may be configured to be selectively opaque. A projection-based system may employ retinal projection technology that projects a graphical image onto a person's retina. A projection system may also be configured to project virtual objects into the physical environment, for example, as a hologram or as a physical surface. In some embodiments, the controller 110 is configured to manage and regulate the user's CGR experience.In some embodiments, the controller 110 includes a suitable combination of software, firmware, and / or hardware. The controller 110 is described in more detail below with reference to FIG. 2. In some embodiments, the controller 110 is a computing device that is local or remote to the scene 105 (e.g., physical setting / environment). For example, the controller 110 is a local server located within the scene 105. In another example, the controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside the scene 105. In some embodiments, the controller 110 is communicatively coupled to the display generation component 120 (e.g., an HMD, a display, a projector, a touch screen, etc.) via one or more wired or wireless communication channels 144 (e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is contained within the housing (e.g., physical housing) of one or more of the display generation component 120 (e.g., an HMD or a portable electronic device including a display and one or more processors), one or more of the input devices 125, one or more of the output devices 155, one or more of the sensors 190, and / or one or more of the peripheral devices 195, or shares the same physical housing or support structure as one or more of the above.

[0053] In some embodiments, display generation component 120 is configured to provide a CGR experience (e.g., at least a visual component of a CGR experience) to a user. In some embodiments, display generation component 120 includes a suitable combination of software, firmware, and / or hardware. Display generation component 120 is described in more detail below with reference to FIG. 3. In some embodiments, functionality of controller 110 is provided by and / or combined with display generation component 120.

[0054] According to some embodiments, the display generation component 120 provides a CGR experience to the user while the user is virtually and / or physically present within the scene 105.

[0055] In some embodiments, the display generation component is worn on a part of the user's body (e.g., head, hand, etc.). Thus, the display generation component 120 includes one or more CGR displays provided for displaying CGR content. For example, the display generation component 120 surrounds the user's field of view. In some embodiments, the display generation component 120 is a handheld device (e.g., a smartphone or tablet) configured to present CGR content, where the user holds the device with a display pointed at the user's field of view and a camera pointed at the scene 105. In some embodiments, the handheld device is optionally located in a housing worn on the user's head. In some embodiments, the handheld device is optionally located on a support (e.g., a tripod) in front of the user. In some embodiments, the display generation component 120 is a CGR chamber, housing, or room configured to present CGR content without the user wearing or holding the display generation component 120. Many user interfaces described with reference to one type of hardware for displaying CGR content (e.g., a handheld device or a device on a tripod) may be implemented on another type of hardware for displaying CGR content (e.g., an HMD or other wearable computing device). For example, a user interface showing an interaction with CGR content that is triggered based on an interaction that occurs in the space in front of a handheld or tripod mounted device may be implemented similarly to an HMD where the interaction occurs in the space in front of the HMD and the CGR content response is displayed via the HMD. Similarly, a user interface showing an interaction with CGR content that is triggered based on movement of a handheld or tripod mounted device relative to the physical environment (e.g., the scene 105 or a part of the user's body (e.g., the user's eye(s), head, or hand)) may be implemented similarly to an HMD where the interaction is triggered by movement of the HMD relative to the physical environment (e.g., the scene 105 or a part of the user's body (e.g., the user's eye(s), head, or hand)).

[0056] While relevant features of operating environment 100 are shown in FIG. 1, those skilled in the art will appreciate from this disclosure that various other features have not been shown for purposes of brevity so as not to obscure more pertinent aspects of the exemplary embodiments disclosed herein.

[0057] 2 is a block diagram of an example of a controller 110, according to some embodiments. While certain features are shown, those skilled in the art will appreciate from this disclosure that for the sake of brevity, various other features are not shown so as to not obscure more pertinent aspects of the embodiments disclosed herein. Thus, by way of non-limiting example, in some embodiments, the controller 110 may include one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., a universal serial bus (USB), FIREWIRE®, THUNDERBOLT®, IEEE 802.3x, IEEE 802.11x, IEEE 802.12b, IEEE 802.13b, IEEE 802.14b, IEEE 802.15b, IEEE 802.16b, IEEE 802.17b, IEEE 802.19b, IEEE 802.21b, IEEE 802.22b, IEEE 802.23b, IEEE 802.24b, IEEE 802.25b, IEEE 802.26b, IEEE 802.27b, IEEE 802.28b, IEEE 802.29b, IEEE 802.30b, IEEE 802.31b, IEEE 802.32b, IEEE 802.33b, IEEE 802.34b, IEEE 802.35b, IEEE 802.36b, IEEE 802.37b, IEEE 802.38b, IEEE 802 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), BLUETOOTH®, ZIGBEE®, or similar type interface), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these and various other components.

[0058] In some embodiments, the one or more communications buses 204 include circuitry for interconnecting and controlling communications between the system components. In some embodiments, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0059] Memory 220 includes high-speed random-access memory, such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDRRAM), or other random-access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile storage devices. Memory 220 optionally includes one or more storage devices located remotely from one or more processing units 202. Memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, memory 220, or the non-transitory computer-readable storage medium of memory 220, stores the following programs, modules, and data structures, or a subset thereof, including optional operating system 230 and CGR experience module 240:

[0060] Operating system 230 includes instructions for handling various basic system services and for performing hardware-dependent tasks. In some embodiments, CGR experience module 240 is configured to manage and coordinate one or more CGR experiences for one or more users (e.g., a single CGR experience for one or more users, or multiple CGR experiences for respective groups of one or more users). To that end, in various embodiments, CGR experience module 240 includes a data acquisition unit 242, a tracking unit 244, an adjustment unit 246, and a data transmission unit 248.

[0061] 1 , and optionally one or more of the input devices 125, the output devices 155, the sensors 190, and / or the peripheral devices 195. To that end, in various embodiments, the data acquisition unit 242 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0062] In some embodiments, the tracking unit 244 is configured to map the scene 105 and track the position of at least the display generation component 120 relative to the scene 105 of FIG. 1, and optionally one or more of the input device 125, the output device 155, the sensor 190, and / or the peripheral device 195. To that end, in various embodiments, the tracking unit 244 includes instructions and / or logic therefor, as well as heuristics and metadata therefor. In some embodiments, the processing unit 244 includes a hand tracking unit 243 and / or an eye tracking unit 245. In some embodiments, the hand tracking unit 243 is configured to track the position of one or more parts of the user's hand and / or the movement of one or more parts of the user's hand relative to the scene 105 of FIG. 1, relative to the display generation component 120, and / or relative to a coordinate system defined relative to the user's hand. The hand tracking unit 243 is described in more detail below with reference to FIG. 4. In some embodiments, eye tracking unit 245 is configured to track the position and movement of a user's gaze (or, more broadly, the user's eyes, face, or head) relative to scene 105 (e.g., relative to the physical environment and / or the user (e.g., the user's hands)), or relative to CGR content displayed via display generation component 120. Eye tracking unit 245 is described in more detail below with reference to FIG.

[0063] In some embodiments, the coordination unit 246 is configured to manage and coordinate the CGR experience presented to the user by the display generation component 120, and optionally by one or more of the output devices 155 and / or peripheral devices 195. To that end, in various embodiments, the coordination unit 246 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0064] In some embodiments, data transmission unit 248 is configured to transmit data (e.g., presentation data, location data, etc.) to at least display generation component 120, and optionally to one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To that end, in various embodiments, data transmission unit 248 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0065] Although the data acquisition unit 242, the tracking unit 244 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the adjustment unit 246, and the data transmission unit 248 are shown as being present on a single device (e.g., the controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 242, the tracking unit 244 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the adjustment unit 246, and the data transmission unit 248 may be located within separate computing devices.

[0066] Additionally, FIG. 2 is intended more to illustrate the functionality of various features that may be present in certain embodiments, as opposed to a structural overview of the embodiments described herein. As will be appreciated by those skilled in the art, items shown separately may be combined and some items may be separated. For example, some functional modules shown separately in FIG. 2 may be implemented within a single module, and various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of specific functions and how functions are allocated among them, will vary from embodiment to embodiment, and in some embodiments will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0067] 3 is a block diagram of an example of a display generation component 120, according to some embodiments. While certain features are shown, those skilled in the art will appreciate from this disclosure that various other features have not been shown for the sake of brevity so as not to obscure more pertinent aspects of the embodiments disclosed herein. To that end, and by way of non-limiting example, in some embodiments the HMD 120 includes one or more processing units 302 (e.g., a microprocessor, an ASIC, an FPGA, a GPU, a CPU, a processing core, etc.), one or more input / output (I / O) devices and sensors 306, one or more communications interfaces 308 (e.g., USB, FIREWIRE®, THUNDERBOLT®, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, infrared, BLUETOOTH®, ZIGBEE®, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more CGR displays 312, one or more optional inward-facing and / or outward-facing image sensors 314, memory 320, and one or more communications buses 304 for interconnecting these and various other components.

[0068] In some embodiments, the one or more communication buses 304 include circuitry to interconnect and control communications between the system components. In some embodiments, the one or more I / O devices and sensors 306 include at least one of an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), and the like.

[0069] In some embodiments, the one or more CGR displays 312 are configured to provide a CGR experience to the user. In some embodiments, the one or more CGR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), MEMS, and / or similar display types. In some embodiments, the one or more CGR displays 312 correspond to a waveguide display, such as diffractive, reflective, polarized, holographic, etc. For example, the HMD 120 includes a single CGR display. In another example, the HMD 120 includes a CGR display for each eye of the user. In some embodiments, the one or more CGR displays 312 can present MR or VR content. In some embodiments, the one or more CGR displays 312 can present MR or VR content.

[0070] In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as an eye tracking camera). In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's hand(s) and optionally the user's arm(s) (and may be referred to as a hand tracking camera). In some embodiments, the one or more image sensors 314 are configured to face forward to acquire image data corresponding to a scene viewed by the user when the HMD 120 is not present (and may be referred to as a scene camera). The one or more optional image sensors 314 may include one or more RGB cameras (e.g., with a complementary metal oxide semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and / or the like.

[0071] Memory 320 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile storage devices. Memory 320 optionally includes one or more storage devices located remotely from one or more processing units 302. Memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, memory 320, or the non-transitory computer-readable storage medium of memory 320, stores the following programs, modules, and data structures, or a subset thereof, including optional operating system 330 and CGR presentation module 340:

[0072] The operating system 330 includes procedures for handling various basic system services and for performing hardware-dependent tasks. In some embodiments, the CGR presentation module 340 is configured to present CGR content to a user via one or more CGR displays 312. As such, in various embodiments, the CGR presentation module 340 includes a data acquisition unit 342, a CGR presentation unit 344, a CGR map generation unit 346, and a data transmission unit 348.

[0073] In some embodiments, the data acquisition unit 342 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the controller 110 of Figure 1. To that end, in various embodiments, the data acquisition unit 342 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0074] In some embodiments, the CGR presentation unit 344 is configured to present CGR content via one or more CGR displays 312. To that end, in various embodiments, the CGR presentation unit 344 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0075] In some embodiments, the CGR map generation unit 346 is configured to generate a CGR map (e.g., a 3D map of a mixed reality scene or a map of a physical environment in which computer-generated objects can be placed) based on the media content data. To that end, in various embodiments, the CGR map generation unit 346 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0076] In some embodiments, data transmission unit 348 is configured to transmit data (e.g., presentation data, position data, etc.) to at least controller 110, and optionally to one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To that end, in various embodiments, data transmission unit 348 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0077] Although the data acquisition unit 342, the CGR presentation unit 344, the CGR map generation unit 346, and the data transmission unit 348 are shown as existing on a single device (e.g., the display generation component 120 of FIG. 1), it should be understood that in other embodiments, any combination of the data acquisition unit 342, the CGR presentation unit 344, the CGR map generation unit 346, and the data transmission unit 348 may be located within separate computing devices.

[0078] Moreover, Figure 3 is intended more to illustrate the functionality of various features that may be present in certain embodiments, as opposed to a structural overview of the embodiments described herein. As will be appreciated by those skilled in the art, items shown separately may be combined and some items may be separated. For example, some functional modules shown separately in Figure 3 may be implemented within a single module, and various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of specific functions and how functions are allocated among them, will vary from embodiment to embodiment, and in some embodiments will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0079] 4 is a schematic diagram of an example embodiment of a hand tracking device 140. In some embodiments, the hand tracking device 140 (FIG. 1) is controlled by a hand tracking unit 243 (FIG. 2) to track the position of one or more parts of a user's hand and / or the movement of one or more parts of a user's hand relative to the scene 105 of FIG. 1 (e.g., relative to a portion of the physical environment surrounding the user, relative to the display generation component 120, or relative to a portion of the user (e.g., the user's face, eyes, or head), and / or relative to a coordinate system defined relative to the user's hand). In some embodiments, the hand tracking device 140 is part of the display generation component 120 (e.g., embedded in or attached to a head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0080] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras, etc.) that captures three-dimensional scene information including at least a hand 406 of a human user. The image sensor 404 captures hand images with sufficient resolution to allow for distinguishing fingers and their respective positions. The image sensor 404 typically captures images of other parts of the user's body, or an image of the entire body, and can have either a dedicated sensor with zoom capabilities or high magnification to capture hand images at a desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with other image sensors that capture the physical environment of the scene 105, or functions as an image sensor that captures the physical environment of the scene 105. In some embodiments, the image sensor 404 is positioned relative to the user or the user's environment such that the field of view of the image sensor, or a portion thereof, is used to define an interaction space in which hand movements captured by the image sensor are processed as inputs to the controller 110.

[0081] In some embodiments, the image sensor 404 outputs a sequence of frames containing 3D map data (and possibly color image data) to the controller 110, which extracts high-level information from the map data. This high-level information is provided, typically via an application program interface (API), to an application running on the controller, which drives the display generation component 120 accordingly. For example, a user can interact with software running on the controller 110 by moving the hand 408 and changing hand postures.

[0082] In some embodiments, the image sensor 404 projects a spot pattern onto a scene including the hand 406 and captures an image of the projected pattern. In some embodiments, the controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on the lateral shift of the spots of the pattern. This approach is advantageous in that it does not require the user to hold or wear any kind of beacon, sensor, or other marker. It gives the depth coordinate of a point in the scene relative to a predefined reference plane at a specific distance from the image sensor 404. In this disclosure, it is assumed that the image sensor 404 defines an orthogonal set of x, y, z axes such that the depth coordinate of a point in the scene corresponds to the z component measured by the image sensor. Alternatively, the hand tracking device 440 can use other 3D mapping methods, such as stereoscopic imaging or time-of-flight measurements, based on single or multiple cameras or other types of sensors.

[0083] In some embodiments, the hand tracking device 140 captures and processes a time sequence of depth maps including the user's hand while the user moves the hand (e.g., the entire hand or one or more fingers). Software running on the image sensor 404 and / or a processor in the controller 110 processes the 3D map data to extract patch descriptors of the hand in these depth maps. The software matches these descriptors to patch descriptors stored in the database 408, based on a previous learning process, to estimate the pose of the hand in each frame. The pose typically includes the 3D positions of the user's wrist joints and fingertips.

[0084] The software can also analyze hand and / or finger trajectories across multiple frames in a sequence to identify gestures. The pose estimation functionality described herein may be interleaved with the motion tracking functionality, whereby patch-based pose estimation is performed only once every two (or more) frames, while tracking is used to discover pose changes that occur across the remaining frames. The pose, motion, and gesture information is provided to an application program running on the controller 110 via the API described above. The program can, for example, move and modify an image presented on the display generation component 120, or perform other functions, in response to the pose and / or gesture information.

[0085] In some embodiments, the software may be downloaded in electronic form to the controller 110, for example over a network, or may alternatively be provided on a tangible non-transitory medium, such as an optical, magnetic, or electronic memory medium. In some embodiments, the database 408 is similarly stored in a memory associated with the controller 110. Alternatively or additionally, some or all of the described functionality of the computer may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). Although the controller 110 is shown in FIG. 4 as a separate unit from the image sensor 440, by way of example, some or all of the processing functionality of the controller may be associated with the image sensor 404 by a suitable microprocessor and software, or by a dedicated circuit within the housing of the hand tracking device 402, or in other ways. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television set, handheld device, or head-mounted device), or with any other suitable computerized device, such as a game console or media player. The sensing functions of the image sensor 404 may similarly be integrated into a computer or other computerized device that is controlled by the sensor output.

[0086] FIG. 4 further includes a schematic diagram of a depth map 410 captured by the image sensor 404, according to some embodiments. The depth map includes a matrix of pixels with respective depth values, as described above. A pixel 412 corresponding to the hand 406 is segmented from the background and wrist in this map. The brightness of each pixel in the depth map 410 is inversely proportional to the depth value, i.e., the measured z-distance from the image sensor 404, with increasing gray levels as depth increases. The controller 110 processes these depth values ​​to identify and segment components of the image (i.e., groups of adjacent pixels) that have characteristics of a human hand. These characteristics can include, for example, the overall size, shape, and frame-to-frame motion of the sequence of depth maps.

[0087] 4 also illustrates, in accordance with some embodiments, a hand skeleton 414 that the controller 110 ultimately extracts from the depth map 410 of the hand 406. In FIG. 4, the skeleton 414 is superimposed on a hand background 416 that was segmented from the original depth map. In some embodiments, key feature points on the hand (e.g., knuckles, fingertips, palm center, end of hand that connects to wrist, etc.) and optionally the wrist or arm connected to the hand are identified and positioned on the hand skeleton 414. In some embodiments, the location and movement of these key feature points over multiple image frames are used by the controller 110 to determine hand gestures performed by the hand or the current state of the hand, according to some embodiments.

[0088] FIG. 5 illustrates an exemplary embodiment of eye tracking device 130 (FIG. 1). In some embodiments, eye tracking device 130 is controlled by eye tracking unit 245 (FIG. 2) to track the position and movement of the user's gaze relative to scene 105 or relative to CGR content displayed via display generation component 120. In some embodiments, eye tracking device 130 is integrated with display generation component 120. For example, in some embodiments, if display generation component 120 is a head-mounted device such as a headset, helmet, goggles, or glasses, or a handheld device disposed on a wearable frame, the head-mounted device includes both components for generating CGR content for viewing by the user and components for tracking the user's gaze relative to the CGR content. In some embodiments, eye tracking device 130 is separate from display generation component 120. For example, if display generation component is a handheld device or a CGR chamber, eye tracking device 130 is, optionally, a separate device from the handheld device or the CGR chamber. In some embodiments, eye tracking device 130 is a head-mounted device or part of a head-mounted device. In some embodiments, the head-mounted eye tracking device 130 is optionally used in conjunction with a head-mounted or non-head-mounted display generation component. In some embodiments, the eye tracking device 130 is not a head-mounted device, and is optionally used in combination with a head-mounted display generation component. In some embodiments, the eye tracking device 130 is not a head-mounted device, and is optionally part of a non-head-mounted display generation component.

[0089] In some embodiments, the display generation component 120 uses a display mechanism (e.g., left and right near-eye display panels) that displays frames including left and right images in front of the user's eyes to provide the user with a 3D virtual view. For example, the head-mounted display generation component may include left and right optical lenses (referred to herein as eyepieces) located between the display and the user's eyes. In some embodiments, the display generation component may include or be coupled to one or more external video cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generation component may have a transparent or semi-transparent display that allows the user to view the physical environment directly and display virtual objects on the transparent or semi-transparent display. In some embodiments, the display generation component projects virtual objects into the physical environment. The virtual objects are projected, for example, onto a physical surface or as a hologram, whereby an individual using the system can observe the virtual objects superimposed on the physical environment. In such cases, separate display panels and image frames for the left and right eyes may not be required.

[0090] As shown in FIG. 5, in some embodiments, the eye tracking device 130 includes at least one eye tracking camera (e.g., an infrared (IR) or near-IR (NIR) camera) and an illumination source (e.g., an IR or NIR light source such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eye. The eye tracking camera may be aimed at the user's eye to receive reflected IR or NIR light from the light source directly from the eye, or alternatively, may be aimed at a "hot" mirror disposed between the user's eye and a display panel that reflects the IR or NIR light from the eye to the eye tracking camera while allowing visual light to pass through. The eye tracking device 130 optionally captures images of the user's eye (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes the images to generate eye tracking information, and communicates the eye tracking information to the controller 110. In some embodiments, the user's eyes are tracked separately by respective eye tracking cameras and illumination sources. In some embodiments, only one eye of the user is tracked by a corresponding eye tracking camera and lighting source.

[0091] In some embodiments, the eye tracking device 130 is calibrated using a device-specific calibration process to determine the parameters of the eye tracking device for the particular operating environment 100, e.g., 3D geometric relationships and parameters of the LEDs, camera, hot mirror (if present), eyepiece, and display screen. The device-specific calibration process may be performed at a factory or another facility prior to delivery of the AR / VR equipment to the end user. The device-specific calibration process may be an automatic calibration process or a manual calibration process. The user-specific calibration process may include estimation of the particular user's eye parameters, e.g., pupil position, central vision position, optical axis, visual axis, eye spacing, etc. According to some embodiments, once the device-specific and user-specific parameters have been determined for the eye tracking device 130, images captured by the eye tracking camera may be processed using a glint-assisted method to determine the user's current visual axis and viewpoint relative to the display.

[0092] As shown in FIG. 5, eye tracking device 130 (e.g., 130A or 130B) includes an eyepiece(s) 520 and an eye tracking system including at least one eye tracking camera 540 (e.g., an infrared (IR) or near-IR (NIR) camera) positioned on the side of the user's face where eye tracking occurs and an illumination source 530 (e.g., an IR or NIR light source such as an array or ring of NIR light emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye(s) 592. The eye tracking camera 540 may be located between the user's eye(s) 592 and the display 510 (e.g., a left or right display panel of a head-mounted display, or a display of a handheld device, a projector, etc.) and may be pointed at a mirror 550 that reflects IR or NIR light from the eye(s) 592 while transmitting visible light (e.g., as shown at the top of FIG. 5), or may be pointed at the user's eye(s) 592 to receive reflected IR or NIR light from the user's eye(s) 592 (e.g., as shown at the bottom of FIG. 5).

[0093] In some embodiments, the controller 110 renders AR or VR frames 562 (e.g., left and right frames for left and right display panels) and provides the frames 562 to the display 510. The controller 110 uses gaze tracking input 542 from the eye tracking camera 540 for various purposes, e.g., in processing the frames 562 for display. The controller 110 optionally estimates the user's viewpoint on the display 510 based on the gaze tracking input 542 obtained from the eye tracking camera 540, using a glint-assisted method or other suitable method. The viewpoint estimated from the gaze tracking input 542 is optionally used to determine the direction in which the user is currently looking.

[0094] Below, some possible use cases of the user's current gaze direction are described, which are not intended to be limiting. As an exemplary use case, the controller 110 can render virtual content differently based on the determined user's gaze direction. For example, the controller 110 can generate virtual content with higher resolution in a central visual area determined from the user's current gaze direction than in a peripheral area. As another example, the controller can position or move virtual content in a view based at least in part on the user's current gaze direction. As another example, the controller can display certain virtual content in a view based at least in part on the user's current gaze direction. As another exemplary use case in an AR application, the controller 110 can orient an external camera to capture the physical environment of the CGR experience and focus in the determined direction. The autofocus mechanism of the external camera can then focus on an object or surface in the environment that the user is currently viewing on the display 510. As another exemplary use case, the eyepiece 520 may be a focusable lens, and the eye tracking information is used by the controller to adjust the focus of the eyepiece 520 so that the virtual object the user is currently looking at has the proper binocular coordination to match the convergence of the user's eyes 592. The controller 110 can leverage the eye tracking information to orient and focus the eyepiece 520 so that a nearby object the user is looking at appears at the correct distance.

[0095] In some embodiments, the eye tracking device is part of a head mounted device that includes a display (e.g., display 510), two eyepieces (e.g., eyepiece(s) 520), an eye tracking camera (e.g., eye tracking camera(s) 540), and a light source (e.g., light source 530 (e.g., IR or NIR LED)) attached to the wearable housing. The light source emits light (e.g., IR or NIR light) toward the user's eye(s) 592. In some embodiments, the light sources may be arranged in a ring or circle around each lens, as shown in FIG. 5. In some embodiments, eight light sources 530 (e.g., LEDs) are arranged around each lens 520 as an example. However, more or fewer light sources 530 may be used, and other arrangements and positions of the light sources 530 may be used.

[0096] In some embodiments, the display 510 emits light in the visible light range and not in the IR or NIR ranges, and therefore does not introduce noise into the eye tracking system. Note that the positions and angles of the eye tracking camera(s) 540 are given by way of example and are not intended to be limiting. In some embodiments, a single eye tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user's face. In some embodiments, a camera 540 with a wide field of view (FOV) and a camera 540 with a narrow FOV may be used on each side of the user's face. In some embodiments, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

[0097] Embodiments of an eye tracking system such as that shown in FIG. 5 may be used, for example, in computer-generated reality (e.g., including virtual reality and / or mixed reality) applications to provide a user with a computer-generated reality (e.g., including virtual reality, augmented reality, and / or augmented virtual) experience.

[0098] FIG. 6 illustrates a glint-assisted eye tracking pipeline according to some embodiments. In some embodiments, the eye tracking pipeline is realized by a glint-assisted eye tracking system (e.g., eye tracking device 130 as shown in FIG. 1 and FIG. 5). The glint-assisted eye tracking system can maintain a tracking state. Initially, the tracking state is off or "no". When in the tracking state, the glint-assisted eye tracking system tracks the pupil contour and glint in the current frame using prior information from the previous frame when analyzing the current frame. When not in the tracking state, the glint-assisted eye tracking system attempts to detect the pupil and glint in the current frame, and if successful, initializes the tracking state to "yes" and continues to the next frame in the tracking state.

[0099] As shown in FIG. 6, an eye tracking camera can capture left and right images of a user's left and right eyes. The captured images are then input into an eye tracking pipeline for processing beginning at 610. As indicated by the arrow returning to element 600, the eye tracking system can continue to capture images of the user's eyes at a rate of, for example, 60-120 frames per second. In some embodiments, each set of captured images may be input into the pipeline for processing. However, in some embodiments, or under some conditions, not all captured frames are processed by the pipeline.

[0100] At 610, if the tracking status is yes for the current captured image, the method proceeds to element 640. If the tracking status is no at 610, the image is analyzed to detect the user's pupil and glint in the image, as shown at 620. If the pupil and glint are successfully detected at 630, the method proceeds to element 640. If not, the method returns to element 610 to process the next image of the user's eye.

[0101] At 640, proceeding from element 410, the current frame is analyzed to track pupils and glints, based in part on prior information from the previous frame. At 640, proceeding from element 630, a tracking state is initialized based on the detected pupils and glints in the current frame. The results of the processing at element 640 are checked to ensure that the tracking or detection results are reliable. For example, the results can be checked to determine whether a sufficient number of glints are successfully tracked or detected in the current frame to perform pupil and gaze estimation. At 650, if the results are not reliable, the tracking state is set to no and the method returns to element 610 to process the next image of the user's eyes. At 650, if the results are reliable, the method proceeds to element 670. At 670, the tracking state is set to yes (if not already yes) and the pupil and glint information is passed to element 680 to estimate the user's gaze point.

[0102] 6 is intended to serve as an example of eye-tracking technology that may be used in a particular implementation. As will be appreciated by those skilled in the art, other eye-tracking technologies, either currently existing or developed in the future, may be used in place of or in combination with the glint-assisted gaze-tracking technology described herein in computer system 101 to provide a user with a CGR experience according to various embodiments.

[0103] In the present disclosure, various input methods are described with respect to interaction with a computer system. If one example is provided using one input device or input method and another example is provided using another input device or input method, it should be understood that each example is compatible with and can be optionally used with the input device or input method described with respect to the other example. Similarly, various output methods are described with respect to interaction with a computer system. If one example is provided using one output device or output method and another example is provided using another output device or output method, it should be understood that each example is compatible with and can be optionally used with the output device or output method described with respect to the other example. Similarly, various methods are described with respect to interaction with a virtual environment or a mixed reality environment via a computer system. If an example is provided using interaction with a virtual environment and another example is provided using a mixed reality environment, it should be understood that each example is compatible with and can be optionally used with the method described with respect to the other example. Thus, the present disclosure discloses embodiments that are combinations of features of multiple examples without exhaustively listing all features of the embodiments in the description of each embodiment. User Interface and Related Processes

[0104] Attention now turns to embodiments of user interfaces ("UI") and associated processes that may be executed in a computer system, such as a portable multifunction device or a head-mounted device, with a display generating component, one or more input devices, and (optionally) one or more cameras.

[0105] 7A-7C show examples of input gestures (e.g., individual small movement gestures performed by moving a user's finger(s) relative to other finger(s) or part(s) of the user's hand, optionally without requiring the user's entire hand or arm to move significantly away from their natural position(s) and posture(s) to perform an action just before or during the gesture) for interacting with a virtual or mixed reality environment, according to some embodiments. The input gestures described with reference to FIGS. 7A-7C are used to illustrate the processes described below, including the process in FIG. 8.

[0106] In some embodiments, the input gestures described with reference to FIGS. 7A-7C are detected by analyzing data and signals captured by a sensor system (e.g., sensor 190 of FIG. 1, image sensor 314 of FIG. 3). In some embodiments, the sensor system includes one or more imaging sensors (e.g., one or more cameras, such as a motion RGB camera, an infrared camera, a depth camera, etc.). For example, the one or more imaging sensors are components of or provide data to a computer system (e.g., computer system 101 of FIG. 1 (e.g., portable electronic device 7100 or HMD as shown in FIG. 7C)) that includes a display generating component (e.g., display generating component 120 of FIGS. 1, 3, and 4 (e.g., a touch screen display that functions as a display and a touch-sensitive surface, a stereoscopic display, a display with a pass-through portion, etc.)). In some embodiments, the one or more imaging sensors include one or more rear-facing cameras on a side of the device opposite the display of the device. In some embodiments, the input gestures are detected by a sensor system of a head-mounted system (e.g., a VR headset that includes a stereoscopic display that provides a left image for the user's left eye and a right image for the user's right eye). For example, one or more cameras that are components of the head-mounted system are mounted on the front and / or underside of the head-mounted system. In some embodiments, one or more imaging sensors are positioned in the space in which the head-mounted system is used (e.g., arrayed around the head-mounted system at various positions in a room) such that the imaging sensors capture images of the head-mounted system and / or a user of the head-mounted system. In some embodiments, the input gesture is detected by a sensor system of a head-up device (e.g., a head-up display, an automobile windshield capable of displaying graphics, a window capable of displaying graphics, a lens capable of displaying graphics). For example, the one or more imaging sensors are mounted on an interior surface of an automobile. In some embodiments, the sensor system includes one or more depth sensors (e.g., a sensor array).For example, the one or more depth sensors include one or more light-based (e.g., infrared) sensors and / or one or more acoustic-based (e.g., ultrasonic) sensors. In some embodiments, the sensor system includes one or more signal emitters, such as light emitters (e.g., infrared emitters) and / or sound emitters (e.g., ultrasonic emitters). For example, while light (e.g., light from an infrared light emitter array having a predetermined pattern) is projected onto a hand (e.g., hand 7200 as described with respect to FIG. 71C), images of the hand under the illumination of the light are captured by one or more cameras, and the captured images are analyzed to determine the position and / or configuration of the hand. By determining input gestures using signals from an image sensor directed at the hand, as opposed to using signals of a touch-sensitive surface or other direct contact or proximity-based mechanism, a user can freely choose to perform large movements or remain relatively still when providing input gestures with their hand without experiencing constraints imposed by a particular input device or input area.

[0107] Portion (A) of FIG. 7A illustrates a tap input of thumb 7106 on index finger 7108 of a user's hand (e.g., on the side of index finger 7108 adjacent thumb 7106). Thumb 7106 moves along an axis indicated by arrow 7110, including moving from an up position 7102 to a touch down position 7104 (e.g., thumb 7106 contacts and remains on index finger 7110), and, optionally, moving from touch down position 7104 to up position 7102 within a threshold time (e.g., tap time threshold) after thumb 7106 contacts index finger 7110. In some embodiments, the tap input is detected without having to lift the thumb from the side of the index finger. In some embodiments, the tap input is detected according to a determination that a downward movement of the thumb is followed by an upward movement of the thumb and the thumb is in contact with the side of the index finger for less than the threshold time. In some embodiments, the tap hold input is detected according to a determination that the thumb moves from the up position 7102 to the touch down position 7104 and remains at the touch down position 7104 for at least a first threshold time (e.g., a tap time threshold or another time threshold longer than the tap time threshold). In some embodiments, the computer system requires that the entire hand remains substantially stationary in a position for at least a first threshold time to detect a tap hold input with the thumb on the index finger. In some embodiments, the touch hold input is detected without requiring the hand to remain substantially stationary (e.g., the entire hand can move while the thumb is resting on the side of the index finger). In some embodiments, the tap hole drag input is detected when the thumb touches the side of the index finger and the entire hand moves while the thumb is stationary on the side of the index finger.

[0108] Part (B) of FIG. 71A illustrates a push or flick input with the thumb 7116 moving across the index finger 7118 (e.g., from the palm side to the back side of the index finger). The thumb 7116 moves across the index finger 7118 (e.g., across the middle phalanx of the index finger 7118) from a retracted position 7112 to an extended position 7114 along the axis indicated by the arrow 7120. In some embodiments, the extended movement of the thumb is accompanied by an upward movement away from the side of the index finger, e.g., as in an upward flick input by the thumb. In some embodiments, the index finger moves in a direction opposite to that of the thumb while the thumb moves forward and upward. In some embodiments, a reverse flick input is performed by the thumb moving from the extended position 7114 to the retracted position 7112. In some embodiments, the index finger moves in a direction opposite to that of the thumb while the thumb moves backward and downward.

[0109] Portion (C) of FIG. 7A illustrates a swipe input by movement of thumb 7126 along index finger 7128 (e.g., along the side of index finger 7128 adjacent to thumb 7126 or along the side of the palm). Thumb 7126 moves along the length of index finger 7128 along an axis indicated by arrow 7130 from a proximal position 7122 of index finger 7128 (e.g., at or near the proximal phalanx of index finger 7118) to a distal position 7124 (e.g., at or near the distal phalanx of index finger 7118) and / or from the distal position 7124 to the proximal position 7122. In some embodiments, the index finger is optionally in an extended state (e.g., substantially straight) or in a bent state. In some embodiments, the index finger moves between an extended state and a bent state during the movement of the thumb in a swipe input gesture.

[0110] Part (D) of FIG. 7A illustrates tap inputs of thumb 7106 on various phalanges of various fingers (e.g., index finger, middle finger, ring finger, and optionally pinky finger). For example, thumb 7106 as shown in part (A) moves from an up position 7102 to a touchdown position as shown in any of 7130-7148 shown in part (D). At touchdown position 7130, thumb 7106 is shown contacting position 7150 on the proximal phalange of index finger 7108. At touchdown position 7134, thumb 7106 is contacting position 7152 on the middle phalange of index finger 7108. At touchdown position 7136, thumb 7106 is contacting position 7154 on the distal phalange of index finger 7108.

[0111] In the touch down positions shown at 7138, 7140, and 7142, the thumb 7106 contacts locations 7156, 7158, and 7160, which correspond to the proximal phalange of the middle finger, the middle phalange of the middle finger, and the distal phalange of the middle finger, respectively.

[0112] In the touchdown positions shown at 7144, 7146, and 7150, the thumb 7106 contacts locations 7162, 7164, and 7166, which correspond to the proximal phalange of the ring finger, the middle phalange of the ring finger, and the distal phalange of the ring finger, respectively.

[0113] In various embodiments, tap inputs by the thumb 7106 on different parts of another finger or different parts of two parallel fingers correspond to different inputs and trigger different actions in the respective user interface contexts. Similarly, in some embodiments, different push or click inputs can be performed by the thumb across different fingers and / or different parts of the fingers to trigger different actions at the respective user interface contacts. Similarly, in some embodiments, different swipe inputs performed by the thumb along different fingers and / or in different directions (e.g., towards the distal or proximal ends of the fingers) trigger different actions in the respective user interface contexts.

[0114] In some embodiments, the computer system processes tap, flick, and swipe inputs as different types of inputs based on the type of thumb movement. In some embodiments, the computer system processes inputs with different finger positions tapped, touched, or swiped by the thumb as different sub-input types (e.g., proximal, intermediate, distal sub-types, or index, middle, ring, or pinky sub-types) of a given input type (e.g., tap input type, flick input type, swipe input type, etc.). In some embodiments, the amount of movement performed by the moving finger (e.g., thumb) and / or other movement measures associated with the finger movement (e.g., velocity, initial velocity, ending velocity, duration, direction, movement pattern, etc.) are used to quantitatively affect the action triggered by the finger input.

[0115] In some embodiments, the computer system recognizes combination input types that combine a series of movements by the thumb, such as a tap-swipe input (e.g., the thumb touches down on another finger, then swipes along the side of the finger), a tap-flick input (e.g., the thumb touches down on another finger, then flicks across the finger from the side of the palm to the back of the finger), and a double-tap input (e.g., two consecutive taps on the side of a finger in approximately the same location).

[0116] In some embodiments, the gesture input is performed by the index finger instead of the thumb (e.g., the index finger performs a tap or swipe on the thumb, or the thumb and index finger move towards each other to perform a pinch gesture). In some embodiments, a wrist movement (e.g., a wrist flick in a horizontal or vertical direction) is performed immediately before, after (e.g., within a threshold time), or simultaneously with the finger movement input to trigger an additional, different, or modified action in the current user interface context compared to a finger movement input without a modified wrist movement input. In some embodiments, a finger input gesture performed with the palm of the user facing the user's face is treated as a different type of gesture than a finger input gesture performed with the palm of the user facing away from the user's face. For example, a tap gesture performed with the palm of the user facing the user performs an action with added (or reduced) privacy protection compared to an action (e.g., the same action) performed in response to a tap gesture performed with the palm of the user facing away from the user's face.

[0117] While one type of finger input may be used to trigger an action type in the examples provided in this disclosure, in other embodiments, other types of finger input are optionally used to trigger the same type of action.

[0118] FIG. 7B illustrates an example user interface context that, in some embodiments, shows a menu 7170 that includes user interface objects 7172-7194.

[0119] In some embodiments, the menu 7170 is displayed in the mixed reality environment (e.g., floating in the air or overlaid on a physical object in the three-dimensional environment and corresponding to an action associated with the mixed reality environment or an action associated with the physical object). For example, the menu 7170 is displayed by a display of a device (e.g., device 7100 (FIG. 7C) or HMD) along with (e.g., overlaid on) at least a portion of a view of the physical environment captured by one or more rear-facing cameras of the device 7100. In some embodiments, the menu 7170 is displayed on a transparent or semi-transparent display of the device (e.g., a head-up display, or HMD) through which the physical environment is visible. In some embodiments, the menu 7170 is displayed in a user interface that includes a pass-through portion (e.g., a transparent or semi-transparent portion through which the physical surroundings are visible, or a portion that displays a camera view of the surrounding physical environment) surrounded by virtual content. In some embodiments, the user's hands performing a gestural input that causes an action to be performed in the mixed reality environment are visible to the user on the display of the device. In some embodiments, the user's hands performing gestural inputs that cause actions to be performed in the mixed reality environment are invisible to the user on the device's display (e.g., the camera that provides the user with a view of the physical world has a different field of view than the camera that captures the user's finger inputs).

[0120] In some embodiments, the menu 7170 is displayed in the virtual reality environment (e.g., floating in virtual space or overlaid on a virtual surface). In some embodiments, the hand 7200 is visible in the virtual reality environment (e.g., an image of the hand 7200 captured by one or more cameras is rendered in the virtual reality setting). In some embodiments, a representation of the hand 7200 (e.g., a cartoon version of the hand 7200) is rendered in the virtual reality setting. In some embodiments, the hand 7200 is invisible (e.g., omitted) in the virtual reality environment. In some embodiments, the device 7100 (FIG. 7C) is invisible in the virtual reality environment (e.g., when the device 7100 is an HMD). In some embodiments, an image of the device 7100 or a representation of the device 7100 is visible in the virtual reality environment.

[0121] In some embodiments, one or more of the user interface objects 7172-7194 are application launch icons (e.g., for performing an action to launch a corresponding application). In some embodiments, one or more of the user interface objects 7172-7194 are controls for performing a respective action within an application (e.g., turn up the volume, turn down the volume, play, pause, fast forward, rewind, initiate communication with a remote device, terminate communication with a remote device, transmit communication with a remote device, start a game, etc.). In some embodiments, one or more of the user interface objects 7172-7194 are representations (e.g., avatars) of respective users of remote devices (e.g., for performing an action to initiate communication with respective users of remote devices). In some embodiments, one or more of the user interface objects 7172-7194 are representations (e.g., thumbnails, two-dimensional images, or album covers) of media (e.g., images, virtual objects, audio files, and / or video files). For example, by activating a user interface object that is a representation of an image, the image is displayed at a location corresponding to a surface (e.g., detected by one or more cameras and displayed in a computer-generated reality view (e.g., a location corresponding to a surface in the physical environment, or a location corresponding to a surface displayed in a virtual space).

[0122] When the thumb of the hand 7200 performs an input gesture as described with respect to FIG. 7A, an action corresponding to the menu 7170 is performed depending on the location and / or type of input gesture detected. For example, in response to an input including a movement of the thumb along the y-axis (e.g., from a proximal position on the index finger to a distal position on the index finger as described with respect to portion (C) of FIG. 7A), the current selection indicator 7198 (e.g., a selector object or a moving visual effect, such as highlighting the object with an outline or changing the appearance of the object) is repeated rightward from the item 7190 to the subsequent user interface object 7192. In some embodiments, in response to an input including a movement of the thumb along the y-axis from a distal position on the index finger to a proximal position on the index finger, the current selection indicator 7198 is repeated leftward from the item 7190 to the previous user interface object 7188. In some embodiments, in response to an input including a thumb tap input on an index finger (e.g., thumb movement along the z-axis as described with respect to portion (A) of FIG. 7A ), a currently selected user interface object 7190 is activated and an action corresponding to the currently selected user interface object 7190 is performed. For example, the user interface object 7190 is an application launch icon, and in response to a tap input while the user interface object 7190 is selected, the application corresponding to the user interface object 7190 is launched and displayed on the display. In some embodiments, in response to an input including thumb movement along the x-axis from a retracted position to an extended position, the current selection indicator 7198 moves upward from the item 7190 to the upper user interface object 7182. Other types of finger inputs provided with respect to one or more of the user interface objects 7172-7194 are possible, optionally causing other types of actions corresponding to the user interface object(s) to be performed in accordance with the input.

[0123] 7C shows a visual indication of a menu 7170 that is visible in a mixed reality view (e.g., an augmented reality view of the physical environment) displayed by a computer system (e.g., device 7100 or an HMD). In some embodiments, a hand 7200 in the physical environment is visible in the displayed augmented reality view (e.g., as part of a view of the physical environment captured by a camera), as illustrated in 7200'. In some embodiments, the hand 7200 is visible through a transparent or semi-transparent display surface on which the menu 7170 is displayed (e.g., device 7100 is a head-up display or HMD with a pass-through portion).

[0124] In some embodiments, as shown in FIG. 7C, the menu 7170 is displayed at a location in the mixed reality environment that corresponds to a predetermined portion of the user's hand (e.g., the tip of the thumb) and has an orientation that corresponds to the orientation of the user's hand. In some embodiments, the menu 7170 is shown moving in the mixed reality environment with the user's hand as the user's hand moves (e.g., moves laterally or rotates) relative to the physical environment (e.g., the user's hand, or the user's eyes, or a camera capturing physical objects or walls surrounding the user). In some embodiments, the menu 7170 moves according to the movement of the user's gaze directed toward the mixed reality environment. In some embodiments, the menu 7170 is displayed at a fixed location on the display regardless of the view of the physical environment shown on the display.

[0125] In some embodiments, the menu 7170 is displayed on the display in response to detecting a ready pose of the user's hand (e.g., thumb resting to the side of the index finger). In some embodiments, the user interface objects displayed in response to detecting a ready pose of the hand vary depending on the current user interface context and / or the user's gaze position within the mixed reality environment.

[0126] 7D-7E show a hand 7200 in an exemplary non-ready (e.g., resting) configuration (FIG. 7D) and an exemplary ready (FIG. 7E) configuration, according to some embodiments. The input gestures described with reference to FIGS. 7D-7E are used to illustrate the processes described below, including the process in FIG.

[0127] In FIG. 7D , hand 7200 is shown in an exemplary non-ready configuration (e.g., a rest configuration (e.g., hand relaxed or in any position, thumb 7202 not resting on index finger 7204)). In an exemplary user interface context, container object 7206 (e.g., application dock, folder, control panel, menu, platter, preview, etc.) containing user interface objects 7208, 7210, and 7212 (e.g., application icons, media objects, controls, menu items, etc.) are displayed in a three-dimensional environment (e.g., a virtual environment or a mixed reality environment) by a display generation component (e.g., display generation component 120 (e.g., touch screen display, stereoscopic projector, heads-up display, HMD, etc.) of a computer system (e.g., computer system 101 of FIG. 1 (e.g., device 7100, HMD, or a system including the above, among other devices)) of the computer system. In some embodiments, the rest configuration of the hand is an example of a hand configuration that is not a ready configuration. For example, other hand configurations that are not necessarily relaxed and stationary and do not meet the criteria for detecting a ready gesture (e.g., thumb resting on index finger (e.g., middle phalanx of index finger)) are also classified as non-ready configurations. For example, if a user is waving their hand in the air, holding an object, clenching a fist, etc., in some embodiments, the computer system determines that the user's hand does not meet the criteria for detecting a ready hand and determines that the user's hand is in a non-ready configuration. In some embodiments, the criteria for detecting a ready hand configuration include detecting that a user has changed their hand configuration, which results in the user's thumb being placed on a predetermined portion of the user's index finger (e.g., middle phalanx of index finger). In some embodiments, the criteria for detecting a ready configuration require that the change in hand gesture results in the user's thumb being placed on a predetermined portion of the user's index finger for at least a first threshold time in order for the computer system to recognize that the hand is in a ready configuration.In some embodiments, if the user does not change his / her hand configuration and does not provide a valid input gesture for at least a second threshold time after entering the ready state configuration, the computer system treats the current hand configuration as a non-ready state configuration. The computer system requires the user to change his / her current hand configuration and then return to the ready state configuration to re-recognize the ready state configuration. In some embodiments, the ready state configuration is user-configurable and user-self-definable, for example, by the user indicating the intended ready state configuration of the hands, and optionally, the acceptable range of variation of the ready state configuration, to the computer system in a gesture setting environment provided by the computer system.

[0128] In some embodiments, the container 7206 is displayed in the mixed reality environment (e.g., as shown in FIGS. 7D and 7E). For example, the container 7206 is displayed by a display of the device 7100 having at least a portion of a view of the physical environment captured by one or more rear-facing cameras of the device 7100. In some embodiments, the hand 7200 in the physical environment is also visible in the displayed mixed reality environment, as shown in 7200b, e.g., with an actual spatial relationship between the hand and the physical environment represented in the displayed view of the mixed reality environment. In some embodiments, the container 7206 and the hand 7200 are displayed in relation to the physical environment displayed via a live feed of a camera located remotely from the user and coupled with the remote physical environment. In some embodiments, the container 7206 is displayed on a transparent or semi-transparent display of the device, where the physical environment surrounding the user (e.g., including the hand 7200 as shown in 7200b) is visible.

[0129] In some embodiments, the container 7206 is displayed in the virtual reality environment (e.g., floating in virtual space). In some embodiments, the hand 7200 is visible in the virtual reality setting (e.g., an image of the hand 7200 captured by one or more cameras is rendered in the virtual reality environment). In some embodiments, a representation of the hand 7200 is visible in the virtual reality environment. In some embodiments, the hand 7200 is invisible (e.g., omitted) in the virtual reality environment. In some embodiments, the device 7100 is invisible in the virtual reality environment. In some embodiments, an image of the device 7100 or a representation of the device 7100 is visible in the virtual reality environment.

[0130] In some embodiments, while the hand 7200 is not in the ready state configuration (e.g., is in any non-ready state configuration or ceases to remain in the ready state configuration) (e.g., due to a change in hand posture or failure to provide a valid input gesture within the threshold time of entering the ready state configuration), the computer system does not perform input gesture recognition (other than recognition of whether the hand has entered the ready state configuration) to perform an action in the current user interface context, and as a result, no action is performed in response to an input gesture performed by the hand 7200 (e.g., a tap of the thumb on the index finger, including movement of the thumb along the axis indicated by arrow 7110, as described with reference to portion (A) of FIG. 7A, movement of the thumb across the index finger along the axis indicated by arrow 7120, as described with reference to portion (B) of FIG. 7A, and / or movement of the thumb on the index finger along the axis indicated by arrow 7130, as described with reference to portion (C) of FIG. 7A). In other words, in order for the computer system to recognize an input gesture as valid and perform a corresponding action in the current user interface context, the user must be in the ready state configuration (e.g., change from a non-ready state configuration to the ready state configuration) and then provide a valid input gesture for the current user interface context (e.g., within a threshold time of the hand entering the ready state configuration). In some embodiments, if a valid input gesture is detected without first detecting the hand in the ready state configuration, the computer system performs certain types of actions (e.g., interacting with (e.g., scrolling or activating) currently displayed user interface objects and prohibits other types of actions (e.g., invoking a new user interface, triggering a system level action (e.g., navigating to a multitasking user interface or an application launching user interface, activating a device functionality control panel, etc.)).These protections help prevent and reduce inadvertent and unintended triggering of actions when the user does not wish to perform an action or a particular type of action within the current user interface context, and avoid unnecessary constraints on the user's free hand movement. In addition, imposing small individual movement requirements on the ready-state configuration does not impose undue physical strain on the user (e.g., by excessively moving the user's arm or hand) and tends to reduce discomfort for the user when interacting with the user interface in a social environment.

[0131] User interface objects 7208-7212 of container 7206 may include, for example, one or more application launch icons, one or more controls for performing actions within an application, one or more representations of a user of a remote device, and / or one or more representations of media (e.g., as described above with respect to user interface objects 7172-7194). In some embodiments, when a user interface object is selected and an input gesture is detected by the computer system without the hands first being found in a ready state configuration, the computer system performs a first action with respect to the selected user interface object in accordance with the input gesture (e.g., launching an application corresponding to the selected application icon, changing a control value of the selected control, initiating communication with the user about the representation of the selected user, initiating playback of a media item corresponding to a representation of the selected media item). When a user interface object is selected and the same input gesture is detected by the computer system with the hands initially found in the ready state configuration, the computer system performs a second action that is different from the first action (e.g., the second action is a system action that is not specific to the currently selected user interface object (e.g., the system action includes displaying a system affordance in response to the hands being found in the ready state configuration and invoking a system menu in response to the input gesture)). In some embodiments, placing the hands in the ready state configuration activates a particular input gesture that is not paired with any functionality in the current user interface context (e.g., a thumb flick gesture), and detecting the newly enabled input gesture after the hands are found in the ready state configuration causes the computer system to perform an additional functionality associated with the newly enabled input gesture.In some embodiments, the computer system, optionally in response to detecting the hand in the ready state configuration, displays a user interface indication (e.g., additional options, system affordances, or a system menu) and allows the user to interact with the user interface indication with the newly enabled input gesture or trigger additional functionality (e.g., a thumb flick gesture detected when a system affordance is displayed causes a system menu to be displayed, a thumb flick gesture detected when a system menu is displayed causes navigation through or expansion of the system menu).

[0132] In FIG. 7E, the hand 7200 is shown in a ready state configuration (e.g., thumb 7202 rests on index finger 7204). Pursuant to a determination that the hand 7200 has moved to the ready state configuration, the computer system displays a system affordance icon 7214 (e.g., in a region of the mixed reality environment corresponding to the tip of the thumb). The system affordance icon 7214 indicates a region in which one or more user interface objects (e.g., a menu of application icons corresponding to different applications, a menu of currently open applications, a device control user interface, etc.) can be displayed and / or accessed (e.g., in response to a thumb flick gesture or other predefined input gesture). In some embodiments, while the system affordance icon 7214 is displayed, the computer system performs an action in response to an input gesture performed by the hand 7200 (e.g., as described below with respect to FIG. 7F).

[0133] In some embodiments, the movement of the hand 7200 from the non-ready state configuration to the ready state configuration is detected by analyzing data captured by a sensor system (e.g., an image sensor or other sensor (e.g., a motion sensor, a touch sensor, a vibration sensor, etc.)) as described above with respect to Figures 7A-7C. In some embodiments, the sensor system includes one or more imaging sensors (e.g., one or more cameras associated with a portable device or a head-mounted system), one or more depth sensors, and / or one or more light emitters.

[0134] In some embodiments, the system affordance icon 7214 is displayed in a mixed reality environment. For example, the system affordance icon 7214 is displayed by a display generating component (e.g., a display of the device 7100 or an HMD) along with at least a portion of a view of the physical environment captured by one or more cameras of the computer system (e.g., one or more rear-facing cameras of the device 7100, or a front or bottom-facing camera of an HMD). In some embodiments, the system affordance icon 7214 is displayed on a transparent or semi-transparent display of the device (e.g., a head-up display, or an HMD with a pass-through portion) through which the physical environment is visible. In some embodiments, the system affordance icon 7214 is displayed in a virtual reality environment (e.g., floating in virtual space).

[0135] In some embodiments, the computer system stops displaying the system affordance icon 7214 in response to detecting that the hand posture has changed and is no longer in the ready state configuration without providing a valid input gesture for the current user interface context. In some embodiments, the computer system stops displaying the system affordance icon 7214 and determines that the criteria for detecting a ready state configuration are no longer met in response to detecting that the hand remains in the ready state posture without providing a valid input gesture for more than a threshold time. In some embodiments, after ceasing to display the system affordance icon 7214, the computer system re-displays the system affordance icon (e.g., with the tip of the thumb in the new hand position) pursuant to a determination that the criteria for detecting a ready state configuration are again met due to a change in the posture of the user's hand.

[0136] In some embodiments, two or more ready state configurations of the hand are optionally defined and recognized by the computer system, with each ready state configuration of the hand causing the computer system to display a different type of affordance, enabling a different set of input gestures and / or actions to be performed in the current user interface context. For example, a second ready state configuration optionally causes all fingers to cooperate in a fist, with the thumb resting on the fingers other than the index finger. When the computer system detects this second ready state configuration, the computer system displays a different system affordance icon from icon 7214, and a subsequent input gesture (e.g., a thumb swipe across the index finger) causes the computer system to initiate a system shutdown operation or display a menu of power options (e.g., shutdown, sleep, suspend, etc.).

[0137] In some embodiments, the system affordance icon displayed in response to the computer detecting the hand in the ready configuration is a home affordance indicating a selection user interface including a plurality of currently installed applications, and is displayed in response to detecting a predetermined input gesture (e.g., a thumb flick input, thumb push input, or other input as described with respect to Figures 7A-7C). In some embodiments, an application dock including a plurality of application icons for launching respective applications is displayed in response to the computer detecting the hand in the ready configuration, and a subsequent activation or selection input gesture with the hand causes the computer system to launch a corresponding application.

[0138] 7F-7G provide various examples of actions that may be performed in response to an input gesture detected with the hands found in a ready state configuration, according to various embodiments. Although FIGS. 7F-7G illustrate different actions that may be performed in response to a user's gaze, it should be understood that in some embodiments, gaze is not a necessary element in the detection of a ready state gesture and / or an input gesture to trigger the execution of their various functions. In some embodiments, the combination of the user's gaze and hand configuration is used by the computer system in conjunction with the current user interface context to determine which action is performed.

[0139] 7F illustrates an example input gesture performed with the hand in a ready configuration and an example response of a displayed three-dimensional environment (e.g., a virtual reality environment or a mixed reality environment) according to some embodiments. In some embodiments, the computer system (e.g., device 7100 or HMD) displays a system affordance (e.g., system affordance icon 7214) at the tip of the thumb to indicate that a ready configuration of the hand has been detected and that the system gesture (e.g., a thumb flick gesture to display a dock or system menu, a thumb tap gesture to activate a voice-based assistant) is enabled to trigger a predefined system action, for example, in addition to other input gestures already available in the current user interface context (e.g., a thumb swipe input to scroll user interface objects, etc.). In some embodiments, the system affordance moves according to the movement of the entire hand and / or thumb in space when providing the gesture input, such that the position of the system affordance remains fixed relative to a predefined portion of the hand (e.g., the tip of the thumb of hand 7200).

[0140] According to some embodiments, portions (A)-(C) of FIG. 7F show a three-dimensional environment 7300 (e.g., a virtual reality environment or a mixed reality environment) displayed by a display generation component of a computer system (e.g., a touchscreen display of device 7100 or a display of a stereoscopic projector or HMD). In some embodiments, device 7100 is a handheld device (e.g., a mobile phone, tablet, or other mobile electronic device) including a display, a touch-sensitive display, or the like. In some embodiments, device 7100 represents a wearable headset including a head-up display, a head-mounted display, or the like.

[0141] In some embodiments, the three-dimensional environment 7300 is a virtual reality environment that includes virtual objects (e.g., user interface objects 7208, 7210, and 7212). In some embodiments, the virtual reality environment does not correspond to a physical environment in which the device 7100 is located. In some embodiments, the virtual reality environment corresponds to the physical environment (e.g., at least some of the virtual objects are displayed in positions in the virtual reality environment that correspond to positions of physical objects in the corresponding physical environment based on positions of the physical objects in the physical environment determined using one or more cameras of the device 7100). In some embodiments, the three-dimensional environment 7300 is a mixed reality environment. In some embodiments, the device 7100 includes one or more cameras, the one or more cameras configured to continuously provide a live view of at least a portion of the surrounding physical environment that is within the field of view of the one or more cameras of the device 7100, and the mixed reality environment corresponds to a portion of the surrounding physical environment that is within the field of view of the one or more cameras of the device 7100. In some embodiments, the mixed reality environment includes at least in part the live view of the one or more cameras of the device 7100. In some embodiments, the mixed reality environment includes one or more virtual objects that are displayed in place of (e.g., superimposed, overlaid, or substituted for) the live camera view (e.g., based on the positions of physical objects in the physical environment determined using one or more cameras, or at positions in the three-dimensional environment 7300 that correspond to the positions of physical objects in the physical environment using a live view of one or more cameras of the device 7100). In some embodiments, the display of the device 7100 includes a head-up display that is at least partially transparent (e.g., less than a threshold transparency, such as less than 25%, 20%, 15%, 10%, or 5%, or has a pass-through portion) so that the user can see at least a portion of the surrounding physical environment through at least partially transparent areas of the display.In some embodiments, the three-dimensional environment 7300 includes one or more virtual objects (e.g., container 7206 including user interface objects 7208, 7210, and 7212) displayed on a display. In some embodiments, the three-dimensional environment 7300 includes one or more virtual objects displayed on a transparent region of the display such that they appear superimposed on a portion of the surrounding physical environment that is visible through the transparent region of the display. In some embodiments, the one or more respective virtual objects are displayed at positions in the three-dimensional environment 7300 that correspond to the positions of the physical objects in the physical environment such that the respective virtual objects are displayed in place of the respective physical objects (e.g., so as to obscure and replace the view of the respective physical objects) (e.g., based on the positions of the physical objects in the physical environment as determined using one or more cameras of the device 7100 monitoring the portions of the physical environment that are visible through the transparent region of the display).

[0142] In some embodiments, a sensor system of a computer system (e.g., one or more cameras of device 7100 or an HMD) tracks the position and / or movement of one or more features of a user, such as the user's hand. In some embodiments, the position and / or movement of the user's hand(s) (e.g., fingers) serves as an input to a computer system (e.g., device 7100 or an HMD). In some embodiments, the user's hand is within the field of view of one or more cameras of a computer system (e.g., device 7100 or an HMD) and the position and / or movement of the user's hand is tracked by a sensor system of the computer system (e.g., a control unit of device 7100 or an HMD) as input to the computer system (e.g., device 7100 or an HMD), but the user's hand is not shown in the three dimensional environment 7300 (e.g., the three dimensional environment 7300 does not include a live view from one or more cameras, the hand is edited out of the live view of one or more cameras, or the user's hand is within the field of view of one or more cameras outside of a portion of the field of view displayed in the live view of the three dimensional environment 7300). In some embodiments, a hand 7200 (e.g., a representation of a user's hand or a representation of a portion of a hand that is within the field of view of one or more cameras of the device 7100) is visible in the three-dimensional environment 7100 (e.g., displayed as a rendered representation, displayed as part of a live camera view, or visible through a pass-through portion of a display), similar to the example shown in FIG. 7F. In FIG. 7F, the hand 7200 is detected to be in a ready configuration (e.g., thumb in the middle of index finger) for providing gesture input (e.g., hand gesture input). In some embodiments, a computer system (e.g., the device 7100 or HMD) determines that the hand 7200 is in a ready configuration by performing image analysis on the live view of one or more cameras. Further details regarding hand ready configurations and input gestures are provided at least in FIGS. 7A-7E and accompanying description and will not be repeated here for brevity.

[0143] According to some embodiments, portion (A) of FIG. 7F illustrates a first type of input gesture (e.g., a thumb flick gesture or a thumb push gesture) that includes movement of the thumb of the hand 7200 across a portion of the index finger of the hand 7200 (e.g., across the middle phalanx from the palmar to the posterior side of the index finger) along an axis indicated by the arrow 7120 (e.g., as illustrated by the transition of the hand configuration from A(1) to A(2) in FIG. 7F). As noted above, in some embodiments, the movement of the hand 7200 while performing the gesture as illustrated in FIG. 7F (e.g., including the movement of the entire hand and the relative movements of the individual fingers during the gesture), and the position of the hand 7200 during the gesture (e.g., including the position of the entire hand and the relative positions of the individual fingers) are tracked by one or more cameras of the device 7100. In some embodiments, the device 7100 detects the gesture performed by the hand 7200 by performing image analysis on the live view of the one or more cameras. In some embodiments, pursuant to a determination that a thumb flick gesture is provided by the hand beginning with the hand ready state configuration, the computer system executes a first action corresponding to the thumb flick gesture (e.g., a system action such as displaying a menu 7170 including multiple application launch icons, or an action corresponding to the current user interface context that would not be enabled if the hand was not initially found in the ready state configuration). In some embodiments, additional gestures are provided by the hand 7200 to interact with the menu. For example, a subsequent thumb flick gesture after displaying the menu 7170 drives the menu 7170 into the three-dimensional environment, causing it to appear in an enhanced state (e.g., with a larger, animated representation of the menu items) in the virtual space. In some embodiments, a subsequent thumb swipe gesture scrolls the selection indicator horizontally within the currently selected row of the menu, and a subsequent thumb push or thumb pull gesture scrolls the selection indicator up or down across different rows of the menu. In some embodiments, a subsequent thumb tap gesture deactivates the currently selected menu item (e.g., an application icon) that is launched in the three-dimensional environment.

[0144] According to some embodiments, portion (B) of FIG. 7F illustrates a second type of gesture (e.g., a thumb swipe gesture) that includes movement of the thumb of hand 7200 along the length of the index finger of hand 7200 along the axis indicated by arrow 7130 (e.g., as shown by the transition of the hand configuration from B(1) to B(2) in FIG. 7F). In some embodiments, the gesture enables interaction with virtual objects (e.g., virtual objects 7208, 7210, and 7212 within container object 7206) in the current user interface context regardless of whether the hand was initially found in the ready state configuration. In some embodiments, different types of interaction with container object 7206 are enabled depending on whether the thumb swipe gesture was initiated from the ready state configuration. In some embodiments, pursuant to a determination that the thumb swipe gesture was provided by the hand originating from the ready state configuration, the computer system performs a second action corresponding to the thumb swipe gesture (e.g., scrolling the view of the container object 7206 to display one or more virtual objects within the container object 7206 that were initially invisible to the user, or another action corresponding to the current user interface context that would not be enabled if the hand was not initially found in the ready state configuration). In some embodiments, additional gestures are provided by the hand 7200 to interact with the container object or to perform system actions. For example, a subsequent thumb flick gesture causes the menu 7170 to be displayed (e.g., as shown in FIG. 7F, part (A)). In some embodiments, a subsequent thumb swipe gesture in a different direction scrolls the view of the container object 7206 in the opposite direction. In some embodiments, a subsequent thumb tap gesture activates a currently selected virtual object within the container 7206. In some embodiments, if the thumb swipe gesture was not initiated from the ready state configuration, in response to the thumb swipe gesture, the selection indicator is shifted through the virtual objects in the container 7206 in the direction of movement of the thumb of the hand 7200.

[0145] According to some embodiments, portion (C) of FIG. 7F illustrates a third type of gesture input (e.g., a thumb tap gesture) (e.g., a tap of the thumb of the hand 7200 on a predetermined portion (e.g., the middle phalanx) of the index finger of the hand 7200) (e.g., by moving the thumb from an up position downward along the axis indicated by the arrow 7110) (e.g., as indicated by the transition of the hand configuration from C(1) to C(2) in FIG. 7F). In some embodiments, the thumb tap gesture requires the thumb to be removed from the index finger to be completed. In some embodiments, the gesture enables an interaction with the virtual objects (e.g., virtual objects 7208, 7210, and 7212 within the container object 7206) in the current user interface context regardless of whether the hand was initially found in the ready state configuration. In some embodiments, different types of interactions with the container object 7206 are enabled depending on whether the thumb tap gesture was initiated from the ready state configuration. In some embodiments, pursuant to a determination that a thumb tap gesture is provided by the hand initiated from the ready state configuration of the hand (e.g., the gesture includes moving the thumb upward away from the index finger prior to tapping the thumb on the index finger), the computer system performs a third action corresponding to the thumb tap gesture (e.g., activation of the voice-based assistant 7302 or a communication channel (e.g., a voice communication application), or another action corresponding to the current user interface context that would not be enabled if the hand was not initially found in the ready state configuration). In some embodiments, additional gestures are provided by the hand 7200 to interact with the voice-based assistant or communication channel. For example, a subsequent thumb flick gesture drives the voice-based assistant or voice communication user interface from a position adjacent to the user's hand to a more distant position in space within the three-dimensional environment. In some embodiments, a subsequent thumb swipe gesture scrolls through different pre-configured functions of the voice-based assistant or scrolls through a list of potential recipients of the voice communication channel.In some embodiments, a subsequent thumb tap gesture causes the voice-based assistant or voice communication channel to be disengaged. In some embodiments, if the thumb tap gesture was not initiated from a ready state configuration, then in response to the thumb tap gesture in portion (C) of FIG. 7E, an action available in the current user interface context is activated (e.g., the currently selected virtual object is activated).

[0146] The example shown in Figure 7F is merely illustrative. By providing additional and / or different actions in response to detection of an input gesture initiated from a hand ready configuration, a user can perform additional functions without cluttering the user interface with controls, further streamlining the user interface by reducing the number of user inputs to perform those functions, and saving time during user interaction with the device.

[0147] The user interface interactions shown in Figures 7A-7F are described independent of the position of the user's gaze. In some embodiments, the interactions are not dependent on the position of the user's gaze or the exact location of the user's gaze within a portion of the three-dimensional environment. However, in some embodiments, the gaze is utilized to modify the system's response behavior, and the actions and user interface feedback are changed in response to different positions of the user when user input is detected. Figure 7G illustrates an example gesture performed with a ready hand and an example response of the displayed three-dimensional environment in response to the user's gaze, according to some embodiments. For example, the left column of the figures (e.g., portions A-0, A-1, A-2, and A-3) illustrate example scenarios in which one or more gesture inputs (e.g., a thumb flick gesture, a thumb swipe gesture, a thumb tap gesture, or a sequence of two or more of the above) are provided by the hand starting from a ready state configuration (e.g., thumb resting on index finger) (e.g., the ready state configuration described in connection with Figures 7E and 7F) while the user's gaze is focused on the user's hand (e.g., as shown in Figure 7G, portion A-0). The right column of the figures (e.g., parts B-0, B-1, B-2, and B-3) illustrate example scenarios in which one or more gesture inputs (e.g., a thumb flick gesture, a thumb swipe gesture, a thumb tap gesture, or a sequence of two or more of these) are provided by the hand starting from a ready state configuration (e.g., thumb resting on index finger) (e.g., the ready state configuration described in connection with Figures 7E and 7F) while the user's gaze is focused on a user interface environment (e.g., a user interface object or a physical object in a three-dimensional environment) other than the user's hand in the ready state configuration (e.g., as shown in Figure 7G, part A-0).In some embodiments, the user interface responses and interactions described with respect to FIG. 7G are performed with respect to other physical objects or predetermined portions thereof (e.g., the top or front surface of the housing of the physical media player device, a physical window in a wall of a room, a physical controller device, etc., as opposed to the user's hand providing the input gesture), thereby enabling special interactions for input gestures provided by the hand starting from the ready state configuration while the user's gaze is focused on said other physical objects or predetermined portions thereof. The diagrams in the left and right columns of the same row of FIG. 7G (e.g., A-1 and B-1, A-2 and B-2, A-3, and B-3) illustrate different user interface responses for the same input gesture provided in the same user interface context depending on whether the user's gaze is focused on the user's hand (or other physical object defined as a physical object being controlled or controlled by a computer system) according to some embodiments. The input gestures described with reference to FIG. 7G are used to illustrate the processes described below, including the process in FIG. 10.

[0148] In some embodiments, as shown in FIG. 7G, a system affordance (e.g., affordance 7214) is optionally displayed at a predetermined location in three dimensions corresponding to the hand in the ready state configuration. In some embodiments, the system affordance is always displayed at a predetermined location (e.g., a static location or a dynamically determined location) whenever the hand is determined to be in the steady state configuration. In some embodiments, the dynamically determined location of the system affordance is fixed relative to the user's hand in the ready state configuration, for example, when the user's entire hand moves while remaining in the ready state configuration. In some embodiments, the system affordance is displayed (e.g., at a static location at or near the tip of the thumb) in response to the user's gaze being directed toward a predetermined physical object (e.g., the user's hand in the ready state configuration, or another predetermined physical object in the environment) while the user's hand is held in the ready state configuration, and is ceased to be displayed in response to the user's hand moving away from the ready state configuration and / or the user's gaze moving away from the predetermined physical object. In some embodiments, the computer system displays a system affordance having a first appearance (e.g., an enlarged, prominent appearance) in response to the user's gaze being directed toward a predetermined physical object (e.g., the user's hand in the ready state configuration or another predetermined physical object in the environment) while the user's hand is held in the ready state configuration, and displays a system affordance having a second appearance (e.g., a reduced, less prominent appearance) in response to the user's hand moving away from the ready state configuration and / or the user's gaze moving away from the predetermined physical object. In some embodiments, the system affordance is displayed (e.g., at a static position at or near the tip of the thumb) in response to an indication that the user is ready to provide input (e.g., the user's hand being raised from a previous elevation relative to the body while the user's hand is held in the ready state configuration) (e.g., regardless of whether the user's gaze is directed toward the user's hand) and is ceased from displaying in response to the user's hand being lowered from the raised state.In some embodiments, the system affordances change appearance (e.g., from a simple indicator to an object menu) in response to an indication that the user is ready to provide input (e.g., the user's hands are in a ready state configuration and / or the user's hands are raised from a previous elevation relative to the body while the user's gaze is focused on the user's hands in the ready state configuration or on a predetermined physical object), and revert the appearance of the system affordances (e.g., to a simple indicator) in response to cessation of the indication that the user is ready to provide input (e.g., the user's hands are lowered from a raised state and / or the user's gaze moves away from the user's hands or on a predetermined physical object). In some embodiments, the indication that the user is ready to provide input includes one or more of the user's fingers touching the physical controller or the user's hand (e.g., the index finger is on the controller or the thumb is on the index finger), the user's hand rising from a low to a high position relative to the user's body (e.g., there is a wrist rotation upwards of the hand in the ready configuration or a flexion of the elbow of the hand in the ready configuration), changing the hand configuration to the ready configuration, etc. In some embodiments, the physical location of the given physical object that is compared to the position of the user's gaze in these embodiments is static with respect to the three-dimensional environment (e.g., also referred to as "anchored to the world"). For example, the system affordances are displayed on a wall in the three-dimensional environment. In some embodiments, the physical location of the given physical object that is compared to the position of the user's gaze in these embodiments is static with respect to the display (e.g., the display generating component) (e.g., also referred to as "anchored to the user"). For example, the system affordances are displayed at the bottom of the display or the user's field of view. In some embodiments, the physical position of a given physical object that is compared to the position of the user's gaze in these embodiments is static with respect to moving parts of the user (e.g., the user's hand) or moving parts of the physical environment (e.g., a car moving on a highway).

[0149] According to some embodiments, portion A-0 of FIG. 7G shows a user 7320 directing his / her gaze at a given physical object (e.g., a hand in a ready configuration) in a physical environment in a three-dimensional environment (e.g., the user's hand 7200 or a representation thereof is visible within the field of view of one or more cameras of device 7100 or through a pass-through or transparent portion of an HMD or head-up display). In some embodiments, device 7100 uses one or more cameras (e.g., front-facing cameras) that are pointed at the user and track the movement of the user's eyes (or track the movement of both of the user's eyes) to determine the direction and / or target of the user's gaze. Further details of an exemplary eye-tracking technique are provided with respect to FIGS. 1-6, and in particular with respect to FIGS. 5 and 6, according to some embodiments. In portion A-0 of FIG. 7G , while the hand 7200 is in the ready state, the user's gaze is directed toward the hand 7200 (e.g., as indicated by dotted lines associating the user's eyeballs 7512 with a representation of the user 7200 or the user's hand 7200′ (e.g., an actual hand or a representation of the hand presented via a display generation component)), so that a system user interface action (e.g., a user interface action associated with the system affordance 7214 or a system menu associated with the system affordance (e.g., menu 7170) rather than a user interface action associated with other areas or elements of the user interface or with individual software applications executing on the device 7100) is performed in response to a gesture performed with the hand 7200. Part B-0 of FIG. 7G shows a user with their line of sight away from a predetermined physical object (e.g., a hand in a ready state or other predetermined physical object) in a three-dimensional environment (e.g., the user's hand 7200 or other predetermined physical object, or a representation thereof that is within the field of view of one or more cameras of device 7100 or visible through a pass-through or transparent portion of an HMD or head-up display) while the user's hand is in a ready state configuration.In some embodiments, the computer system requires that the user's gaze remains on a predefined physical object (e.g., the user's hand or another predefined physical object) while the hand input gesture is being processed in order to provide a system response corresponding to the input gesture. In some embodiments, the computer system requires that the user's gaze remains on the predefined physical object for more than a threshold time and with a predefined stability (e.g., remains substantially stationary or maintains less than a threshold amount of movement for a threshold time) in order to provide a system response corresponding to the input gesture; for example, the gaze can optionally move away from the predefined input gesture after the time and stability requirements are met and before the input gesture is completed.

[0150] In one example, portion A-1 of Figure 7G illustrates a thumb flick gesture beginning with the hand in a ready state configuration and including a forward movement of the thumb across the index finger of hand 7200 along the axis indicated by arrow 7120. In response to the thumb flick gesture beginning with the ready state configuration of portion A-1 of Figure 7G, and pursuant to a determination that the user's gaze is directed toward a predetermined physical object (e.g., hand 7200 as viewed in the real world or via a display generation component of the computer system), the computer system displays a system menu 7170 (e.g., a menu of application icons) within the three-dimensional environment (e.g., replacing system affordance 7214 with the tip of the user's thumb).

[0151] In another example, part A-2 of FIG. 7G illustrates a thumb swipe gesture that starts from a ready state configuration and includes moving the thumb of the hand 7200 along the length of the index finger of the hand 7200 along the axis indicated by the arrow 7130. In this example, the hand gesture of part A-2 of FIG. 7G is performed while the computer system is displaying a system menu (e.g., menu 7170) (e.g., the system menu 7170 is displayed in response to a thumb flick gesture described herein with reference to part A-1 of FIG. 7G). In response to the thumb swipe gesture of part A-2 of FIG. 7G, starting from the ready state configuration and pursuant to a determination that the user's gaze is directed toward a predetermined physical object (e.g., hand 7200) (e.g., the gaze meets predetermined location, duration, and stability requirements), the computer system moves a current selection indicator 7198 on the system menu (e.g., menu 7170) in the direction of movement of the thumb of the hand 7200 (e.g., to an adjacent user interface object on the menu 7170). In some embodiments, the thumb swipe gesture is one of a sequence of two or more input gestures that are initiated with the hand in a ready configuration and represent a continuous series of user interactions with a system user interface element (e.g., a system menu or a system control object, etc.). Thus, in some embodiments, the requirement that the gaze remain on a given physical object (e.g., the user's hand) is, optionally, applied only to the initiation of a first input gesture (e.g., the thumb flick gesture of part A-1 of FIG. 7G) and is not imposed on subsequent input gestures as long as the user's gaze is directed at a system user interface element during the subsequent input gesture. For example, following a determination that the user's gaze is directed at the user's hand, or following a determination that the user's gaze is directed at a system menu located at a fixed position relative to the user's hand (e.g., the tip of the thumb), the computer system performs a system action in response to the thumb swipe gesture (e.g., navigate within a system menu).

[0152] In yet another example, portion A-3 of FIG. 7G illustrates a thumb tap gesture that begins with the hand in a ready configuration and includes moving the thumb of hand 7200 to tap on the index finger of hand 7200 (e.g., by moving the thumb from the index finger to a raised position and then moving from the raised position downward along the axis indicated by arrow 7110 to a position of the hand where the thumb again contacts the index finger). In this example, the thumb tap gesture of portion A-3 of FIG. 7G is performed while the computer system is displaying a system menu (e.g., menu 7170) and a current selection indicator is displayed on the corresponding user interface object (e.g., in response to a thumb swipe gesture described herein with reference to portion A-2 of FIG. 7G). In response to the thumb tap gesture of part A-3 of FIG. 7G, in response to the user's gaze being directed toward a predetermined physical object (e.g., hand 7200), the currently selected user interface object 7190 is activated and an action corresponding to the user interface object 7190 is performed (e.g., the display of menu 7170 is stopped, a user interface object 7306 (e.g., a preview or control panel, etc.) associated with the user interface object 7190 is displayed, and / or an application corresponding to the user interface object 7190 is launched). In some embodiments, the user interface object 7306 is displayed at a location in the three-dimensional environment that corresponds to the position of the hand 7200. In some embodiments, the thumb tap gesture is one of a sequence of two or more input gestures that are initiated with the hand in a ready configuration and represent a continuous series of user interactions with system user interface elements (e.g., a system menu or a system control object, etc.).Thus, in some embodiments, the requirement for the gaze to remain on a given physical object (e.g., the user's hand) optionally applies only to the initiation of a first input gesture (e.g., the thumb flick gesture of part A-1 of FIG. 7G) and is not imposed on subsequent input gestures (e.g., the thumb swipe gesture of part A-2 and the thumb tap gesture of part A-3 of FIG. 7G) so long as the user's gaze is directed at a system user interface element during the subsequent input gesture. For example, following a determination that the user's gaze is directed at the user's hand, or that the user's gaze is directed at a system menu located at a fixed position relative to the user's hand (e.g., the tip of the thumb), the computer system performs a system action in response to the thumb tap gesture (e.g., activates a currently selected user interface object in the system menu).

[0153] In contrast to portion A-0 of FIG. 7G, portion B-0 of FIG. 7G shows a user with their gaze away from a given physical object (e.g., a hand in a ready configuration) in the three-dimensional environment (e.g., the user's hand 7200 or a representation thereof is within the field of view of one or more cameras of the device 7100, but the user's gaze is not on the hand 7100 (e.g., directly or through a pass-through or transparent portion of an HMD or head-up display, or through a camera view)). Instead, the user's gaze is directed toward a container 7206 (e.g., a row of a menu) of a user interface object (e.g., as described herein with reference to the container 7206 of FIGS. 7D-7E) or the displayed user interface in general (e.g., the user's gaze does not meet the stability and duration requirements of a particular location or object in the three-dimensional environment). Following a determination that the user's gaze is away from the predetermined physical object (e.g., hand 7200), the computer system does not perform a system user interface action (e.g., a user interface action associated with the system affordance 7214 or a function of displaying a system menu (e.g., a menu of application icons) associated with the system affordance (e.g., as shown in part A-1 of FIG. 7G )) in response to the thumb flick gesture performed using the hand 7200. Optionally, instead of a system user interface action, a user interface action associated with another region or element of the user interface or associated with an individual software application running on the computer system (e.g., device 100 or HMD) is performed in response to a thumb flick gesture performed using the hand 7200 while the user's gaze is directed away from the predetermined physical object (e.g., ready hand 7200). In one example, as shown in FIG. 7G , the entire user interface including the container 7206 is scrolled upwards in accordance with the upward thumb flick gesture.In another example, a user interface action associated with container 7206 is executed in response to a gesture performed using hand 7200 instead of a system user interface action (e.g., displaying a system menu) while the user's gaze is directed from hand 7200 to container 7206.

[0154] Similar to portion A-1 of FIG. 7G, portion B-1 of FIG. 7G also illustrates a thumb flick gesture by hand 7200 starting from the ready state configuration. In contrast to the behavior illustrated in portion A-1 of FIG. 7G, in portion B-1 of FIG. 7G, the computer system does not display a system menu in response to the thumb flick gesture in accordance with a determination that the user's gaze is not directed toward a predetermined physical object (e.g., the user's hand in the ready state configuration). Instead, the user interface is scrolled upward in accordance with the thumb flick gesture, e.g., container 7206 is moved upward in the three-dimensional environment in accordance with the movement of the thumb of hand 7200 across the index finger of hand 7200 and in accordance with the user's gaze being directed from hand 7200 to container 7206. In this example, no system action is performed, but the system affordance 7214 indicating that a system action is available (e.g., because the user's hands are in the ready configuration) remains displayed next to the user's thumb, and the system affordance 7214 moves with the user's thumb during the input gesture as the user interface is scrolled upward in response to the input gesture.

[0155] Similar to portion A-2 of FIG. 7G, portion B-2 of FIG. 7G also illustrates a thumb swipe gesture by hand 7200 initiated from a ready state configuration. In contrast to portion A-2 of FIG. 7G, the thumb swipe gesture of portion B-2 of FIG. 7G is performed while a system menu (e.g., menu 7170) is not displayed (e.g., because menu 7170 was not displayed in response to a flick gesture described herein with reference to portion B-1 of FIG. 7G) and the user's gaze is not focused on a predetermined physical object (e.g., the user's hand). In response to the thumb swipe gesture of portion B-2 of FIG. 7G, a current selection indicator within container 7206 is scrolled in the direction of movement of the thumb of hand 7200 pursuant to a determination that the user's gaze is directed away from the predetermined physical object (e.g., hand 7200) and toward container 7206.

[0156] Similar to portion A-3 of FIG. 7G, portion B-3 of FIG. 7G also illustrates a thumb tap gesture by hand 7200 initiated from a ready state configuration. In contrast to portion A-3 of FIG. 7G, the thumb tap gesture of portion B-3 of FIG. 7G is performed while a system menu (e.g., menu 7170) is not displayed (e.g., because menu 7170 was not displayed in response to a flick gesture described herein with reference to portion B-1 of FIG. 7G) and the user's gaze is not focused on a predetermined physical object (e.g., the user's hand). In response to the thumb tap gesture of portion A-3 of FIG. 7G, the computer system does not perform a system operation pursuant to the user's gaze being directed away from a predetermined physical object (e.g., hand 7200, in the absence of a system user interface element in response to a previously received input gesture). In response to a determination that the user's gaze is directed toward container 7206, a currently selected user interface object within container 7206 is activated, and the currently selected user interface object is activated. An action corresponding to the face object is performed (e.g., container 7206 is deactivated, and user interface 7308 corresponding to the activated user interface object is displayed). In this example, no system action is performed, but system affordance 7214 indicating that a system action is available (e.g., because the user's hand is in the ready configuration) remains displayed next to the user's thumb, and system affordance 7214 moves with the user's thumb during the input gesture as the user interface scrolls upward in response to the input gesture. In some embodiments, because user interface object 7308 was activated from container 7206, user interface object 7308 is displayed at a location in the three-dimensional environment that does not correspond to the location of hand 7200, in contrast to user interface object 7306 of FIG. 7G part A-3.

[0157] It should be understood that in the example shown in FIG. 7G, the computer system treats the user's hand 7200 as a predefined physical object whose position (e.g., compared to the user's gaze) is used to determine whether a system action should be performed in response to a predefined gesture input. Although the position of the hand 7200 appears different on the display in the examples shown in the left and right columns of FIG. 7G, this is an indication that the position of the user's gaze has changed relative to the three-dimensional environment, and does not necessarily impose any restrictions on the position of the user's hand relative to the three-dimensional environment. In fact, in most situations, the user's entire hand is often not fixed in position during an input gesture, and the gaze is compared to the moving physical position of the user's hand to determine whether the gaze is focused on the user's hand. In some embodiments, when another physical object in the user's environment other than the user's hand is used as the predefined physical object for determining whether a system action should be performed in response to a predefined gesture input, the gaze is compared to the physical position of the physical object even if the physical object is moving relative to the environment or the user and / or the user is moving relative to the physical object.

[0158] In the example shown in FIG. 7G, according to some embodiments, whether the user's gaze is directed at a predetermined physical location (e.g., focused on a physical object (e.g., the user's hand in a ready state configuration or a system user interface object displayed in a fixed position relative to the predetermined physical object)) is used in conjunction with whether the input gesture is initiated from the hand in a ready state configuration to determine whether to perform a system action (e.g., display a system user interface or a system user interface object) or to perform an action within the current context of the three-dimensional environment without performing a system action. 7H-7J show example behaviors of a displayed three-dimensional environment (e.g., a virtual reality or mixed reality environment) that depend on whether the user is ready to provide gesture input (e.g., the user's hands meet certain requirements (e.g., elevated to a predetermined altitude and placed in a ready state configuration for at least a threshold time) in conjunction with the user's gaze meeting certain requirements (e.g., the gaze is focused on an activatable virtual object and meets stability and duration requirements) in accordance with some embodiments. The input gestures described with reference to FIGS. 7H-7J are used to illustrate the processes described below, including the process in FIG. 11.

[0159] FIG. 7H illustrates an exemplary computer-generated environment that corresponds to a physical environment. As described herein with reference to FIG. 7H, the computer-generated environment may be a virtual reality environment, an augmented reality environment, or a computer-generated environment displayed on a display, where the computer-generated environment is displayed on the display so as to be superimposed on a view of the physical environment that is visible through a transparent portion of the display. As shown in FIG. 7H, a user 7502 is standing (e.g., holding a device 7100 or wearing an HMD) in a physical environment (e.g., scene 105) in which a computer system (e.g., computer system 101) is operated. In some embodiments, as in the example shown in FIG. 7H, the device 7100 is a handheld device (e.g., a mobile phone, tablet, or other mobile electronic device) including a display, a touch-sensitive display, or the like. In some embodiments, the device 7100 represents, and is optionally replaced by, a wearable headset including a head-up display, a head-mounted display, or the like. In some embodiments, the physical environment includes one or more physical surfaces and physical objects (e.g., walls of a room, (e.g., represented by a shaded 3D box 7504)) surrounding the user 7502.

[0160] In the example shown in part (B) of FIG. 7H, a computer-generated three-dimensional environment corresponding to a physical environment (e.g., a portion of the physical environment that is within the field of view of one or more cameras of device 7100 or visible through a transparent portion of a display of device 7100) is displayed on device 7100. The physical environment includes a physical object 7504 represented by an object 7504' in the computer-generated environment shown on the display (e.g., the computer-generated environment is a virtual reality environment that includes a virtual representation of the physical object 7504, the computer-generated environment is an augmented reality environment that includes a representation 7504' of the physical object 7504 as part of a live view of one or more cameras of device 7100, or the physical object 7504 is visible through a transparent portion of a display of device 7100). Additionally, the computer-generated environment shown on the display includes virtual objects 7506, 7508, and 7510. The virtual object 7508 is displayed such that it appears attached to the object 7504' (e.g., overlaying the flat front surface of the physical object 7504). Virtual object 7506 is displayed to appear to be attached to a wall of the computer-generated environment (e.g., overlapping a portion of a wall or a representation of a wall in the physical environment). Virtual object 7510 is displayed to appear to be attached to a floor of the computer-generated environment (e.g., overlapping a portion of the floor or a representation of a floor in the physical environment). In some embodiments, virtual objects 7506, 7508, and 7510 are activatable user-interface objects that perform object-specific actions when activated by user input. In some embodiments, the computer-generated environment also includes virtual objects that are not activatable by user input, but are displayed to improve the aesthetic qualities of the computer-generated environment and provide information to the user.Part (C) of Figure 7H illustrates that, according to some embodiments, the computer-generated environment shown on device 7100 is a three-dimensional environment, and as the perspective of device 7100 with respect to the physical environment changes (e.g., as the viewing angle of device 7100 or one or more cameras of device 7100 with respect to the physical environment changes in response to movement and / or rotation of device 7100 within the physical environment), the perspective of the computer-generated environment displayed on device 7100 changes accordingly (e.g., including changes in physical surfaces and objects (e.g., walls, floors, physical object 7504) and virtual objects 7506, 7508, and 7510).

[0161] FIG. 7I illustrates an exemplary behavior of a computer-generated environment in response to a user directing his / her gaze to a respective virtual object in the computer-generated environment while the user 7502 is not ready to provide a gesture input (e.g., the user's hand is not in a ready state configuration). As illustrated in parts (A)-(C) of FIG. 7I, the user holds his / her left hand 7200 in a state other than the ready state (e.g., in a position other than a first predefined ready state configuration) to provide a gesture input. In some embodiments, the computer system determines that the user's hand is in a predefined ready state for providing a gesture input according to detecting that a predefined portion of the user's finger is touching a physical control element (e.g., the thumb is touching the center of the index finger, or the index finger is touching a physical controller). In some embodiments, the computer system determines that the user's hand is in a predefined ready state for providing gesture input according to detecting that the user's hand is elevated above a predefined altitude relative to the user (e.g., the hand is elevated in response to arm rotation about an elbow joint, or wrist rotation about a wrist, or fingers elevated relative to the hand, etc.). In some embodiments, the computer system determines that the user's hand is in a predefined ready state for providing gesture input according to detecting that the posture of the user's hand has changed to a predefined configuration (e.g., thumb is placed between index finger, fingers are closed in a fist, etc.). In some embodiments, multiple of the above requirements are combined to determine whether the user's hand is in a ready state for providing gesture input. In some embodiments, the computer system also requires that the user's entire hand is stationary (e.g., less than a threshold amount of movement without a threshold time) to determine that the hand is ready to provide gesture input.According to some embodiments, if the user's hands are not found to be in a ready state to provide gesture input and the user's gaze is focused on an activatable virtual object, subsequent movements of the user's hands (e.g., free movements or movements mimicking a pre-defined gesture) are not recognized and / or treated as user input directed at the virtual object that is the focus of the user's gaze.

[0162] In this example, a representation of the hand 7200 is displayed in the computer-generated environment. The computer-generated environment does not include a representation of the user's right hand (e.g., because the right hand is not within the field of view of one or more cameras of the device 7100). Furthermore, in some embodiments, for example in the example shown in FIG. 7I where the device 7100 is a handheld device, the user can see portions of the surrounding physical environment separately from any representation of the physical environment displayed on the device 7100. For example, portions of the user's hand are visible to the user outside of the display of the device 7100. In some embodiments, the device 7100 in these examples represents or can be replaced by a headset having a display (e.g., a head-mounted display) that completely blocks the user's view of the surrounding physical environment. In some such embodiments, no part of the physical environment is directly visible to the user, but instead the physical environment is visible to the user via a representation of a portion of the physical environment displayed by the device. In some embodiments, the user's hand(s) are not visible to the user, either directly or via the display of device 7100, while the current state of the user's hands is continuously or periodically monitored by the device to determine whether the user's hand(s) have entered a state ready for providing a gesture input. In some embodiments, the device displays an indicator of whether the user's hands are in a state ready for providing an input gesture, provides feedback to the user, and alerts the user to adjust the position of their hands if they wish to provide an input gesture.

[0163] In portion (A) of FIG. 7I, the user's gaze is directed toward the virtual object 7506 (e.g., as indicated by the dotted line connecting the user's eyeball 7512 and a representation of the virtual object 7506). In some embodiments, the device 7100 uses one or more cameras (e.g., a front-facing camera) that are pointed toward the user and track the movement of the user's eyes (or track the movement of both of the user's eyes) to determine the direction and / or subject of the user's gaze. Further details of eye tracking or gaze tracking techniques are provided in FIGS. 1-6, particularly FIGS. 5-6, and the accompanying description. In portion (A) of FIG. 7I, in response to the user directing his or her gaze toward the virtual object 7506, pursuant to a determination that the user's hands are not ready to provide a gesture input (e.g., the left hand is not stationary and is not held in a first predetermined ready state configuration for more than a threshold time), no action is performed on the virtual object 7506. Similarly, in portion (B) of FIG. 7I , the user's gaze has a left virtual object 7506 and is currently directed toward virtual object 7508 (e.g., as indicated by the dotted line connecting a representation of the user's eyeball 7512 and virtual object 7508). In response to the user directing his gaze toward virtual object 7508 in accordance with a determination that the user's hands are not ready to provide gestural input, no action is performed on virtual object 7508. Similarly, in portion (C) of FIG. 7I , the user's gaze is directed toward virtual object 7510 (e.g., as indicated by the dotted line connecting a representation of the user's eyeball 7512 and virtual object 7510). In response to the user directing his gaze toward virtual object 7510 in accordance with a determination that the user's hands are not ready to provide gestural input, no action is performed on virtual object 7510.In some embodiments, requiring the user's hand to be in a ready state to provide gestural input in order to trigger a visual change indicating that a virtual object under the user's gaze is activatable by gestural input is advantageous because it tends to prevent unnecessary visual changes in the displayed environment when the user does not wish to interact with a particular virtual object in the environment but merely wishes to explore the environment (e.g., gazing briefly or purposefully at various virtual objects for a period of time), thereby reducing the user's visual fatigue and confusion, and thus the user's errors, when using a computer system to experience a computer-generated three-dimensional environment.

[0164] In contrast to the exemplary scenario shown in Figure 7I, Figure 7J illustrates exemplary behavior of a computer-generated environment in response to a user directing their gaze to respective virtual objects in the computer-generated environment while the user is ready to provide gesture input, according to some embodiments. As shown in parts (A)-(C) of Figure 7J, the user holds their hands in a first ready-state configuration for providing gesture input (e.g., thumb resting on index finger, hand elevated above a predetermined altitude relative to the user's body) while virtual objects 7506, 7608, and 7510 are displayed in the three-dimensional environment.

[0165] In portion (A) of Figure 7J, the user's gaze is directed toward virtual object 7506 (e.g., as indicated by a dotted line connecting a representation of the user's eyeball 7512 and virtual object 7506). In response to the user directing his or her gaze toward virtual object 7506 pursuant to a determination that the user's left hand is ready to provide gesture input (e.g., the gaze meets duration and stability requirements at virtual object 7506), the computer system provides visual feedback indicating that virtual object 7506 is activatable by gesture input (e.g., virtual object 7506 is highlighted, enlarged, or augmented with additional information or user interface detail to indicate that virtual object 7506 is interactable (e.g., one or more actions associated with virtual object 7506 may be performed in response to the user's gesture input)). Similarly, in portion (B) of Figure 7J, the user's gaze has moved away from virtual object 7506 and is now directed at virtual object 7508 (e.g., as indicated by the dotted line connecting the representation of the user's eyeball 7512 and virtual object 7508). In response to the user directing their gaze at virtual object 7508 pursuant to a determination that the user's hands are ready to provide gesture input (e.g., the gaze meets duration and stability requirements at virtual object 7508), the computer system provides visual feedback indicating that virtual object 7506 is activatable by gesture input (e.g., virtual object 7508 is highlighted, enlarged, or augmented with additional information or user interface detail to indicate that virtual object 7508 is interactable (e.g., one or more actions associated with virtual object 7508 may be performed in response to the user's gesture input)). Similarly, in portion (C) of FIG. 7J, the user's gaze has moved away from virtual object 7508 and is now directed toward virtual object 7510 (e.g., as indicated by the dotted line connecting the representation of the user's eyeball 7512 and virtual object 7510).Following a determination that the user's hands are ready to provide gesture input, the computer system, in response to detecting the user's gaze toward the virtual object 7510, provides visual feedback indicating that the virtual object 7506 can be activated by the gesture input (e.g., the virtual object 7510 is highlighted, enlarged, or augmented with additional information or user interface detail to indicate that the virtual object 7510 can be interacted with (e.g., indicating that one or more actions associated with the virtual object 7510 can be performed in response to the gesture input)).

[0166] In some embodiments, while the visual feedback indicating that the virtual object is activatable by gesture input is displayed, and in response to detecting a gesture input initiated from the user's ready hand, the computer system performs an action corresponding to the virtual object that is the subject of the user's gaze in accordance with the user's gesture input. In some embodiments, the visual feedback indicating that the respective virtual object is activatable by gesture input ceases to be displayed in response to the user's gaze moving away from the respective virtual object and / or the user's hand ceasing to be ready for providing gesture input without providing a valid gesture input.

[0167] In some embodiments, each virtual object (e.g., virtual objects 7506, 7508, or 7510) corresponds to an application (e.g., each virtual object is an application icon), and actions associated with each virtual object that are available to be performed include launching the corresponding application, performing one or more actions within the application, or displaying a menu of actions to be performed with respect to or within the application. For example, if the respective virtual object corresponds to a media player application, the one or more actions may include increasing an output volume of the media (e.g., in response to a thumb swipe gesture or a pinch-and-twist gesture in a first direction), decreasing an output volume (e.g., in response to a thumb swipe gesture or a pinch-and-twist gesture in a second direction opposite the first direction), toggling (e.g., play or pause) playback of the media (e.g., in response to a thumb tap gesture), fast forwarding, rewinding, browsing media for playback (e.g., in response to multiple successive thumb swipe gestures in the same direction), or otherwise controlling media playback (e.g., menu navigation in response to a thumb flick gesture followed by a thumb swipe gesture). In some embodiments, each virtual object is a simplified user interface for controlling a physical object (e.g., an electronic device, a smart speaker, a smart lamp, etc.) beneath the corresponding virtual object, and a wrist or thumb flick gesture detected while a visual indication that the respective virtual object is interactable is displayed causes the computer system to stop displaying an enhanced user interface for controlling the physical object (e.g., indicating an on / off button, the currently playing media album, and additional playback and output adjustment controls, etc.).

[0168] In some embodiments, visual feedback indicating that the virtual object is interactable (e.g., in response to user input, including gesture input and other types of input such as audio input and touch input) includes displaying one or more user interface objects or information, prompts that were not displayed before the user's gaze was directed onto the virtual object. In one example, each virtual object is a virtual window overlaid on a physical wall represented in a three-dimensional environment, and in response to the user directing his or her gaze toward the virtual window while the user's hands are ready to provide gesture input, the computer system displays a location and / or date and time associated with the virtual scenery visible through the virtual window to indicate that the scenery may be altered (e.g., through changes in location, date and time, season, etc., following subsequent gesture input by the user). In another example, each virtual object includes a displayed still photograph (e.g., each virtual object is a picture frame), and in response to the user directing their gaze toward the displayed photograph while the user's hands are ready to provide gesture input, the computer system displays a multi-frame photograph or video clip associated with the displayed still photograph to indicate that the photograph is interactable, and optionally, to indicate that the photograph may be modified (e.g., through browsing a photo album in accordance with subsequent gesture input by the user).

[0169] 7K-7M show example views that change in response to detecting a three-dimensional environment (e.g., a virtual reality environment or a mixed reality environment) that changes in response to detecting a grip of a user's hand on a housing of a display generating component of a computer system (e.g., computer system 101 (e.g., a handheld device or HMD) of FIG. 1) while the display generating component of the computer system (e.g., display generating component 120 of FIG. 1, FIG. 3, and FIG. 4) is in a predetermined position relative to a user of the device (e.g., when the user first enters the computer-generated reality experience (e.g., when the user holds the device in front of his / her eyes or when the user wears an HMD on his / her head)). The change in view of the three-dimensional environment is not determined entirely by the computer system without user input, but rather by the user (e.g., by changing his / her grip on the device or the housing of the display generating component) to form an initial transition into the computer-generated reality experience. The input gestures described with reference to FIG. 7G are used to illustrate the processes described below, including the process in FIG. 12.

[0170] Part (A) of FIG. 7K illustrates a physical environment 7800 in which a user (e.g., user 7802) is using a computer system. The physical environment 7800 includes one or more physical surfaces (e.g., walls, floors, surfaces of physical objects, etc.) and physical objects (e.g., physical object 7504, the user's hands, body, etc.). Part (B) of FIG. 7K illustrates an example view 7820 of a three-dimensional environment (also referred to as a "first view 7820" or "first view 7820" of the three-dimensional environment) displayed by a display generating component of a computer system (e.g., device 7100 or HMD). In some embodiments, the first view 7820 is displayed when the display generating component (e.g., the display of device 7100 or HMD) is positioned in a predetermined position relative to the user 7802. For example, in FIG. 7K, the display of device 7100 is positioned in front of the user's eyes. In another example, the computer system determines that the display generating component is located at a predetermined position relative to the user in accordance with a determination that the display generating component (e.g., an HMD) is located on the user's head such that the user can view the physical environment only through the display generating component. In some embodiments, the computer system determines that the display generating component is located at a predetermined position relative to the user in accordance with a determination that the user is seated in front of a head-up display of the computer system. In some embodiments, placing the display generating component at a predetermined position relative to the user or placing the user at a predetermined position relative to the display generating component allows the user to view content (e.g., real or virtual content) through the display generating component. In some embodiments, when the display generating component and the user are in a predetermined relative position, the user's view of the physical environment may be at least partially (or completely) blocked by the display generating component.

[0171] In some embodiments, the placement of the display generating components of the computer system is determined based on an analysis of data captured by a sensor system. In some embodiments, the sensor system includes one or more sensors that are components of the computer system (e.g., an internal component enclosed within the same housing as the display generating components of the device 7100 or HMD). In some embodiments, the sensor system is an external system and is not enclosed in the same housing as the display generating components of the computer system (e.g., the sensor is an external camera that provides captured image data to the computer system for data analysis).

[0172] In some embodiments, the sensor system includes one or more imaging sensors (e.g., one or more cameras) that track the movement of a user and / or a display generating component of the computer system. In some embodiments, the one or more imaging sensors track the position and / or movement of one or more features of the user, such as the user's hand(s) and / or the user's head, to detect the placement of the display generating component relative to the user or a predetermined portion of the user (e.g., head, eyes, etc.). For example, image data is analyzed in real time to determine if the user is holding the display of device 7100 in front of the user's eyes or if the user is wearing a head mounted display on the user's head. In some embodiments, the one or more imaging sensors track the line of sight of the user's eyes to determine where the user is looking (e.g., whether the user is looking at the display). In some embodiments, the sensor system includes one or more touch-based sensors (e.g., attached to the display) that detect the grip of the user's hands against the display, such as holding device 7100 with one or both hands and / or on the edge of the device, or holding a head mounted display with both hands and placing the head mounted device on the user's head. In some embodiments, the sensor system includes one or more motion sensors (e.g., accelerometers) and / or position sensors (e.g., gyroscopes, GPS sensors, and / or proximity sensors) that detect movement and / or position information (e.g., position, height, and / or orientation) of a display of the electronic device to determine placement of the display relative to a user. For example, the movement and / or position data is analyzed to determine whether a mobile device is elevated and facing the user's eyes, or whether a head mounted display is lifted and placed on the user's head. In some embodiments, the sensor system includes one or more infrared sensors that detect the positioning of the head mounted display on the user's head. In some embodiments, the sensor system includes a combination of different types of sensors that provide data for determining placement of a display generation component relative to a user.For example, the grip of a user's hand on the housing of the display generating component, movement and / or orientation information of the display generating component, and user gaze information may be analyzed in combination to determine the placement of the display generating component relative to the user.

[0173] In some embodiments, based on an analysis of data captured by the sensor system, it is determined that the display of the electronic device is positioned at a predetermined position relative to the user. In some embodiments, the predetermined position of the display relative to the user indicates that the user is about to initiate a virtual and immersive experience using the computer system (e.g., to start a three-dimensional movie, enter a three-dimensional virtual world, etc.). For example, the sensor data indicates that the user is holding the mobile device in the user's palms while the user's gaze is directed at the display screen, or while the user holds and raises the head-mounted device on the user's head with both hands (e.g., the hand configuration shown in FIG. 7K). In some embodiments, the computer system allows the user to adjust the position of the display generation components relative to the user (e.g., move the HMD for a comfortable fit and for the display to be properly aligned with the eyes), and changes in hand grip and position during this period do not trigger any change in the first view displayed. In some embodiments, the initial hand grip monitored for changes is not a grip to hold the display generation components, but a touch of the hand or fingers on a particular portion of the display generation components (e.g., a switch or control to turn on the HMD or to start displaying virtual content). In some embodiments, the combination of placing the HMD on the user's head with a hand grip and activating a control that initiates the immersive experience is the first hand grip that is monitored for changes.

[0174] In some embodiments, as shown in portion (B) of Figure 7K, in response to detecting that the display is in a predetermined position relative to the user, a first view 7820 of the three-dimensional environment is displayed by a display generation component of the computer system. In some embodiments, the first view 7820 of the three-dimensional environment is a welcome / introduction user interface. In some embodiments, the first view 7820 includes a pass-through portion that includes a representation of at least a portion of the physical environment 7800 surrounding the user 7802.

[0175] In some embodiments, the pass-through portion is a transparent or semi-transparent (e.g., see-through) portion of the display generating component that surrounds the user's 7802 field of view and reveals at least a portion of the physical environment 7800 within the field of view. For example, the pass-through portion is a portion of a head mounted display that is semi-transparent (e.g., less than 50%, 40%, 30%, 20%, 15%, 10%, or 5% opacity) or transparent through which the user can view the real world surrounding the user without removing the display generating component. In some embodiments, the pass-through portion gradually transitions from semi-transparent or transparent to fully opaque as the welding / introduction user interface changes to an immersive virtual or mixed reality environment, for example, in response to a subsequent change in the user's hand grip indicating that the user is ready to proceed into the fully immersive environment.

[0176] In some embodiments, the pass-through portion of the first view 7820 displays a live feed of images or video of at least a portion of the physical environment 7800 captured by one or more cameras (e.g., a rear-facing camera(s) associated with a mobile device or a head-mounted display, or other camera that provides image data to an electronic device). For example, the pass-through portion includes all or a portion of a display screen that displays a live image or video of the physical environment 7800. In some embodiments, the one or more cameras are pointed at a portion of the physical environment that is directly in front of the user (e.g., behind the display generation component). In some embodiments, the one or more cameras are pointed at a portion of the physical environment that is not directly in front of the user (e.g., in a different physical environment, or to the side or behind the user).

[0177] In some embodiments, the first view of the three-dimensional environment 7820 includes three-dimensional virtual reality (VR) content. In some embodiments, the VR content includes one or more virtual objects that correspond to one or more physical objects (e.g., shelves and / or walls) in the physical environment 7800. For example, at least some of the virtual objects are displayed at positions in the virtual reality environment that correspond to the positions of the physical objects in the corresponding physical environment 7800 (e.g., the positions of the physical objects in the physical environment are determined using one or more cameras). In some embodiments, the VR content does not correspond to the physical environment 7800 seen through the pass-through portion and / or is displayed independently of the physical objects in the pass-through portion. For example, the VR content includes virtual user interface elements (e.g., a virtual dock or virtual menu that includes user interface objects) or other virtual objects that are unrelated to the physical environment 7800.

[0178] In some embodiments, the first view of the three-dimensional environment 7820 includes three-dimensional augmented reality (AR) content. In some embodiments, one or more cameras (e.g., rear-facing camera(s) of a mobile device or associated with a head-mounted display, or other camera that provides image data to a computer system) continuously provide a live view of at least a portion of the surrounding physical environment 7800 that is within the field of view of the one or more cameras, and the AR content corresponds to that portion of the surrounding physical environment 7800 that is within the field of view of the one or more cameras. In some embodiments, the AR content includes at least a portion of the live view of the one or more cameras. In some embodiments, the AR content includes one or more virtual objects that are positioned on (e.g., overlaid on or appear to occlude) the portion of the live view. In some embodiments, the virtual objects are displayed at positions in the virtual environment 7820 that correspond to the positions of the corresponding objects in the physical environment 7800. For example, each virtual object is displayed in place of (e.g., overlaid on, obscures, and / or replaces) the view of the corresponding physical object in the physical environment 7800.

[0179] In some embodiments, in the first view of the three-dimensional environment 7820, the pass-through portion (e.g., representing at least a portion of the physical environment 7800) is surrounded by virtual content (e.g., VR and / or AR content). For example, the pass-through portion does not overlap with the virtual content on the display. In some embodiments, in the first view of the three-dimensional virtual environment 7820, the VR and / or AR virtual content is displayed in place of the pass-through portion (e.g., overlaid on top of or replacing the displayed content). For example, the virtual content (e.g., a virtual dock or a virtual starting menu listing multiple virtual user interface elements) is overlaid or occluded by a portion of the physical environment 7800 displayed through a semi-transparent or transparent pass-through portion. In some embodiments, the first view of the three-dimensional environment 7820 initially includes only the pass-through portion without any virtual content. For example, when a user first holds the device in the user's palm (e.g., as shown in FIG. 7L) or when the user first places the head mounted display on the user's head, the user views a portion of the physical environment within the field of view of the user's eyes or within the field of view of the live feed camera through the see-through portion. The virtual content (e.g., a welcome / introduction user interface with virtual menus / icons) then gradually fades in to overlay or block the pass-through portion over a period of time while the user's hand grip remains unchanged. In some embodiments, the welcome / introduction user interface remains displayed (e.g., in a steady state having both the virtual content and the pass-through portion showing the physical world) as long as the user's hand grip does not change.

[0180] In some embodiments, activating a user's virtual immersive experience temporarily blocks the user's current view of the surrounding real world by the display generation component (e.g., by the presence of a display in front of the user and the mute function of the head-mounted display). This occurs at a point before the start of the user's virtual immersive experience. By having a pass-through portion in the welcome / introduction user interface, the transition from viewing the physical environment surrounding the user to the user's virtual immersive experience benefits from a better controlled and smoother transition (e.g., a cognitively gentle transition). This allows the user themselves more control over how long it takes to be ready for the fully immersive experience after seeing the welcome / introduction user interface, as opposed to having a computer system or content provider dictate when to transition to the fully immersive experience for all users.

[0181] 7M illustrates another exemplary view 7920 of the three-dimensional environment (also referred to as "second view 7920 of the three-dimensional environment" or "second view 7920") displayed by a display generating component of the computer system (e.g., on a display of the device 7100). In some embodiments, the second view 7920 of the three-dimensional environment replaces the first view 7820 of the three-dimensional environment in response to detecting a change in the grip of the user's hand on the housing of the display generating component of the computer system (e.g., a change from the hand configuration of part (B) of FIG. 7K (e.g., a two-handed grip) to the hand configuration of part (B) of FIG. 7L (e.g., a one-handed grip)) that meets a first predetermined criterion (e.g., a criterion corresponding to detection of a sufficient decrease in the user's control or protection).

[0182] In some embodiments, the change in grip of the user's hand(s) is detected by a sensor system as described above with reference to FIG. 7K. For example, one or more imaging sensors track the movement and / or position of the user's hand to detect the change in grip of the user's hand. In another example, one or more touch-based sensors on the display detect the change in grip of the user's hand.

[0183] In some embodiments, the first predefined criteria for a change in the user's hand grip requires a change in the total number of hands detected on the display (e.g., from two to one hand, one to no hand, or two to no hand), a change in the total number of fingers in contact with the display generating component (e.g., from eight fingers to six fingers, four fingers to two fingers, two fingers to no fingers), a change from hand contact to no hand contact on the display generating component, a change in contact location (e.g., from palm to fingers), and / or a change in contact intensity on the display (e.g., resulting from a change in hand posture, orientation, relative grip of different fingers on the display generating component, etc.). In some embodiments, the change in hand grip on the display does not cause a change in the predefined position of the display relative to the user (e.g., the head mounted display is still on the user's head and covering the user's eyes). In some embodiments, the change in hand grip represents that the user is (e.g., gradually or purposefully) releasing their hands from the display and is ready to be immersed in the virtual immersive experience.

[0184] In some embodiments, the initial hand grip monitored for a change is not a grip to hold the display generation component, but a hand or finger touching a particular portion of the display generation component (e.g., a switch or control to turn on the HMD or start displaying virtual content), and the first predetermined criterion for a change in the user's hand grip requires that a finger touching the particular portion of the display generation component (e.g., a finger that activates a switch or control to turn on the HMD or start displaying virtual content) stops touching the particular portion of the display generation component.

[0185] In some embodiments, the second view 7920 of the three-dimensional environment replaces at least a portion of the pass-through portion in the first view 7820 with virtual content. In some embodiments, the virtual content of the second view 7920 of the three-dimensional environment includes VR content (e.g., virtual objects 7510 (e.g., virtual user interface elements or system affordances)), AR content (e.g., virtual objects 7506 (e.g., a virtual window overlaid on a live view of a wall captured by one or more cameras)), and / or virtual objects 7508 (e.g., a picture or virtual controls displayed in place of or overlaid on a portion or all of a representation 7504′ of a physical object 7504 in the physical environment).

[0186] In some embodiments, replacing the first view 7820 with the second view 7920 includes increasing the opacity of the pass-through portion (e.g., when the pass-through portion is implemented in a semi-transparent or transparent state of the display) so that virtual content superimposed on the semi-transparent or transparent portion of the display is more visible and more color-saturated. In some embodiments, the virtual content of the second view 7920 provides a more immersive experience for the user than the virtual content in the first view 7820. For example, the virtual content of the first view 7820 is displayed in front of the user, while the virtual content of the second view 7920 includes a three-dimensional world represented in a panoramic or 360-degree view that is visible to the user as the user turns his head and / or walks around. In some embodiments, the second view 7920 includes a smaller pass-through portion that displays a smaller or lesser portion of the physical environment 7800 surrounding the user than the first view 7820. For example, the pass-through portion of the first view 7820 may show an actual window in one of the walls of a room in which the user is located, and the pass-through portion of the second view 7920 may show the window in one of the walls replaced with a virtual window, resulting in a reduced area of ​​the pass-through portion in the second view 7920.

[0187] FIG. 7M illustrates yet another exemplary third view 7821 (e.g., the first view 7820 or a modified version thereof, or a different view, of the three-dimensional environment) displayed by the display generating component of the computer system in response to again detecting the initial hand grip configuration on the housing of the display generating component (e.g., after the second view 7920 is displayed in response to detecting a required change in hand grip, as shown in FIG. 7L). In some embodiments, the third view 7821 re-establishes the pass-through portion in response to detecting another change in the user's hand grip on the display generating component of the computer system (e.g., a change from the hand configuration of FIG. 7L, or a change from no hand grip to the hand configuration of FIG. 7M). In some embodiments, the change in the user's hand grip represents a re-establishment of the user's hand grip on the housing of the display generating component, indicating that the user wishes to leave the virtual immersive experience (e.g., partially or completely, gradually, or immediately).

[0188] In some embodiments, the sensor system detects a change in the total number of hands detected on the housing of the display generating components (e.g., from one hand to two hands, or from no hands to two hands), an increase in the total number of fingers touching the housing of the display generating components, a change from no hand contact to hand contact on the housing of the display generating components, a change in contact location (e.g., from finger(s) to palm), and / or a change in contact intensity on the housing of the display generating components. In some embodiments, the re-establishment of the user's hand grip changes the position and / or orientation of the display generating components (e.g., changes the position and angle of the device 7100 with respect to the environment in part (A) of FIG. 7M compared to the angle in part (A) of FIG. 7L). In some embodiments, the change in the user's hand grip causes a change in the user's perspective with respect to the physical environment 7800 (e.g., the viewing angle of the device 7100 or one or more cameras of the device 7100 with respect to the physical environment 7800 changes). As a result, the perspective of the displayed third view 7821 changes accordingly (e.g., including changing the perspective of the pass-through portion and / or virtual objects on the display).

[0189] In some embodiments, the pass-through portion of the third view 7821 is the same as the pass-through portion of the first view 7820, or at least is increased relative to the pass-through portion (if any) in the second view 7920. In some embodiments, the pass-through portion of the third view 7821 shows a different perspective of the physical objects 7504 in the physical environment 7800 as compared to the pass-through portion of the first view 7820. In some embodiments, the pass-through portion of the third view 7821 is a see-through portion of a display generating component that is transparent or semi-transparent. In some embodiments, the pass-through portion of the third view 7821 displays a live feed from one or more cameras configured to capture image data of at least a portion of the physical environment 7800. In some embodiments, there is no virtual content displayed with the pass-through portion in the third view 7821. In some embodiments, the virtual content is paused or made semi-transparent or reduced color saturation in the third view 7821 and displayed simultaneously with the pass-through portion in the third view 7821. Once the third view is displayed, the user can resume the fully immersive experience by again changing their hand grip, as described with respect to Figures 7K-7L.

[0190] 7N-7P show example views of a three-dimensional virtual environment that change in response to detecting a change in a user's position relative to an object (e.g., an obstacle or target) in the physical environment surrounding the user, according to some embodiments. The input gestures described with reference to Figures 7N-7P are used to illustrate the processes described below, including the process in Figure 13.

[0191] In part (A) of FIG. 7N, a user 7802 holds a device 7100 within a physical environment 7800. The physical environment includes one or more physical surfaces and physical objects (e.g., walls, floors, physical object 7602). The device 7100 displays a virtual three-dimensional environment 7610 without displaying a pass-through portion that shows the physical environment surrounding the user. In some embodiments, the device 7100 can be replaced by an HMD or other computer system that includes a display generating component that blocks the user's view of the physical environment when displaying the virtual environment 7610. In some embodiments, the HMD or display generating component of the computer system surrounds at least the user's eyes, and the user's view of the physical environment is partially or completely blocked by the virtual content displayed by the display generating component and other physical barriers formed by the display generating component or its housing.

[0192] Part (B) of Figure 7N shows a first view 7610 of the three-dimensional environment displayed by a display generation component (also referred to as a "display") of a computer system (e.g., device 7100 or HMD).

[0193] In some embodiments, the first view 7610 is a three-dimensional virtual environment that provides an immersive virtual experience (e.g., a three-dimensional movie or game). In some embodiments, the first view 7610 includes three-dimensional virtual reality (VR) content. In some embodiments, the VR content includes one or more virtual objects that correspond to one or more physical objects in a physical environment that does not correspond to the physical environment 7800 surrounding the user. For example, at least some of the virtual objects are displayed at positions in the virtual reality environment that correspond to positions of the physical objects in the physical environment that are separate from the physical environment 7800. In some embodiments, the first view includes virtual user interface elements (e.g., a virtual dock or virtual menu that includes user interface objects) or other virtual objects that are unrelated to the physical environment 7800.

[0194] In some embodiments, the first view 7610 includes 100% virtual content (e.g., virtual objects 7612 and virtual surfaces 7614 (e.g., virtual walls and floors)) that does not include and is distinct from any representation of the physical environment 7800 surrounding the user 7802. In some embodiments, the virtual content in the first view 7610 (e.g., virtual objects 7612 and virtual surfaces 7614) does not correspond to or visually communicate the presence, location, and / or physical structure of any physical objects in the physical environment 7800. In some embodiments, the first view 7610 optionally includes a virtual representation that indicates the presence and location of a first physical object in the physical environment 7800 but does not visually communicate the presence, location, and / or physical structure of a second physical object in the physical environment 7800, both of which are within the user's field of view if the user's view is not blocked by the display generation component. In other words, the first view 7610 includes virtual content that replaces the display of at least some physical objects or portions thereof that would be present in the user's normal field of view (e.g., the user's field of view without a display generating component positioned in front of the user).

[0195] 7O illustrates another exemplary view 7620 of the three-dimensional virtual environment (also referred to as "second view of the three-dimensional environment 7620," "second view of the virtual environment 7620," or "second view 7620") displayed by a display generation component of a computer system. In some embodiments, a sensor system detects that a user 7802 moves toward a physical object 7602 in the physical environment 7800, and sensor data acquired by the sensor system is analyzed to determine whether the distance between the user 7802 and the physical object 7602 is within a predetermined threshold distance (e.g., an arm's length, or the user's normal walking length). In some embodiments, when a portion of physical object 7602 is determined to be within a threshold distance to user 7602, the appearance of the view of the virtual environment is changed to indicate a physical characteristic of the portion of physical object 7602 (e.g., showing portion 7604 of physical object 7602 in second view 7620 of part (B) of FIG. 7O that is within a threshold distance from user 7802, and not showing other portions of physical object 7602 that are in the same field of view of the user as portion 7604 but that are not within the user's threshold distance). In some embodiments, instead of replacing the portion of the virtual content with a direct or camera view of portion 7604 of the physical object, visual characteristics (e.g., opacity, color, texture, virtual material, etc.) of the portion of the virtual content at a location corresponding to portion 7604 of the physical object are altered to indicate a physical characteristic (e.g., size, color, pattern, structure, contour, shape, surface, etc.) of portion 7604 of the physical object. The change in the virtual content at the location corresponding to portion 7604 of the physical object 7602 is not applied to other portions of the virtual content, including portions of the virtual content at locations corresponding to portions of the physical object 7602 that are outside of portion 7604. In some embodiments, the computer system provides a blend (e.g., a smooth visual transition) between the portion of the virtual content at the location corresponding to portion 7604 of the physical object and the portion of the virtual content just outside of the location corresponding to portion 7604 of the physical object.

[0196] In some embodiments, physical object 7602 is a static object in physical environment 7800, such as a wall, a chair, or a table. In some embodiments, physical object 7602 is an object that moves within physical environment 7800, such as another person or a dog in physical environment 7800 that moves relative to user 7802 while user 7802 is stationary relative to physical environment 7800 (e.g., a user's pet moves around while the user sits on the couch watching a movie).

[0197] In some embodiments, while a user 7802 is enjoying a three-dimensional immersive virtual experience (e.g., including a panoramic three-dimensional display with surround sound effects or other virtual perceptions, etc.), if real-time analysis of sensor data from a sensor system coupled to a computer system indicates that the user 7802 is close enough to a physical object 7602 (e.g., by either the user's movement towards the physical object or the physical object's movement towards the user), the user 7802 may benefit from receiving an alert that blends in with the virtual environment in a smooth and less intrusive manner, allowing the user to make a more informed decision about whether to alter their movement and / or discontinue / continue the immersive experience without losing the immersive quality of the experience.

[0198] In some embodiments, the second view 7620 is displayed when analysis of the sensor data indicates that the user 7802 is within a threshold distance of at least a portion of a physical object 7602 in the physical environment 7800 (e.g., the physical object 7602 has a range that may be potentially visible to the user based on the user's field of view relative to the virtual environment). In some embodiments, given a location of a portion of a physical object relative to the user in the physical environment 7800, that portion of the physical object would be visible within the user's field of view if the display had a pass-through portion or if the display generation component was not present in front of the user.

[0199] In some embodiments, the portion 7604 in the second view 7620 of the virtual environment includes a semi-transparent visual representation of the corresponding portion of the physical object 7602. For example, the semi-transparent representation overlays the virtual content. In some embodiments, the portion 7604 in the second view 7620 of the virtual environment includes a glassy appearance of the corresponding portion of the physical object 7602. For example, when the user 7802 approaches a table located in a room while enjoying the immersive virtual experience, the portion of the table closest to the user is shown with a glassy semi-transparent see-through appearance that overlays the virtual content (e.g., a virtual ball or a virtual grass field in the virtual view), and the virtual content behind that portion of the table is visible through that portion of the glassy table. In some embodiments, the second view 7620 of the virtual environment shows a predetermined distortion or other visual effect (e.g., visual effects such as shimmer, wave, sparkle, darken, blur, swirl, or different text effects) applied to the portion 7604 corresponding to the portion of the physical object 7602 closest to the user 7802.

[0200] In some embodiments, when the user moves toward and within a threshold distance of the corresponding portion of the physical object 7602, the second view 7620 of the virtual environment momentarily replaces the first view 7610 to provide a timely warning to the user. In some embodiments, the second view 7620 of the virtual environment is gradually displayed, for example, using a fade-in / fade-out effect, to provide a smoother transition and less confusing / disturbing user experience. In some embodiments, the computer system allows the user to navigate in the three-dimensional environment by moving in the physical environment, and changes the view of the three-dimensional environment presented to the user to reflect the computer-generated movement in the three-dimensional environment. For example, as shown in FIG. 7N and FIG. 7O, when the user is walking toward the physical object 7602, the user perceives his / her movement as moving in the same direction toward the virtual object 7612 in the three-dimensional virtual environment (e.g., the virtual object 7612 becomes closer and larger). In some embodiments, the virtual content presented to the user is independent of the user's movement within the physical environment and does not change according to the user's movement within the physical environment except when the user comes within a threshold distance of a physical object in the physical environment.

[0201] FIG. 7P illustrates yet another exemplary view 7630 of a three-dimensional environment (also referred to as a "third view 7630 of the three-dimensional environment," "third view 7630 of the virtual environment," or "third view 7630") displayed by a display generating component of a computer system (e.g., device 7100 or HMD). In some embodiments, after displaying the second view 7620 of the virtual three-dimensional environment as described with reference to FIG. 7O, as the user 7802 continues to move toward a physical object 7602 in the physical environment 7800, analysis of the sensor data indicates that the distance between the user 7802 and the portion 7606 of the physical object 7602 is less than a predetermined threshold distance. In response, the display transitions from the second view 7620 to the third view 7630. In some embodiments, depending on the structure (e.g., size, shape, length, width, etc.) and relative positions of the user and physical object 7602, portion 7606 and portion 7604 of physical object 7602 that were within a predetermined threshold distance of the user when the user was at various locations in physical environment 7800 are optionally completely separate non-overlapping portions of the physical object, portion 7606 optionally completely surrounds portion 7604, portion 7606 and portion 7604 optionally partially overlap, or portion 7604 optionally completely surrounds portion 7606. In some embodiments, portions or the whole of one or more other physical objects may be visually represented, or cease to be represented, in the currently displayed view of the virtual three-dimensional environment as the user moves around the room relative to those physical objects, depending on whether those physical objects are inside or outside the user's predetermined threshold distance.

[0202] In some embodiments, the computer system optionally allows the user to pre-select a subset of physical objects in the physical environment 7800 for which the distance between the user and a given physical object is monitored and visual changes are applied to the virtual environment. For example, the user may wish to pre-select furniture and pets as a subset of physical objects and not select clothes, curtains, etc. as a subset of physical objects and not apply visual changes to the virtual environment to alert the user to the presence of the clothes or curtains even when the user is walking. In some embodiments, the computer system allows the user to pre-designate one or more physical objects that will always be visually represented in the virtual environment, regardless of whether the user is within a threshold distance of the physical object, by applying visual effects (e.g., transparency, opacity, shine, refractive index, etc.) to portions of the virtual environment that correspond to the respective positions of the physical objects. These visual indications allow the user to orient themselves relative to the real world even when the user is immersed in the virtual world and feel more secure and stable when exploring the virtual world.

[0203] In some embodiments, as shown in FIG. 7P , the third view 7630 includes a rendering of a portion 7606 of the physical object 7602 that is within a threshold distance from the user as the user approaches the physical object. In some embodiments, the computer system optionally further increases values ​​of display properties of visual effects applied to portions of the virtual environment that depict physical properties of the corresponding portions of the physical object 7602 in accordance with the reduced distance between the user and the portion of the physical object. For example, the computer system optionally increases the refractive index, color saturation, visual effects, opacity, and / or transparency of the portions of the virtual environment that correspond to the portions of the physical object in the third view 7630 as the user gradually approaches that portion of the physical object. In some embodiments, as the user 7802 approaches the physical object 7602, the spatial extent of the visual effects increases, causing the corresponding portions of the physical object 7602 to appear larger in the user's field of view of the virtual environment. For example, as the user 7802 approaches a physical object 7602 in the physical environment 7800, portion 7606 of the third view 7630 appears to gradually increase in size from the virtual object 7612 and extend toward the user, compared to portion 7604 of the second view 7620, for at least two reasons: (1) more of the physical object 7602 comes within a predetermined distance of the user; and (2) the same portion of the physical object 7602 (e.g., portion 7604) occupies a larger portion of the user's field of view of the virtual environment because it is closer to the user's eyes.

[0204] In some embodiments, the computer system defines a gestural input that causes portions of physical objects that are partially located within a threshold distance of the user (e.g., all portions potentially visible within the user's field of view of the virtual environment), or all physical objects potentially visible within the user's field of view of the virtual environment, to be visually displayed in the virtual environment by modifying the display characteristics of the virtual environment at locations corresponding to those portions or all physical objects (e.g., a user raises one or both arms to a predetermined height relative to the user's body within a threshold time (e.g., a sharp, sudden movement that is a muscle reflex to prevent falling or hitting something)). This feature is useful for allowing the user to quickly reorient themselves when they are certain of their body's position in the physical environment, without leaving the immersive experience entirely.

[0205] Additional explanation regarding Figures 7A-7P is provided below with reference to methods 8000, 9000, 10000, 11000, 12000, and 13000 described with respect to Figures 8-13 below.

[0206] FIG. 8 is a flowchart of an exemplary method 8000 for interacting with a three-dimensional environment using predefined input gestures, according to some embodiments. In some embodiments, the method 8000 is performed on a computer system (e.g., computer system 101 of FIG. 1) including a display generating component (e.g., display generating component 120 of FIG. 1, FIG. 3, and FIG. 4) (e.g., a head-up display, a display, a touch screen, a projector, etc.) and one or more cameras (e.g., a camera facing downward at the user's hand (e.g., color sensors, infrared sensors, and other depth-sensing cameras)) or a camera facing forward from the user's head). In some embodiments, the method 8000 is performed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of the computer system 101 (e.g., control unit 110 of FIG. 1A). Some operations of the method 8000 are optionally combined and / or the order of some operations is optionally changed.

[0207] In method 8000, a computer system displays (8002) a view of a three-dimensional environment (e.g., a virtual or mixed reality environment). While displaying the view of the three-dimensional environment, the computer system detects (8004) the movement of a user's thumb over the user's index finger of a user's first hand (e.g., a left or right hand that is not gloved or covered or resting on an input device / surface) using one or more cameras (e.g., using one or more cameras positioned at the bottom edge of the HMD as opposed to using a touch-sensitive glove, or a touch-sensitive surface on a hand-controlled input device, or other non-image-based means (e.g., sound waves, etc.). This is shown, for example, in FIG. 7A and the accompanying description (e.g., thumb tap, thumb swipe, and thumb flick gestures). In some embodiments, the user's hand or a graphical representation thereof is displayed in the view of the three-dimensional environment (e.g., in a pass-through portion of the display generation component, or as part of an augmented reality view of the physical environment surrounding the user). In some embodiments, the user's hand or a graphical representation thereof is not shown in the view of the three-dimensional environment or is displayed in a portion of the display that is outside the view of the three-dimensional environment (e.g., a separate or floating window). Benefits of using one or more cameras, particularly cameras that are part of the HMD, include the spatial position and size of the user's hand being viewed by the user as if it were naturally present in the physical or virtual environment with which the user is interacting, giving the user an intuitive sense of scale, orientation, and fixed position for perceiving the three-dimensional environment on the display without requiring additional calculations to match the space of the input device to the three-dimensional environment and / or to scale, rotate, and translate the representation of the user's hand before placing it in the displayed three-dimensional environment.Referring back to FIG. 8 , in response to detecting (8006) using the one or more cameras to move a user's thumb over the user's index finger (as opposed to a more exaggerated gesture such as waving a finger or hand in the air or sliding across the touch-sensitive surface), and pursuant to a determination that the movement is a swipe of the thumb over the index finger of a first hand in a first direction (e.g., movement along a first axis (e.g., the x-axis) of the x- and y-axes, where the movement along the x-axis is movement along the length of the index finger and the movement along the y-axis is movement across the index finger (substantially perpendicular to the movement along the length of the index finger), the computer system performs a first action (e.g., change a selected user interface object within the displayed user interface (e.g., cycle through the selection of an item in the first direction through a list of items corresponding to the first direction (in a left-right column of items)), adjust the position of the user interface object within the displayed user interface), and 7B, 7C, and 7F, and the accompanying description.8, in response to detecting (8006) a movement of the user's thumb on the user's index finger using the one or more cameras (e.g., as opposed to a more exaggerated gesture of waving a finger or hand in the air or sliding across the touch-sensitive surface), and in accordance with a determination that the movement is a tap of the thumb on the index finger at a first location (e.g., at a first portion of the index finger, such as the distal phalanges, middle phalanges, and / or proximal phalanges) of the index finger of a first hand (including a touchdown and liftoff of the thumb on the index finger within a threshold time), the computer system performs a second action that is different from the first action (e.g., performs an action corresponding to a currently selected user interface object and / or changes the selected user interface object in the displayed user interface). In some embodiments, performing the first / second action includes changing a view of the three-dimensional user interface, the change being dependent on the current operational context. In other words, each gesture triggers a different action and changes the view of the three-dimensional environment in a different way depending on the current action context (e.g., the object the user is looking at, the direction the user is facing, the last function performed immediately before the current gesture, and / or the currently selected object), as shown, for example, in Figures 7B, 7C, and 7F and the accompanying description.

[0208] In some embodiments, in response to detecting a movement of the user's thumb on the user's index finger using the one or more cameras, pursuant to a determination that the movement is a swipe of the thumb on the index finger of the first hand in a second direction (e.g., the y-axis) substantially perpendicular to the first direction (e.g., movement along a second axis (e.g., the y-axis) of the x- and y-axes, where movement along the x-axis is movement along the length of the index finger and movement along the y-axis is movement in a direction across the index finger (substantially perpendicular to movement along the length of the index finger)), the computer system performs a third action different from the first and second actions (e.g., altering a selected user interface object within a displayed user interface (e.g., in the second direction within a list of items corresponding to the second direction (e.g., altering a multi-line item in a 2D menu)). the third action may be to repeat a selection (up or down an item, or up or down a vertically arranged list), adjust a position of a user interface object within a displayed user interface (move an object (e.g., up or down) within the user interface in a direction corresponding to the second direction, and / or adjust a system setting (e.g., volume) of the device). In some embodiments, the third action is distinct from the first action and / or the second action. In some embodiments, a swipe in a first sub-direction (e.g., along the circumference of the index finger) in a second direction (e.g., away from the palm along the circumference of the index finger) corresponds to performing the third action in one manner, and a swipe in a second sub-direction (e.g., towards the palm along the circumference of the index finger) corresponds to performing the third action in another manner.

[0209] In some embodiments, in response to detecting a movement of the user's thumb on the user's index finger using one or more cameras, in accordance with a determination that the movement is a movement of the thumb on the index finger in a third direction (not a tap of the thumb on the index finger) different from the first direction (and the second direction), the computer system performs a fourth action different from the first action, different from the second action (and different from the third action). In some embodiments, the third direction is an upward direction from the index finger away from the index finger (e.g., as opposed to tapping on the side of the index finger), and the gesture is a flick of the thumb from the side of the index finger away from the index finger and the palm. In some embodiments, this upward flick gesture using the thumb across the middle of the index finger pushes the currently selected user interface object into a three-dimensional environment and initiates an immersive experience (e.g., a 3D video or 3D virtual experience, a panoramic viewing mode, etc.) corresponding to the currently selected user interface object (e.g., an animated icon, an app icon, an image, etc.). In some embodiments, while the immersive experience is ongoing, swiping downward across the middle of the index finger toward the palm (e.g., movement in one of the sub-directions of the second direction, as opposed to tapping with the middle of the index finger) pauses, stops, and / or changes the immersive experience to a reduced immersive state (e.g., non-full screen, 2D mode, etc.).

[0210] In some embodiments, performing the first action includes increasing a value corresponding to the first action (e.g., a value of a system setting, a value indicating a position of and / or selection of at least a portion of a user interface (e.g., a user interface object), and / or a value corresponding to the selected content or portion of content). For example, increasing the value includes increasing a volume, moving an object in an incremental direction (e.g., upward and / or to the right), and / or adjusting a position (e.g., within a list and / or content item) to a next or another advanced position pursuant to a determination that a swipe of the thumb on an index finger in a first direction (e.g., along the length of the index finger or along the circumference of the index finger) moves toward a first predetermined portion of the index finger (e.g., toward the tip of the index finger or toward the back of the index finger). In some embodiments, performing the first motion further includes decreasing a value corresponding to the first motion in accordance with a determination that a swipe of the thumb on the index finger in a first direction (e.g., along the length of the index finger or along the circumference of the index finger) moves away from a first predetermined portion of the index finger (e.g., away from the tip of the index finger or away from the rear side (backward) of the index finger) toward a second predetermined portion of the index finger (e.g., toward the base of the index finger or toward the front side (backward) of the index finger) (e.g., decreasing the value includes decreasing a volume, moving the object in a decreasing direction (e.g., downward and / or leftward), and / or adjusting a position (e.g., within a list and / or content item) to a previous or another preceding position). In some embodiments, the direction of a swipe of the thumb on the index finger in the second direction also determines the direction of the third motion, similar to how a swipe direction in the first direction determines the direction of the first motion.

[0211] In some embodiments, performing the first action includes adjusting a value corresponding to the first action (e.g., a value of a system setting, a value indicative of a position and / or selection of at least a portion of a user interface (e.g., a user interface object), and / or a value corresponding to the selected content or portion of content) by an amount corresponding to an amount of thumb movement on the index finger. In some embodiments, the thumb movement is measured relative to a threshold position on the index finger, and the value corresponding to the first action is adjusted between a plurality of discrete altitudes according to which threshold position is reached. In some embodiments, the thumb movement is measured continuously, and the value corresponding to the first action is continuously and dynamically adjusted based on a current position of the thumb on (e.g., along or around) the index finger. In some embodiments, the speed of thumb movement is used to determine a threshold used to determine a magnitude of the action and / or when different discrete values ​​of the action are triggered.

[0212] In some embodiments, in response to detecting a movement of the user's thumb on the user's index finger using one or more cameras, in accordance with a determination that the movement is a tap of the thumb on the index finger at a second location (e.g., a portion of the index finger and / or a finger double) different from the first location on the index finger, the computer system performs a fifth action different from the second action (e.g., performs an action corresponding to a currently selected user interface object and / or changes a selected user interface object within a displayed user interface). In some embodiments, the fifth action is different from the first action, the third action, and / or the fourth action. In some embodiments, tapping the middle part of the index finger activates a currently selected object, and tapping the tip of the index finger minimizes / stops / closes a currently active application or experience. In some embodiments, detecting a tap of the thumb on the index finger does not need to detect a lift-off of the thumb from the index finger, and movement of the thumb or the entire hand while the thumb remains on the index finger can be treated as a movement combined with a thumb tap-hold input, for example, to drag an object.

[0213] In some embodiments, the computer system detects a swipe of the user's thumb over the user's middle finger using the one or more cameras (e.g., while detecting the user's index finger extended away from the middle finger). In response to detecting a swipe of the user's thumb over the user's middle finger, the computer system performs a sixth action. In some embodiments, the sixth action is different from the first action, the second action, the third action, the fourth action, and / or the fifth action. In some embodiments, one or more different actions are performed in accordance with a determination that a swipe of the user's thumb over the middle finger includes movement of the thumb along the length of the middle finger (e.g., from the base toward the tip of the middle finger, or vice versa), and that a swipe of the user's thumb over the middle finger includes movement of the thumb along the length of the middle finger from the tip of the middle finger toward the base of the middle finger, and / or movement of the thumb from the palm side of the middle finger across the middle finger to the top of the middle finger.

[0214] In some embodiments, the computer system detects a tap of the user's thumb on the user's middle finger using the one or more cameras (e.g., while detecting the user's index finger extended away from the middle finger). In response to detecting a tap of the user's thumb on the user's middle finger, the computer system performs a seventh action. In some embodiments, the seventh action is different from the first action, the second action, the third action, the fourth action, the fifth action, and / or the sixth action. In some embodiments, the different action is performed pursuant to a determination that the tap of the user's thumb on the middle finger is at a first location on the middle finger and the tap of the user's thumb on the middle finger is at a second location different from the first location on the middle finger. In some embodiments, an upward flick from the first and / or second location on the middle finger causes the device to perform other actions different from the first, second, ..., and / or seventh action.

[0215] In some embodiments, the computer system displays a visual indication (e.g., a menu of selectable options, a dial for adjusting a value, an avatar of a digital assistant, displaying a selection indicator of a currently selected object, highlighting an interaction object, etc.) of the motion context of a thumb gesture (e.g., a swipe / tap / flick of the thumb over the other fingers of the hand) within the three-dimensional environment (e.g., when the device detects, using one or more cameras, that the user's hand is in or has entered a pre-defined ready state (e.g., thumb resting on the side of the index finger or hovering above the side of the index finger and / or flicking the wrist with the back of the thumb pointing up / on the side of the index finger) and / or when the thumb side of the hand is pointing upward towards the camera, and displays multiple user interface objects within the three-dimensional environment, which user interface objects are responsive to the swipe and tap gestures of the thumb over the other fingers of the hand). Performing the first action (or the second, third, etc. action) includes displaying a visual change in the three-dimensional environment corresponding to the performance of the first action (or the second, third, etc. action) (e.g., displaying the visual change includes activating each user interface object of the plurality of user interface objects and causing an action associated with the each user interface object).

[0216] In some embodiments, while displaying a visual indication of the thumb gesture's operational context (e.g., while displaying a plurality of user interface objects in a three-dimensional environment in response to detecting a user's hand in a predefined ready state), the computer system detects, using one or more cameras, a movement of the user's first hand (e.g., a movement of the entire hand in the physical environment relative to the camera as opposed to a relative internal movement of the fingers) (e.g., detects the movement of the hand while the hand remains in the ready state) (e.g., detects the movement and / or rotation of the hand / wrist in the three-dimensional environment). In response to detecting the movement of the first hand, the computer system changes a displayed position of the visual indication of the thumb gesture's operational context (e.g., a plurality of user interface objects) in the three-dimensional environment according to the detected change in the position of the hand (e.g., maintains the display of a plurality of user interface objects within a predefined distance of the hand) (e.g., a menu of objects is attached to the tip of the thumb during the movement of the hand). In some embodiments, the visual indication is a system affordance (e.g., an indicator for an application launching user interface or a dock). In some embodiments, the visual indication is a dock including a plurality of application launching icons. In some embodiments, the dock changes as the hand configuration (e.g., thumb position, index / middle finger position) changes. In some embodiments, the visual indication disappears when the hand moves out of the microgesture orientation (e.g., thumb up below the shoulder). In some embodiments, the visual indication reappears when the hand moves into the microgesture orientation. In some embodiments, the visual indication appears in response to a gesture (e.g., swiping thumb over index finger while the user is looking at the hand). In some embodiments, the visual indication resets (e.g., disappears) after a hand inactivity time threshold (e.g., 8 seconds). Further details are described, for example, with respect to FIGS. 7D-7F and 9 and the accompanying description.

[0217] In some embodiments, the computer system uses one or more cameras to detect the movement of a user's thumb over the user's index finger of the user's second hand (e.g., different from the first hand) either (e.g., in a two-handed gesture scenario) (e.g., while detecting the movement of the user's thumb over the user's index finger of the first hand) or (e.g., in a one-handed gesture scenario) without detecting the movement of the user's thumb over the user's index finger of the first hand. In response to detecting, using the one or more cameras, a movement of the user's thumb on the user's index finger of the second hand, pursuant to a determination that the movement is a swipe of the thumb on the index finger of the second hand in a first direction (e.g., along the length of the index finger, around the index finger, or up and away from the side of the index finger), the computer system performs an eighth action that is different from the first action, and pursuant to a determination that the movement is a tap of the thumb on the index finger of the second hand at a first location (at a first portion of the index finger, such as the distal phalange, middle phalange, and / or proximal phalange), on the index finger of the second hand, the computer system performs a ninth action that is different from the second action (and the eighth action). In some embodiments, the eighth and / or ninth action are different from the first action, the second action, the third action, the fourth action, the fifth action, the sixth action, and / or the seventh action. In some embodiments, when both hands are used to perform a two-handed gesture, the movements of the thumbs of both hands are treated as simultaneous inputs and used together to determine which function is triggered. For example, when the thumbs move from the tips of the index fingers to the bases of the index fingers of both hands (hands facing each other), the device expands the currently selected object, and when the thumbs move from the bases of the index fingers to the tips of the index fingers of both hands (hands facing each other), the device minimizes the currently selected object.In some embodiments, when the thumbs tap the index fingers of both hands simultaneously, the device activates the currently selected object in a first manner (e.g., start a video recording using a camera app), when the thumbs tap the index fingers of the left hand, the device activates the currently selected object in a second manner (e.g., perform autofocus using the camera app), and when the thumbs tap the index fingers of the right hand, the device activates the currently selected object in a third manner (e.g., take a snapshot using the camera app).

[0218] In some embodiments, in response to detecting using the one or more cameras a movement of the user's thumb over the user's index finger (e.g., as opposed to a more exaggerated gesture such as waving a finger or hand in the air or sliding across the touch-sensitive surface), pursuant to a determination that the movement includes a touchdown of the thumb on the index finger of the first hand followed by a wrist flick gesture of the first hand (e.g., an upward movement of the first hand relative to the wrist of the first hand), the computer system performs a tenth action (e.g., providing an input for operating a selected user interface object, providing an input for selecting an object (e.g., a virtual object selected and / or held by the user), and / or providing an input for discarding the object) that is distinct from the first action (e.g., distinct from each or a subset of the first through ninth actions that correspond to other types of movement patterns of the user's fingers). In some embodiments, while the device detects that the user's gaze is directed at a selectable object within the three-dimensional environment (e.g., a photo file icon, a video file icon, a notification banner, etc.), the device detects a touchdown of the user's thumb on the index finger followed by an upward wrist flick gesture, and the device launches an experience corresponding to the object (e.g., open a photo in mid-air, start a 3D movie, open a magnified notification, etc.).

[0219] In some embodiments, in response to detecting, using one or more cameras, a movement of the user's thumb over the user's index finger (e.g., as opposed to a more exaggerated gesture such as waving a finger or hand in the air or sliding across a touch-sensitive surface), in accordance with a determination that the movement includes a touchdown of the thumb on the index finger of the first hand followed by a hand rotation gesture of the first hand (e.g., a rotation of at least a portion of the first hand relative to the wrist of the first hand), the computer system performs an eleventh action (e.g., adjusts an amount, value corresponding to the amount of hand rotation) that is distinct from the first action (e.g., distinct from each or a subset of the first through tenth actions that correspond to other types of movement patterns of the user's fingers). For example, the eleventh action rotates a virtual object (e.g., selected and / or held by the user (e.g., using gaze)) or a user interface object (e.g., a virtual dial control) in accordance with the hand rotation gesture.

[0220] In some embodiments, while displaying the view of the three-dimensional environment, the computer system detects a movement of the palm of the user's first hand toward the user's face. In accordance with a determination that the movement of the palm of the user's first hand toward the user's face satisfies the invocation criterion, the computer system performs a twelfth operation different from the first operation (e.g., different from each of the first to eleventh operations or a subset corresponding to other types of movement patterns of the user's fingers) (e.g., displaying a user interface object associated with the virtual assistant, and / or displaying an image (e.g., a virtual representation of the user, a camera view of the user, a magnified view of the three-dimensional environment, and / or a magnified view of an object (e.g., a virtual object and / or a real object in the three-dimensional environment)) at a position corresponding to the palm of the first hand). In some embodiments, the invocation criterion includes a criterion that is met in accordance with a determination that the distance between the palm of the user and the user's face has decreased below a threshold distance. In some embodiments, the invocation criterion includes a criterion that is met in accordance with a determination that the fingers of the hand are extended.

[0221] It should be understood that the particular order described for the operations in FIG. 8 is merely an example, and the order described is not intended to indicate that the operations are the only order in which they can be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that other process details described herein with respect to other methods described herein (e.g., methods 9000, 10000, 11000, 12000, and 13000) can also be applied in a similar manner to method 8000 described above in connection with FIG. 8. For example, the contacts, gestures, gaze inputs, physical objects, and user interface objects, and animations described above with reference to method 8000 optionally have one or more characteristics of the contacts, gestures, gaze inputs, physical objects, and user interface objects, and animations described herein with reference to other methods described herein (e.g., methods 9000, 10000, 11000, 12000, and 13000). For the sake of brevity, those details will not be repeated here.

[0222] FIG. 9 is a flowchart of an exemplary method 9000 for interacting with a three-dimensional environment using predefined input gestures, according to some embodiments. In some embodiments, the method 9000 is performed on a computer system (e.g., computer system 101 of FIG. 1) including a display generating component (e.g., display generating component 120 of FIG. 1, FIG. 3, and FIG. 4) (e.g., a head-up display, a display, a touch screen, a projector, etc.) and one or more cameras (e.g., a camera facing downward at the user's hand (e.g., color sensors, infrared sensors, and other depth-sensing cameras)) or a camera facing forward from the user's head). In some embodiments, the method 9000 is performed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of the computer system 101 (e.g., control unit 110 of FIG. 1A). Some operations of the method 9000 are optionally combined and / or the order of some operations is optionally changed.

[0223] In method 9000, a computer system displays (9002) a view of a three-dimensional environment (e.g., a virtual environment, or an augmented reality environment). While displaying the three-dimensional environment, the computer system detects a hand at a first position corresponding to a portion of the three-dimensional environment (e.g., detects a hand at a position in the physical environment that makes the hand visible to the user according to the user's current view of the three-dimensional environment (e.g., the user's hand is moving to a position at or near where it intersects the user's line of sight)). In some embodiments, a representation or image of the user's hand is displayed in the portion of the three-dimensional environment in response to detecting the hand at the first position in the physical environment. In response to detecting a hand in a first position corresponding to a portion of the three-dimensional environment (9004), and in accordance with a determination that the hand is being held in a first predetermined configuration (e.g., using a camera or a touch-sensitive glove or touch-sensitive finger attachment to detect a predetermined ready state, such as the thumb resting on the index finger), a hand gesture (e.g., a visual indication, such as a system affordance (e.g., system affordance 7214 in Figures 7E, 7F, and 7G), a dock, a menu, an avatar for a voice-based virtual assistant, or the like, in response to a hand gesture input within the three-dimensional environment (e.g., in response to a hand gesture input, such as a hand gesture input, ... 7D , display a visual indication of a first operational context for gesture input using a hand gesture in the three dimensional environment (e.g., displaying additional information about a user interface element in the three dimensional environment that can be manipulated), and not display a visual indication of a first operational context for gesture input using a hand gesture in the three dimensional environment pursuant to a determination that the hand is not maintained in the first predetermined configuration (e.g., displaying a representation of the hand in the portion of the three dimensional environment without displaying a visual indication proximate to the representation of the hand, as shown in FIG. 7D ).

[0224] In some embodiments, a visual indication of a first motion context for gesture input using a hand gesture is displayed at a location in a portion of the three-dimensional environment that corresponds to the first location (e.g., the detected hand location). For example, the visual indication (e.g., a home affordance or a dock, etc.) is displayed at a location at and / or within a predetermined distance of the detected hand location. In some embodiments, the visual indication is displayed at a location that corresponds to a particular part of the hand (e.g., above an upper part of the detected hand, below a lower part of the detected hand, and / or over the hand).

[0225] In some embodiments, while displaying the visual indication on the portion of the three-dimensional environment, the computer system detects a change in the position of the hand from a first position to a second position (e.g., detects a movement and / or rotation of the hand within the three-dimensional environment (e.g., while the hand is in a first predetermined configuration, or some other predetermined configuration that indicates a ready state of the hand)). In response to detecting the change in the position of the hand from the first position to the second position, the computer system changes the displayed position of the visual indication according to the detected change in the position of the hand (e.g., maintains the display of the visual indication within a predetermined distance of the hand within the three-dimensional environment).

[0226] In some embodiments, the visual indication includes a set of one or more user interface objects. In some embodiments, the visual indicator is a system affordance icon (e.g., system affordance 7120 of FIG. 7E) that indicates an area in which one or more user interface objects can be displayed and / or accessed. For example, as shown in part (A) of FIG. 7F, when the thumb of hand 7200 moves in direction 7120 across the index finger, the display of visual indicator 7214 is replaced with the display of a set of user interface objects 7170. In some embodiments, the presence of a system affordance icon near the user's hand indicates that the next gesture provided by the hand will cause a system-level action (e.g., a non-object or application-specific action, an action performed by an operating system independent of an application), and the absence of a system affordance icon near the user's hand indicates that the next gesture provided by the hand will cause an application or object-specific action (e.g., an action that is specific to or at the currently selected object or application, e.g., an action performed by the application). In some embodiments, the device displays a system affordance near the hand pursuant to a determination that the user's gaze is directed toward the ready hand.

[0227] In some embodiments, the one or more user interface objects include a plurality of application launch icons (e.g., the one or more user interface objects are a dock that includes a row of application launch icons for a plurality of frequently used applications or experiences), and activation of each of the application launch icons causes an action associated with the corresponding application to be performed (e.g., launches the corresponding application).

[0228] In some embodiments, while displaying the visual indication, the computer system detects a change in the hand configuration from a first predefined configuration to a second predefined configuration (e.g., detects a change in the position of the thumb relative to another finger, such as moving across another finger). In response to detecting the change in the detected hand configuration from the first predefined configuration to the second predefined configuration, the computer system displays (e.g., in addition to and / or in place of displaying the visual indication) a first set of user interface objects (e.g., a home area or an application launching user interface), and activation of each user interface object of the first set of user interface objects causes an operation associated with the respective user interface object to be executed. In some embodiments, the visual indicator is a system affordance icon (e.g., system affordance 7214 of FIGS. 7E and 7F ) indicating an area in which the home area or the application launching user interface can be displayed and / or accessed. For example, as shown in portion (A) of FIG. 7F , when the thumb of the hand 7200 moves across the index finger in the direction 7120, the display of the visual indicator 7214 is replaced with the display of a set of user interface objects 7170. In some embodiments, at least some of the user interface objects in the first set of user interface objects are application launch icons, and activation of the application launch icons launches the corresponding applications.

[0229] In some embodiments, while displaying the visual indication, the computer system determines whether the hand movement satisfies the interaction criteria (e.g., the interaction criteria is met according to a determination that at least one finger and / or thumb of the hand moves a distance exceeding a threshold distance and / or moves according to a predefined gesture) during a time window (e.g., a time window of 5 seconds, 8 seconds, 15 seconds, etc. from the time the visual indication was displayed in response to detecting a ready hand at a first location). Following a determination that the hand movement does not satisfy the interaction criteria during the time window, the computer system stops displaying the visual indication. In some embodiments, the device redisplays the visual indication when the hand is again detected in the first predefined configuration within the user's field of view after the user's hand leaves the user's field of view or the user's hand changes to another configuration that is not the first or other predefined configuration corresponding to a ready hand state.

[0230] In some embodiments, while displaying the visual indication, the computer system detects a change in the hand configuration from a first predefined configuration to a second predefined configuration that meets the input criteria (e.g., the hand configuration has changed, but the hand is still within the user's field of view). For example, the detected change is a change in the position of the thumb (e.g., a change relative to another finger, such as contact with and / or release of contact from another finger, movement along the length of and / or movement across another finger), and / or a change in the position of the index finger and / or middle finger of the hand (e.g., an extension of the fingers and / or other movement of the fingers relative to the hand). In response to detecting a change in the hand configuration from a first predetermined configuration to a second configuration that meets the input criteria (e.g., pursuant to a determination that the user's hands have changed from a configuration that is a starting state for a first accepted gesture to a starting state for a second accepted gesture), the computer system adjusts a visual indication (e.g., adjusts a selected respective user interface object of the set of one or more user interface objects from the first respective user interface object to a second respective user interface object, changes a displayed position of the one or more user interface objects, and / or displays and / or ceases displaying the respective user interface object of the one or more user interface objects).

[0231] In some embodiments, while displaying the visual indication, the computer system detects a change in the hand configuration from a first predefined configuration to a third configuration that does not meet the input criteria (e.g., the configuration does not meet the input criteria according to a determination that at least a portion of the hand is outside the user's field of view). In some embodiments, the device determines that the third configuration does not meet the input criteria according to a determination that the user's hand has changed from a configuration that is a starting state of a first accepted gesture to a state that does not correspond to a starting state of any accepted gesture. In response to detecting a change in the detected hand configuration from the first predefined configuration to the third configuration that does not meet the input criteria, the computer system stops displaying the visual indication.

[0232] In some embodiments, after ceasing to display the visual indication, the computer system detects a change in the hand configuration to the first predetermined configuration (and the hand being within the user's field of view), and in response to detecting a change in the detected hand configuration to the first predetermined configuration, the computer system re-displays the visual indication.

[0233] In some embodiments, in response to detecting a hand at a first position corresponding to a portion of the three-dimensional environment, and in accordance with a determination that the hand is not being held in the first predetermined configuration, the computer system performs an action other than displaying a visual indication of the first operational context for gesture input using the hand gesture (e.g., displaying a representation of the hand without a visual indication and / or providing a prompt indicating that the hand is not being held in the first predetermined configuration).

[0234] It should be understood that the particular order described for the operations in FIG. 9 is merely an example, and the order described is not intended to indicate that the operations are the only order in which they can be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that other process details described herein with respect to other methods described herein (e.g., methods 8000, 10000, 11000, 12000, and 13000) can also be applied in a similar manner to method 9000 described above in connection with FIG. 9. For example, the contacts, gestures, gaze inputs, physical objects, and user interface objects, and animations described above with reference to method 9000 optionally have one or more characteristics of the contacts, gestures, gaze inputs, physical objects, and user interface objects, and animations described herein with reference to other methods described herein (e.g., methods 8000, 10000, 11000, 12000, and 13000). For the sake of brevity, those details will not be repeated here.

[0235] FIG. 10 is a flowchart of an exemplary method 10000 for interacting with a three-dimensional environment using predefined input gestures, according to some embodiments. In some embodiments, the method 10000 is performed on a computer system (e.g., computer system 101 of FIG. 1) including a display generating component (e.g., display generating component 120 of FIG. 1, FIG. 3, and FIG. 4) (e.g., a head-up display, a display, a touch screen, a projector, etc.) and one or more cameras (e.g., a camera facing downward at the user's hand (e.g., color sensors, infrared sensors, and other depth-sensing cameras)) or a camera facing forward from the user's head). In some embodiments, the method 10000 is performed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of the computer system 101 (e.g., control unit 110 of FIG. 1A). Some operations of the method 10000 are optionally combined and / or the order of some operations is optionally changed.

[0236] In method 10000, a computer system displays (10002) a three-dimensional environment (e.g., an augmented reality environment) that includes displaying a representation of a physical environment (e.g., displaying a camera view of the physical environment surrounding a user, or including a pass-through portion that causes the physical environment surrounding a user to be displayed within a displayed user interface or virtual environment). While displaying the representation of the physical environment, the computer system detects (10004) a gesture (e.g., using a camera or one or more motion sensors) (e.g., a gesture including a predetermined movement of a user's hand, fingers, wrist, or arm, or a predetermined resting posture of the hand that differs from the hand's natural resting posture). In response to detecting the gesture (10006), and in accordance with a determination that the user's gaze is directed to a location (e.g., in the three-dimensional environment) corresponding to a predetermined physical location in the physical environment (e.g., the user's hand) (e.g., in accordance with a determination that the gaze is directed to and remains at that location while the gesture is initiated and completed, or in accordance with a determination that the gaze is directed to the hand while the hand is in the final state of the gesture (e.g., the hand's ready state (e.g., a predetermined resting posture of the hand)), the computer system displays a system user interface in the three-dimensional environment (e.g., a user interface including visual indications of interaction options available in the three-dimensional environment and / or selectable options, the user interface being displayed in response to the gesture and not displayed prior to detection of the gesture (e.g., when the gaze was directed). This is shown, for example, in portions A-1, A-2, and A-3 of FIG. 7G, where an input gesture by the hand 7200 causes interactions with system user interface elements such as system affordances 7214, system menus 7170, and application icons 7190. In some embodiments, the location corresponding to the predetermined physical location is a representation (e.g., a video image or a graphical abstraction) of the predetermined physical location in the three-dimensional environment (a user's hand, which is movable within the physical environment, or a physical object, which is stationary within the physical environment).In some embodiments, the system user interface includes one or more application icons (e.g., when activated, each application icon launches a corresponding application). In response to detecting the gesture (10006), and in accordance with a determination that the user's gaze is not directed to a location (e.g., in the three-dimensional environment) corresponding to a predetermined physical location in the physical environment (e.g., in accordance with a determination that the gaze is directed and / or remains in another location, or that the gaze is not detected between the beginning and completion of the gesture, or in accordance with a determination that the gaze is not directed to the hand while the hand is in the final state of the gesture (e.g., the hand is in a ready state (e.g., a predetermined resting posture of the hand)), an action is performed within the current context of the three-dimensional environment without displaying the system user interface. This is shown, for example, in Figures 7G, ​​B-1, B-2, and B-3. In some embodiments, the action includes a first action that changes a state of the electronic device (e.g., changing an output volume of the device) that does not generate a visual change in the three-dimensional environment. In some embodiments, the action includes a second action that displays the hand making a gesture and does not cause further interaction with the three-dimensional environment. In some embodiments, the action includes an action for changing a state of a virtual object that the gaze is currently directed at. In some embodiments, the action includes an action for changing a state of a virtual object with which the user last interacted in the three-dimensional environment. In some embodiments, the action includes an action for changing a state of a currently selected virtual object that has input focus.

[0237] In some embodiments, the computer system displays a system affordance (e.g., a home affordance indicating readiness to detect one or more system gestures that cause a user interface for system-level (as opposed to application-level) operations to be displayed) at a predetermined location relative to a location corresponding to the predetermined physical location. In some embodiments, the location corresponding to the predetermined physical location is a location within the three-dimensional environment. In some embodiments, the location corresponding to the predetermined physical location is a location on the display. In some embodiments, as long as the predetermined location of the system affordance is a location within the displayed three-dimensional environment, the system affordance remains displayed even if the location corresponding to the predetermined physical location is no longer visible in the displayed portion of the three-dimensional environment (e.g., the system affordance continues to be displayed even if the predetermined physical location moves out of the field of view of one or more cameras of the electronic device). In some embodiments, the system affordance is displayed at a predetermined, fixed location of the user's hand, wrist, or fingers relative to a representation of the user's hand, wrist, or fingers in the three-dimensional environment (e.g., superimposed on or replacing a representation of a portion of the user's hand, wrist, or fingers or at a fixed location offset from the user's hand, wrist, or fingers). In some embodiments, the system affordances are displayed at a predetermined location relative to a location that corresponds to a predetermined physical location, regardless of whether the user's gaze remains directed at the location in the three-dimensional environment (e.g., the system affordances remain displayed for a predetermined timeout period even after the user's gaze moves away from the user's ready hand or after the gesture is completed).

[0238] In some embodiments, displaying the system affordance at a predetermined location relative to a location corresponding to the predetermined physical location includes detecting movement at a location corresponding to the predetermined physical location in the three dimensional environment (e.g., detecting that a position of the user's hand shown in the three dimensional environment has changed as the user's head or hand moves), and, in response to detecting movement of the location corresponding to the predetermined physical location in the three dimensional environment, moving the system affordance within the three dimensional environment such that the relative position of the system affordance and the location corresponding to the predetermined physical location remain unchanged in the three dimensional environment (e.g., as the position of the user's hand changes in the three dimensional environment, the system affordance follows the position of the user's hand (e.g., the system affordance is displayed at a location corresponding to the top of the user's thumb in the displayed view of the three dimensional environment)).

[0239] In some embodiments, the system affordance is displayed at a predetermined position relative to a position corresponding to a predetermined physical location (e.g., sometimes referred to as a "predetermined relative position") pursuant to a determination that the user's gaze is directed to a position corresponding to the predetermined physical location. In some embodiments, the system affordance is displayed at a predetermined relative position pursuant to a determination that the user's gaze is directed to a proximal position of the predetermined physical location (e.g., within a predetermined threshold distance of the predetermined physical location). In some embodiments, the system affordance is not displayed when the user's gaze is not directed to the predetermined physical location (e.g., when the user's gaze moves away from the predetermined physical location or is at least a predetermined distance away from the predetermined physical location). In some embodiments, while displaying the system affordance at a predetermined position relative to a position corresponding to the predetermined physical location in the three-dimensional environment, the device detects that the user's gaze moves away from the position corresponding to the predetermined physical location, and in response to detecting that the user's gaze moves away from the position corresponding to the predetermined physical location in the three-dimensional environment, the device stops displaying the system affordance at the predetermined position in the three-dimensional environment.

[0240] In some embodiments, displaying the system affordance at a predetermined location relative to a location corresponding to a predetermined physical location within the three-dimensional environment includes displaying the system affordance having a first appearance (e.g., shape, size, color, etc.) in accordance with a determination that the user's gaze is not directed to the location corresponding to the predetermined physical location, and displaying the system affordance having a second appearance different from the first appearance in accordance with a determination that the user's gaze is directed to the location corresponding to the predetermined physical location. In some embodiments, the system affordance has the first appearance while the user's gaze is away from the location corresponding to the predetermined physical location. In some embodiments, the system affordance has the second appearance while the user's gaze is directed to the location corresponding to the predetermined physical location. In some embodiments, the system affordance changes from the first appearance to the second appearance when the user's gaze moves (e.g., at least within a threshold distance) to the location corresponding to the predetermined physical location, and changes from the second appearance to the first appearance when the user's gaze moves (e.g., at least a threshold distance away) away from the location corresponding to the predetermined physical location.

[0241] In some embodiments, the system affordance is displayed at a predetermined location relative to a location corresponding to a predetermined physical location pursuant to a determination that the user is ready to perform the gesture. In some embodiments, determining that the user is ready to perform the gesture includes detecting an indication that the user is ready to perform the gesture, for example, by detecting that the predetermined physical location (e.g., the user's hand, wrist, or finger(s)) is in a predetermined configuration (e.g., a predetermined posture relative to a device in a physical environment). In one example, the system affordance is displayed at a predetermined location relative to a displayed representation of the user's hand in a three-dimensional environment when the device detects that the user has placed the hand in a predetermined ready state (e.g., a particular position and / or orientation of the hand) in the physical environment in addition to detecting a line of sight to the ready hand. In some embodiments, the predetermined configuration requires that the predetermined physical location (e.g., the user's hand) have a particular location relative to the electronic device or one or more input devices of the electronic device, such as being within the field of view of one or more cameras.

[0242] In some embodiments, displaying the system affordance at a predetermined location relative to a location corresponding to the predetermined physical location includes displaying the system affordance having a first appearance according to a determination that the user is not ready to perform the gesture. In some embodiments, determining that the user is not ready to perform the gesture includes detecting an indication that the user is not ready to perform the gesture (e.g., detecting that the user's hands are not in a predetermined ready state). In some embodiments, determining that the user is not ready includes failing to detect an indication that the user is ready (e.g., failing to detect or being unable to detect that the user's hands are in a predetermined ready state if the user's hands are outside the field of view of one or more cameras of the electronic device). Detecting an indication of the user's readiness to perform the gesture is described in further detail herein with reference to FIG. 7E and related discussion. In some embodiments, displaying the system affordance at a predetermined location relative to a location corresponding to the predetermined physical location further includes displaying the system affordance having a second appearance different from the first appearance pursuant to a determination that the user is ready to perform a gesture (e.g., pursuant to detecting an indication that the user is ready to perform a gesture, as described herein with reference to FIG. 7E and related description). Those skilled in the art will recognize that the presence or absence of a system affordance, and the particular appearance of the system affordance, may be modified depending on what information is intended to be conveyed to the user in that particular context (e.g., depending on what action(s) are performed in response to the gesture and / or whether additional criteria need to be met for the gesture to invoke a system user interface).In some embodiments, while the system affordance is displayed at a predetermined position relative to a position corresponding to a predetermined physical location in the three-dimensional environment, the device detects a change in the user's hand from a first state to a second state, and in response to detecting the change from the first state to the second state, in accordance with a determination that the first state is a ready state and the second state is not a ready state, the device displays a system affordance having a second appearance (e.g., a change from the first appearance), and in accordance with a determination that the first state is not a ready state and the second state is a ready state, the device displays a system affordance having a first appearance (e.g., a change from the second appearance). In some embodiments, if the computer system does not detect the user's gaze on the user's hand and the user's hand is not in the ready state configuration, the computer system does not display the system affordance, or, optionally, displays the system affordance having the first appearance. If a subsequent input gesture is detected (e.g., if the system affordance is not displayed or is displayed in the first appearance), the computer system does not perform a system action corresponding to the input gesture, or optionally performs an action within the current user interface context corresponding to the input gesture. In some embodiments, if the computer system detects a user's gaze on the user's hand but the hand is not in the ready state configuration, the computer system does not display a system affordance, or optionally displays a system affordance having a first appearance or a second appearance. If a subsequent gesture input is detected (e.g., if the system affordance is not displayed or is displayed in the first appearance or a second appearance), the computer system does not perform a system action corresponding to the input gesture, or optionally displays a system user interface (e.g., a dock or a system menu). In some embodiments, if the computer system does not detect a user's gaze on the user's hand but the hand is in the ready state configuration, the computer system does not display a system affordance, or optionally displays a system affordance having a first appearance or a second appearance.If a subsequent gesture input is detected (e.g., if the system affordance is not displayed or is displayed with the first or second appearance), the computer system does not perform a system action corresponding to the input gesture, or, optionally, performs an action within the current user interface context. In some embodiments, if the computer system detects the user's gaze on the user's hand and the hand is in the ready state configuration, the computer system displays a system affordance having the second or third appearance. If a subsequent gesture input is detected (e.g., if the system affordance has the second or third appearance), the computer system performs a system action (e.g., displays a system user interface). In some embodiments, two or more of the above are combined in the same implementation.

[0243] In some embodiments, the predefined physical location is the user's hand, and determining that the user is ready to perform a gesture (e.g., the hand is currently in a predefined ready state or a start gesture has been detected) includes determining that a predefined portion of the hand (e.g., a designated finger) is in contact with a physical control element. In some embodiments, the physical control element is a controller (e.g., a respective input device) separate from the user (e.g., the ready state is a state in which the user's thumb is in contact with a touch-sensitive strip or ring attached to the user's index finger). In some embodiments, the physical control element is a different portion of the user's hand (e.g., the ready state is a state in which the thumb is in contact with the top side of the index finger (e.g., near the second knuckle)). In some embodiments, the device detects whether the hand is in the predefined ready state using a camera and displays the hand in the ready state in a view of the three-dimensional environment. In some embodiments, the device detects whether the hand is in the predefined ready state using a physical control element that is touch-sensitive and communicatively coupled to the electronic device to transmit touch input to the electronic device.

[0244] In some embodiments, the predetermined physical location is the user's hand, and determining that the user is ready to perform the gesture includes determining that the hand is elevated above a predetermined elevation relative to the user. In some embodiments, determining that the hand is elevated includes determining that the hand is positioned above a particular lateral plane relative to the user (e.g., above the user's waist, i.e., closer to the user's head than the user's feet). In some embodiments, determining that the hand is elevated includes determining that the user's wrist or elbow is bent at least a particular amount (e.g., within a 90 degree angle). In some embodiments, the device uses a camera to detect whether the hand is in a predefined ready state, and optionally displays the ready hand in a view of the three-dimensional environment. In some embodiments, the device uses one or more sensors (e.g., motion sensors) attached to the user's hand, wrist, or arm to detect whether the hand is in a predefined ready state, and is communicatively coupled to the electronic device to transmit movement input to the electronic device.

[0245] In some embodiments, the predefined physical location is the user's hand, and determining that the user is ready to perform a gesture includes determining that the hand is in a predefined configuration. In some embodiments, the predefined configuration requires that a corresponding finger of the hand (e.g., thumb) is in contact with a different portion of the user's hand (e.g., an opposing finger such as an index finger, or a predefined portion of the opposing index finger such as the middle phalanx or middle joint). In some embodiments, the predefined configuration requires that the hand is above a particular lateral plane (e.g., above the user's waist), as described above. In some embodiments, the predefined configuration requires flexion (e.g., radial flexion) of the wrist toward the thumb side and away from the pinky side (e.g., without axial rotation of the arm). In some embodiments, when the hand is in the predefined configuration, one or more fingers are in a natural resting position (e.g., curved), and the entire hand is tilted or moved away from the natural resting position of the hand, wrist, or arm to indicate that the user is ready to perform a gesture. One skilled in the art will recognize that the particular predefined ready state used may be selected to have an intuitive and natural user interaction and may require any combination of the criteria described above. In some embodiments, when a user desires merely to view the three-dimensional environment rather than provide input to and interact with the three-dimensional environment, the predetermined configuration differs from the natural resting position of the user's hands (e.g., resting relaxed on the lap, on a tabletop, or at the side of the body). The change from the natural resting position to the predetermined configuration is intentional and requires the intentional movement of the user's hands into the predetermined configuration.

[0246] In some embodiments, the location corresponding to the predetermined physical location is a fixed location in the three-dimensional environment (e.g., the corresponding predetermined physical location is a fixed location in the physical environment). In some embodiments, the physical environment is the user's frame of reference. That is, one skilled in the art will recognize that a location referred to as a "fixed" location in the physical environment may not be an absolute location in space, but is fixed relative to the user's frame of reference. In some examples, if the user is in a room of a building, the location is a fixed location in the three-dimensional environment that corresponds to (e.g., is a representation of) a fixed location in the room (e.g., on a wall, floor, or ceiling of the room). In some examples, if the user is inside a moving vehicle, the location is a fixed location in the three-dimensional environment that corresponds to (e.g., represents a fixed location) a fixed location along the interior of the vehicle. In some embodiments, the location is fixed relative to content displayed in the three-dimensional environment, and the displayed content corresponds to a fixed, predetermined physical location in the physical environment.

[0247] In some embodiments, the location corresponding to the predetermined physical location is a fixed location relative to the display of the three-dimensional environment (e.g., relative to the display generating component). In some embodiments, the location is fixed relative to the user's viewpoint of the three-dimensional environment (e.g., the location is fixed relative to the display of the three-dimensional environment by the display generating component), regardless of the particular content displayed within the three-dimensional environment, which is typically updated as the user's viewpoint changes (e.g., in response to or with the user's viewpoint change). In some examples, the location is a fixed location along an edge of the display of the three-dimensional environment (e.g., within a predetermined distance of the edge). In some examples, the location is centered relative to the display of the three-dimensional environment (e.g., centered within a display area along a bottom, top, left edge, or right edge of the display of the three-dimensional environment).

[0248] In some embodiments, the predetermined physical location is a fixed location on the user. In some examples, the predetermined physical location is a hand or finger of the user. In some such examples, the location corresponding to the predetermined physical location includes a displayed representation of the user's hand or finger within the three-dimensional environment.

[0249] In some embodiments, after displaying the system user interface in the three-dimensional environment, the computer system detects a second gesture (e.g., a second gesture performed by the user's hand, wrist, finger(s), or arm) while displaying the system user interface (e.g., after detecting the first gesture and a gaze directed at a position corresponding to the predetermined physical location). In response to detecting the second gesture, the system user interface (e.g., an application launching user interface) is displayed. In some embodiments, the second gesture is a continuation of the first gesture. For example, the first gesture is a swipe gesture (e.g., by the movement of the user's thumb over the user's index finger on the same hand) and the second gesture is a continuation of the swipe gesture (e.g., continued movement of the thumb over the index finger) (e.g., the second gesture starts from the end position of the first gesture without resetting the start position of the second gesture to the start position of the first gesture). In some embodiments, the second gesture is a repetition of the first gesture (e.g., after performing the first gesture, the starting location of the second gesture is reset to within a predetermined distance of the starting location of the first gesture, and the second gesture again retraces the movement of the first gesture within a predetermined tolerance). In some embodiments, displaying the home user interface includes extending system affordances from the predetermined location to a location corresponding to the predetermined physical location to occupy a larger portion of the displayed three-dimensional environment and to show additional user interface objects and options.In some embodiments, the system affordances are indicators without the respective content, and the respective content (e.g., a dock with a row of application icons of recently used or frequently used applications) replaces the indicator in response to a first swipe gesture by the hand, a two-dimensional grid of all application icons of installed applications replaces the dock in response to a second swipe gesture by the hand, and a three-dimensional working environment with interactive application icons floating at different depths and positions within the three-dimensional working environment replaces the two-dimensional grid in response to a third swipe gesture by the hand.

[0250] In some embodiments, the current context of the three-dimensional environment includes a display of an indication of the received notification (e.g., an initial display of a subset of information about the received notification), and performing an action within the current context of the three-dimensional environment includes displaying an enlarged notification that includes additional information about the received notification (e.g., displaying information beyond the initially displayed subset). In some embodiments, the current context of the three-dimensional environment is determined based on a location where the gaze is currently directed. In some embodiments, when a notification is received and displayed in the three-dimensional environment and a user's gaze is detected to be directed at the notification (and not in a location that corresponds to a predetermined physical location (e.g., the user's hand)), the device determines that the current context is interacting with the notification and displays the enlarged notification content in the three-dimensional environment in response to detecting a user gesture (e.g., an upward flick gesture with a thumb or wrist).

[0251] In some embodiments, the current context of the three-dimensional environment includes a display of one or more photographs (e.g., one or more respective thumbnails of the one or more photographs), and performing an operation within the current context of the three-dimensional environment includes displaying at least one of the one or more photographs within the three-dimensional environment (e.g., displaying in an augmented manner (e.g., magnified, animated, enhanced, 3D, etc.)). In some embodiments, the current context of the three-dimensional environment is determined based on a location to which the gaze is currently directed. In some embodiments, when an image is displayed in the three-dimensional environment and a user's gaze is detected to be directed at the image (and not in a location corresponding to a predetermined physical location (e.g., the user's hand)), the device determines that the current context is interacting with the image and, in response to detecting a user gesture (e.g., an upward flick gesture with a thumb or wrist), displays the augmented image in the three-dimensional environment.

[0252] It should be understood that the particular order described for the operations in FIG. 10 is merely an example, and the order described is not intended to indicate that the operations are the only order in which they can be performed. Those skilled in the art will recognize various ways to reorder the operations described herein. In addition, it should be noted that other process details described herein with respect to other methods described herein (e.g., methods 8000, 9000, 11000, 12000, and 13000) can also be applied in a manner similar to method 10000 described above in connection with FIG. 10. For example, the contacts, gestures, gaze inputs, physical objects, and user interface objects, and animations described above with reference to method 10000 optionally have one or more characteristics of the contacts, gestures, gaze inputs, physical objects, and user interface objects, and animations described herein with reference to other methods described herein (e.g., methods 8000, 9000, 11000, 12000, and 13000). For the sake of brevity, those details will not be repeated here.

[0253] FIG. 11 is a flowchart of an exemplary method 11000 for interacting with a three-dimensional environment using a predefined input gesture, according to some embodiments. In some embodiments, the method 11000 is performed on a computer system (e.g., computer system 101 of FIG. 1) including a display generating component (e.g., display generating component 120 of FIG. 1, FIG. 3, and FIG. 4) (e.g., a head-up display, a display, a touch screen, a projector, etc.) and one or more cameras (e.g., a camera facing downward at the user's hand (e.g., color sensors, infrared sensors, and other depth-sensing cameras)) or a camera facing forward from the user's head). In some embodiments, the method 11000 is performed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of the computer system 101 (e.g., control unit 110 of FIG. 1A). Some operations of the method 11000 are optionally combined and / or the order of some operations is optionally changed. ...

Claims

1. 1. A method comprising:

1. A computer system including a display generation component and one or more input devices, comprising: displaying a three dimensional environment, including displaying a representation of a physical environment; detecting a first gesture made by a user's hand while displaying the three-dimensional environment including the representation of the physical environment; In response to detecting the first gesture, displaying a system user interface within the three-dimensional environment in accordance with a determination that the user's gaze is directed toward a representation of the user's hand within the three-dimensional environment; suppressing a display of the system user interface within the three dimensional environment in accordance with a determination that the user's gaze is not directed toward the representation of the user's hand within the three dimensional environment; A method comprising:

2. Displaying the system user interface within the three dimensional environment, Detecting a movement of the user's hand within the physical environment; and 2. The method of claim 1, comprising: in response to detecting a movement of the user's hand within the physical environment, moving the system user interface within the three-dimensional environment such that a relative position of the system user interface and the representation of the user's hand within the three-dimensional environment does not change.

3. A method as described in claim 1 or 2, wherein the system user interface is displayed pursuant to a determination that the user's hand has been raised beyond a predetermined height relative to the user.

4. A method according to any one of claims 1 to 3, wherein the system user interface is displayed pursuant to a determination that the user's hand is in a predetermined shape after the first gesture.

5. After displaying the system user interface within the three-dimensional environment, Detecting a second gesture; and 5. The method of claim 1, further comprising: in response to detecting the second gesture, displaying within the three-dimensional environment an application launch user interface including application launch icons corresponding to a plurality of applications. In response to detecting the first gesture, 6. The method of claim 1, further comprising: performing an action in a current context of the three-dimensional environment without displaying the system user interface in accordance with a determination that the user's gaze is not directed toward the representation of the user's hand in the three-dimensional environment.

7. 7. The method of claim 6, wherein the current context of the three-dimensional environment includes displaying an indication of a received notification, and performing the action in the current context of the three-dimensional environment includes displaying an enlarged notification that includes additional information about the received notification.

8. 7. The method of claim 6, wherein the current context of the three-dimensional environment includes displaying an indication of one or more photographs, and performing the action in the current context of the three-dimensional environment includes displaying at least one of the one or more photographs within the three-dimensional environment.

9. The method of claim 1, wherein the system user interface is a control user interface for the computer system.

10. A method according to any one of claims 1 to 9, wherein detecting the first gesture includes detecting movement of the palm of the user's hand relative to the user's face.

11. A computer-readable storage medium storing executable instructions that, when executed by a computer system having one or more processors and a display generating component, cause the computer system to perform the method of any one of claims 1 to 10.

12. 1. A computer system comprising: one or more processors; A display generating component; A memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 1 to 10; A computer system comprising:

Images