SYSTEM, METHOD, AND GRAPHICAL USER INTERFACE FOR UPDATING A DEVICE'S DISPLAY OF A USER'S BODY - Patent application

Abstract

To provide a method for optionally compensating or replacing a conventional method of a computer-generative reality experience.SOLUTION: While an electronic device is worn over a predefined portion of the body of a user, the electronic device displays, via a display generation component provided on the electronic device opposite the predefined portion of the body of the user, a graphical representation of an exterior view of a body part that corresponds to the predefined portion of the body of the user. The electronic device detects a change in position of the electronic device relative to the predefined portion of the body of the user. The electronic device, in response to detection of the change in the position of the electronic device relative to the predefined portion of the body of the user, modifies the graphical representation of the exterior view of the body part that corresponds to the predefined portion of the body of the user in accordance with the detected change in position of the electronic device relative to the predefined portion of the body of the user.SELECTED DRAWING: Figure 7H

Description

[Technical Field]

[0001] (Related Applications) This application is a continuation of U.S. Patent Application No. 17 / 483,744, filed September 23, 2021, which claims priority to U.S. Provisional Patent Application No. 63 / 083,911, filed September 26, 2020, each of which is incorporated by reference in its entirety.

[0002] (Technical field) The present disclosure generally relates to computer systems having a display generation component, including but not limited to electronic devices that provide virtual reality and mixed reality experiences via a display, and one or more input devices that provide computer-generated experiences. [Background technology]

[0003] The development of computer systems for augmented reality has progressed significantly in recent years. Exemplary augmented reality environments include at least some virtual elements that replace or augment the physical world. Input devices such as cameras, controllers, joysticks, touch-sensitive surfaces, and touchscreen displays for computer systems and other electronic computing devices are used to interact with the virtual / augmented reality environment. Exemplary virtual elements include virtual objects, which include digital images, videos, text, icons, and control elements such as buttons and other graphics.

[0004] However, methods and interfaces for interacting with environments (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) that include at least some virtual elements are cumbersome, inefficient, and limited. For example, systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to achieve a desired result in an augmented reality environment, and systems that make manipulating virtual objects complex, tedious, and error-prone create a significant cognitive load for users and detract from their experience with the virtual / augmented reality environment. In addition, these methods are unnecessarily time-consuming, thereby wasting energy. This latter consideration is particularly important in battery-operated devices. Summary of the Invention

[0005] Thus, there is a need for a computer system having improved methods and interfaces for providing users with computer-generated experiences that make interaction with the computer system more efficient and intuitive for the user. Such methods and interfaces can optionally complement or replace conventional methods of providing users with computer-generated reality experiences. Such methods and interfaces reduce the number, extent, and / or type of inputs from the user by helping the user understand the connection between the input provided and the device response to that input, thereby creating a more efficient human-machine interface.

[0006] The above-mentioned deficiencies and other problems associated with user interfaces for computer systems having a display generation component and one or more input devices are reduced or eliminated by the disclosed system. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is a portable device (e.g., a notebook computer, a tablet computer, or a handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device such as a wristwatch or a head-mounted device). In some embodiments, the computer system has a touchpad. In some embodiments, the computer system has one or more cameras. In some embodiments, the computer system has a touch-sensitive display (also known as a "touch screen" or "touchscreen display"). In some embodiments, the computer system has one or more eye-tracking components. In some embodiments, the computer system has one or more hand-tracking components. In some embodiments, the computer system has one or more output devices in addition to the display generation component, the output devices including one or more tactile output generators and one or more audio output devices. In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or instruction sets stored in the memory for performing a plurality of functions. In some embodiments, a user interacts with the GUI through stylus and / or finger contacts and gestures on the touch-sensitive surface, the user's eye and hand movements in space relative to the GUI or the user's body as captured by cameras and other motion sensors, and voice input as captured by one or more audio input devices.In some embodiments, the functions performed through the interactions optionally include image editing, drawing, presenting, word processing, spreadsheet creation, game playing, making phone calls, video conferencing, emailing, instant messaging, training support, digital photography, digital videography, web browsing, digital music playback, note taking, and / or digital video playback, and executable instructions to perform those functions are optionally contained on a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0007] There is a need for electronic devices with improved methods and interfaces for interacting with three-dimensional environments. Such methods and interfaces can complement or replace conventional methods for interacting with three-dimensional environments. Such methods and interfaces reduce the number, extent, and / or type of input from a user, creating a more efficient human-machine interface.

[0008] According to some embodiments, a method is performed on an electronic device configured to be worn on a predetermined portion of a user's body, the electronic device having one or more processors, a memory, and a display generation component disposed on the electronic device on an opposite side of the user's body from the predetermined portion. The method includes displaying, via the display generation component, a graphical representation of the appearance of a body portion corresponding to the predetermined portion of the user's body while the electronic device is worn on the predetermined portion of the user's body. The method further includes detecting a change in position of the electronic device relative to the predetermined portion of the user's body. In response to detecting the change in position of the electronic device relative to the predetermined portion of the user's body, the method includes modifying the graphical representation of the appearance of the body portion corresponding to the predetermined portion of the user's body according to the detected change in position of the electronic device relative to the predetermined portion of the user's body.

[0009] It should be noted that the various embodiments described above can be combined with any other embodiment described herein. The features and advantages described herein are not exhaustive, and many additional features and advantages will become apparent to those skilled in the art, particularly in light of the drawings, specification, and claims. Furthermore, it should be noted that the language used in this specification has been selected solely for the purposes of readability and explanation, and not to define or limit the subject matter of the present invention. [Brief explanation of the drawings]

[0010] For a better understanding of the various described embodiments, reference should be made to the following Detailed Description of the Invention in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout:

[0011] [Figure 1] FIG. 1 is a block diagram illustrating a computer system operating environment for providing a CGR experience, according to some embodiments.

[0012] [Figure 2] FIG. 1 is a block diagram illustrating a controller of a computer system configured to manage and coordinate a user's CGR experience, according to some embodiments.

[0013] [Figure 3] FIG. 1 is a block diagram illustrating display generation components of a computer system configured to provide a user with visual components of a CGR experience, according to some embodiments.

[0014] [Figure 4] FIG. 1 is a block diagram illustrating a hand tracking unit of a computer system configured to capture a user's gesture input, according to some embodiments.

[0015] [Figure 5]FIG. 1 is a block diagram illustrating an eye-tracking unit of a computer system configured to capture a user's gaze input, according to some embodiments.

[0016] [Figure 6] 1 is a flowchart illustrating a glint-assisted gaze tracking pipeline, according to some embodiments.

[0017] [Figure 7A] FIG. 1 illustrates a device with a display, according to some embodiments.

[0018] [Figure 7B] 10A-10C illustrate exemplary user interfaces for updating the device's display relative to the user's body, according to some embodiments. [Figure 7C] 10A-10C illustrate exemplary user interfaces for updating the device's display relative to the user's body, according to some embodiments. [Figure 7D] 10A-10C illustrate exemplary user interfaces for updating the device's display relative to the user's body, according to some embodiments. [Figure 7E] 10A-10C illustrate exemplary user interfaces for updating the device's display relative to the user's body, according to some embodiments. [Figure 7F] 10A-10C illustrate exemplary user interfaces for updating the device's display relative to the user's body, according to some embodiments. [Figure 7G] 10A-10C illustrate exemplary user interfaces for updating the device's display relative to the user's body, according to some embodiments. [Figure 7H] 10A-10C illustrate exemplary user interfaces for updating the device's display relative to the user's body, according to some embodiments. [Figure 7I]10A-10C illustrate exemplary user interfaces for updating the device's display relative to the user's body, according to some embodiments. [Figure 7J] 10A-10C illustrate exemplary user interfaces for updating the device's display relative to the user's body, according to some embodiments.

[0019] [Figure 8A] FIG. 1 is a flow diagram of a process for providing a user with a computer-generated experience that makes interacting with a computing system more efficient and intuitive for the user, according to some embodiments. [Figure 8B] FIG. 1 is a flow diagram of a process for providing a user with a computer-generated experience that makes interacting with a computing system more efficient and intuitive for the user, according to some embodiments. [Figure 8C] FIG. 1 is a flow diagram of a process for providing a user with a computer-generated experience that makes interacting with a computing system more efficient and intuitive for the user, according to some embodiments. [Figure 8D] FIG. 1 is a flow diagram of a process for providing a user with a computer-generated experience that makes interacting with a computing system more efficient and intuitive for the user, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0020] The present disclosure relates to a user interface that provides a computer-generated reality (CGR) experience to a user, according to some embodiments.

[0021] The systems, methods, and GUIs described herein improve user interface interaction with virtual / augmented reality environments in several ways.

[0022] In some embodiments, the systems, methods, and GUIs described herein improve the visual information available to a user and others in the surrounding environment by displaying animated portions of the user's body that are covered by the display. For example, they make it easier to provide a user with a computer-generated experience that makes interaction with a computing system more efficient and intuitive for the user and others in the surrounding environment.

[0023] Figures 1-6 illustrate an exemplary computer system for providing a CGR experience to a user. Figures 7A-7J illustrate exemplary user interfaces for providing a user with a computer-generated experience that makes interacting with a computing system more efficient and intuitive for the user. Figures 8A-8D illustrate a flow diagram of a method for providing a user with a computer-generated experience that makes interacting with a computing system more efficient and intuitive for the user. The user interfaces of Figures 7A-7J are used to illustrate the processes of Figures 8A-8D.

[0024] 1, a CGR experience is provided to a user via an operating environment 100 that includes a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), a display generation component 120 (e.g., a head-mounted device (HMD), a display, a projector, a touchscreen, etc.), one or more input devices 125 (e.g., an eye-tracking device 130, a hand-tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a tactile output generator 170, and other output devices 180), one or more sensors 190 (e.g., an image sensor, a light sensor, a depth sensor, a tactile sensor, an orientation sensor, a proximity sensor, a temperature sensor, a position sensor, a motion sensor, a velocity sensor, etc.), and optionally one or more peripheral devices 195 (e.g., a consumer electronics device, a wearable device, etc.). In some embodiments, one or more of the input device 125, the output device 155, the sensor 190, and the peripheral device 195 are integrated with the display generation component 120 (e.g., in a head-mounted or handheld device).

[0025] When describing a CGR experience, various terms are used to individually refer to several related but distinct environments that a user senses and / or can interact with (e.g., using inputs detected by computer system 101 that cause the computer system generating the CGR experience to generate audio, visual, and / or haptic feedback corresponding to various inputs provided to computer system 101 generating the CGR experience). The following is a subset of these terms:

[0026] Physical Environment: The physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic systems. A physical environment, such as a physical park, includes physical objects such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment through their senses, such as sight, touch, hearing, taste, and smell.

[0027] Computer-Generated Reality: In contrast, a computer-generated reality (CGR) environment refers to a wholly or partially simulated environment that people sense and / or interact with via electronic systems. In a CGR, a subset of a person's body movements or representations thereof are tracked, and in response, one or more properties of one or more virtual objects simulated within the CGR environment are adjusted to behave according to at least one law of physics. For example, a CGR system may detect a person's head rotation and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. In some circumstances (e.g., for accessibility reasons), adjustments to property(ies) of virtual object(s) in a CGR environment may be made in response to representations of body movements (e.g., voice commands). A person may sense and / or interact with a CGR object using any one of these senses, including sight, hearing, touch, taste, and smell. For example, a person may perceive and / or interact with audio objects that create a 3D or spatially expansive audio environment, providing the perception of a point sound source in 3D space. In another example, audio objects may enable audio transparency, selectively incorporating ambient sounds from the physical environment, with or without computer-generated audio. In some CGR environments, a person may perceive and / or interact with only audio objects.

[0028] Examples of CGR include virtual reality and mixed reality.

[0029] Virtual Reality: A virtual reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes multiple virtual objects that a person can sense and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence in the computer-generated environment and / or through a simulation of a subset of the person's physical movements in the computer-generated environment.

[0030] Mixed Reality: A mixed reality (MR) environment refers to a simulated environment designed to incorporate sensory input from or representations of a physical environment in addition to including computer-generated sensory input (e.g., virtual objects), as opposed to a VR environment designed to be based entirely on computer-generated sensory input. On the virtual continuum, a mixed reality environment is anywhere between, but not including, a fully physical environment on the one hand and a virtual reality environment on the other. In some MR environments, computer-generated sensory input may respond to changes in sensory input from the physical environment. Some electronic systems for presenting MR environments may also track the position and / or orientation relative to the physical environment to allow virtual objects to interact with real objects (i.e., physical items or representations thereof from the physical environment). For example, the system may account for movement so that a virtual tree appears stationary relative to the physical ground.

[0031] Examples of mixed reality include augmented reality and augmented virtuality.

[0032] Augmented reality: An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are overlaid on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on the transparent or translucent display, whereby a person using the system perceives the virtual objects overlaid on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system composites the images or videos with virtual objects and presents the composite on the opaque display. A person uses the system to indirectly view the physical environment through the images or videos of the physical environment and perceive the virtual objects overlaid on the physical environment. As used herein, videos of a physical environment shown on an opaque display are referred to as "pass-through video," meaning that the system captures images of the physical environment using one or more image sensors and uses those images in presenting the AR environment on the opaque display. Alternatively, the system may include a projection system that projects virtual objects, e.g., as holograms, into the physical environment or onto a physical surface, such that a person using the system perceives the virtual objects superimposed on the physical environment. An augmented reality environment also refers to a simulated environment in which a representation of the physical environment is transformed by computer-generated sensory information. For example, when providing pass-through video, the system may distort one or more sensor images to impart a selected perspective (e.g., viewpoint) different from the perspective captured by the imaging sensor. As another example, the representation of the physical environment may be distorted by graphically altering (e.g., magnifying) a portion thereof, thereby rendering the altered portion a non-photorealistic, altered version of the originally captured image. As a further example, the representation of the physical environment may be distorted by graphically removing or obscuring a portion thereof.

[0033] Augmented Virtual: An augmented virtual (AV) environment refers to a simulated environment in which the virtual or computer-generated environment incorporates one or more sensory inputs from the physical environment. The sensory inputs may be representations of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, while people with faces are realistically recreated from images taken of physical people. As another example, virtual objects may adopt the shape or color of physical items imaged by one or more imaging sensors. As a further example, virtual objects may adopt shadows that match the position of the sun in the physical environment.

[0034] Hardware: There are many different types of electronic systems that enable a person to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed over a person's eyes (e.g., similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have one or more speaker(s) and an integrated opaque display. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. A head-mounted system may have a transparent or translucent display rather than an opaque display. A transparent or translucent display may have a medium through which light representing an image is directed toward a person's eye. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser-scanned light source, or any combination of these technologies. The medium may be a light guide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, the transparent or translucent display may be configured to be selectively opaque. A projection-based system may employ retinal projection technology that projects a graphical image onto a person's retina. The projection system may also be configured to project virtual objects into the physical environment, for example, as holograms or as physical surfaces. In some embodiments, the controller 110 is configured to manage and coordinate the user's CGR experience.In some embodiments, controller 110 includes a suitable combination of software, firmware, and / or hardware. Controller 110 is described in more detail below with reference to FIG. 2. In some embodiments, controller 110 is a computing device that is local or remote to scene 105 (e.g., the physical environment). For example, controller 110 is a local server located within scene 105. In another example, controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside scene 105. In some embodiments, controller 110 is communicatively coupled to display generation component 120 (e.g., an HMD, a display, a projector, a touchscreen, etc.) via one or more wired or wireless communication channels 144 (e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is contained within the housing (e.g., physical housing) of, or shares the same physical housing or support structure as, one or more of the display generation component 120 (e.g., an HMD or a portable electronic device including a display and one or more processors), one or more of the input devices 125, one or more of the output devices 155, one or more of the sensors 190, and / or one or more of the peripheral devices 195.

[0035] In some embodiments, display generation component 120 is configured to provide a CGR experience (e.g., at least a visual component of the CGR experience) to a user. In some embodiments, display generation component 120 includes a suitable combination of software, firmware, and / or hardware. Display generation component 120 is described in more detail below with reference to FIG. 3. In some embodiments, functionality of controller 110 is provided by and / or combined with display generation component 120.

[0036] According to some embodiments, the display generation component 120 provides a CGR experience to the user while the user is virtually and / or physically present within the scene 105.

[0037] In some embodiments, the display generation component is worn on a part of the user's body (e.g., on their head, their hand, etc.). Thus, display generation component 120 includes one or more CGR displays provided for displaying CGR content. For example, in various embodiments, display generation component 120 surrounds the user's field of view. In some embodiments, display generation component 120 is a handheld device (e.g., a smartphone or tablet) configured to present CGR content, where the user holds the device with a display pointed toward the user's field of view and a camera pointed toward scene 105. In some embodiments, the handheld device is optionally located within a housing worn on the user's head. In some embodiments, the handheld device is optionally located on a support (e.g., a tripod) in front of the user. In some embodiments, display generation component 120 is a CGR chamber, housing, or room configured to present CGR content without the user wearing or holding display generation component 120. Many user interfaces described with reference to one type of hardware for displaying CGR content (e.g., a handheld device or a tripod-mounted device) may be implemented on another type of hardware for displaying CGR content (e.g., an HMD or other wearable computing device). For example, a user interface showing interactions with CGR content triggered based on interactions occurring in the space in front of a handheld or tripod-mounted device may be implemented similarly to an HMD in which the interactions occur in the space in front of the HMD and the CGR content responses are displayed via the HMD. Similarly, a user interface showing interactions with CGR content triggered based on movement of a handheld or tripod-mounted device relative to the physical environment (e.g., the scene 105 or a part of the user's body (e.g., the user's eye(s), head, or hand)) may be implemented similarly to an HMD in which the interactions are triggered by movement of the HMD relative to the physical environment (e.g., the scene 105 or a part of the user's body (e.g., the user's eye(s), head, or hand)).

[0038] While relevant features of operating environment 100 are shown in FIG. 1, those skilled in the art will understand from this disclosure that various other features have not been shown for the sake of brevity so as not to obscure more relevant aspects of the exemplary embodiments disclosed herein.

[0039] 2 is a block diagram of an example controller 110, according to some embodiments. While certain features are shown, those skilled in the art will understand from this disclosure that various other features are not shown for the sake of brevity so as not to obscure more relevant aspects of the embodiments disclosed herein. Thus, by way of non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., Universal Serial Bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), BLUETOOTH, ZIGBEE, or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these and various other components.

[0040] In some embodiments, one or more communication buses 204 include circuitry that interconnects and controls communication between system components. In some embodiments, one or more I / O devices 206 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0041] Memory 220 includes high-speed random-access memory, such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDR RAM), or other random-access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 220 optionally includes one or more storage devices located remotely from one or more processing units 202. Memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, memory 220, or its non-transitory computer-readable storage medium, stores the following programs, modules, and data structures, or a subset thereof, including optional operating system 230 and CGR experience module 240:

[0042] Operating system 230 includes instructions for handling various basic system services and for performing hardware-dependent tasks. In some embodiments, CGR experience module 240 is configured to manage and coordinate one or more CGR experiences for one or more users (e.g., a single CGR experience for one or more users, or multiple CGR experiences for respective groups of one or more users). To that end, in various embodiments, CGR experience module 240 includes a data acquisition unit 242, a tracking unit 244, an adjustment unit 246, and a data transmission unit 248.

[0043] In some embodiments, data acquisition unit 242 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least display generation component 120 of FIG. 1 and, optionally, one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To that end, in various embodiments, data acquisition unit 242 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0044] In some embodiments, tracking unit 244 is configured to map scene 105 and track the position of at least display generation component 120, and optionally one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195, relative to scene 105 of FIG. 1 . To that end, in various embodiments, tracking unit 244 includes instructions and / or logic therefor, as well as heuristics and metadata therefor. In some embodiments, tracking unit 244 includes hand tracking unit 245 and / or eye tracking unit 243. In some embodiments, hand tracking unit 245 is configured to track the position of one or more parts of a user's hand and / or the movement of one or more parts of a user's hand relative to scene 105 of FIG. 1 , relative to display generation component 120, and / or relative to a coordinate system defined relative to the user's hand. Hand tracking unit 245 is described in more detail below with respect to FIG. 4 . In some embodiments, eye tracking unit 243 is configured to track the position and movement of the user's gaze (or, more broadly, the user's eyes, face, or head) relative to scene 105 (e.g., relative to the physical environment and / or the user (e.g., the user's hands)) or relative to CGR content displayed via display generation component 120. Eye tracking unit 243 is described in more detail below with respect to FIG. 5.

[0045] In some embodiments, adjustment unit 246 is configured to manage and adjust the CGR experience presented to the user by display generation component 120, and optionally by one or more of output devices 155 and / or peripheral devices 195. To that end, in various embodiments, adjustment unit 246 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0046] In some embodiments, data transmission unit 248 is configured to transmit data (e.g., presentation data, location data, etc.) to at least display generation component 120, and optionally to one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To that end, in various embodiments, data transmission unit 248 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0047] Although the data acquisition unit 242, the tracking unit 244 (e.g., including the eye tracking unit 243 and the hand tracking unit 245), the adjustment unit 246, and the data transmission unit 248 are shown as being present on a single device (e.g., the controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 242, the tracking unit 244 (e.g., including the eye tracking unit 243 and the hand tracking unit 245), the adjustment unit 246, and the data transmission unit 248 may be located within separate computing devices.

[0048] Furthermore, Figure 2 is intended more to illustrate the functionality of various features that may be present in particular embodiments, as opposed to a structural overview of the embodiments described herein. As will be recognized by those skilled in the art, items shown separately may be combined and some items may be separated. For example, some functional modules shown separately in Figure 2 may be implemented within a single module, and various functions of a single functional block may be performed by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of specific functions and how functions are allocated among them, will vary depending on implementation and, in some embodiments, will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0049] 3 is a block diagram of an example of a display generation component 120, according to some embodiments. While certain features are shown, those skilled in the art will understand from this disclosure that, for the sake of brevity, various other features are not shown so as not to obscure more relevant aspects of the embodiments disclosed herein. To that end, by way of non-limiting example, in some embodiments, the HMD 120 includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, infrared, BLUETOOTH, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more CGR displays 312, one or more optional inward-facing and / or outward-facing image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these and various other components.

[0050] In some embodiments, the one or more communication buses 304 include circuitry that interconnects and controls communication between the system components. In some embodiments, the one or more I / O devices and sensors 306 include at least one of an inertial measurement unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0051] In some embodiments, one or more CGR displays 312 are configured to provide a CGR experience to a user. In some embodiments, one or more CGR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conduction electron-emissive element display (SED), field-emission display (FED), quantum dot light-emitting diode (QD-LED), MEMS, and / or similar display types. In some embodiments, one or more CGR displays 312 correspond to a waveguide display, such as a diffractive, reflective, polarized, holographic, etc. For example, the HMD 120 includes a single CGR display. In another example, the HMD 120 includes a CGR display for each eye of the user. In some embodiments, one or more CGR displays 312 are capable of presenting MR or VR content. In some embodiments, one or more CGR displays 312 are capable of presenting MR or VR content.

[0052] In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as eye-tracking cameras). In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's hand(s) and optionally the user's arm(s) (and may be referred to as hand-tracking cameras). In some embodiments, the one or more image sensors 314 are configured to face forward (and may be referred to as scene cameras) to acquire image data corresponding to a scene that the user would view if the HMD 120 were not present. The one or more optional image sensors 314 may include one or more RGB cameras (e.g., with a complementary metal-oxide semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and / or the like.

[0053] Memory 320 includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 320 optionally includes one or more storage devices located remotely from one or more processing units 302. Memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, memory 320, or its non-transitory computer-readable storage medium, stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 330 and a CGR presentation module 340:

[0054] The operating system 330 includes instructions for handling various basic system services and for performing hardware-dependent tasks. In some embodiments, the CGR presentation module 340 is configured to present CGR content to a user via one or more CGR displays 312. To that end, in various embodiments, the CGR presentation module 340 includes a data acquisition unit 342, a CGR presentation unit 344, a CGR map generation unit 346, and a data transmission unit 348.

[0055] In some embodiments, the data acquisition unit 342 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the controller 110 of Figure 1. To that end, in various embodiments, the data acquisition unit 342 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0056] In some embodiments, the CGR presentation unit 344 is configured to present CGR content via one or more CGR displays 312. To that end, in various embodiments, the CGR presentation unit 344 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0057] In some embodiments, the CGR map generation unit 346 is configured to generate a CGR map (e.g., a 3D map of a mixed reality scene or a map of a physical environment in which computer-generated objects can be placed to generate a computer-generated reality) based on the media content data. To that end, in various embodiments, the CGR map generation unit 346 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0058] In some embodiments, data transmission unit 348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least controller 110, and optionally to one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To that end, in various embodiments, data transmission unit 348 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0059] Although the data acquisition unit 342, the CGR presentation unit 344, the CGR map generation unit 346, and the data transmission unit 348 are shown as residing on a single device (e.g., the display generation component 120 of FIG. 1), it should be understood that in other embodiments, any combination of the data acquisition unit 342, the CGR presentation unit 344, the CGR map generation unit 346, and the data transmission unit 348 may be located within separate computing devices.

[0060] Furthermore, Figure 3 is intended more to illustrate the functionality of various features that may be present in particular implementations, as opposed to a structural overview of the embodiments described herein. As will be recognized by those skilled in the art, items shown separately can be combined and some items can be separated. For example, some functional modules shown separately in Figure 3 can be implemented within a single module, and various functions of a single functional block can be performed by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of specific functions and how functions are allocated among them, will vary from implementation to implementation and, in some embodiments, will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0061] 4 is a schematic diagram of an example embodiment of hand tracking device 140. In some embodiments, hand tracking device 140 (FIG. 1) is controlled by hand tracking unit 245 (FIG. 2) to track the position of one or more parts of a user's hand and / or the movement of one or more parts of a user's hand relative to scene 105 of FIG. 1 (e.g., relative to parts of the physical environment surrounding the user, relative to display generation component 120, or relative to parts of the user (e.g., the user's face, eyes, or head), and / or relative to the user's hand). In some embodiments, hand tracking device 140 is part of display generation component 120 (e.g., embedded in or attached to a head-mounted device). In some embodiments, hand tracking device 140 is separate from display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0062] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras) that captures three-dimensional scene information including at least the hand 406 of a human user. The image sensor 404 captures images of the hand with sufficient resolution to allow for differentiation of the fingers and their respective positions. The image sensor 404 typically captures images of other parts of the user's body, or all of the body, and can have either zoom capabilities or a dedicated sensor with high magnification to capture images of the hand at a desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with or functions as an image sensor that captures the physical environment of the scene 105. In some embodiments, the image sensor 404 is positioned relative to the user or the user's environment such that the field of view of the image sensor, or a portion thereof, is used to define an interaction space in which hand movements captured by the image sensor are processed as inputs to the controller 110.

[0063] In some embodiments, image sensor 404 outputs a sequence of frames containing 3D map data (and possibly color image data) to controller 110, which extracts high-level information from the map data. This high-level information is typically provided via an application program interface (API) to an application running on the controller, which drives display generation component 120 accordingly. For example, a user can interact with software running on controller 110 by moving hand 408 and changing hand posture.

[0064] In some embodiments, the image sensor 404 projects a spot pattern onto a scene containing the hand 406 and captures an image of the projected pattern. In some embodiments, the controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on the lateral shift of the pattern's spots. This approach is advantageous in that it does not require the user to hold or wear any type of beacon, sensor, or other marker. This provides depth coordinates of points in the scene relative to a predetermined reference plane at a specific distance from the image sensor 404. In this disclosure, the image sensor 404 is assumed to define a set of orthogonal x, y, and z axes such that the depth coordinate of a point in the scene corresponds to the z-component measured by the image sensor. Alternatively, the hand tracking device 440 can use other 3D mapping methods, such as stereoscopic imaging or time-of-flight measurements, based on single or multiple cameras or other types of sensors.

[0065] In some embodiments, the hand tracking device 140 captures and processes a time sequence of depth maps containing the user's hand while the user moves the hand (e.g., the entire hand or one or more fingers). Software running on the image sensor 404 and / or a processor in the controller 110 processes the 3D map data to extract patch descriptors of the hand in these depth maps. The software matches these descriptors to patch descriptors stored in the database 408, based on a previous training process, to estimate the pose of the hand in each frame. The pose typically includes the 3D positions of the user's hand joints and fingertips.

[0066] The software can also analyze hand and / or finger trajectories across multiple frames in a sequence to identify gestures. The pose estimation functionality described herein may be interleaved with motion tracking functionality, whereby patch-based pose estimation is performed only once every two (or more) frames, while tracking is used to discover pose changes that occur across the remaining frames. The pose, motion, and gesture information is provided to an application program running on the controller 110 via the API described above. This program can, for example, move and modify an image presented on the display generation component 120 or perform other functions in response to the pose and / or gesture information.

[0067] In some embodiments, the software may be downloaded to the controller 110 in electronic form, for example, over a network, or alternatively may be provided on a tangible, non-transitory medium, such as an optical, magnetic, or electronic memory medium. In some embodiments, the database 408 is similarly stored in memory associated with the controller 110. Alternatively, or additionally, some or all of the described functionality of the computer may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). While the controller 110 is shown in FIG. 4 as, by way of example, a separate unit from the image sensor 440, some or all of the controller's processing functions may be performed by a suitable microprocessor and software, or by dedicated circuitry within the housing of the hand tracking device 402, or otherwise associated with the image sensor 404. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television set, handheld device, or head-mounted device) or using any other suitable computerized device, such as a game console or media player. The sensing function of the image sensor 404 may likewise be integrated into a computer or other computerized device that is controlled by the sensor output.

[0068] FIG. 4 also includes a schematic diagram of a depth map 410 captured by the image sensor 404, according to some embodiments. The depth map includes a matrix of pixels having respective depth values, as described above. A pixel 412 corresponding to the hand 406 is segmented from the background and wrist in this map. The intensity of each pixel in the depth map 410 is inversely proportional to the depth value, i.e., the measured z-distance from the image sensor 404, with increasing depth resulting in darker shades. The controller 110 processes these depth values ​​to identify and segment components of the image (i.e., groups of adjacent pixels) that have characteristics of a human hand. These characteristics can include, for example, the overall size, shape, and frame-to-frame motion of the depth map sequence.

[0069] Figure 4 also schematically illustrates a hand skeleton 414 that the controller 110 ultimately extracts from the depth map 410 of the hand 406, according to some embodiments. In Figure 4, the skeleton 414 is superimposed on a hand background 416 that was segmented from the original depth map. In some embodiments, key feature points on the hand (e.g., knuckles, fingertips, center of the palm, end of the hand where it connects to the wrist, etc.), and optionally the wrist or arm connected to the hand, are identified and located on the hand skeleton 414. In some embodiments, the positions and movements of these key feature points over multiple image frames are used by the controller 110 to determine hand gestures performed by the hand or the current state of the hand, according to some embodiments.

[0070] FIG. 5 illustrates an exemplary embodiment of eye tracking device 130 (FIG. 1). In some embodiments, eye tracking device 130 is controlled by eye tracking unit 243 (FIG. 2) to track the position and movement of a user's gaze relative to scene 105 or relative to CGR content displayed via display generation component 120. In some embodiments, eye tracking device 130 is integrated with display generation component 120. For example, in some embodiments, if display generation component 120 is a head-mounted device such as a headset, helmet, goggles, or glasses, or a handheld device disposed on a wearable frame, the head-mounted device includes both components for generating CGR content for viewing by the user and components for tracking the user's gaze relative to the CGR content. In some embodiments, eye tracking device 130 is separate from display generation component 120. For example, if the display generation component is a handheld device or a CGR chamber, eye tracking device 130 is optionally a device separate from the handheld device or the CGR chamber. In some embodiments, eye tracking device 130 is a head-mounted device or part of a head-mounted device. In some embodiments, head-mounted eye tracking device 130 is optionally used in conjunction with head-mounted or non-head-mounted display generation components. In some embodiments, eye tracking device 130 is not a head-mounted device and is optionally used in conjunction with head-mounted display generation components. In some embodiments, eye tracking device 130 is not a head-mounted device and is optionally part of non-head-mounted display generation components.

[0071] In some embodiments, the display generation component 120 uses a display mechanism (e.g., left and right near-eye display panels) that displays frames including left and right images in front of the user's eyes to provide a 3D virtual view to the user. For example, a head-mounted display generation component may include left and right optical lenses (referred to herein as eyepieces) positioned between the display and the user's eyes. In some embodiments, the display generation component may include or be coupled to one or more external video cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generation component may have a transparent or translucent display that allows the user to view the physical environment directly and display virtual objects on the transparent or translucent display. In some embodiments, the display generation component projects virtual objects into the physical environment. The virtual objects are projected, for example, onto a physical surface or as a hologram, allowing an individual using the system to observe the virtual objects superimposed on the physical environment. In such cases, separate display panels and image frames for the left and right eyes may not be required.

[0072] As shown in FIG. 5 , in some embodiments, the eye tracking device 130 includes at least one eye tracking camera (e.g., an infrared (IR) or near-IR (NIR) camera) and an illumination source (e.g., an IR or NIR light source such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eyes. The eye tracking camera may be aimed at the user's eyes to receive reflected IR or NIR light from the light source directly from the eyes, or alternatively, it may be aimed at a “hot” mirror positioned between the user's eyes and a display panel that reflects the IR or NIR light from the eyes to the eye tracking camera while allowing visual light to pass through. The eye tracking device 130 optionally captures images of the user's eyes (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes the images to generate eye tracking information, and communicates the eye tracking information to the controller 110. In some embodiments, the user's eyes are tracked separately by their respective eye tracking cameras and illumination sources. In some embodiments, only one eye of the user is tracked by a corresponding eye-tracking camera and lighting source.

[0073] In some embodiments, the eye tracking device 130 is calibrated using a device-specific calibration process to determine the eye tracking device's parameters for the particular operating environment 100, such as the 3D geometric relationships and parameters of the LEDs, camera, hot mirror (if present), eyepiece, and display screen. The device-specific calibration process may be performed at a factory or another facility prior to delivery of the AR / VR equipment to the end user. The device-specific calibration process may be an automatic or manual calibration process. The user-specific calibration process may include estimation of a particular user's eye parameters, such as pupil position, central vision position, optical axis, visual axis, eye spacing, etc. According to some embodiments, once the device-specific and user-specific parameters for the eye tracking device 130 have been determined, images captured by the eye tracking camera can be processed using glint-assisted methods to determine the user's current visual axis and viewpoint relative to the display.

[0074] As shown in FIG. 5, eye tracking device 130 (e.g., 130A or 130B) includes an eye tracking system including eyepiece(s) 520, at least one eye tracking camera 540 (e.g., an infrared (IR) or near-IR (NIR) camera) positioned on the side of the user's face where eye tracking occurs, and an illumination source 530 (e.g., an IR or NIR light source such as an array or ring of NIR light emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye(s) 592. The eye tracking camera 540 may be located between the user's eye(s) 592 and the display 510 (e.g., the left or right display panel of a head-mounted display, or the display of a handheld device, a projector, etc.) and may be pointed at a mirror 550 that reflects IR or NIR light from the eye(s) 592 while transmitting visible light (e.g., as shown in the top of Figure 5), or may be pointed at the user's eye(s) 592 to receive reflected IR or NIR light from the user's eye(s) 592 (e.g., as shown in the bottom of Figure 5).

[0075] In some embodiments, controller 110 renders AR or VR frames 562 (e.g., left and right frames for left and right display panels) and provides frames 562 to display 510. Controller 110 uses gaze tracking input 542 from eye tracking camera 540 for various purposes, such as in processing frames 562 for display. Controller 110 optionally estimates the user's viewpoint on display 510 based on gaze tracking input 542 obtained from eye tracking camera 540, using a glint-assisted method or other suitable method. The viewpoint estimated from gaze tracking input 542 is optionally used to determine the user's current looking direction.

[0076] Some possible use cases of the user's current gaze direction are described below, but are not intended to be limiting. As an exemplary use case, the controller 110 can render virtual content differently based on the determined user's gaze direction. For example, the controller 110 may generate virtual content with higher resolution in a central visual area determined from the user's current gaze direction than in a peripheral area. As another example, the controller may position or move virtual content within a view based at least in part on the user's current gaze direction. As another example, the controller may display particular virtual content within a view based at least in part on the user's current gaze direction. As another exemplary use case in an AR application, the controller 110 can orient an external camera to capture the physical environment of the CGR experience and focus in the determined direction. The external camera's autofocus mechanism can then focus on an object or surface within the environment the user is currently viewing on the display 510. As another exemplary use case, eyepiece 520 may be a focusable lens, and eye-tracking information is used by the controller to adjust the focus of eyepiece 520 so that the virtual object the user is currently looking at has the proper binocular coordination to match the convergence of the user's eyes 592. Controller 110 can utilize the eye-tracking information to orient and focus eyepiece 520 so that close objects the user is looking at appear at the correct distance.

[0077] In some embodiments, the eye tracking device is part of a head-mounted device that includes a display (e.g., display 510), two eyepieces (e.g., eyepiece(s) 520), an eye tracking camera (e.g., eye tracking camera(s) 540), and a light source (e.g., light source 530 (e.g., IR LED or NIR LED)) attached to the wearable housing. The light source emits light (e.g., IR or NIR light) toward the user's eye(s) 592. In some embodiments, the light sources may be arranged in a ring or circle around each lens, as shown in FIG. 5. In some embodiments, eight light sources 530 (e.g., LEDs) are arranged around each lens 520, as an example. However, more or fewer light sources 530 may be used, and other arrangements and positions of the light sources 530 may be used.

[0078] In some embodiments, the display 510 emits light in the visible light range and not in the IR or NIR range, thereby not introducing noise into the eye-tracking system. Note that the positions and angles of the eye-tracking camera(s) 540 are given as examples and are not intended to be limiting. In some embodiments, a single eye-tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 can be used on each side of the user's face. In some embodiments, a camera 540 with a wider field of view (FOV) and a camera 540 with a narrower FOV can be used on each side of the user's face. In some embodiments, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) can be used on each side of the user's face.

[0079] Embodiments of an eye tracking system such as that shown in FIG. 5 may be used, for example, in computer-generated reality, virtual reality, and / or mixed reality applications to provide a user with a computer-generated reality, virtual reality, augmented reality, and / or augmented virtual experience.

[0080] Figure 6 illustrates a glint-assisted eye tracking pipeline, according to some embodiments. In some embodiments, the eye tracking pipeline is implemented by a glint-assisted eye tracking system (e.g., eye tracking device 130 as shown in Figures 1 and 5). The glint-assisted eye tracking system can maintain a tracking state. Initially, the tracking state is off or "no." When in the tracking state, the glint-assisted eye tracking system tracks the pupil contour and glint in the current frame using prior information from the previous frame when analyzing the current frame. When not in the tracking state, the glint-assisted eye tracking system attempts to detect the pupil and glint in the current frame, and if successful, initializes the tracking state to "yes" and continues in the tracking state to the next frame.

[0081] As shown in FIG. 6, an eye-tracking camera can capture left and right images of a user's left and right eyes. The captured images are then input into an eye-tracking pipeline for processing beginning at 610. As indicated by the arrow returning to element 600, the eye-tracking system can continue to capture images of the user's eyes at a rate of, for example, 60-120 frames per second. In some embodiments, each set of captured images may be input into the pipeline for processing. However, in some embodiments, or under some conditions, not all captured frames are processed by the pipeline.

[0082] At 610, if the tracking status is yes for the currently captured image, the method proceeds to element 640. If the tracking status is no at 610, the image is analyzed to detect the user's pupil and glint in the image, as shown at 620. If the pupil and glint are successfully detected at 630, the method proceeds to element 640. If not, the method returns to element 610 to process the next image of the user's eyes.

[0083] At 640, proceeding from element 410, the current frame is analyzed to track pupils and glints based in part on prior information from the previous frame. At 640, proceeding from element 630, a tracking state is initialized based on the detected pupils and glints in the current frame. The results of the processing at element 640 are checked to ensure that the tracking or detection results are reliable. For example, the results can be checked to determine whether a sufficient number of glints are successfully tracked or detected in the current frame to perform pupil and gaze estimation. At 650, if the results are not reliable, the tracking state is set to no and the method returns to element 610 to process the next image of the user's eyes. At 650, if the results are reliable, the method proceeds to element 670. At 670, the tracking state is set to yes (if not already yes) and pupil and glint information is passed to element 680 to estimate the user's gaze point.

[0084] 6 is intended to serve as an example of eye-tracking technology that may be used in particular implementations. As will be recognized by those skilled in the art, other eye-tracking technologies, now existing or developed in the future, may be used in place of or in combination with the glint-assisted eye-tracking technology described herein in computer system 101 to provide a user with a CGR experience according to various embodiments.

[0085] In this disclosure, various input methods are described with respect to interaction with a computer system. Where one example is provided using one input device or input method and another example is provided using a different input device or input method, it should be understood that each example may be compatible with, and optionally utilize, the input device or input method described with respect to the other example. Similarly, various output methods are described with respect to interaction with a computer system. Where one example is provided using one output device or output method and another example is provided using a different output device or output method, it should be understood that each example may be compatible with, and optionally utilize, the output device or output method described with respect to the other example. Similarly, various methods are described with respect to interaction with a virtual environment or a mixed reality environment via a computer system. Where an example is provided using interaction with a virtual environment and another example is provided using a mixed reality environment, it should be understood that each example may be compatible with, and optionally utilize, the method described with respect to the other example. Thus, the present disclosure discloses embodiments that are combinations of features of multiple examples, without exhaustively listing all features of the embodiments in the description of each exemplary embodiment. User Interface and Related Processes

[0086] We now turn our attention to embodiments of user interfaces (“UIs”) and associated processes that may be executed in a computer system, such as a portable multifunction device or a head-mounted device, that includes a display generation component, one or more input devices, and (optionally) one or more cameras.

[0087] 7A-7J show exemplary user interfaces for updating a device's display relative to a user's body, according to some embodiments. The user interfaces in these figures are used to illustrate the processes described below, including the processes of FIGS. 8A-8D . For ease of explanation, some of the embodiments are discussed with reference to operations performed on a device having touch-sensitive display system 112. However, similar operations are optionally performed on a device having display 450 and a separate touch-sensitive surface 451 in response to detecting contact on touch-sensitive surface 451 while displaying the illustrated user interface on display 450, along with a focus selector.

[0088] 7A shows a device with a display, according to some embodiments. In some embodiments, device 700 is computer system 301. In some embodiments, device 700 includes a display 712 (e.g., a touch-sensitive display) and side input buttons 702. In some embodiments, device 700 is a wearable device (e.g., a watch, a headset, etc.). Device 700 is attached to (e.g., worn by) a band 706 designed to be worn by a user.

[0089] 7B-7J illustrate exemplary user interfaces for updating the display of a device relative to a user's body, according to some embodiments. FIG. 7 illustrates a user's arm including two tattoos: a sun tattoo 716 and a star tattoo 714. The arm is shown from two different perspectives, including a first perspective view from above the arm and a second perspective view looking at the inside of the arm, to illustrate that the tattoos are located on different parts of the arm. In the following description, tattoos are used to illustrate user interface features based on adjusting the device 700 relative to the tattoo (e.g., adjusting the position of the device 700). The user does not need to have tattoos for the device 700 to provide the following user interfaces. For example, the following description applies to any body part or feature of the user (e.g., eyes, eyebrows, hair, face, freckles, etc.) that is covered by the device when worn by the user. Descriptions referring to "tattoos" may also refer to any other body part or feature of the user.

[0090] 7C shows a band 706 worn at a first position on a user's arm, attached to, or integrated with, device 700. At the first position on the user's arm, device 700 covers a portion of star tattoo 714, and display 712 of device 700 displays a graphical representation of the physical features of the user's arm covered by device 700. For example, in FIGS. 7C and 7D , displaying the graphical representation includes displaying an animated version of the star tattoo (e.g., a portion of animated star tattoo 718) that corresponds to the portion of star tattoo 714 covered by device 700. More specifically, in FIG. 7C , a graphical representation of two of the points of the star is displayed by device 700 in the approximate location where the two points of the star would appear on the user's arm (e.g., as if the user could see through device 700). In addition to animated star tattoo 718, display 712 also displays a graphical representation of the portion of the user's arm covered by device 700. The graphical representation of the user's arm is optionally animated, in which case the display 712 of the device 700 displays an animated version of the user's arm. In this manner, the display 712 displays the portion of the user's arm that is covered by the display 712 (e.g., including the portion of the star tattoo 714).

[0091] In some embodiments, band 706 is adjustable (e.g., movable) after being placed on the user's body. For example, the user moves band 706 to the left (e.g., toward the user's hand), as shown by the change from FIG. 7C to FIG. 7D . Band 706 can be adjusted in multiple ways, including being moved left, right, up (e.g., vertically, away from the arm), down (e.g., toward the arm), rotated, or any combination of these movements. As band 706 is adjusted (e.g., and device 700 is moved relative to the user's body), the displayed portion of animated star tattoo 718 and the graphical representation of the user's arm are updated to correspond to the current position of device 700. For example, as device 700 is moved to cover a larger portion of star tattoo 714, a larger portion of animated star tattoo 718 is displayed.

[0092] 7D shows the band 706 in a second position on the user's arm after it has been moved to the left (e.g., toward the hand) of the first position shown in FIG. 7C. The display 712 is updated to show the new portion of the user's arm (e.g., the user's wrist) that the device 700 covers. Furthermore, the device 700 and its display 712 now completely cover (e.g., overlap) the star tattoo 714. The display 712 displays (e.g., in real time) an animated star tattoo 718 at the location where the star tattoo 714 is relative to the device 700. In some embodiments, one or more visual effects are superimposed on (or applied to) the graphical representation (e.g., the animated star tattoo 718) to generate an animation of the graphical representation. For example, the one or more visual effects cause the graphical representation (e.g., the animated star tattoo 718) to move, increase in size, rotate, flash, fade, blur, and / or change color. In some embodiments, the animated movement is based on the body part (or body feature) covered by the device. For example, if the device covers the user's eyes and / or eyebrows, the animation may include the eyes moving, blinking, and / or changing focus or gaze direction, and / or the eyebrows moving (e.g., to show different facial expressions).

[0093] FIG. 7E shows a user rotating band 706 to change the position of device 700 on the user's body. For example, the user rotates the band so that the display covers the top of the user's wrist. In some embodiments, as the user rotates band 706, device 700 continues to update display 712 to show a graphical representation of the part of the user's body that device 700 and its display 712 currently cover. In some embodiments, as the user rotates band 706, device 700 generates an animation, such as a fade, blur, or darkening of the display over time (e.g., in an animation), as shown by the shading of display 712 in FIG. 7E. In some embodiments, the generated animation is applied to the entire display 712. In some embodiments, the generated animation is applied to a portion of the display (e.g., the portion including animated star tattoo 718). For example, device 700 blurs (or darkens) the animated star tattoo 718 on display 712.

[0094] FIG. 7F shows band 706 after being rotated to a third position, with display 712 overlaying the portion of the user's body including sun tattoo 716. Display 712 updates to display sun tattoo 720 and a graphical representation of the appearance of the portion of the user's arm underlying device 712. The graphical representation optionally includes animated sun tattoo 720. In some embodiments, animated sun tattoo 720 is a different color than sun tattoo 716. As described with reference to FIG. 7D , in some embodiments, device 700 animates the tattoo to move, increase in size, rotate, flash, and / or change color on display 712. In some embodiments, the animation of animated sun tattoo 720 is a different animation than the animation of animated star tattoo 718.

[0095] 7G shows another perspective top view of display 712 covering a portion of a user's arm including sun tattoo 716. From the top view, an animated sun tattoo 720 is displayed on display 712. FIG. 7G shows the user lifting band 706 from the user's arm (e.g., to remove the band).

[0096] FIG. 7H shows the device 700 as it is being removed from the user's body. In some embodiments, in response to lifting the band 706 from the user's arm, the display 712 shows an animation (e.g., a different animation than the animation for rotating the band described in FIG. 7E). As shown in FIG. 7H, the movement of the device 700 causes only a subset of the graphical representations 720 of the tattoos 716 to be displayed. In another example, different movements of the band 706 generate different animations (e.g., based on the direction and orientation of the movement). For example, as the device moves away from the user's body (e.g., is removed), the animation includes a blurring animation of at least a portion of the content currently displayed on the display 712. In yet another example, as the device moves away from the user's body (e.g., is removed), the device gradually obscures at least a portion of the graphical representation of a predetermined part of the user's body (e.g., with an overlay whose characteristics change as the distance between the device and a predetermined part of the user's body changes). As the device is lifted from the user's arm, the display 712 continues to update the portion of the animated sun tattoo 720 to correspond to the part of the user's body that the device 700 is currently over.

[0097] In FIG. 7I , device 700 continues to be removed from the user's body (e.g., by sliding the device toward the user's hand and pulling the device away from the arm). Display 712 continues to display a graphical representation of the appearance of the portion of the user's body corresponding to the portion of the user's body covered by the device. Device 700 continues to animate display 712 (e.g., by blurring the display), as described with reference to FIG. 7H . As seen in FIG. 7I , device 700 no longer overlies the portion of the user's arm including sun tattoo 716, and therefore display 712 no longer displays a graphical representation including animated sun tattoo 720.

[0098] 7J, device 700 (e.g., mounted on band 706) has been removed from the user's wrist. In some embodiments, following a determination that device 700 has been removed from the user's body, display 712 is updated. In some embodiments, display 712 dims (e.g., increases shading or changes color). In some embodiments, display 712 is turned off.

[0099] 8A-8D are flowcharts of an example method 800 for interacting with a three-dimensional environment using predefined input gestures, according to some embodiments. In some embodiments, method 800 is performed on a computer system (e.g., computer system 101 of FIG. 1 ), also referred to as an electronic device or wearable electronic device, that includes a display generation component (e.g., display generation component 120 of FIGS. 1 , 3, and 4 ) (e.g., a heads-up display, a display, a touchscreen, a projector, etc.) and one or more cameras (e.g., a camera (e.g., a camera facing downward at the user's hand or facing forward from the user's head (e.g., color sensors, infrared sensors, and other depth-sensing cameras)). In some embodiments, method 800 is performed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of computer system 101 (e.g., control unit 110 of FIG. 1A ). Some operations of method 800 are optionally combined and / or the order of some operations is optionally changed.

[0100] As described below, method 800 provides an intuitive way to provide a user with a computer-generated experience to make interaction with a computing system more efficient and intuitive for the user. The method improves the visual feedback provided while performing actions associated with adjusting the position of the device. For battery-operated electronic devices, making the device appear more responsive to user input conserves power and increases the time between battery charges. The improved visual feedback further improves device usability (e.g., by helping the user provide appropriate inputs when operating / interacting with the device and reducing user errors).

[0101] In some embodiments, the electronic device is configured to be worn over a predetermined portion of the user's body, such as a portion of the user's arm near or adjacent to the user's wrist, or over the user's eye. In some embodiments, the display generation component of the electronic device is located on the electronic device on a side or surface of the device opposite the side of the device closest to the predetermined portion of the user's body. For example, in FIGS. 7B-7J , the display 712 is located on a side or surface of the electronic device opposite the side of the electronic device closest to the user's arm and faces outward (away from the user's body). While the electronic device is worn over the predetermined portion of the user's body, the electronic device displays (802) a graphical representation of the appearance of a body part corresponding to the predetermined portion of the user's body via the display generation component.

[0102] In some embodiments, the graphical representation is a simulated representation or imitation of the appearance of a body part corresponding to a predetermined portion of the user's body, which may be a realistically rendered representation of the appearance of the body part or a caricatured representation of the appearance of the body part (e.g., a graphical representation of the appearance of the body part that intentionally simplifies one or more features of the appearance of the body part and / or intentionally exaggerates one or more features of the appearance of the body part). For example, as shown in FIGS. 7C-7H , a graphical representation of an outer portion of a user's arm and a tattoo on the user's arm is displayed on device 700. In some embodiments, the graphical representation is animated (e.g., by applying visual effects to the graphical representation on the display). In some embodiments, the predetermined portion of the user's body includes at least a portion of the user's face.

[0103] For example, the electronic device may be a headset or head-mounted display (HMD) (e.g., configured to be worn on a user's head), which includes a display generation component that faces outward (away from the user's body), and the predetermined portion of the user's body includes the user's eyes, eyebrows, nose, mouth, hair, freckles, eyeglasses, and / or any other features present on the user's face. For example, the display generation component of the HMD displays a graphical representation of the user's eyes (and / or other features) when the HMD is worn over the user's eyes (and / or other features). In some embodiments, the graphical representation of the user's eyes is animated to cause the graphical representation of the user's eyes to appear to blink (or wink), change size (e.g., the graphical representation of the user's eyes becomes larger or smaller), or otherwise exhibit different facial expressions (e.g., the graphical representation of the user's eyebrows rises or falls relative to the position of the graphical representation of the user's eyes).

[0104] The electronic device detects a change in the position of the electronic device relative to the predetermined portion of the user's body (804). In response to detecting the change in the position of the electronic device relative to the predetermined portion of the user's body, the electronic device modifies (806) a graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body according to the detected change in the position of the electronic device relative to the predetermined portion of the user's body. For example, as shown in FIGS. 7C-7H , as the device 700 is adjusted (e.g., repositioned) relative to the user's arm, the graphical representation on the display 712 is updated to show the portion of the user's arm and the portion of the tattoo currently covered by the device 700. In some embodiments, the electronic device detects the change in the position of the electronic device relative to the predetermined portion of the user's body.

[0105] For example, an HMD changes position relative to a user's eyes in several situations, including when a user places the HMD on the user's head, when the user removes the HMD from the user's head, and when the user adjusts the HMD while it is on the user's head, e.g., when pulling the HMD away from the user's face (and eyes), moving the HMD left or right (relative to a neutral position corresponding to the HMD placed in a predetermined position over the user's eyes), and moving the HMD up or down (or any combination of these movements). In some embodiments, following a determination that the HMD is changing position in any of these ways, a display generation component on the HMD is updated to modify a graphical representation of the user's eyes displayed on the HMD. As described below, the modification to the graphical representation of the user's eyes is based on (e.g., corresponds to) the type of change or adjustment in the position of the HMD relative to the user's eyes.

[0106] Updating the display to show a portion of the user's body that is hidden behind the electronic device provides a more intuitive way for the user and others in the surrounding environment viewing the display to know where the electronic device is located relative to the user's body, even when that portion of the user's body is covered and not visible to the user (or others in the surrounding environment). This improves the experience of interacting with the wearable device by reducing the cognitive load on the user and others in the surrounding environment to imagine or guess what is behind the electronic device. Providing improved visual feedback to the user improves device usability, increases the efficiency of the user-device interface (e.g., by helping the user provide appropriate inputs when operating / interacting with the device, thereby reducing user errors), and further reduces device power consumption and improves battery life by allowing the user to use the device more quickly and efficiently.

[0107] In some embodiments, the detected change in position of the electronic device is movement of the device away from a predetermined portion of the user's body (808). For example, FIGS. 7G-7H show device 700 as it is being removed from a user's arm. In some embodiments, movement of the device away from a predetermined portion of the user's body includes moving the electronic device away from the user's body without removing the electronic device from the user's body. In some embodiments, the detected change in position begins as movement of the device away from a predetermined portion of the user's body, and following a determination that the device is located at least a threshold distance away from the user's body, the detected change in position corresponds to removing the device from the user's body.

[0108] According to the above example of an HMD configured to be worn over a user's eyes, in some embodiments, the user moves the HMD away from the user's eyes (e.g., pulls the HMD forward, away from the user's face). In some embodiments, the user moves the HMD away from the user's eyes without removing the HMD from the user's head. For example, the user creates more space (e.g., a greater distance) between the user's face and the HMD. In some embodiments, the user pulls the HMD straight away from the user's face (e.g., without moving the HMD left, right, up, or down, relative to the user's face looking straight ahead).

[0109] Tracking the movement of the electronic device relative to the user's body provides a more accurate representation of parts of the user's body that are obscured by the electronic device. Accurately displaying a representation of parts of the user's body that are obscured by the electronic device provides an intuitive way for the user and others in the surrounding environment viewing the display to know where the electronic device is located relative to the user's body, even when the user's body, or at least said parts of the user's body, are covered and not visible to the user. This improves the experience of interacting with the wearable device by reducing the cognitive load on the user and others in the surrounding environment to imagine what is underneath the electronic device. Providing improved visual feedback to the user improves device usability, increases the efficiency of the user-device interface (e.g., by helping the user provide appropriate inputs when operating / interacting with the device, thereby reducing user errors), and further reduces device power consumption and improves battery life by allowing the user to use the device more quickly and efficiently.

[0110] In some embodiments, modifying the graphical representation according to the detected change in position of the electronic device relative to the predetermined portion of the user's body includes ceasing to display at least a portion (or all) of the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body at a predetermined distance away from the predetermined portion of the user's body (810). For example, as shown in FIG. 7J , the display 712 ceases to display the graphical representation of the user's arm when the device is moved away from the user's arm (e.g., at least a predetermined distance). For example, as described above, the device ceases to display at least a portion of the graphical representation of the appearance of the body part when the change in position corresponds to removing the device from the user's body after the device is moved at least a threshold amount (a predetermined distance away from the body).

[0111] In the above example of an HMD configured to be worn over a user's eyes, in some embodiments, the user repositions the HMD by removing it from the user's head. In some embodiments, pursuant to a determination that the HMD has been removed a predetermined distance from the user's eyes, the display generation component stops displaying the graphical representation of the user's eyes.

[0112] Ceasing to display a representation of the user's body part when the electronic device is detached provides an intuitive transition that indicates to the user and others in the surrounding environment viewing the display that the electronic device has been detached from the user's body and that the display no longer displays a representation of the user's body part when the device is detached from the user's body part. This improves the experience of interacting with the wearable device by reducing the cognitive load on the user and others in the surrounding environment to recognize the current location of the electronic device relative to the user's body. Providing improved visual feedback to the user improves device usability, increases the efficiency of the user-device interface (e.g., by helping the user provide appropriate inputs when operating / interacting with the device, reducing user errors), and further reduces device power consumption and improves battery life by allowing the user to use the device more quickly and efficiently.

[0113] In some embodiments, modifying the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body in accordance with the detected change in the position of the electronic device relative to the predetermined portion of the user's body includes maintaining a position of at least a portion of the graphical representation (of the appearance of the body part corresponding to the predetermined portion of the user's body) relative to the predetermined portion of the user's body (812). For example, while the position of the electronic device changes, the electronic device maintains the graphical representation of an anatomical feature of the user's body in the same position relative to that anatomical feature of the user's body (e.g., the graphical representation of the user's eye or arm is updated or moved to remain displayed on the display generation component in a position that covers the anatomical feature). For example, as shown in FIGS. 7C-7D , when the device 700 is adjusted to a different portion of the user's arm, the graphical representation of the portion of the user's arm that corresponds to the device's current position is displayed on the display 712. Thus, as device 700 moves to completely cover star tattoo 714, an animated version of star tattoo 718 is generated and displayed on the user's arm in a position corresponding to the position of star tattoo 714, and therefore animated version of star tattoo 718 continues to match and cover the position of star tattoo 714 as device 700 moves.

[0114] According to the above example of an HMD configured to be worn over a user's eyes, in some embodiments, as a user moves (e.g., adjusts) the HMD relative to the user's eyes, the display generation component updates to display a graphical representation of the user's eyes at a position on the display corresponding to the current position of the user's eyes (which are currently covered by the HMD). For example, as a user moves the HMD to the user's left, the graphical representation of the user's eyes does not appear to move with the HMD (e.g., to a third party viewing the display generation component). Instead, the graphical representation of the user's eyes appears to remain in a fixed position corresponding to the position in front of the user's actual eyes. Thus, even if the user moves the HMD to a different location on the user's face, the graphical representation of the user's eyes will appear to a third party viewing the display generation component in the position where the user's eyes would be found if the user's eyes were not currently covered by the HMD.

[0115] Maintaining the relative positions of features in the graphical representation of a user's body parts as the electronic device is moved to cover different parts of the user's body provides an intuitive way for the user, and others in the surrounding environment viewing the display, to know where the electronic device is located relative to the user's body, even when the user's body is covered and not visible to the user. This improves the experience of interacting with a wearable device by reducing the cognitive load on the user and others in the surrounding environment to imagine what is underneath the electronic device. Providing improved visual feedback to the user improves device usability, increases the efficiency of the user-device interface (e.g., by helping the user provide appropriate inputs when operating / interacting with the device, thereby reducing user errors), and further reduces device power consumption and improves battery life by allowing the user to use the device more quickly and efficiently.

[0116] In some embodiments, modifying the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body in accordance with the detected change in position of the electronic device relative to the predetermined portion of the user's body includes (e.g., gradually) modifying the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body to include a subset of less than all of the graphical representations of the appearance of the body part corresponding to the predetermined portion of the user's body (814). An example of such modification of the graphical representation is shown in and discussed above with respect to FIG. 7H. In another example, the graphical representation becomes more abstract (e.g., less detailed) as the electronic device changes position relative to the predetermined portion of the user's body.

[0117] In some embodiments, the graphical representation of the appearance of the predetermined part of the user's body includes graphical representations (e.g., depictions) of a plurality of anatomical features of the user's body (e.g., eyes, eyebrows, nose, mouth, tattoos). In some embodiments, modifying the displayed graphical representation to include a subset of less than all of the graphical representations of the predetermined part of the user's body includes reducing the number of the graphical representations of the plurality of anatomical features (e.g., eliminating one or more of the graphical representations of the anatomical features in the plurality of anatomical features).

[0118] In some embodiments, modifying the displayed graphical representation to include a subset of less than all of the graphical representation of the predetermined part of the user's body is performed in accordance with a determination that a change in the position of the electronic device relative to the predetermined part of the user's body is along a first direction (away from) the predetermined part of the user's body (e.g., the modifying is performed while the user is removing the electronic device). For example, the graphical representation shows a less detailed version (having fewer or less detailed features) of the predetermined part of the user's body as the electronic device is moved away from the user's body.

[0119] For example, as the user moves the HMD away from the user's eyes (e.g., pulling the HMD forward away from the user's face), the graphical representation of the user's eyes becomes less detailed. For example, when the HMD is positioned in a neutral position (e.g., properly positioned on the user's face), the graphical representation includes a graphical representation of the user's eyes and a graphical representation of the user's eyebrows (and / or a graphical representation of other features associated with the user's eyes or a graphical representation of the user's face surrounding the user's eyes). As the HMD is moved further away from the user's eyes (e.g., by the user pulling the HMD away from the user's face), the graphical representation is updated to become more abstract (e.g., by removing or obscuring one or more features within the graphical representation). For example, the graphical representation is updated to lessen the graphical representation of the user's eyes so that the modified graphical representation includes only a graphical representation of the user's eyebrows. In another embodiment, the graphical representation is updated to replace the display of the graphical representation of the user's eyes with a simplified graphical representation of the eyes (e.g., a line drawing of the eyes, a two-dimensional representation of the eyes instead of a simulated three-dimensional representation of the eyes, and / or a monotone or two-tone representation of the eyes instead of a representation of the eyes with more colors, such as 5, 10, or 256 colors).

[0120] In some embodiments, the method includes detecting a subsequent change in position of the electronic device relative to the predetermined portion of the user's body, where the subsequent change in position of the electronic device relative to the predetermined portion of the user's body is opposite to the first direction. For example, the subsequent change includes moving the electronic device closer to the user's body (e.g., placing the electronic device on the user's body). In some embodiments, the electronic device increases the number of graphical representations of anatomical features of the user's body in response to the subsequent change in position of the electronic device relative to the predetermined portion of the user's body (e.g., reversing the modification(s) of the displayed graphical representation made while the user moved the electronic device away from the predetermined portion of the user's body). For example, the graphical representation shows more details of the predetermined portion of the user's body (having more features or more detailed features) as the electronic device is moved closer to the user's body.

[0121] For example, after a user pulls the HMD away from the user's eyes, the user subsequently brings the HMD in the opposite direction (e.g., back toward the user's eyes). As the user moves the HMD closer to the user's face (and eyes), the modified graphical representation that included only the user's eyebrows (and not a graphical representation of the user's eyes) is updated to increase the level of detail present in the modified graphical representation. For example, the modified graphical representation is updated from displaying only a graphical representation of the user's eyebrows to displaying a graphical representation that includes the user's eyes and the user's eyebrows as the user moves the HMD closer to the user's face.

[0122] The progressive display of less detailed (or more detailed) representations of a given part of the user's body visually indicates whether the electronic device is moving farther away (or closer) from the user's body, providing an intuitive way for the user and others in the surrounding environment viewing the display to know how closely the electronic device is located to the user's body, even when the given part of the user's body is covered and not visible to the user. This improves the experience of interacting with a wearable device by reducing the cognitive load on the user and others in the surrounding environment to imagine what is underneath the electronic device. Providing improved visual feedback to the user improves device usability, increases the efficiency of the user-device interface (e.g., by helping the user provide appropriate inputs when operating / interacting with the device, thereby reducing user errors), and further reduces device power consumption and improves battery life by allowing the user to use the device more quickly and efficiently.

[0123] In some embodiments, modifying the graphical representation of the appearance of the body part corresponding to the predetermined part of the user's body in accordance with the detected change in position of the electronic device relative to the predetermined part of the user's body includes obscuring at least a portion of the graphical representation of the appearance of the body part corresponding to the predetermined part of the user's body (816). For example, as described above with reference to FIGS. 7G-7H , when the device 700 changes position relative to the user's arm (e.g., is moved away from the user's arm), the display 712 obscures (e.g., with an overlay) the graphical representation of the user's arm and animated sun tattoo 720 shown in FIG. 7H . Optionally, one or more visual characteristics of the obscuring (e.g., one or more of blur, brightness, saturation, contrast, etc.) change as the distance between the device and the predetermined part of the user's body changes.

[0124] In some embodiments, obscuring the displayed graphical representation is performed in accordance with a determination that a change in position of the electronic device relative to a predetermined portion of the user's body is along a second direction (e.g., different from the first direction) away from the predetermined portion of the user's body.

[0125] For example, as described above, when the user moves the HMD away from the user's face (e.g., by pulling the HMD straight away from the face without moving the HMD left, right, up, or down), the graphical representation is updated to be less detailed. In some embodiments, when the user moves the HMD in another direction, such as up (e.g., as if the HMD were positioned on the user's forehead above the user's eyes), the graphical representation is modified to display a visual effect applied to the graphical representation. In some embodiments, the modified graphical representation is an obscured version of the graphical representation (e.g., generated by fading, blurring, or vignetting at least a portion of the graphical representation). For example, by the user moving the HMD upward, the graphical representation of the user's eyes becomes blurred as the position of the HMD changes. It will be understood that statements describing generating particular visual effects according to particular directions or types of movement are merely examples, and that any of the visual effects (or combinations of visual effects) described herein may be applied to the graphical representation when the electronic device is moved in any other possible direction. Furthermore, as described in more detail below, movement of the HMD in multiple directions (e.g., movement with components in the left, up, and away directions) results in a combination of visual effects, each corresponding to a distinct direction of movement.

[0126] Displaying an obscured view of a representation of a part of a user's body to indicate that the electronic device is moving relative to the user's body provides an intuitive way for the user and others in the surrounding environment viewing the display to know that the electronic device is being adjusted relative to the user's body. This improves the experience of interacting with a wearable device by reducing the cognitive load on the user and others in the surrounding environment to know that the electronic device is being adjusted. Providing improved visual feedback to the user improves device usability, increases the efficiency of the user-device interface (e.g., by helping the user provide appropriate inputs when operating / interacting with the device, reducing user errors), and further reduces device power consumption and improves battery life by allowing the user to use the device more quickly and efficiently.

[0127] In some embodiments, obscuring the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body includes fading out at least a portion of the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body 818. For example, in some embodiments, the obscuring includes applying a vignette effect to the graphical representation by fading out peripheral portions of the graphical representation (e.g., by reducing brightness and / or saturation).

[0128] Continuing with the above example, in some embodiments, when the user moves the HMD in another direction, such as upwards (e.g., as if the HMD were positioned on the user's forehead above the user's eyes), the graphical representation is modified to apply a fade-out visual effect to at least a portion of the graphical representation. For example, as the position of the HMD changes by the user moving the HMD upwards, the graphical representation of the user's eyes (e.g., and / or eyebrows) is faded.

[0129] Displaying a faded view of a representation of a part of the user's body to indicate that the electronic device is moving relative to the user's body provides an intuitive way for the user and others in the surrounding environment viewing the display to know that the position of the electronic device is being adjusted relative to the user's body or a predetermined part of the user's body. This improves the experience of interacting with a wearable device by reducing the cognitive load on the user and others in the surrounding environment to know that the electronic device is being adjusted. Providing improved visual feedback to the user improves device usability, increases the efficiency of the user-device interface (e.g., by helping the user provide appropriate inputs when operating / interacting with the device, reducing user errors), and further reduces device power consumption and improves battery life by allowing the user to use the device more quickly and efficiently.

[0130] In some embodiments, obscuring the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body includes blurring at least a portion of the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body (820).

[0131] Continuing with the above example, in some embodiments, when the user moves the HMD in another direction, such as to the left, the graphical representation is modified to apply a blurring effect to at least a portion of the graphical representation. For example, as the position of the HMD changes by the user moving the HMD to the left, the graphical representation of the user's eyes (e.g., and / or eyebrows) is blurred.

[0132] Displaying a blurred view of a representation of a user's body part to indicate that the electronic device is moving relative to the user's body or a predetermined part of the user's body provides an intuitive way for the user and others in the surrounding environment viewing the display to know that the electronic device is being adjusted relative to the user's body. This improves the experience of interacting with a wearable device by reducing the cognitive load on the user and others in the surrounding environment to know that the electronic device is being adjusted. Providing improved visual feedback to the user improves device usability, increases the efficiency of the user-device interface (e.g., by helping the user provide appropriate inputs when operating / interacting with the device, reducing user errors), and further reduces device power consumption and improves battery life by allowing the user to use the device more quickly and efficiently.

[0133] In some embodiments, obscuring the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body includes vignetting the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body (822).

[0134] Continuing with the above example, as the user moves the HMD to the right, the graphical representation is modified to apply a vignette effect to at least a portion of the graphical representation. For example, as the position of the HMD changes by the user moving the HMD to the right, the graphical representation of the user's eyes (e.g., and / or eyebrows) is displayed with a vignette effect such that the center of the graphical representation of the user's eyes appears to be in focus, while the outer portions of the graphical representation of the user's eyes appear blurred, faded, and / or less intense (in color).

[0135] Displaying a vignette effect overlaid on a graphical representation of a predetermined part of a user's body to indicate that the electronic device is moving relative to the user's body provides an intuitive way for the user and others in the surrounding environment viewing the display to know that the electronic device is being adjusted relative to the user's body. This improves the experience of interacting with a wearable device by reducing the cognitive load on the user and others in the surrounding environment to know that the electronic device is being adjusted. Providing improved visual feedback to the user improves device operability, increases the efficiency of the user-device interface (e.g., by helping the user provide appropriate inputs when operating / interacting with the device, thereby reducing user errors), and further reduces device power consumption and improves battery life by allowing the user to use the device more quickly and efficiently.

[0136] In some embodiments, the electronic device displays (824) a graphical representation (e.g., as an animated graphical representation) of the appearance of a body part corresponding to the predetermined part of the user's body, before the user puts on the electronic device, and while the user is wearing the electronic device. In some embodiments, the display of the graphical representation is a modified version of the graphical representation (e.g., by applying visual effects to the graphical representation). For example, the graphical representation appears to be dimmed, blurred, or moving (e.g., animated) as the device moves. In some embodiments, the electronic device determines that the user is putting on the electronic device according to a determination that the electronic device is within a predetermined distance from the predetermined part of the user's body, as described below with reference to step 826.

[0137] In some embodiments, when a user puts on the HMD (e.g., in a predetermined position on the user's head (e.g., a use position, an operation position, or an appropriate position) designated as a position where representations of physical features are displayed with distinct appearances, and optionally where the HMD functions as an AR or VR viewing device for viewing an AR or VR environment), a graphical representation of the user's eyes appears on the display generation component of the HMD. For example, the display generation component of the HMD displays a modified version of the graphical representation of the user's eyes (e.g., before the HMD reaches a predetermined position on the user's head). For example, an obscured version (e.g., a darkened version) of the graphical representation of the user's eyes is displayed to indicate that the HMD is still being adjusted (e.g., being moved into position). In some embodiments, an animated version of the graphical representation of the user's eyes is displayed, such as displaying the graphical representation of the user's eyes as if the eyes are moving (e.g., looking around) as the HMD changes position while the user is putting it on the user's head.

[0138] Once the electronic device is in place on the user's body, displaying a representation of the part of the user's body currently obscured by the electronic device provides an intuitive transition to the user and others in the surrounding environment viewing the display that the electronic device is being placed on the user's body and that the electronic device will continue to update the display to show the part of the user's body that is covered by the electronic device. This improves the experience of interacting with the wearable device by reducing the cognitive load on the user and others in the surrounding environment to imagine what is underneath the electronic device. Providing improved visual feedback to the user improves device usability, increases the efficiency of the user-device interface (e.g., by helping the user provide appropriate inputs when operating / interacting with the device, reducing user errors), and further reduces device power consumption and improves battery life by allowing the user to use the device more quickly and efficiently.

[0139] In some embodiments, the electronic device begins displaying (826) a graphical representation of the appearance of a body part corresponding to the predetermined part of the user's body when the electronic device is a predetermined distance away from the predetermined part of the user's body. For example, the electronic device does not display the graphical representation of the appearance of a body part corresponding to the predetermined part of the user's body until the electronic device is in close proximity (e.g., within 3 inches, or within 6 inches) to the predetermined part of the user's body. In some embodiments, the electronic device uses one or more sensors (e.g., camera(s) 311 and / or orientation sensor(s) 312 of FIGS. 3B-3C ) to determine its distance from the predetermined part of the user's body.

[0140] In some embodiments, when a user puts on the HMD (e.g., at a predetermined position on the user's head designated as a position where representations of physical features are displayed with distinct appearances, and optionally where the HMD functions as an AR or VR viewing device for viewing an AR or VR environment), the graphical representation of the user's eyes appears on the display generation component of the HMD only after the HMD is within a predetermined distance of the predetermined position on the user's head (e.g., above the user's eyes). For example, the display generation component does not display the graphical representation of the user's eyes when the HMD is lifted by the user. As the user continues to move the HMD toward the user's eyes, pursuant to a determination that the HMD is sufficiently close (e.g., within a predetermined distance) to the user's eyes (e.g., at a predetermined position), the display generation component begins to display the graphical representation of the user's eyes (or a modified version of the graphical representation of the user's eyes, such as the obscured version described above).

[0141] When the electronic device gets close enough to the user's body, starting to display a graphical representation of the part of the user's body that is obscured by the electronic device provides an intuitive transition to the user and others in the surrounding environment viewing the display that the electronic device is close to the user's body and that the display will continue to update to show the part of the user's body that is obscured by the electronic device. This improves the experience of interacting with the wearable device by reducing the cognitive load on the user and others in the surrounding environment to imagine what is underneath the electronic device. Providing improved visual feedback to the user improves device usability, increases the efficiency of the user-device interface (e.g., by helping the user provide appropriate inputs when operating / interacting with the device, reducing user errors), and further reduces device power consumption and improves battery life by allowing the user to use the device more quickly and efficiently.

[0142] In some embodiments, the electronic device detects (828) a second change in the position of the electronic device relative to the predetermined portion of the user's body, the second change in the position of the electronic device being toward the predetermined portion of the user's body. In some embodiments, the electronic device, in response to detecting the second change in the position of the electronic device relative to the predetermined portion of the user's body, modifies the graphical representation of the appearance of the body portion corresponding to the predetermined portion of the user's body according to the second change in the position of the electronic device relative to the predetermined portion of the user's body, gradually, over time, or according to the progression of the change in position toward the predetermined portion of the user's body. In some embodiments, the graphical representation is first displayed in an initial state pursuant to a determination that the electronic device is within a predetermined distance from the portion of the user's body. In some embodiments, the graphical representation is not displayed before the predetermined distance is reached. For example, the display may display another user interface, be blanked / dimmed, or turned off before displaying the graphical representation. When the user wears the electronic device, the electronic device gradually transitions to displaying a final state graphical representation (e.g., a graphical representation of a predetermined part of the user's body) by blurring, fading in, reducing vignettes, or reducing the level of abstraction of the graphical representation as the user wears the electronic device.

[0143] In some embodiments, when a user puts on the HMD (e.g., at a predetermined position on the user's head designated as a position where representations of physical features are displayed with distinct appearances, and optionally where the HMD functions as an AR or VR viewing device for viewing an AR or VR environment), a graphical representation of the user's eyes gradually appears on the display generation component. For example, one or more visual effects are applied to the graphical representation, such as fading in the graphical representation, so that it becomes clearer (e.g., more focused) according to the distance the HMD is moved. For example, the closer the HMD is moved to the predetermined position, the more focused the graphical representation becomes. In this manner, the graphical representation of the user's eyes appears modified (e.g., blurred, darkened, or otherwise obscured) until the HMD is placed in position, at which point the graphical representation of the user's eyes appears unmodified (e.g., at a neutral position where an unmodified graphical representation of the user's eyes is displayed at a predetermined brightness and a predetermined amount of detail (and focus)).

[0144] Gradually updating the display of a representation of a portion of the user's body to show more detail in the representation as the electronic device changes position relative to the user's body provides an intuitive transitional effect that indicates the electronic device is moving closer to the user's body, allowing the user and others in the surrounding environment viewing the display to know where the electronic device is located relative to the user's body even as the device is adjusted. This improves the experience of interacting with a wearable device by reducing the cognitive load on the user and others in the surrounding environment to imagine what is underneath the electronic device. Providing improved visual feedback to the user improves device usability, increases the efficiency of the user-device interface (e.g., by helping the user provide appropriate inputs when operating / interacting with the device, reducing user errors), and further reduces device power consumption and improves battery life by allowing users to use the device more quickly and efficiently.

[0145] In some embodiments, modifying the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body in accordance with a second change in the position of the electronic device relative to the predetermined portion of the user's body includes (e.g., gradually) modifying the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body to include a greater number of graphical representations of anatomical features as the electronic device moves toward the predetermined portion of the user's body (830). For example, the graphical representation becomes less abstract (e.g., more detailed) as the device is moved closer to the predetermined portion of the user's body.

[0146] For example, as the user moves the HMD toward the user's eyes (e.g., moving the HMD inward and closer to the user's face), the graphical representation of the user's eyes gradually includes more detail (e.g., as the HMD moves closer to the user's eyes). For example, the graphical representation displays additional features in addition to the graphical representation of the user's eyes, such as a graphical representation of the user's eyebrows (and / or other features associated with the user's eyes or a graphical representation of the user's face surrounding the user's eyes). In some embodiments, displaying a more detailed graphical representation includes displaying a focused (not blurred or faded) graphical representation. In some embodiments, the graphical representation is modified to become more detailed in proportion to the amount of movement (e.g., distance) of the HMD relative to the user's eyes. For example, as the HMD is moved closer to the user's eyes, the graphical representation becomes gradually less blurred.

[0147] Gradually displaying a more detailed view of a representation of the user's body, showing more features on the display as the device approaches the user's body, provides an intuitive way for the user and others in the surrounding environment viewing the display to know that the electronic device is approaching the user's body, even when the user's body is covered and not visible to the user. This improves the experience of interacting with a wearable device by reducing the cognitive load on the user and others in the surrounding environment to imagine what is underneath the electronic device. Providing improved visual feedback to the user improves device usability, increases the efficiency of the user-device interface (e.g., by helping the user provide appropriate inputs when operating / interacting with the device, reducing user errors), and further reduces device power consumption and improves battery life by allowing the user to use the device more quickly and efficiently.

[0148] In some embodiments, modifying the graphical representation of the appearance of the body part corresponding to the predetermined part of the user's body in accordance with a second change in the position of the electronic device relative to the predetermined part of the user's body includes fading in at least a portion of the graphical representation of the appearance of the body part corresponding to the predetermined part of the user's body (832).

[0149] Continuing with the above example, in some embodiments, when the user moves the HMD in another direction, such as downward (e.g., as if the HMD is being moved from the top of the user's head / forehead and pulled down over the user's eyes) or towards the user's eyes (e.g., inward, closer to the user's face), the graphical representation is modified to apply a fade-in visual effect to at least a portion of the graphical representation. For example, as the position of the HMD changes by the user moving the HMD downward and / or inward, the graphical representation of the user's eyes (e.g., and / or eyebrows) fades in (e.g., changes from a faded state to a non-faded state).

[0150] Providing a visual effect to fade in view of a representation of a user's body part provides an intuitive way for the user and others in the surrounding environment viewing the display to know that the electronic device is being adjusted relative to the user's body. This improves the experience of interacting with a wearable device by reducing the cognitive load on the user and others in the surrounding environment to know that the electronic device is being adjusted. Providing improved visual feedback to the user improves device usability, increases the efficiency of the user-device interface (e.g., by helping the user provide appropriate inputs when operating / interacting with the device, reducing user errors), and further reduces device power consumption and improves battery life by allowing the user to use the device more quickly and efficiently.

[0151] In some embodiments, modifying the graphical representation of the appearance of the body part corresponding to the predetermined part of the user's body in accordance with a second change in the position of the electronic device relative to the predetermined part of the user's body includes reducing blurring of at least a portion of the graphical representation of the appearance of the body part corresponding to the predetermined part of the user's body (834).

[0152] Continuing with the above example, in some embodiments, when the user moves the HMD in another direction (toward the user's eyes), such as to the right (e.g., as if the HMD were positioned to the left of the user's eyes and moved to the right and worn over the user's eyes), the graphical representation is modified to reduce the blurry visual effect for at least a portion of the graphical representation. For example, as the position of the HMD changes by the user moving the HMD to the right (and toward the user's eyes), the graphical representation of the user's eyes (e.g., and / or eyebrows) becomes less blurry (e.g., more focused).

[0153] Reducing the blur effect displayed over representations of a user's body parts provides an intuitive way for the user and others in the surrounding environment viewing the display to know that the electronic device is adjusted to the user's body. This improves the experience of interacting with the wearable device by reducing the cognitive load on the user and others in the surrounding environment to know that the electronic device is adjusted. Providing improved visual feedback to the user improves device usability, increases the efficiency of the user-device interface (e.g., by helping the user provide appropriate inputs when operating / interacting with the device, reducing user errors), and further reduces device power consumption and improves battery life by allowing the user to use the device more quickly and efficiently.

[0154] In some embodiments, modifying the graphical representation of the appearance of the body part corresponding to the predetermined part of the user's body in accordance with a second change in the position of the electronic device relative to the predetermined part of the user's body includes reducing a vignetting effect in the graphical representation of the appearance of the body part corresponding to the predetermined part of the user's body (836).

[0155] Continuing with the above example, in some embodiments, when the user moves the HMD in another direction, such as to the left (e.g., as if the HMD were positioned to the right of the user's eyes and moved to the left so that it was worn over the user's eyes), the graphical representation is modified to reduce the vignetting effect for at least a portion of the graphical representation. For example, when the position of the HMD changes by the user moving the HMD to the right (and toward the user's eyes), the graphical representation of the user's eyes (e.g., and / or eyebrows) is displayed with a reduced vignetting effect (e.g., a vignetting / blur applied to an outer portion of the display of the graphical representation is removed so that the entire graphical representation appears in focus).

[0156] Reducing the vignetting effect displayed over representations of the user's body parts provides an intuitive way for the user and others in the surrounding environment viewing the display to know that the electronic device is adjusted relative to the user's body. This improves the experience of interacting with the wearable device by reducing the cognitive load on the user and others in the surrounding environment to know that the electronic device is adjusted. Providing improved visual feedback to the user improves device usability, increases the efficiency of the user-device interface (e.g., by helping the user provide appropriate inputs when operating / interacting with the device, reducing user errors), and further reduces device power consumption and improves battery life by allowing the user to use the device more quickly and efficiently.

[0157] In the above examples, the movement of the HMD is defined by the direction of movement (up, down, left, right) relative to the user's eyes. For example, in the example associated with steps 816-822, the direction of movement associated with a particular effect is generated from movement in the opposite direction to the example associated with steps 830-836 for that particular effect. This is because when the HMD is moved to the left, away from the user's eyes, a more obscured graphic representation is displayed, while movement of the HMD to the right, toward the user's eyes (and back to the HMD's neutral position), generates a less obscured graphic representation. Thus, an obscured graphic representation becomes more obscured as the HMD moves away from the user's eyes (whether in the up, down, left, or right direction, as described in steps 816-822), and the obscured graphic representation becomes less obscured (e.g., clearer and more focused) as the HMD moves toward the user's eyes (e.g., as described in steps 830-836). The graphical representation of the user's eyes is considered unmodified (e.g., sharp and in focus) when the HMD is worn in place over the user's eyes. The graphical representation of the user's eyes is obscured using any visual effect or combination of visual effects when the HMD is moved away from its neutral position.

[0158] In some embodiments, the predetermined portion of the user's body includes the user's eyes (e.g., both eyes) 838. As described in the examples above, in some embodiments, the electronic device is a headset or HMD configured to be worn over the user's eyes, and the electronic device generates a graphical representation of the user's eyes for display on a display generation component located on a side or surface of the electronic device opposite the side of the device that is closest to the predetermined portion of the user's body (e.g., facing away from the body).

[0159] Displaying a representation of a user's eyes when the user's eyes are covered by an electronic device provides a more intuitive interaction between others viewing the display without being able to see the user's eyes. By displaying a graphic representation of the user's eyes to others, interaction between others and the user is improved by allowing others to feel more engaged with the user. Enabling others to see a graphic representation of the user's eyes makes it easier for others to interact with a user wearing the device. Providing improved visual feedback to the user and others improves device usability, increases the efficiency of the user-device interface (e.g., by assisting the user in providing appropriate inputs when operating / interacting with the device, thereby reducing user errors), and further reduces device power consumption and improves battery life by allowing users to use the device more quickly and efficiently.

[0160] In some embodiments, the detected change in position of the electronic device relative to the predetermined portion of the user's body is a first change in position of the electronic device relative to the predetermined portion of the user's body (840). In some embodiments, the electronic device detects a second change in position of the electronic device relative to the predetermined portion of the user's body, the second change in position of the electronic device relative to the predetermined portion of the user's body being opposite to the first change. In some embodiments, the electronic device, in response to the second change in position of the electronic device relative to the predetermined portion of the user's body, reverses (e.g., gradually reverses) the modification of the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body over a period of time or according to the progression of the second change in position away from the predetermined portion of the user's body. For example, the first change in position includes removing the electronic device from the user's body (e.g., moving the electronic device away from the user's body) and the second change in position includes putting the electronic device on the user's body (e.g., moving the electronic device toward the user's body). In some embodiments, the visual effect applied to the graphical representation of the predetermined portion of the user's body is reversed between the two (opposite) changes in position. For example, as the device is removed from the user's body, the modification may include gradually abstracting (providing less detail) the graphical representation of the user's body, and then, once the device is placed back on the user's body, the modification may include gradually adding more detail to the graphical representation of the user's body.

[0161] In some embodiments, a gesture (or series of gestures) for adjusting the position of the electronic device includes multiple components, including one or more components including moving the electronic device toward the user's body and one or more components including moving the electronic device away from the user's body. As the multiple components of the gesture alternate between types of movement, the modifications to the graphical representation are smoothly (e.g., flowing). For example, the electronic device smoothly transitions between increasing modifications (e.g., levels of abstraction) of the graphical representation of the body part's appearance (as the electronic device is moved away from the predetermined part of the body) and decreasing modifications of the graphical representation of the body part's appearance (as the device is moved toward the predetermined part of the user's body) as the position of the electronic device moves.

[0162] The multiple components of a gesture for adjusting the position of the electronic device may include, for example, two or more of an adjustment to the vertical alignment of the electronic device relative to a predetermined portion of the user's body, an adjustment to the horizontal alignment of the electronic device relative to a predetermined portion of the user's body, an adjustment to the distance between the electronic device and a predetermined portion of the user's body, and an adjustment to the orientation of the electronic device relative to the predetermined portion of the user's body (e.g., tilt relative to a vertical axis and / or a horizontal axis). Further, in some embodiments, for each separate component of the multiple components, the corresponding modification to the graphical representation is smoothly transitioned as that component of the gesture changes over time. Thus, different simultaneous components of the gesture may result in simultaneous transitions in different types of modifications to the graphical representation, such as, without limitation, fading, blurring, abstraction, or vignetting.

[0163] For example, after a user moves the HMD in a first direction relative to the user's eyes, the user subsequently moves the HMD in the opposite direction. For example, the user repositions the HMD by moving it upward (e.g., above the user's forehead), which obscures (e.g., blurs, fades, vignettes, etc.) the graphical representation of the user's eyes; after the user repositions the HMD upward, the user moves the HMD downward (e.g., back to a position above the user's eyes), which undoes the modification to the graphical representation, thereby modifying the graphical representation of the user's eyes in the opposite direction (e.g., to become less obscured). In this manner, the graphical representation is dynamically modified such that as the HMD is repositioned away from the user's eyes, the graphical representation becomes more obscured using one or more visual effects, and as the HMD is repositioned in position above the user's eyes, the graphical representation returns to its unmodified state (e.g., in the unmodified state, the graphical representation is an unobscured representation of the user's eyes).

[0164] Generating different corrections to a representation of the user's body, such that the corrections appear to increase or decrease based on the direction (or type) of movement of the electronic device, creates an intuitive way for the user to recognize how the device is moving based on the displayed corrections. Providing improved visual feedback to the user improves device operability, increases the efficiency of the user-device interface (e.g., by helping the user provide appropriate inputs when operating / interacting with the device, thereby reducing user errors), and further reduces device power consumption and improves battery life by allowing the user to use the device more quickly and efficiently.

[0165] In some embodiments, the electronic device modifies the graphical representation in a first manner in accordance with a determination that a change in the position of the electronic device relative to the predetermined portion of the user's body is along a first direction (842). For example, the change in the position of the electronic device includes moving the device away from the predetermined portion of the user's body (e.g., shifting the device away from the user's eyes), and the modification includes applying a visual effect to the graphical representation, such as blurring, applying a vignette, and / or changing the position (z-height) of the predetermined portion of the user's body (e.g., the user's eyes) on the display.

[0166] In some embodiments, the electronic device modifies the graphical representation in a second manner different from the first manner in accordance with a determination that a change in the position of the electronic device relative to the predetermined portion of the user's body is along a second direction different from (e.g., not parallel to) the first direction. In some embodiments, the second direction is defined along a third axis different from the first direction (along the first axis). In this manner, the visual effect applied to the graphical representation depends on how the electronic device is moved. For example, tilting the electronic device generates a first visual effect, moving the electronic device toward (or away from) the user's body generates a second visual effect (different from the first effect), shifting the electronic device up or down (e.g., along the y-axis) generates a third visual effect (different from the first and second visual effects), and removing the electronic device from the predetermined portion of the user's body generates a fourth visual effect.

[0167] For example, when a user moves the HMD by tilting it (e.g., moving the left side of the HMD up while moving the right side of the HMD down), the graphical representation is modified using a first visual effect, such as fading the graphical representation of the user's eyes. When a user pulls the HMD away from the user's eyes (e.g., forward), the graphical representation is modified using a second visual effect different from the first visual effect, such as blurring the graphical representation of the user's eyes. When a user moves the HMD up or down (e.g., to position the HMD on the user's forehead or nose), the graphical representation is modified using a third visual effect different from the first and second visual effects, such as reducing the number of features displayed in the graphical representation (e.g., showing a graphical representation of the user's eyebrows without showing a graphical representation of the user's eyes). When a user removes the HMD from the user's head, the graphical representation is modified using a fourth visual effect, such as (e.g., gradually) dimming the graphical representation until the graphical representation is no longer visible. The association of which visual effect is associated with a particular position change is for illustrative purposes only, and it will be understood that a first position change (e.g., in a first direction) may be associated with any one or more visual effects described herein.

[0168] Applying different visual effects to a representation of a user's body based on how the position of an electronic device is changing, such that each visual effect reflects a different adjustment to the position of the electronic device, creates an intuitive way for a user to recognize how the device is moving based on the visual effects displayed. Providing improved visual feedback to a user improves device usability, increases the efficiency of the user-device interface (e.g., by helping the user provide appropriate inputs when operating / interacting with the device, thereby reducing user errors), and further reduces device power consumption and improves battery life by allowing a user to use the device more quickly and efficiently.

[0169] In some embodiments, the electronic device, in accordance with determining that a change in position of the electronic device relative to a predetermined portion of the user's body has a component along the first direction and a component along the second direction, modifies the graphical representation in a first manner based on the component along the first direction and modifies the graphical representation in a second manner based on the component along the second direction (844). In some embodiments, the modification in the first manner and the modification in the second manner occur gradually and simultaneously (e.g., both modifications are applied simultaneously), as described above. For example, when a user removes an electronic device (e.g., a headset) from the user's head, there is a change in position in at least two directions: away from the user's body and down over the user's body. In this example, both visual effects are displayed when the user removes the electronic device, e.g., both shifting the user's eyes (to reflect the change in position down over the user's body) and blurring the user's eyes (to reflect the change in position away from the user's body) are displayed as the headset is removed from the user's head.

[0170] In some embodiments, the user moves the HMD in multiple directions simultaneously (e.g., within the same motion). In response to the user's movement in multiple directions, visual effects associated with each direction of movement are simultaneously applied to the graphical representation of the user's eyes. For example, if the user pulls the HMD forward away from the user's eyes and moves the HMD upward away from the user's eyes, the graphical representation of the user's eyes is blurred (e.g., proportional to the amount of movement as the HMD is pulled away) and the number of features displayed within the graphical representation is reduced (e.g., proportional to the amount of movement of the HMD upward). In some embodiments, other combinations of movements that result in changes in the position of the HMD (e.g., along multiple axes) generate corresponding combinations of visual effects that are simultaneously applied to the graphical representation of the user's eyes.

[0171] Combining different visual effects on a representation of the user's body based on how the position of the electronic device is changing, such that the combination of visual effects simultaneously reflects different types of adjustments to the position of the electronic device, creates an intuitive way for the user to recognize how the device is moving based on the combination of visual effects displayed. Providing improved visual feedback to the user improves device usability, increases the efficiency of the user-device interface (e.g., by helping the user provide appropriate inputs when operating / interacting with the device, thereby reducing user errors), and further reduces device power consumption and improves battery life by allowing the user to use the device more quickly and efficiently.

[0172] In some embodiments, the electronic device modifies the graphical representation in a third manner (846) in accordance with determining that the change in position of the electronic device relative to the predetermined portion of the user's body is a rotation of the electronic device relative to the predetermined portion of the user's body. In some embodiments, detecting a rotation of the electronic device relative to the predetermined portion of the user's body includes detecting an adjustment of the position of the electronic device relative to the user's body. For example, in FIG. 7E , when the device 700 is rotated to wrap the band around the user's arm, the display 712 modifies the animated graphical representation using a modification (a third manner) that is different from the modification displayed when the device 700 is moved toward or away from the user's wrist (e.g., as shown in FIGS. 7C-7D ) and different from the modification displayed when the device 700 is moved away from the arm (e.g., as shown in FIGS. 7G-7H ). In some embodiments, different modifications (e.g., visual effects) are displayed when the change in position is along different axes.

[0173] In the examples below, the axes are defined as follows (relative to the user's eyes as a given part of the user's body): the x-axis refers to forward or backward movement (e.g., away from or toward the eye), the y-axis refers to upward or downward movement (e.g., above or below the eye), and the z-axis refers to left and right movement. For example, tilting the device (e.g., moving up or down relative to the y-axis) causes a different visual effect than pivoting (rotating) the electronic device (e.g., movement along the z-axis) and a different visual effect than pushing or pulling movement (e.g., moving forward or backward along the x-axis). In some embodiments, tilting the electronic device corresponds to (e.g., causes the electronic device to generate a visual effect) that turns off or dims the display and shows eyebrows instead of eyes. In some embodiments, wrapping the electronic device corresponds to (e.g., causes the electronic device to generate a visual effect) that shifts the representation of the user's body parts on the display so that the representation matches the position of the user's body parts (e.g., the position of a graphical representation of the user's eyes is moved to match the user's actual eyes). For example, when the user moves the HMD by tilting it (e.g., moving the left side of the HMD up while moving the right side of the HMD down), the graphical representation of the user's eyes becomes dimmer (or darker). When the user rotates the HMD (e.g., by moving the HMD right or left along the z-axis), the graphical representation is modified by changing the location of the graphical representation of the user's eyes on the display. For example, the graphical representation of the user's eyes is moved on the display such that the position of the graphical representation of the user's eyes remains in a position corresponding to the user's actual eyes that are covered by the display.

[0174] Distinguishing between multiple visual effects based on the type (or direction) of movement that causes a change in the position of the electronic device creates an intuitive way for a user to recognize how the device is moving based on which visual effect is displayed. Providing improved visual feedback to the user improves device usability, increases the efficiency of the user-device interface (e.g., by helping the user provide appropriate inputs when operating / interacting with the device, thereby reducing user errors), and further reduces device power consumption and improves battery life by allowing the user to use the device more quickly and efficiently.

[0175] In some embodiments, modifying the graphical representation in a first manner includes displaying, via the display generation component, an overlay that obscures at least a portion of the graphical representation of the appearance of the body part corresponding to the predetermined part of the user's body. For example, while the electronic device is being repositioned (e.g., as the user adjusts the position of the electronic device), displaying the overlay includes fluidly showing, such as by blurring, fading, or dimming, the graphical representation of the appearance of the body part to indicate that the user is adjusting the position of the electronic device. In some embodiments, the overlay includes showing text or pictograms over at least a portion of the graphical representation of the appearance of the body part corresponding to the predetermined part of the user's body.

[0176] For example, in some embodiments, in response to a user changing the position of the HMD (e.g., adjusting the HMD on the user's head), an overlay is displayed over the graphical representation of the user's eyes to indicate (e.g., to a third party viewing the display) that the user is adjusting the HMD. In some embodiments, the graphical representation of the user's eyes is animated to make the user's eyes appear to be closed.

[0177] In some embodiments, the electronic device detects a trigger (e.g., an input) at the electronic device (e.g., with or without detecting a change in the position of the electronic device relative to a predetermined part of the user's body). In some embodiments, the trigger includes an input selected from the group consisting of an input detected in front of a display generating component of the electronic device (e.g., a hand detected in front of a face (e.g., in front of the electronic device while the electronic device is on the user's face)), an input selecting a physical control (e.g., a button) of the electronic device, and an input detected at an edge of the electronic device.

[0178] In some embodiments, in response to detecting the trigger, the electronic device modifies the graphical representation of the appearance of the predetermined part of the user's body in accordance with the detected trigger, including displaying, via the display generation component, an overlay that obscures at least a portion of the graphical representation of the appearance of the body part corresponding to the predetermined part of the user's body. For example, the trigger is a trigger to pause a current session, and in response to the trigger, the electronic device displays an overlay over at least a portion of the graphical representation displayed by the display generation component of the electronic device to indicate that the current session is paused.

[0179] For example, in response to a user placing their hands in front of an HMD worn on the user's head, a graphical representation of the user's eyes is obscured on the display (e.g., with an overlay placed over the graphical representation of the user's eyes). In some embodiments, the obscured graphical representation of the user's eyes indicates to a third party viewing the display generating component that the user is not currently engaged with the third party (e.g., the user's session with the third party is paused).

[0180] In some embodiments, the graphical representation of the appearance of a body part corresponding to a predetermined part of the user's body moves as the corresponding part of the user's body moves when the device is worn on the predetermined part of the user. For example, the graphical representation of an eye moves to look around and / or open and close based on the movement and opening and closing of the user's eyes as detected by a camera or other biometric sensor of or in communication with the electronic device.

[0181] 8A-8D are merely examples, and are not intended to represent the only order in which the operations may be performed. Those skilled in the art will recognize various ways to reorder the operations described herein.

[0182] The foregoing has been described with reference to specific embodiments for purposes of explanation. However, the exemplary discussion above is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. These embodiments were chosen and described in order to best explain the principles of the invention and its practical application, and thereby enable others skilled in the art to best utilize the invention and the various described embodiments with various modifications suited to the particular uses contemplated.

Claims

1. 1. An electronic device configured to be worn on a predetermined portion of a user's body, the electronic device having one or more processors, a memory, and a display generation component disposed on the electronic device on an opposite side of the user's body from the predetermined portion, displaying, via the display generation component, a graphical representation of the appearance of a body part corresponding to the predetermined portion of the user's body while the electronic device is worn on the predetermined portion of the user's body; detecting a trigger on the electronic device, the trigger comprising a gesture or series of gestures for adjusting a position of the electronic device relative to the predetermined part of the user's body; in response to detecting the trigger including the gesture or series of gestures for adjusting the position of the electronic device relative to the predetermined part of the user's body, modifying the graphical representation of the appearance of the body part corresponding to the predetermined part of the user's body in accordance with the detected trigger, including obscuring a portion of the graphical representation of the appearance of the body part corresponding to the predetermined part of the user's body.

2. 10. The method of claim 1, wherein detecting the trigger at the electronic device is performed without detecting a change in position of the electronic device relative to the predetermined part of the user's body.

3. The method of claim 1 , wherein detecting the trigger at the electronic device includes detecting a user input to pause a current session of the electronic device.

4. The method of claim 1 , wherein detecting the trigger on the electronic device comprises detecting a user input on an edge of the electronic device.

5. 10. The method of claim 1, Detecting a first change in position of the electronic device relative to the predetermined portion of the user's body; in response to detecting the first change in the position of the electronic device relative to the predetermined portion of the user's body, modifying the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body in accordance with the detected first change in the position of the electronic device relative to the predetermined portion of the user's body.

6. 6. The method of claim 5, wherein detecting the first change in the position of the electronic device relative to the predetermined portion of the user's body comprises detecting a movement of the electronic device away from a discrete position of the electronic device relative to the predetermined portion of the user's body; modifying the graphical representation in accordance with the detected first change in the position of the electronic device relative to the predetermined part of the user's body includes gradually obscuring one or more features of the graphical representation of the appearance of the body part as the position of the electronic device changes relative to the discrete position of the electronic device.

7. 7. The method of claim 6, wherein modifying the graphical representation according to the detected first change in the position of the electronic device relative to the predetermined part of the user's body comprises: ceasing to display at least a portion of the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body at a location a predetermined distance away from the predetermined portion of the user's body.

8. 2. The method of claim 1, wherein obscuring the portion of the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body comprises fading out at least a portion of the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body.

9. 2. The method of claim 1, wherein obscuring the portion of the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body comprises blurring at least a portion of the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body.

10. 2. The method of claim 1, wherein obscuring the portion of the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body comprises dimming the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body.

11. 2. The method of claim 1, wherein obscuring the portion of the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body comprises displaying an overlay that obscures at least a portion of the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body.

12. 2. The method of claim 1, wherein obscuring the portion of the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body comprises displaying text over at least a portion of the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body.

13. 2. The method of claim 1, wherein obscuring the portion of the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body comprises displaying a pictogram over at least a portion of the graphical representation of the appearance of the body part corresponding to the predetermined portion of the user's body.

14. 10. The method of claim 1, detecting a second change in position of the electronic device relative to the predetermined portion of the user's body, the second change in position of the electronic device changing the position of the electronic device relative to the predetermined portion of the user's body; in response to detecting the second change in the position of the electronic device relative to the predetermined portion of the user's body, displaying an overlay indicating that a user of the electronic device is adjusting the electronic device.

15. The method of claim 1 , wherein the predetermined portion of the user's body includes two eyes.

16. 1. An electronic device comprising: a display generating component disposed on the electronic device opposite the predetermined portion of the user's body; one or more input devices; one or more processors; a memory for storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising: displaying, via the display generation component, a graphical representation of the appearance of a body part corresponding to the predetermined portion of the user's body while the electronic device is worn on the predetermined portion of the user's body; detecting a trigger on the electronic device, the trigger comprising a gesture or series of gestures for adjusting a position of the electronic device relative to the predetermined part of the user's body; and in response to detecting the trigger including the gesture or series of gestures for adjusting the position of the electronic device relative to the predetermined part of the user's body, modifying the graphical representation of the appearance of the body part corresponding to the predetermined part of the user's body in accordance with the detected trigger, including obscuring a portion of the graphical representation of the appearance of the body part corresponding to the predetermined part of the user's body.

17. 17. An electronic device according to claim 16, wherein the one or more programs contain instructions for carrying out the method according to any one of claims 2 to 15.

18. When executed by an electronic device including and / or in communication with a display generation component located on the electronic device opposite a predetermined portion of a user's body and one or more input devices, the electronic device: displaying, via the display generation component, a graphical representation of the appearance of a body part corresponding to the predetermined portion of the user's body while the electronic device is worn on the predetermined portion of the user's body; detecting a trigger on the electronic device, the trigger comprising a gesture or series of gestures for adjusting a position of the electronic device relative to the predetermined part of the user's body; and in response to detecting the trigger including the gesture or series of gestures for adjusting the position of the electronic device relative to the predetermined part of the user's body, modifying the graphical representation of the appearance of the body part corresponding to the predetermined part of the user's body in accordance with the detected trigger, including obscuring a portion of the graphical representation of the appearance of the body part corresponding to the predetermined part of the user's body.

19. 19. A computer program according to claim 18, comprising instructions for carrying out the method according to any one of claims 2 to 15.

Images