Device, method, and graphical user interface for gaze-based navigation

Abstract

To provide an electronic device having improved method and interface for navigation of a user interface.SOLUTION: An electronic device enlarges an item of a content in accordance with detection of a visual line of a user, scrolls a text of the content item in accordance with determination that the user is reading the content item, navigates user interfaces in accordance with detection of motion of a user head and detection of a user visual line, and displays enlarged contents related to the content part in accordance with the detection of the motion of the user head and the detection of the user visual line.SELECTED DRAWING: Figure 8A

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 63 / 001,210, filed March 27, 2020, and U.S. Provisional Patent Application No. 63 / 131,754, filed December 29, 2020, which are incorporated by reference in their entireties.

[0002] It generally relates to a computer system having a display generation component and one or more input devices that present a graphical user interface, including but not limited to electronic devices that present a graphical user interface via the display generation component, that implement gaze-based navigation. [Background technology]

[0003] The development of computer systems for augmented reality has progressed significantly in recent years. Exemplary augmented reality environments include at least some virtual elements that replace or augment the physical world. Input devices such as cameras, controllers, joysticks, touch-sensitive surfaces, and touchscreen displays for computer systems and other electronic computing devices are used to interact with the virtual / augmented reality environment. Exemplary virtual elements include digital images, video, text, icons, and virtual objects such as buttons and other graphics.

[0004] However, methods and interfaces for interacting with environments (e.g., applications, augmented reality environments, mixed reality environments, and virtual reality environments) that include at least some virtual elements are cumbersome, inefficient, and limited. For example, systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to achieve a desired result in an augmented reality environment, and systems in which manipulating virtual objects is complex and error-prone create a significant cognitive burden for users and detract from the experience of the virtual / augmented reality environment. In addition, these methods are unnecessarily time-consuming, thereby wasting energy. This latter consideration is particularly important in battery-operated devices. Summary of the Invention

[0005] Therefore, there is a need for a computer system having improved methods and interfaces for providing users with computer-generated experiences that make interaction with the computer system more efficient and intuitive for the user. Such methods and interfaces can complement or replace conventional methods of providing users with computer-generated reality experiences. Such methods and interfaces reduce the number, extent, and / or type of inputs from the user by helping the user understand the connection between the input provided and the device response to that input, thereby creating a more efficient human-machine interface.

[0006] The above-mentioned deficiencies and other problems associated with user interfaces for computer systems having a display generation component and one or more input devices are reduced or eliminated by the disclosed system. In some embodiments, the computer system is a desktop computer with an associated display. In some embodiments, the computer system is a portable device (e.g., a notebook computer, a tablet computer, or a handheld device). In some embodiments, the computer system is a personal electronic device (e.g., a wearable electronic device such as a wristwatch or a head-mounted device). In some embodiments, the computer system has a touchpad. In some embodiments, the computer system has one or more cameras. In some embodiments, the computer system has a touch-sensitive display (also known as a "touch screen" or "touchscreen display"). In some embodiments, the computer system has one or more eye-tracking components. In some embodiments, the computer system has one or more hand-tracking components. In some embodiments, the computer system has one or more output devices in addition to the display generation component, the output devices including one or more tactile output generators and one or more audio output devices. In some embodiments, the computer system has a graphical user interface (GUI), one or more processors, memory, and one or more modules, programs, or instruction sets stored in the memory for performing a plurality of functions. In some embodiments, a user interacts with the GUI through stylus and / or finger contacts and gestures on the touch-sensitive surface, the movement of the user's eyes and hands in space relative to the GUI or the user's body as captured by cameras and other movement sensors, and voice input as captured by one or more audio input devices.In some embodiments, the functions performed through the interactions optionally include image editing, drawing, presenting, word processing, spreadsheet creation, game playing, making phone calls, video conferencing, emailing, instant messaging, training support, digital photography, digital videography, web browsing, digital music playback, note taking, and / or digital video playback, and executable instructions to perform those functions are optionally contained on a non-transitory computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0007] There is a need for electronic devices with improved methods and interfaces for navigating user interfaces. Such methods and interfaces can complement or replace conventional methods for interacting with graphical user interfaces. Such methods and interfaces reduce the number, extent, and / or type of input from a user, creating a more efficient human-machine interface.

[0008] In some embodiments, the electronic device enhances an item of content in accordance with detecting a user's gaze. In some embodiments, the electronic device scrolls text through the content item in accordance with determining that the user is reading the content item. In some embodiments, the electronic device navigates through a user interface in accordance with detecting a user's head movement and detecting a user's gaze. In some embodiments, the electronic device displays enhanced content related to a portion of the content in accordance with detecting a user's head movement and detecting a user's gaze.

[0009] It should be noted that the various embodiments described above can be combined with any other embodiment described herein. The features and advantages described herein are not exhaustive, and many additional features and advantages will become apparent to those skilled in the art, particularly in light of the drawings, specification, and claims. Furthermore, it should be noted that the language used in this specification has been selected solely for the purposes of readability and explanation, and not to define or limit the subject matter of the present invention. [Brief explanation of the drawings]

[0010] For a better understanding of the various described embodiments, reference should be made to the following Detailed Description of the Invention in conjunction with the following drawings, in which like reference numerals refer to corresponding parts throughout:

[0011] [Figure 1] FIG. 1 is a block diagram illustrating a computer system operating environment for providing a CGR experience, according to some embodiments.

[0012] [Figure 2] FIG. 1 is a block diagram illustrating a controller of a computer system configured to manage and coordinate a user's CGR experience, according to some embodiments.

[0013] [Figure 3] FIG. 1 is a block diagram illustrating display generation components of a computer system configured to provide a user with visual components of a CGR experience, according to some embodiments.

[0014] [Figure 4] FIG. 1 is a block diagram illustrating a hand tracking unit of a computer system configured to capture a user's gesture input, according to some embodiments.

[0015] [Figure 5]FIG. 1 is a block diagram illustrating an eye-tracking unit of a computer system configured to capture a user's gaze input, according to some embodiments.

[0016] [Figure 6] 1 is a flowchart illustrating a glint-assisted gaze tracking pipeline, according to some embodiments.

[0017] [Figure 7A] 1A-1C illustrate examples of how an electronic device provides navigation of a user interface according to detection of a user's gaze, according to some embodiments. [Figure 7B] 1A-1C illustrate examples of how an electronic device provides navigation of a user interface according to detection of a user's gaze, according to some embodiments. [Figure 7C] 1A-1C illustrate examples of how an electronic device provides navigation of a user interface according to detection of a user's gaze, according to some embodiments. [Figure 7D] 1A-1C illustrate examples of how an electronic device provides navigation of a user interface according to detection of a user's gaze, according to some embodiments. [Figure 7E] 1A-1C illustrate examples of how an electronic device provides navigation of a user interface according to detection of a user's gaze, according to some embodiments. [Figure 7F] 1A-1C illustrate examples of how an electronic device provides navigation of a user interface according to detection of a user's gaze, according to some embodiments. [Figure 7G] 1A-1C illustrate examples of how an electronic device provides navigation of a user interface according to detection of a user's gaze, according to some embodiments. [Figure 7H]1A-1C illustrate examples of how an electronic device provides navigation of a user interface according to detection of a user's gaze, according to some embodiments. [Figure 7I] 1A-1C illustrate examples of how an electronic device provides navigation of a user interface according to detection of a user's gaze, according to some embodiments. [Figure 7J] 1A-1C illustrate examples of how an electronic device provides navigation of a user interface according to detection of a user's gaze, according to some embodiments. [Figure 7K] 1A-1C illustrate examples of how an electronic device provides navigation of a user interface according to detection of a user's gaze, according to some embodiments.

[0018] [Figure 8A] 1 is a flowchart illustrating a method for enhancing an item of content according to detection of a user's gaze, according to some embodiments. [Figure 8B] 1 is a flowchart illustrating a method for enhancing an item of content according to detection of a user's gaze, according to some embodiments. [Figure 8C] 1 is a flowchart illustrating a method for enhancing an item of content according to detection of a user's gaze, according to some embodiments. [Figure 8D] 1 is a flowchart illustrating a method for enhancing an item of content according to detection of a user's gaze, according to some embodiments. [Figure 8E] 1 is a flowchart illustrating a method for enhancing an item of content according to detection of a user's gaze, according to some embodiments. [Figure 8F] 1 is a flowchart illustrating a method for enhancing an item of content according to detection of a user's gaze, according to some embodiments. [Figure 8G] 1 is a flowchart illustrating a method for enhancing an item of content according to detection of a user's gaze, according to some embodiments.

[0019] [Figure 9A] 10 is a flowchart illustrating a method for scrolling text of a content item pursuant to a determination that a user is reading the content item, according to some embodiments. [Figure 9B] 10 is a flowchart illustrating a method for scrolling text of a content item pursuant to a determination that a user is reading the content item, according to some embodiments. [Figure 9C] 10 is a flowchart illustrating a method for scrolling text of a content item pursuant to a determination that a user is reading the content item, according to some embodiments. [Figure 9D] 10 is a flowchart illustrating a method for scrolling text of a content item pursuant to a determination that a user is reading the content item, according to some embodiments. [Figure 9E] 10 is a flowchart illustrating a method for scrolling text of a content item pursuant to a determination that a user is reading the content item, according to some embodiments. [Figure 9F] 10 is a flowchart illustrating a method for scrolling text of a content item pursuant to a determination that a user is reading the content item, according to some embodiments. [Figure 9G] 10 is a flowchart illustrating a method for scrolling text of a content item pursuant to a determination that a user is reading the content item, according to some embodiments. [Figure 9H] 10 is a flowchart illustrating a method for scrolling text of a content item pursuant to a determination that a user is reading the content item, according to some embodiments. [Figure 9I] 10 is a flowchart illustrating a method for scrolling text of a content item pursuant to a determination that a user is reading the content item, according to some embodiments. [Figure 9J]10 is a flowchart illustrating a method for scrolling text of a content item pursuant to a determination that a user is reading the content item, according to some embodiments. [Figure 9K] 10 is a flowchart illustrating a method for scrolling text of a content item pursuant to a determination that a user is reading the content item, according to some embodiments. [Figure 9L] 10 is a flowchart illustrating a method for scrolling text of a content item pursuant to a determination that a user is reading the content item, according to some embodiments.

[0020] [Figure 10A] 1 is a flowchart illustrating a method for navigating through a user interface according to detection of a user's head movement and detection of a user's gaze, according to some embodiments. [Figure 10B] 1 is a flowchart illustrating a method for navigating through a user interface according to detection of a user's head movement and detection of a user's gaze, according to some embodiments. [Figure 10C] 1 is a flowchart illustrating a method for navigating through a user interface according to detection of a user's head movement and detection of a user's gaze, according to some embodiments. [Figure 10D] 1 is a flowchart illustrating a method for navigating through a user interface according to detection of a user's head movement and detection of a user's gaze, according to some embodiments. [Figure 10E] 1 is a flowchart illustrating a method for navigating through a user interface according to detection of a user's head movement and detection of a user's gaze, according to some embodiments. [Figure 10F] 1 is a flowchart illustrating a method for navigating through a user interface according to detection of a user's head movement and detection of a user's gaze, according to some embodiments. [Figure 10G]1 is a flowchart illustrating a method for navigating through a user interface according to detection of a user's head movement and detection of a user's gaze, according to some embodiments.

[0021] [Figure 11A] 10 is a flowchart illustrating a method for displaying augmented content related to portions of content according to detection of a user's head movement and detection of a user's gaze, according to some embodiments. [Figure 11B] 10 is a flowchart illustrating a method for displaying augmented content related to portions of content according to detection of a user's head movement and detection of a user's gaze, according to some embodiments. [Figure 11C] 10 is a flowchart illustrating a method for displaying augmented content related to portions of content according to detection of a user's head movement and detection of a user's gaze, according to some embodiments. [Figure 11D] 10 is a flowchart illustrating a method for displaying augmented content related to portions of content according to detection of a user's head movement and detection of a user's gaze, according to some embodiments. [Figure 11E] 10 is a flowchart illustrating a method for displaying augmented content related to portions of content according to detection of a user's head movement and detection of a user's gaze, according to some embodiments. [Figure 11F] 10 is a flowchart illustrating a method for displaying augmented content related to portions of content according to detection of a user's head movement and detection of a user's gaze, according to some embodiments. [Figure 11G] 10 is a flowchart illustrating a method for displaying augmented content related to portions of content according to detection of a user's head movement and detection of a user's gaze, according to some embodiments. [Figure 11H] 10 is a flowchart illustrating a method for displaying augmented content related to portions of content according to detection of a user's head movement and detection of a user's gaze, according to some embodiments. [Figure 11I]10 is a flowchart illustrating a method for displaying augmented content related to portions of content according to detection of a user's head movement and detection of a user's gaze, according to some embodiments. [Figure 11J] 10 is a flowchart illustrating a method for displaying augmented content related to portions of content according to detection of a user's head movement and detection of a user's gaze, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0022] The present disclosure relates to a user interface that provides a computer-generated reality (CGR) experience to a user, according to some embodiments.

[0023] The systems, methods, and GUIs described herein provide improved ways for electronic devices to provide navigation based on a user's gaze and / or the user's head movements.

[0024] In some embodiments, the computer system enhances an item of content based on detecting, via an eye-tracking device in communication with the computer system, that a user's gaze indicates that the user is reading the item of content. The user's gaze is detected, for example, using a camera (e.g., a camera integrated with a head-worn device, a camera mounted away from the user (e.g., in a CGR room), or a camera mounted toward the user as the user views an image presented by a display generation component (e.g., an array of cameras mounted above a display screen or touchscreen)) as opposed to a touch-sensitive surface or other physical controller. The computing system optionally determines which item of content the user is reading among multiple displayed items of content based on the detected gaze position, and enhances the item of content the user is reading. By enhancing the individual item of content in response to detecting that the user is reading the individual item of content, the user can continue reading the content item without using any input device other than the eye-tracking device and can provide input requesting enhancement of the individual item of content, thereby allowing the user to access the enhanced content more naturally and efficiently. Furthermore, gaze-based navigation requires only the user's eye and / or head movements, making computing systems accessible to people with impaired motor control.

[0025] In some embodiments, the computer system scrolls items of content based on detecting, via an eye-tracking device in communication with the computer system, that a user's gaze indicates that the user has finished reading a discrete portion of the item of content. The user's gaze is detected, for example, using a camera (e.g., a camera integrated with a head-worn device, a camera mounted away from the user (e.g., in a CGR room), or a camera mounted toward the user while the user is viewing an image presented by a display generation component (e.g., an array of cameras mounted above a display screen or touchscreen)) as opposed to a touch-sensitive surface or other physical controller. The computing system optionally determines that the user has finished reading a portion of content, such as a discrete line of text in the content, based on the detected gaze position. Scrolling discrete items of content in response to detecting the user's gaze allows the user to continue reading the content item without using an input device other than the eye-tracking device and to provide input requesting scrolling of the item of content, thereby allowing the user to read the content more naturally and efficiently. Furthermore, gaze-based scrolling requires only the user's eye and / or head movement, making the computing system accessible to people with impaired motor control.

[0026] In some embodiments, the computer system navigates the user interface based on detecting the user's gaze and / or head movements via one or more image sensors in communication with the computer system. The user's gaze and head movements are detected, for example, using a camera (e.g., a camera integrated with a head-worn device, a camera installed away from the user (e.g., in a CGR room), or a camera installed toward the user while the user is viewing images presented by a display generation component (e.g., an array of cameras mounted above a display screen or touchscreen)) as opposed to a touch-sensitive surface or other physical controller. The computing system optionally determines, based on the one or more image sensors, that the user is looking at a discrete portion of the user interface corresponding to a navigation location other than the current navigation location and / or that the user is tilting or directing their head toward a discrete portion of the user interface. In response to detecting the user's gaze and head movements directed toward a discrete portion of the user interface, the electronic device optionally navigates to the discrete portion of the user interface. By navigating a user interface in response to detecting a user's gaze and / or head movements, a user can navigate the user interface and provide inputs required to enhance individual items of content without using input devices other than the eye-tracking device, thereby allowing the user to interact with the user interface more naturally and efficiently. Furthermore, gaze-based navigation requires only the user's eye and / or head movements, making computing systems accessible to people with impaired motor control.

[0027] In some embodiments, the computer system presents augmented content associated with a portion of an item of content (e.g., a word, an image, etc.) based on detecting, via one or more image sensors in communication with the computer system, that a user's gaze and / or head movement is directed toward the portion of the content. The user's gaze and / or head movement is detected, for example, using a camera (e.g., a camera integrated with a head-worn device, a camera installed away from the user (e.g., in a CGR room), or a camera installed toward the user as the user views images presented by a display generation component (e.g., an array of cameras mounted above a display screen or touchscreen)), as opposed to a touch-sensitive surface or other physical controller. The computing system optionally uses the one or more image sensors to determine that the user's gaze is directed toward the portion of content and that the user's head is moving forward relative to the user's body. In response to the user's gaze and / or head movement, the electronic device displays augmented content (e.g., a definition, an augmented image, a website preview) associated with the portion of the content. By presenting augmented content in response to detecting a user's gaze and / or head movement, the user can view the augmented content without using an input device other than the eye tracking device to provide input requesting the presentation of the augmented content, thereby allowing the user to access the augmented content more naturally and efficiently. Furthermore, presenting augmented content in response to detecting a user's gaze and / or head movement requires only the user's eye and / or head movement, making the computing system accessible to people with impaired motor control.

[0028] Figures 1-6 illustrate an exemplary computer system for providing a CGR experience to a user. Figures 7A-7K illustrate examples of how an electronic device provides user interface navigation according to detection of a user's gaze, according to some embodiments. Figures 8-11 are flow diagrams of methods for providing navigation according to detection of a user's gaze, according to various embodiments. The user interfaces of Figures 7A-7K are used to explain the processes of Figures 8-11.

[0029] 1, a CGR experience is provided to a user via an operating environment 100 that includes a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor of a portable electronic device or a remote server), a display generation component 120 (e.g., a head-mounted device (HMD), a display, a projector, a touchscreen, etc.), one or more input devices 125 (e.g., an eye-tracking device 130, a hand-tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a tactile output generator 170, and other output devices 180), one or more sensors 190 (e.g., an image sensor, a light sensor, a depth sensor, a tactile sensor, an orientation sensor, a proximity sensor, a temperature sensor, a position sensor, a motion sensor, a velocity sensor, etc.), and optionally one or more peripheral devices 195 (e.g., a consumer electronics device, a wearable device, etc.). In some embodiments, one or more of the input device 125, the output device 155, the sensor 190, and the peripheral device 195 are integrated with the display generation component 120 (e.g., within a head-mounted or handheld device).

[0030] When describing a CGR experience, various terms are used to individually refer to several related, but distinct, environments that a user senses and / or with which the user can interact (e.g., using inputs detected by computer system 101 that cause the computer system generating the CGR experience to generate audio, visual, and / or haptic feedback corresponding to various inputs provided to computer system 101 generating the CGR experience). The following is a subset of these terms:

[0031] Physical Environment: The physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic systems. A physical environment, such as a physical park, includes physical objects such as physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment through their senses of sight, touch, hearing, taste, smell, etc.

[0032] Computer-Generated Reality: In contrast, a computer-generated reality (CGR) environment refers to a wholly or partially mimicked environment that people sense and / or interact with via electronic systems. In a CGR, a subset of a person's body movements or representations thereof are tracked, and in response, one or more properties of one or more virtual objects simulated within the CGR environment are adjusted to behave according to at least one law of physics. For example, a CGR system may detect a person's head rotation and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. In some circumstances (e.g., for accessibility reasons), adjustments to property(ies) of virtual object(s) in a CGR environment may be made in response to representations of body movements (e.g., voice commands). A person may sense and / or interact with a CGR object using any one of these senses, including sight, hearing, touch, taste, and smell. For example, a person may sense and / or interact with audio objects that create a 3D or spatially expansive audio environment that provides the perception of a point sound source in 3D space. In another example, audio objects may enable audio transparency that selectively incorporates ambient sounds from the physical environment, with or without computer-generated audio. In some CGR environments, a person may sense and / or interact with only audio objects.

[0033] Examples of CGR include virtual reality and mixed reality.

[0034] Virtual Reality: A virtual reality (VR) environment refers to an emulated environment designed to be based entirely on computer-generated sensory input for one or more senses. A VR environment includes multiple virtual objects that a person can sense and / or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person can sense and / or interact with virtual objects in a VR environment through a simulation of the person's presence in the computer-generated environment and / or through a simulation of a subset of the person's physical movements in the computer-generated environment.

[0035] Mixed Reality: A mixed reality (MR) environment refers to a mimetic environment designed to incorporate sensory input from or representations of a physical environment in addition to including computer-generated sensory input (e.g., virtual objects), as opposed to a VR environment designed to be based entirely on computer-generated sensory input. On the virtuality continuum, a mixed reality environment is anywhere between, but not including, a complete physical environment at one end and a virtual reality environment at the other. In some MR environments, computer-generated sensory input may respond to changes in sensory input from the physical environment. Some electronic systems for presenting MR environments may also track the position and / or orientation relative to the physical environment to allow virtual objects to interact with real objects (i.e., physical items or representations thereof from the physical environment). For example, the system may account for movement so that a virtual tree appears stationary relative to the physical ground.

[0036] Examples of mixed reality include augmented reality and augmented virtuality.

[0037] Augmented reality: An augmented reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed on a physical environment or a representation thereof. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person can directly view the physical environment. The system may be configured to present virtual objects on the transparent or translucent display, whereby a person using the system perceives the virtual objects superimposed on the physical environment. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system composites the images or videos with virtual objects and presents the composite on the opaque display. A person uses the system to indirectly view the physical environment through the images or videos of the physical environment and perceive the virtual objects superimposed on the physical environment. As used herein, videos of a physical environment shown on an opaque display are referred to as "pass-through video," meaning that the system captures images of the physical environment using one or more image sensors and uses those images in presenting the AR environment on the opaque display. Alternatively, the system may include a projection system that projects virtual objects, e.g., as holograms, into the physical environment or onto a physical surface, so that a person using the system perceives the virtual objects superimposed on the physical environment. Augmented reality environments also refer to simulated environments in which a representation of the physical environment is transformed by computer-generated sensory information. For example, when providing pass-through video, the system may distort one or more sensor images to impose a selected perspective (e.g., viewpoint) different from the perspective captured by the image sensor. As another example, the representation of the physical environment may be distorted by graphically altering (e.g., enlarging) portions thereof, thereby rendering the altered portions a non-photorealistic, altered version of the originally captured image. As a further example, the representation of the physical environment may be distorted by graphically removing or obscuring portions thereof.

[0038] Augmented Virtual: An augmented virtual (AV) environment refers to a mimicking environment in which a virtual or computer-generated environment incorporates one or more sensory inputs from a physical environment. The sensory inputs may be representations of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, while people with faces are realistically recreated from images taken of physical people. As another example, virtual objects may adopt the shape or color of physical items imaged by one or more imaging sensors. As a further example, virtual objects may adopt shadows that match the position of the sun in the physical environment.

[0039] Hardware: There are many different types of electronic systems that enable a person to sense and / or interact with various CGR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed over a person's eyes (e.g., similar to contact lenses), headphones / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have one or more speaker(s) and an integrated opaque display. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment. A head-mounted system may have a transparent or translucent display rather than an opaque display. A transparent or translucent display may have a medium through which light representing an image is directed toward a person's eye. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser-scanned light source, or any combination of these technologies. The medium may be a light guide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, the transparent or translucent display may be configured to be selectively opaque. A projection-based system may employ retinal projection technology that projects graphical images onto a person's retina. The projection system may also be configured to project virtual objects into the physical environment, for example, as holograms or as physical surfaces. In some embodiments, the controller 110 is configured to manage and coordinate the user's CGR experience.In some embodiments, controller 110 includes a suitable combination of software, firmware, and / or hardware. Controller 110 is described in more detail below with reference to FIG. 2. In some embodiments, controller 110 is a computing device that is local or remote to scene 105 (e.g., the physical environment). For example, controller 110 is a local server located within scene 105. In another example, controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside scene 105. In some embodiments, controller 110 is communicatively coupled to display generation component 120 (e.g., an HMD, a display, a projector, a touchscreen, etc.) via one or more wired or wireless communication channels 144 (e.g., BLUETOOTH, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). In another example, the controller 110 is contained within the housing (e.g., physical housing) of, or shares the same physical housing or support structure as, one or more of the display generation component 120 (e.g., an HMD or a portable electronic device including a display and one or more processors), one or more of the input devices 125, one or more of the output devices 155, one or more of the sensors 190, and / or one or more of the peripheral devices 195.

[0040] In some embodiments, display generation component 120 is configured to provide a CGR experience (e.g., at least a visual component of the CGR experience) to a user. In some embodiments, display generation component 120 includes a suitable combination of software, firmware, and / or hardware. Display generation component 120 is described in more detail below with reference to FIG. 3. In some embodiments, functionality of controller 110 is provided by and / or combined with display generation component 120.

[0041] According to some embodiments, the display generation component 120 provides a CGR experience to the user while the user is virtually and / or physically present within the scene 105.

[0042] In some embodiments, the display generation component is worn on a part of the user's body (e.g., head, hand, etc.). Thus, the display generation component 120 includes one or more CGR displays provided for displaying CGR content. For example, the display generation component 120 surrounds the user's field of view. In some embodiments, the display generation component 120 is a handheld device (e.g., a smartphone or tablet) configured to present CGR content, where the user holds the device with a display pointed toward the user's field of view and a camera pointed toward the scene 105. In some embodiments, the handheld device is optionally located in a housing worn on the user's head. In some embodiments, the handheld device is optionally located on a support (e.g., a tripod) in front of the user. In some embodiments, the display generation component 120 is a CGR chamber, housing, or room configured to present CGR content without the user wearing or holding the display generation component 120. Many user interfaces described with reference to one type of hardware for displaying CGR content (e.g., a handheld device or a device on a tripod) may be implemented on another type of hardware for displaying CGR content (e.g., an HMD or other wearable computing device). For example, a user interface showing interactions with CGR content triggered based on interactions occurring in the space in front of a handheld or tripod-mounted device may be implemented similarly to an HMD where the interactions occur in the space in front of the HMD and the CGR content responses are displayed via the HMD. Similarly, a user interface showing interactions with CGR content triggered based on movement of a handheld or tripod-mounted device relative to the physical environment (e.g., scene 105 or a part of the user's body (e.g., the user's eye(s), head, or hand)) may be implemented similarly to an HMD where the interactions are triggered by movement of the HMD relative to the physical environment (e.g., scene 105 or a part of the user's body (e.g., the user's eye(s), head, or hand)).

[0043] While relevant features of operating environment 100 are shown in FIG. 1, those skilled in the art will understand from this disclosure that various other features have not been shown for the sake of brevity so as not to obscure more pertinent aspects of the exemplary embodiments disclosed herein.

[0044] 2 is a block diagram of an example controller 110, according to some embodiments. While certain features are shown, those skilled in the art will understand from this disclosure that various other features are not shown for the sake of brevity so as not to obscure more pertinent aspects of the embodiments disclosed herein. Thus, by way of non-limiting example, in some embodiments, controller 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., Universal Serial Bus (USB), FIREWIRE®, THUNDERBOLT®, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), Infrared (IR), BLUETOOTH, ZIGBEE®, or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these and various other components.

[0045] In some embodiments, one or more communication buses 204 include circuitry that interconnects and controls communications between system components. In some embodiments, one or more I / O devices 206 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, etc.

[0046] Memory 220 includes high-speed random-access memory, such as dynamic random-access memory (DRAM), static random-access memory (SRAM), double-data-rate random-access memory (DDRRAM), or other random-access solid-state memory devices. In some embodiments, memory 220 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile storage devices. Memory 220 optionally includes one or more storage devices located remotely from one or more processing units 202. Memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, memory 220, or its non-transitory computer-readable storage medium, stores the following programs, modules, and data structures, or a subset thereof, including optional operating system 230 and CGR experience module 240:

[0047] Operating system 230 includes procedures for handling various basic system services and for performing hardware-dependent tasks. In some embodiments, CGR experience module 240 is configured to manage and coordinate one or more CGR experiences for one or more users (e.g., a single CGR experience for one or more users, or multiple CGR experiences for each group of one or more users). To that end, in various embodiments, CGR experience module 240 includes a data acquisition unit 242, a tracking unit 244, an adjustment unit 246, and a data transmission unit 248.

[0048] 1 , and optionally one or more of input devices 125, output devices 155, sensors 190, and / or peripheral devices 195. To that end, in various embodiments, data acquisition unit 242 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0049] In some embodiments, tracking unit 244 is configured to map scene 105 and at least display generation component 120 relative to scene 105 of FIG. 1 , and optionally track the positions of one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To that end, in various embodiments, tracking unit 244 includes instructions and / or logic therefor, as well as heuristics and metadata therefor. In some embodiments, processing unit 244 includes hand tracking unit 243 and / or eye tracking unit 245. In some embodiments, hand tracking unit 243 is configured to track the position of one or more parts of a user's hand and / or the movement of one or more parts of a user's hand relative to scene 105 of FIG. 1 , relative to display generation component 120, and / or relative to a coordinate system defined relative to the user's hand. Hand tracking unit 243 is described in more detail below with reference to FIG. 4. In some embodiments, eye tracking unit 245 is configured to track the position and movement of a user's gaze (or, more broadly, the user's eyes, face, or head) relative to scene 105 (e.g., relative to the physical environment and / or the user (e.g., the user's hands)) or relative to CGR content displayed via display generation component 120. Eye tracking unit 245 is described in more detail below with reference to FIG. 5.

[0050] In some embodiments, adjustment unit 246 is configured to manage and adjust the CGR experience presented to the user by display generation component 120, and optionally by one or more of output devices 155 and / or peripheral devices 195. To that end, in various embodiments, adjustment unit 246 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0051] In some embodiments, data transmission unit 248 is configured to transmit data (e.g., presentation data, location data, etc.) to at least display generation component 120, and optionally to one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To that end, in various embodiments, data transmission unit 248 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0052] Although the data acquisition unit 242, the tracking unit 244 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the adjustment unit 246, and the data transmission unit 248 are shown as being present on a single device (e.g., the controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 242, the tracking unit 244 (e.g., including the eye tracking unit 243 and the hand tracking unit 244), the adjustment unit 246, and the data transmission unit 248 may be located within separate computing devices.

[0053] Furthermore, Figure 2 is intended more to illustrate the functionality of various features that may be present in particular embodiments, as opposed to a structural overview of the embodiments described herein. As will be recognized by those skilled in the art, items shown separately may be combined and some items may be separated. For example, some functional modules shown separately in Figure 2 may be implemented within a single module, and various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of specific functionality and how functions are allocated among them, will vary from embodiment to embodiment and, in some embodiments, will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0054] 3 is a block diagram of an example of a display generation component 120, according to some embodiments. While certain features are shown, those skilled in the art will understand from this disclosure that, for the sake of brevity, various other features are not shown so as not to obscure more pertinent aspects of the embodiments disclosed herein. To that end, by way of non-limiting example, in some embodiments, the HMD 120 includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, infrared, BLUETOOTH, ZIGBEE, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more CGR displays 312, one or more optional inward-facing and / or outward-facing image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these and various other components.

[0055] In some embodiments, the one or more communication buses 304 include circuitry for interconnecting and controlling communication between the system components. In some embodiments, the one or more I / O devices and sensors 306 include at least one of an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0056] In some embodiments, one or more CGR displays 312 are configured to provide a CGR experience to a user. In some embodiments, one or more CGR displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conduction electron-emissive element display (SED), field-emission display (FED), quantum dot light-emitting diode (QD-LED), MEMS, and / or similar display types. In some embodiments, one or more CGR displays 312 correspond to a waveguide display, such as a diffractive, reflective, polarized, holographic, etc. For example, the HMD 120 includes a single CGR display. In another example, the HMD 120 includes a CGR display for each eye of the user. In some embodiments, one or more CGR displays 312 are capable of presenting MR or VR content. In some embodiments, one or more CGR displays 312 are capable of presenting MR or VR content.

[0057] In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face, including the user's eyes (and may be referred to as eye-tracking cameras). In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's hand(s) and optionally the user's arm(s) (and may be referred to as hand-tracking cameras). In some embodiments, the one or more image sensors 314 are configured to face forward to acquire image data corresponding to a scene viewed by the user when the HMD 120 is not present (and may be referred to as scene cameras). The one or more optional image sensors 314 may include one or more RGB cameras (e.g., with a complementary metal-oxide semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), one or more infrared (IR) cameras, one or more event-based cameras, and / or the like.

[0058] Memory 320 includes high-speed random-access memory, such as DRAM, SRAM, DDR RAM, or other random-access solid-state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile storage devices. Memory 320 optionally includes one or more storage devices located remotely from one or more processing units 302. Memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, memory 320, or its non-transitory computer-readable storage medium, stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 330 and a CGR presentation module 340:

[0059] The operating system 330 includes procedures for handling various basic system services and instructions for performing hardware-dependent tasks. In some embodiments, the CGR presentation module 340 is configured to present CGR content to a user via one or more CGR displays 312. To that end, in various embodiments, the CGR presentation module 340 includes a data acquisition unit 342, a CGR presentation unit 344, a CGR map generation unit 346, and a data transmission unit 348.

[0060] In some embodiments, the data acquisition unit 342 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the controller 110 of Figure 1. To that end, in various embodiments, the data acquisition unit 342 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0061] In some embodiments, the CGR presentation unit 344 is configured to present CGR content via one or more CGR displays 312. To that end, in various embodiments, the CGR presentation unit 344 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0062] In some embodiments, the CGR map generation unit 346 is configured to generate a CGR map (e.g., a 3D map of a mixed reality scene or a map of a physical environment in which computer-generated objects can be placed) based on the media content data. To that end, in various embodiments, the CGR map generation unit 346 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0063] In some embodiments, data transmission unit 348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least controller 110, and optionally to one or more of input device 125, output device 155, sensor 190, and / or peripheral device 195. To that end, in various embodiments, data transmission unit 348 includes instructions and / or logic therefor, as well as heuristics and metadata therefor.

[0064] Although the data acquisition unit 342, the CGR presentation unit 344, the CGR map generation unit 346, and the data transmission unit 348 are shown as residing on a single device (e.g., the display generation component 120 of FIG. 1), it should be understood that in other embodiments, any combination of the data acquisition unit 342, the CGR presentation unit 344, the CGR map generation unit 346, and the data transmission unit 348 may be located within separate computing devices.

[0065] Furthermore, Figure 3 is intended more to illustrate the functionality of various features that may be present in particular embodiments, as opposed to a structural overview of the embodiments described herein. As will be recognized by those skilled in the art, items shown separately may be combined and some items may be separated. For example, some functional modules shown separately in Figure 3 may be implemented within a single module, and various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules, as well as the division of specific functions and how functions are allocated among them, will vary from embodiment to embodiment and, in some embodiments, will depend in part on the particular combination of hardware, software, and / or firmware selected for a particular implementation.

[0066] 4 is a schematic diagram of an example embodiment of hand tracking device 140. In some embodiments, hand tracking device 140 (FIG. 1) is controlled by hand tracking unit 243 (FIG. 2) to track the position of one or more parts of a user's hand and / or the movement of one or more parts of a user's hand relative to scene 105 of FIG. 1 (e.g., relative to parts of the physical environment surrounding the user, relative to display generation component 120, or relative to parts of the user (e.g., the user's face, eyes, or head), and / or relative to a coordinate system defined relative to the user's hand). In some embodiments, hand tracking device 140 is part of display generation component 120 (e.g., embedded in or attached to a head-mounted device). In some embodiments, hand tracking device 140 is separate from display generation component 120 (e.g., located in a separate housing or attached to a separate physical support structure).

[0067] In some embodiments, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and / or color cameras) that captures three-dimensional scene information including at least the hand 406 of a human user. The image sensor 404 captures hand images with sufficient resolution to allow for differentiation of the fingers and their respective positions. The image sensor 404 typically captures images of other parts of the user's body, or all of the body, and can have either zoom capabilities or a dedicated sensor with high magnification to capture hand images at a desired resolution. In some embodiments, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some embodiments, the image sensor 404 is used in conjunction with or functions as an image sensor that captures the physical environment of the scene 105. In some embodiments, the image sensor 404 is positioned relative to the user or the user's environment such that the image sensor or portions of its field of view are used to define an interaction space in which hand movements captured by the image sensor are processed as inputs to the controller 110.

[0068] In some embodiments, image sensor 404 outputs a sequence of frames containing 3D map data (and possibly color image data) to controller 110, which extracts high-level information from the map data. This high-level information is provided, typically via an application program interface (API), to an application running on the controller, which drives display generation component 120 accordingly. For example, a user can interact with software running on controller 110 by moving hand 408 and changing hand posture.

[0069] In some embodiments, the image sensor 404 projects a spot pattern onto a scene containing the hand 406 and captures an image of the projected pattern. In some embodiments, the controller 110 calculates the 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on the lateral shift of the pattern's spots. This approach is advantageous in that it does not require the user to hold or wear any type of beacon, sensor, or other marker. This provides depth coordinates of points in the scene relative to a predetermined reference plane at a specific distance from the image sensor 404. In this disclosure, the image sensor 404 is assumed to define an orthogonal set of x, y, and z axes such that the depth coordinate of a point in the scene corresponds to the z-component measured by the image sensor. Alternatively, the hand tracking device 440 can use other 3D mapping methods, such as stereoscopic imaging or time-of-flight measurements, based on single or multiple cameras or other types of sensors.

[0070] In some embodiments, the hand tracking device 140 captures and processes a time sequence of depth maps containing the user's hand while the user moves the hand (e.g., the entire hand or one or more fingers). Software running on the image sensor 404 and / or a processor in the controller 110 processes the 3D map data to extract patch descriptors of the hand in these depth maps. The software matches these descriptors with patch descriptors stored in the database 408, based on a previous training process, to estimate the pose of the hand in each frame. The pose typically includes the 3D positions of the user's wrist joints and fingertips.

[0071] The software can also analyze hand and / or finger trajectories across multiple frames in a sequence to identify gestures. The pose estimation functionality described herein may be interleaved with motion tracking functionality, whereby patch-based pose estimation is performed only once every two (or more) frames, while tracking is used to discover pose changes that occur across the remaining frames. The pose, motion, and gesture information is provided to an application program running on the controller 110 via the API described above. This program can, for example, move and modify an image presented on the display generation component 120 or perform other functions in response to the pose and / or gesture information.

[0072] In some embodiments, the software may be downloaded to the controller 110 in electronic form, for example, over a network, or alternatively may be provided on a tangible, non-transitory medium, such as an optical, magnetic, or electronic memory medium. In some embodiments, the database 408 is similarly stored in memory associated with the controller 110. Alternatively, or additionally, some or all of the described functionality of the computer may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable digital signal processor (DSP). While the controller 110 is shown in FIG. 4 as, by way of example, a separate unit from the image sensor 440, some or all of the processing functionality of the controller may be implemented by a suitable microprocessor and software, or by dedicated circuitry within the housing of the hand tracking device 402, or otherwise associated with the image sensor 404. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with the display generation component 120 (e.g., in a television set, handheld device, or head-mounted device) or using any other suitable computerized device, such as a game console or media player. The sensing function of the image sensor 404 may likewise be integrated into a computer or other computerized device that is controlled by the sensor output.

[0073] FIG. 4 also includes a schematic diagram of a depth map 410 captured by the image sensor 404, according to some embodiments. The depth map includes a matrix of pixels having respective depth values, as described above. A pixel 412 corresponding to the hand 406 is segmented from the background and wrist in this map. The intensity of each pixel in the depth map 410 is inversely proportional to the depth value, i.e., the measured z-distance from the image sensor 404, with increasing depth resulting in darker shades. The controller 110 processes these depth values ​​to identify and segment components of the image (i.e., groups of adjacent pixels) that have characteristics of a human hand. These characteristics can include, for example, the overall size, shape, and frame-to-frame motion of the depth map sequence.

[0074] Figure 4 also schematically illustrates a hand skeleton 414 that the controller 110 ultimately extracts from the depth map 410 of the hand 406, according to some embodiments. In Figure 4, the skeleton 414 is superimposed on a hand background 416 that was segmented from the original depth map. In some embodiments, key feature points on the hand (e.g., knuckles, fingertips, center of the palm, end of the hand where it connects to the wrist, etc.), and optionally the wrist or arm connected to the hand, are identified and located on the hand skeleton 414. In some embodiments, the location and movement of these key feature points over multiple image frames are used by the controller 110 to determine hand gestures performed by the hand or the current state of the hand, according to some embodiments.

[0075] FIG. 5 illustrates an exemplary embodiment of eye tracking device 130 (FIG. 1). In some embodiments, eye tracking device 130 is controlled by eye tracking unit 245 (FIG. 2) to track the position and movement of a user's gaze relative to scene 105 or relative to CGR content displayed via display generation component 120. In some embodiments, eye tracking device 130 is integrated with display generation component 120. For example, in some embodiments, if display generation component 120 is a head-mounted device such as a headset, helmet, goggles, or glasses, or a handheld device disposed on a wearable frame, the head-mounted device includes both components for generating CGR content for viewing by the user and components for tracking the user's gaze relative to the CGR content. In some embodiments, eye tracking device 130 is separate from display generation component 120. For example, if the display generation component is a handheld device or a CGR chamber, eye tracking device 130 is optionally a device separate from the handheld device or the CGR chamber. In some embodiments, eye tracking device 130 is a head-mounted device or part of a head-mounted device. In some embodiments, head-mounted eye tracking device 130 is optionally used in conjunction with head-mounted or non-head-mounted display generation components. In some embodiments, eye tracking device 130 is not a head-mounted device, and is optionally used in conjunction with head-mounted display generation components. In some embodiments, eye tracking device 130 is not a head-mounted device, and is optionally part of non-head-mounted display generation components.

[0076] In some embodiments, the display generation component 120 uses a display mechanism (e.g., left and right near-eye display panels) that displays frames including left and right images in front of the user's eyes to provide the user with a 3D virtual view. For example, a head-mounted display generation component may include left and right optical lenses (referred to herein as eyepieces) positioned between the display and the user's eyes. In some embodiments, the display generation component may include or be coupled to one or more external video cameras that capture video of the user's environment for display. In some embodiments, the head-mounted display generation component may have a transparent or translucent display that allows the user to view the physical environment directly and display virtual objects on the transparent or translucent display. In some embodiments, the display generation component projects virtual objects into the physical environment. The virtual objects are projected, for example, onto a physical surface or as a hologram, allowing an individual using the system to observe the virtual objects superimposed on the physical environment. In such cases, separate display panels and image frames for the left and right eyes may not be required.

[0077] As shown in FIG. 5 , in some embodiments, the gaze tracking device 130 includes at least one eye tracking camera (e.g., an infrared (IR) or near-IR (NIR) camera) and an illumination source (e.g., an IR or NIR light source such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eyes. The eye tracking camera may be aimed at the user's eyes to receive reflected IR or NIR light from the light source directly from the eyes, or alternatively, may be aimed at a “hot” mirror positioned between the user's eyes and a display panel that reflects the IR or NIR light from the eyes to the eye tracking camera while allowing visual light to pass through. The gaze tracking device 130 optionally captures images of the user's eyes (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes the images to generate gaze tracking information, and communicates the gaze tracking information to the controller 110. In some embodiments, the user's eyes are tracked separately by their respective eye tracking cameras and illumination sources. In some embodiments, only one eye of the user is tracked by a corresponding eye-tracking camera and lighting source.

[0078] In some embodiments, the eye tracking device 130 is calibrated using a device-specific calibration process to determine the eye tracking device's parameters for the particular operating environment 100, such as the 3D geometric relationships and parameters of the LEDs, camera, hot mirror (if present), eyepiece, and display screen. The device-specific calibration process may be performed at a factory or another facility prior to delivery of the AR / VR equipment to the end user. The device-specific calibration process may be an automatic or manual calibration process. The user-specific calibration process may include estimation of a particular user's eye parameters, such as pupil position, central vision position, optical axis, visual axis, eye spacing, etc. According to some embodiments, once the device-specific and user-specific parameters for the eye tracking device 130 have been determined, images captured by the eye tracking camera can be processed using glint-assisted methods to determine the user's current visual axis and viewpoint relative to the display.

[0079] As shown in FIG. 5, eye tracking device 130 (e.g., 130A or 130B) includes an eyepiece(s) 520 and a gaze tracking system including at least one eye tracking camera 540 (e.g., an infrared (IR) or near-IR (NIR) camera) positioned on the side of the user's face where eye tracking occurs and an illumination source 530 (e.g., an IR or NIR light source such as an array or ring of NIR light emitting diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eye(s) 592. The eye tracking camera 540 may be positioned between the user's eye(s) 592 and the display 510 (e.g., the left or right display panel of a head-mounted display, or the display of a handheld device, a projector, etc.) and may be directed at a mirror 550 that reflects IR or NIR light from the eye(s) 592 while transmitting visible light (e.g., as shown at the top of FIG. 5), or may be directed at the user's eye(s) 592 to receive reflected IR or NIR light from the user's eye(s) 592 (e.g., as shown at the bottom of FIG. 5).

[0080] In some embodiments, controller 110 renders AR or VR frames 562 (e.g., left and right frames for left and right display panels) and provides frames 562 to display 510. Controller 110 uses gaze tracking input 542 from eye tracking camera 540 for various purposes, such as in processing frames 562 for display. Controller 110 optionally estimates the user's viewpoint on display 510 based on gaze tracking input 542 obtained from eye tracking camera 540, using a glint-assisted method or other suitable method. The viewpoint estimated from gaze tracking input 542 is optionally used to determine the direction the user is currently looking.

[0081] Several possible use cases of the user's current gaze direction are described below, but are not intended to be limiting. As an exemplary use case, the controller 110 can render virtual content differently based on the determined user's gaze direction. For example, the controller 110 can generate virtual content with higher resolution in a central visual area determined from the user's current gaze direction than in a peripheral area. As another example, the controller can position or move virtual content within a view based at least in part on the user's current gaze direction. As another example, the controller can display particular virtual content within a view based at least in part on the user's current gaze direction. As another exemplary use case in an AR application, the controller 110 can orient an external camera to capture the physical environment of the CGR experience and focus in the determined direction. The external camera's autofocus mechanism can then focus on an object or surface within the environment the user is currently viewing on the display 510. As another exemplary use case, eyepiece 520 may be a focusable lens, and eye-tracking information is used by the controller to adjust the focus of eyepiece 520 so that the virtual object the user is currently looking at has the proper binocular coordination to match the convergence of the user's eyes 592. Controller 110 can utilize the eye-tracking information to orient and focus eyepiece 520 so that close objects the user is looking at appear at the correct distance.

[0082] In some embodiments, the eye tracking device is part of a head-mounted device that includes a display (e.g., display 510), two eyepieces (e.g., eyepiece(s) 520), an eye tracking camera (e.g., eye tracking camera(s) 540), and a light source (e.g., light source 530 (e.g., IR or NIR LED)) attached to the wearable housing. The light source emits light (e.g., IR or NIR light) toward the user's eye(s) 592. In some embodiments, the light sources may be arranged in a ring or circle around each lens, as shown in FIG. 5. In some embodiments, eight light sources 530 (e.g., LEDs) are arranged around each lens 520, as an example. However, more or fewer light sources 530 may be used, and other arrangements and positions of the light sources 530 may be used.

[0083] In some embodiments, the display 510 emits light in the visible light range and not in the IR or NIR ranges, and therefore does not introduce noise into the gaze tracking system. Note that the positions and angles of the eye tracking camera(s) 540 are given by way of example and are not intended to be limiting. In some embodiments, a single eye tracking camera 540 is located on each side of the user's face. In some embodiments, two or more NIR cameras 540 may be used on each side of the user's face. In some embodiments, a camera 540 with a wide field of view (FOV) and a camera 540 with a narrow FOV may be used on each side of the user's face. In some embodiments, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

[0084] Embodiments of an eye tracking system such as that shown in FIG. 5 may be used, for example, in computer-generated reality, virtual reality, and / or mixed reality applications to provide a user with a computer-generated reality, virtual reality, augmented reality, and / or augmented virtual experience.

[0085] FIG. 6 illustrates a glint-assisted gaze tracking pipeline according to some embodiments. In some embodiments, the gaze tracking pipeline is implemented by a glint-assisted gaze tracking system (e.g., eye tracking device 130 as shown in FIGS. 1 and 5). The glint-assisted gaze tracking system can maintain a tracking state. Initially, the tracking state is off or "no." When in the tracking state, the glint-assisted gaze tracking system tracks the pupil contour and glint in the current frame using prior information from the previous frame when analyzing the current frame. When not in the tracking state, the glint-assisted gaze tracking system attempts to detect the pupil and glint in the current frame, and if successful, initializes the tracking state to "yes" and continues to the next frame in the tracking state.

[0086] As shown in FIG. 6, an eye-tracking camera can capture left and right images of a user's left and right eyes. The captured images are then input into an eye-tracking pipeline for processing beginning at 610. As indicated by the arrow returning to element 600, the eye-tracking system can continue to capture images of the user's eyes at a rate of, for example, 60-120 frames per second. In some embodiments, each set of captured images may be input into the pipeline for processing. However, in some embodiments, or under some conditions, not all captured frames are processed by the pipeline.

[0087] At 610, if the tracking status is yes for the currently captured image, the method proceeds to element 640. If the tracking status is no at 610, the image is analyzed to detect the user's pupil and glint in the image, as shown at 620. If the pupil and glint are successfully detected at 630, the method proceeds to element 640. If not, the method returns to element 610 to process the next image of the user's eyes.

[0088] At 640, proceeding from element 410, the current frame is analyzed to track pupils and glints based in part on prior information from the previous frame. At 640, proceeding from element 630, a tracking state is initialized based on the detected pupils and glints in the current frame. The results of the processing at element 640 are checked to ensure that the tracking or detection results are reliable. For example, the results can be checked to determine whether a sufficient number of glints are successfully tracked or detected in the current frame to perform pupil and gaze estimation. At 650, if the results are not reliable, the tracking state is set to no and the method returns to element 610 to process the next image of the user's eyes. At 650, if the results are reliable, the method proceeds to element 670. At 670, the tracking state is set to yes (if not already yes) and pupil and glint information is passed to element 680 to estimate the user's gaze point.

[0089] 6 is intended to serve as an example of eye-tracking technology that may be used in a particular implementation. As will be recognized by those skilled in the art, other eye-tracking technologies, now existing or developed in the future, may be used in place of or in combination with the glint-assisted eye-tracking technology described herein in computer system 101 to provide a user with a CGR experience according to various embodiments.

[0090] In this disclosure, various input methods are described with respect to interaction with a computer system. Where one example is provided using one input device or input method and another example is provided using a different input device or input method, it should be understood that each example is compatible with, and may optionally be used with, the input device or input method described with respect to the other example. Similarly, various output methods are described with respect to interaction with a computer system. Where one example is provided using one output device or output method and another example is provided using a different output device or output method, it should be understood that each example is compatible with, and may optionally be used with, the output device or output method described with respect to the other example. Similarly, various methods are described with respect to interaction with a virtual environment or a mixed reality environment via a computer system. Where an example is provided using interaction with a virtual environment and another example is provided using a mixed reality environment, it should be understood that each example is compatible with, and may optionally be used with, the method described with respect to the other example. Thus, this disclosure discloses embodiments that are combinations of features of multiple examples, without exhaustively listing all features of the embodiments in the description of each embodiment. User Interface and Related Processes

[0091] We now turn our attention to embodiments of user interfaces (“UIs”) and associated processes that may be executed in a computer system, such as a portable multifunction device or a head-mounted device, equipped with a display generation component, one or more input devices, and (optionally) one or more cameras.

[0092] 7A-7K are diagrams illustrating examples of how an electronic device may provide navigation of a user interface according to detection of a user's gaze, according to some embodiments.

[0093] 7A illustrates an electronic device 101 displaying a content browsing user interface via a display generation component 120. As described above with reference to FIGS. 1-6 , the electronic device 101 optionally includes the display generation component 120 (e.g., a touchscreen) and multiple image sensors 314. The image sensors 314 optionally include one or more of a visible light camera, an infrared camera, a depth sensor, or any other sensor that the electronic device 101 can use to capture one or more images of a user or a portion of a user while the user is interacting with the electronic device 101.

[0094] As shown in FIG. 7A , the user interface includes a content viewing area 700. The content viewing area 700 includes representations of multiple content items, including portions of images 702a-d and text 704a-d associated with each item. FIG. 7A also includes an indication 708 of an event received at the electronic device and a visual indication 706 of the event displayed in response. For example, the indication of the event is an indication generated by an application accessible to the electronic device 101, such as an indication of an incoming message from another electronic device or a user account on another electronic device. The visual indication of the notification 706 optionally includes text and / or an image associated with the event, such as the text of a message received at the electronic device 101 and an image associated with the user account that sent the message. Thus, in some embodiments, absent the exceptions described below, the device 101 displays the visual indication 706 of the notification (e.g., indication 708) via the display generation component 120 in response to receiving an indication of a notification-generating event.

[0095] 7B shows the display generation component 120 displaying the content viewing area 700. In FIG. 7B, the electronic device 101 detects a user's gaze 716p (e.g., using one or more image sensors 314) (e.g., the user's gaze position is aligned with text 704a) and determines that the user's gaze indicates that the user is reading the text 704a of one of the items of content. In some embodiments, the electronic device 101 determines whether the user is reading by tracking the user's eye movements and determining whether the movements match eye movements consistent with reading. For example, when people read, the eyes optionally move gradually along the line of text in the direction of the text in the associated language (e.g., from left to right for Latin, as shown in FIG. 7B) while reading the line of text, and after the user reads the line of text, the eyes optionally move in the opposite direction (e.g., from right to left) along the line just read before looking at the next line. The electronic device 101 optionally compares the detected eye movements of the user with this pattern to determine whether the user is reading.

[0096] In response to detecting that the user is reading the text 704a of the content item, as shown in FIG. 7B, the electronic device 101 updates the user interface as shown in FIG. 7C. As shown in FIG. 7C, the electronic device 101 displays the image 702a and text 704a of the item of content, along with an additional portion 710a of the text of the content and another image 712a included in the content, in an extended content area 714 overlaid on the original content browsing user interface 700. Thus, in some embodiments, the extended content area 714 includes portions of the content (e.g., text 710a, image 712a) that were not visible in the content browsing user interface 700. The device 101 optionally modifies the displayed user interface in other ways, which are described in more detail below.

[0097] As described above, the electronic device 101 can use the image sensor 314 to determine whether the user's eye movements while viewing the text 704a correspond to the user reading, as shown in Figure 7B. In some embodiments, if the electronic device 101 detects that the user is simply looking at the text 704a of the content item without reading it, or is skimming / scanning the text without reading it, the electronic device 101 continues to display the user interface shown in Figure 7B instead of updating the display to display the user interface shown in Figure 7C. However, in some embodiments, the electronic device 101 ceases updating the user interface in response to detecting the user looking at the text 704a without reading it, and updates the user interface as shown in Figure 7C in response to detecting the user skimming or scanning the text 704a.

[0098] 7B, the content viewing area 700 of the user interface includes representations of multiple items of content, such as an image 702a and text 704a of a first content item and an image 702b and text 704b of another content item. As shown in FIG. 7C, when the electronic device 101 updates the user interface to include another area 714 with additional portions of the first content item, the other content items (e.g., content items that the user has not determined to be reading) are not expanded (e.g., the electronic device 101 does not display the additional portions of the other content items).

[0099] Similarly, in Figure 7B, if the user starts reading the text 704b of the second content item instead of reading the text 704a of the first content item, the electronic device 101 updates its user interface to include a user interface area with an additional portion of the second content item in a manner similar to how the electronic device 101 displays the additional area 714 of the first content item shown in Figure 7C. The electronic device 101, optionally, similarly expands any other items of content that the user begins reading (optionally without expanding the other items of content) while displaying the content viewing user interface 700 shown in Figure 7B.

[0100] As shown in FIG. 7C , while displaying the first item of content in the expanded content area 714 of the user interface, the electronic device 101 continues to display the content viewing area 700 of the user interface (and other items of content within the content viewing area 700). The content viewing area 700 (and the items of content displayed therein) are displayed with a de-highlighted appearance while displaying additional portions 710a and 712a of the first content item in the expanded content area 714. For example, as shown in FIG. 7C , the content viewing area 700 (and the content items displayed therein) are displayed in a darker color and smaller size compared to how the content viewing area 700 was displayed before expanding the first content item, as shown in FIG. 7B . In some embodiments, the electronic device 101 displays the content viewing area 700 (and the content items displayed therein) with increased transparency while displaying the expanded content area 714 compared to how the content viewing area 700 (and the content items displayed therein) was displayed before displaying the expanded content area 714.

[0101] 7C , the extended content area 714, which includes the first content item, is displayed over the content viewing area 700 of the user interface. In some embodiments, the z-height of the extended content area 714 in the user interface is the same as the z-height of the content viewing area 700 before the extended content area 714 was displayed (e.g., the z-height of the content viewing area 700 in FIG. 7B ). In some embodiments, when updating the user interface from the user interface shown in FIG. 7B to the user interface shown in FIG. 7C , the electronic device 101 optionally moves the content viewing area 700 to a lower z-height in the user interface.

[0102] As shown in Figure 7C, in some embodiments, the electronic device 101 displays the content item in the expanded content area 714 of the user interface in the same position (e.g., the same horizontal and / or vertical position within the display area) as the content item was displayed in the content viewing user interface 700 as shown in Figure 7B (e.g., the image 702a and the text 704a are displayed in the same horizontal and / or vertical position within the display area as in Figures 7B and 7C). Also, as shown in Figure 7C, the width of the content item is optionally the same, and the content item is displayed in the expanded content area 714 as when the content item was displayed in the content viewing area 700 shown in Figure 7B. As shown in Figure 7C, the height of the content changes to reveal additional portions 710a and 712a of the content when displayed in the expanded content area 714. In some embodiments, the height of the content remains the same between Figures 7B and 7C. In some embodiments, the size of the text of the content item remains the same between Figures 7B and 7C, thereby avoiding jarring movement or resizing of the text that may distract the user in some situations.

[0103] 7B to 7C , electronic device 101 displays the content in expanded content area 714 in the center (or other predetermined location) of the display area of ​​display generation component 120. In some embodiments, electronic device 101 initially displays the content in expanded content area 714 in the same location that the content item was displayed in content viewing area 700 of FIG. 7B , and gradually moves the content of expanded content area 714, and the expanded content area itself 714, toward the center (or other predetermined location) of the display area as the user continues to read the content item in expanded content area 714 (e.g., moves the expanded content area more and more as the user reads more of the content in the expanded content area).

[0104] 7B to the user interface of FIG. 7C, electronic device 101 also optionally increases the portion of the user interface for displaying the first item of content. For example, the amount of display area for displaying the content of the first content item of FIG. 7B is optionally smaller than the amount of display area for displaying the content of the first content item of FIG. 7C. As a result, more of the first content item can be displayed in expanded content area 714 than was displayed in content viewing area 700.

[0105] In some embodiments, the electronic device 101 gradually animates the transition from the user interface shown in Figure 7B to the user interface shown in Figure 7C. The electronic device optionally progresses the animation at a rate proportional to the speed at which the user reads the content item's text 704a (e.g., the more text 704a the user reads, the more transition to Figure 7C is completed). For example, when the user reads the portion of the content item's text 704a shown in Figure 7B, the electronic device optionally displays a portion of the animation. As another example, when the user reads most of the content item's text 704a, the electronic device 101 displays most of the animation.

[0106] 7C , when the user stops reading the content item text 704a or 710a and looks away from the content item (e.g., looks at location 716c in the user interface), the electronic device 101 optionally returns the user interface to the user interface shown in FIG. 7B . In some embodiments, the electronic device 101 instead navigates between the user interfaces displayed in FIGS. 7B and 7C according to one or more steps of method 1000 described below. In some embodiments, the electronic device 101 returns the user interface from the user interface shown in FIG. 7C to the user interface shown in FIG. 7B in response to detecting (e.g., using the image sensor 314) that the user has stopped reading the content item text 704a and / or 710a. Furthermore, in some embodiments, when the user looks away from the content item (e.g., to location 716c) while the electronic device 101 is displaying an animation updating the user interface from the user interface shown in FIG. 7B to the user interface shown in FIG. 7C , the electronic device returns the user interface to the user interface shown in FIG. 7B .

[0107] In some embodiments, the electronic device 101 ceases displaying or generating 706 an indication of an event while the user is reading an item of content displayed in the user interface (e.g., in the expanded content area 714 or the content viewing area 700). For example, as shown in FIG. 7A , in response to detecting an indication of an event 708 while the user is not reading any of the content items in the content viewing area 700, the electronic device 101 displays a visual indication of a notification of the event 706. As shown in FIG. 7C , in response to receiving the indication of the event 708 while the electronic device 101 detects the user's gaze 716b reading a portion 704a of the content item, the electronic device 101 ceases displaying the visual indication of the notification 706 displayed in FIG. 7A . In some embodiments, once the user stops reading the content item and the user is not reading another content item, the electronic device 101 displays the indication of the event 706 detected while the user was reading. In some embodiments, the electronic device 101 does not display a visual indication 706 of the event, even when the device 101 determines that the user has stopped reading content in the user interface.

[0108] In some embodiments, the electronic device 101 scrolls the text of the content item as the user reads the content item. In Figure 7C, the electronic device 101 displays a first line 724a of the text of the content item in a first position 722a in the user interface (e.g., directly below the image 702a). The electronic device 101 also displays a second line 724b of the text in a second position 722b in the user interface (e.g., directly below position 722a).

[0109] In response to detecting the user's gaze 724a, indicating that the user has finished reading the first line of text 716b, the electronic device 101 scrolls the text as shown in FIG. 7D. As a result of the scrolling, the second line of text 724b is displayed in the first position 722a, and a third line of text 724c, after the second line of text 724b, is displayed in the second position 722b, as shown in FIG. 7D. The electronic device 101 also scrolls up the remainder of the content item, such as the text 710a and the image 712a. In some embodiments, the positions 722a and 722b are fixed relative to the rest of the user interface. In some embodiments, after scrolling the second line of text 724b to the user interface position 722a, the electronic device stops displaying the first line of text 724a. In some embodiments, the image 712a does not scroll.

[0110] 7C , in some embodiments, the electronic device 101 does not scroll the content until the electronic device detects that the user has finished reading the line of text 724a. While detecting that the user is reading the line of text 724a without finishing the line of text, the electronic device 101 maintains the current scrolling state of the content in the user interface (e.g., stops scrolling the content). Furthermore, the electronic device 101 optionally stops scrolling the content in response to detecting that the user has looked at the content item without reading, looked elsewhere in the user interface, or looked away from the user interface. In some embodiments, the electronic device 101 also stops scrolling the content item in response to detecting that the user is skimming or skimming the text 724a rather than reading it completely. In some embodiments, the electronic device 101 scrolls the text while the user is skimming or skimming the text. For example, while detecting using the image sensor 314 that a user is scanning or skimming the content item, in response to detecting that the user has scanned or skimmed the first line of text 724a, the electronic device 101 optionally scrolls the text to the position shown in Figure 7D. In summary, in some embodiments, if the user's gaze does not correspond to the user completing reading the line of text 724a, the electronic device 101 stops scrolling the content item.

[0111] In some embodiments, the electronic device 101 scrolls the text in response to detecting that the user has finished reading the first line of text 724a, which includes detecting the user's gaze 716b moving backward (e.g., from right to left for Latin, as shown) along the line 724a that the user just read. In some embodiments, the user looks backward along the line just read before looking at the next line 724a—thus, detecting the user's gaze 716b moving backward along the line of text 724a indicates that the user is about to read the next line of text 724b. In some embodiments, the electronic device 101 times the scrolling of the text 704a so that the second line of text 724b moves to the first position 722a at the same time (e.g., within a time threshold of 0.1, 0.3, 0.5 seconds, etc.) that the user's gaze 716b moves to the beginning of the first position 722a, as shown in FIG. 7D . In this way, the user can read the second line of text 724b in the same location 722a as the first line of text 724a that the user read without having to move their eyes down to see the second location 722a in the user interface.

[0112] In some embodiments, the user skips one or more lines of text while reading the content. For example, assume that the user finishes reading a first line of text 724a and, instead of beginning to read a second line of text 724b, begins to read a portion of text 710a of the content item. In response to detecting, using the user's gaze, that the user is reading portion of text 710a of the content item, the electronic device 101 optionally scrolls portion 710a to position 722a. In some embodiments, the rate at which the electronic device 101 scrolls portion of text 710a to position 722a is the same as or slower than the rate at which the electronic device 101 scrolls line 724b to position 722a (e.g., in response to detecting that the user has finished reading the first line of text 724a). In some embodiments, as the user continues to read the content item from portion 710a, the user's current reading position within the user interface gradually moves to position 722a within the user interface (e.g., partway through after reading the first line of text in portion 710a, another partway through after reading the second line of text in portion 710a, etc.).

[0113] In some embodiments, the electronic device 101 scrolls the image of the content item in response to detecting a user's gaze on the image. Figure 7D illustrates detecting a user's gaze 716d directed at the image 712a of the content item. In response to detecting a user's gaze 716d directed at the image 712a as shown in Figure 7D, the electronic device 101 scrolls the content to display the image 712a at position 722a (e.g., the vertical center or other predetermined portion of the image 712a), as shown in Figure 7E.

[0114] While displaying image 712a at position 722a, the electronic device detects a user's gaze 710b directed toward a portion of text 716e of the content item displayed below image 712a, as shown in Figure 7E. In response, electronic device 101 scrolls text 710b until it overlaps image 712a (e.g., without scrolling image 712a within the user interface), as shown in Figure 7F. In some embodiments, device 101 scrolls text 710b so that the portion of text 710b directed by the user's gaze appears at position 722a within the user interface. In some embodiments, the device scrolls text 710b so that the portion of text 710b directed by the user's gaze appears at another scroll position within the user interface overlaid on image 712a. In some embodiments, rather than scrolling text 710b to the position described with reference to FIG. 7F in response to detecting that the user is looking at text 710b, as shown in FIG. 7E, electronic device 101 scrolls text 710b to the position described with reference to FIG. 7F in response to different criteria, such as in response to detecting that a predetermined amount of time has elapsed since scrolling image 712a to position 722a, as shown in FIG. 7E, or in response to detecting that the user is reading rather than simply looking at text 710b.

[0115] 7F , electronic device 101 uses the user's gaze to detect that the user is reading portion 712a of text superimposed on image 710b. In response to detecting that the user is reading portion 712a of text superimposed on image 710b, electronic device 101 optionally scrolls image 712a up so that text 710b is no longer superimposed on the image. Optionally, electronic device 101 also scrolls text 704a and 710a up (and optionally stops displaying some or all of text 704a and 710a) in a similar manner as it scrolls image 712a up. Scrolling of the content items optionally continues based on the user's gaze, as described with reference to FIGS. 7C-7F .

[0116] As shown in Figures 7C-7F, in some embodiments, electronic device 101 scrolls text vertically. In some embodiments, electronic device 101 scrolls text horizontally as the user reads the text. Additionally, in some embodiments, electronic device 101 moves the text (and / or the entire area 714 in which content items are displayed) to "follow" (e.g., to stay aligned with) the user's head movements and / or gaze. In this manner, for example, electronic device 101 can display content at a predetermined location within the user interface, such as the center of the user interface, at the location where the content item was displayed in content viewing area 700 as shown in Figure 7A, or at another predetermined location.

[0117] In some embodiments, other ways of scrolling content are possible. Scrolling directions optionally include both vertical and horizontal scrolling of text. For example, if a user is reading a digital comic book, a panel of the comic book optionally scrolls vertically and horizontally as needed to display the panel the user is currently reading at a given position within the user interface. In some embodiments, while displaying a digital comic book, electronic device 101 optionally does not display a speech bubble and / or one or more images of the digital comic book until the user has finished reading the currently displayed speech bubble of the comic (e.g., a speech bubble or other portion of the digital comic book immediately preceding or otherwise preceding a hidden portion of the digital comic book that is displayed in response to the user reading a preceding portion of the digital comic book).

[0118] In some embodiments, electronic device 101 presents audio content associated with the text and / or images of the content item, such as music, sound effects, and / or a recording of the text being read aloud. Electronic device 101 optionally plays the audio content at a speed coordinated with the speed at which the user reads the content.

[0119] Also, in some embodiments, electronic device 101 may store an indication of the content the user is reading and / or a timestamp of when the user was reading the content. Later, in response to a request to present an indication of the item of content the user was reading at a particular time, electronic device 101 may optionally present an indication of the content item the user was reading at a particular time. In some embodiments, electronic device 101 may also store the user's reading progress of an item of content, and when the user returns to the item of content, electronic device 101 optionally presents an indication of the portion of the content the user last read and / or scrolls the content to the portion the user last read.

[0120] In some embodiments, while displaying content items in the extended content area 714 and scrolling the content according to the user's gaze, in response to detecting using the image sensor 314 that the user has closed their eyes for a predetermined amount of time (e.g., 0.5 seconds, 1 second, 2 seconds), the electronic device scrolls the content item to the beginning, such as the state of the content item shown in FIG. 7C.

[0121] In some embodiments, in response to detecting a user scanning or skimming a content item, the electronic device 101 optionally displays one or more keywords of the content item with different visual characteristics than the rest of the content item, for example, the keywords are displayed in bold and / or in a different color and / or size than the rest of the text.

[0122] As described with reference to Figures 7C-7F, electronic device 101 scrolls text downward in response to detecting that the user is reading the text. In some embodiments, input other than gaze detection is required to scroll the text upward (e.g., device 101 optionally does not scroll the content item up in response to gaze input). For example, electronic device optionally scrolls text upward in response to detecting input from an input device other than image sensor 314, such as a directional input detected on a touchscreen, mouse, keyboard, motion sensor, or other input device for scrolling a content item up within a user interface. In this way, device 101 can avoid cumbersome and unnecessary forward and backward scrolling that can occur if both upward and downward scrolling were performed based on gaze.

[0123] 7A-7C, in response to detecting that the user is reading portion 704a of the content item, electronic device 101 displays region 714 overlaid on content viewing area 700. As shown in FIG. 7C, region 714 is visually emphasized relative to content viewing area 700 while the user is looking at region 714, for example, by displaying region 714 overlaid on region 700 and / or displaying region 714 with full color, clarity (e.g., no blur), and / or size while displaying content viewing area 700 with dull colors (e.g., reduced saturation), reduced clarity (e.g., blurred), and / or low saturation.

[0124] In some embodiments, electronic device 101 navigates between content viewing area 700 of the user interface and area 714 of the user interface according to the user's gaze and / or head movements. Returning to FIG. 7C , while displaying content items in area 714 superimposed on content viewing area 700, electronic device 101 detects the user's gaze 716c on content viewing area 700. In some embodiments, electronic device 101 also detects user's head movements that meet one or more criteria, such as the user's head movements (e.g., leaning or turning right) forward (relative to the user's body) and / or toward content viewing area 700. In response to detecting the user's gaze 716c and head movements, electronic device 101 begins returning to the content viewing user interface shown in FIG. 7A and / or 7B, as described in more detail below.

[0125] Electronic device 101 optionally animates the transition between displaying region 714 superimposed on content viewing area 700 as shown in FIG. 7C and displaying the content viewing area as shown in FIG. 7B. In some embodiments, the animation progresses at a speed proportional to the amount and / or duration of head movement forward and / or toward content viewing area 700 and / or the duration of the user's gaze on content viewing area 700. FIGS. 7G and 7H show intermediate frames of the animation. In response to detecting the user's head movement forward and / or toward content viewing area 700 and the user's gaze 716c in FIG. 7C, electronic device 101 begins the animation, including displaying the user interface shown in FIG. 7G. As shown in FIG. 7G, electronic device 101 begins de-emphasizing (e.g., darkening and / or blurring) region 714 and / or emphasizing (e.g., brightening and / or increasing clarity) content viewing area 700. In some embodiments, the electronic device 101 modifies other properties of the regions 714 and 700, such as transparency, brightness, saturation, and / or size.

[0126] In response to continued detection of head movement forward and / or toward the content viewing area 700 and the user's gaze 716g on the content viewing area 700, as shown in FIG. 7G, the electronic device 101 progresses through the animation, including displaying the user interface shown in FIG. 7H. As shown in FIG. 7H, the electronic device 101 de-emphasizes (e.g., darkens and / or blurs) the region 714 and / or continues to highlight (e.g., brightens and / or increases clarity) the content viewing area 700. The electronic device 101 also displays the region 714 at a reduced size and displays the content viewing area 700 at an increased size compared to the size at which the regions 714 and 700 were previously displayed in the animation, as shown in FIG. 7G. In some embodiments, the electronic device 101 changes other characteristics of the regions 714 and 700, such as transparency, brightness, saturation, and / or size. In some embodiments, in response to detecting a user's gaze 716c directed toward the content viewing area 700 (as shown in FIG. 7C ), without detecting head movement forward and / or toward the content viewing area 700, the electronic device 101 begins displaying the animations shown in FIGS. 7G and 7H . However, the electronic device optionally does not navigate back to the content viewing area 700, as shown in FIG. 7B , unless both the user's gaze 716c directed toward the content viewing area 700 and a head movement that meets the criteria are detected. In some embodiments, in response to detecting a user's gaze moving back and forth between areas 700 and 714 without the corresponding head movement optionally required to initiate navigation between areas 700 and 714, the device 101 changes only one or more visual characteristics other than the sizes of the two areas, without displaying or animating any changes in the size and / or movement of the two areas.For example, while the user's gaze is directed towards region 714, region 714 is displayed with full clarity, color, etc., and region 700 is displayed with less clarity (e.g., blurred), color (e.g., reduced saturation such as grayscale), etc., and in response to the user's gaze being directed towards region 700 without a corresponding head movement, device 101 displays region 700 with full clarity, color, etc., and region 714 is displayed with less clarity (e.g., blurred), color (e.g., grayscale), etc., and vice versa.

[0127] As shown in FIG. 7H, in response to continuing to detect head movement forward and / or toward the content viewing area 700 and the user's gaze 716h on the content viewing area 700 (e.g., in response to detecting that the user's gaze and head movement meet one or more predetermined criteria), the electronic device 101 continues the animation to display the content viewing area 700 and stops displaying the expanded content area 714, as shown in FIG. 7B. As shown in FIG. 7B, the content viewing area 700 is displayed at full size, full color, full clarity, and / or minimal transparency, and area 714 is no longer displayed. Thus, in response to detecting the user's head movement forward and / or toward the content viewing area 700 and the user's gaze on the content viewing user interface 700, the electronic device 101 navigates from area 714 back to the content viewing area 700.

[0128] 7C , in response to detecting a user's head movement that does not meet the navigation criteria for returning to region 700, or detecting no head movement at all, electronic device 101 continues to display region 714 superimposed on content viewing region 700. Additionally, in response to detecting user 716c's gaze on content viewing region 700 without detecting head movement, electronic device 101 abandons navigation back to content viewing region 700. Furthermore, if the user performs a head movement that meets the navigation criteria (e.g., moving forward or moving toward content viewing region 700) but continues to look at region 714 instead of looking at content viewing region 700, electronic device 101 abandons navigation back to content viewing region 700.

[0129] In some embodiments, electronic device 101 reverses the animation of navigating from region 714 to region 700 and cancels navigation back to content viewing area 700 if the user looks at region 714 while the animation is being displayed. As shown in FIG. 7G , electronic device 101 detects the user's gaze 700 on region 714 while displaying the animation of navigating from region 716f to content viewing area 714. In response to detecting the user's gaze 716f, electronic device 101 optionally stops the navigation back to content viewing area 700 and displays the user interface shown in FIG. 7C . As shown in FIG. 7C , region 714 is displayed with full color, clarity, and / or size and / or minimal transparency, and content viewing area 700 is displayed with a reduced size, darker color, greater blur, and / or greater transparency compared to the appearance of regions 714 and 700 shown in FIG. 7G .

[0130] 7A-7H, the electronic device can navigate between regions of the user interface according to the user's gaze and / or head movements, including ceasing to display text. In some embodiments, some text (e.g., visual indications of notifications 706, text displayed in expanded content area 714, etc.) cannot be dismissed unless it is first read. For example, visual indications of notifications and / or terms and conditions 706 are displayed until electronic device 101 detects (e.g., using image sensor 314) that the user has read the content of those items. In some embodiments, in response to detecting that the user has read the indication of notification 706, the electronic device displays an application associated with the notification.

[0131] In some embodiments, the electronic device 101 displays augmented content related to the portion of the content item in response to detecting that the user is looking at the portion of the content item and has moved their head forward (e.g., relative to the user's body). For example, the augmented content may be a definition of a word included in the content item, a preview of a hyperlink, or an enlarged (e.g., enlarged or uncropped) version of an image.

[0132] For example, referring to Figure 7C, the electronic device 101 detects a user's gaze 716a while the user moves their head forward. As shown in Figure 7C, the user's gaze 716a is located at an individual word 718a, "Lorem," in the content item. In response to detecting the user's gaze 716a on the individual word 718a for a predetermined amount of time and / or moving the user's head for a predetermined amount of time and / or a predetermined distance while the user's gaze 716a is on the individual word 718a, the electronic device 101 displays the word 718a in a modified appearance, such as displaying the word 718a in bold typeface, as shown in Figure 7I.

[0133] In response to continuing to detect the user's gaze 716i on word 718a and forward head movement, as shown in Figure 7I, the electronic device 101 displays augmented content 720a associated with word 718a, as shown in Figure 7J. Thus, in response to the user's gaze and head movement, the electronic device 101 first displays an indication that the electronic device 101 is about to display the augmented content 720 (e.g., by changing the appearance of the word at which the gaze is directed), and then, if the head movement is maintained or continued and the gaze is held on word 718a, the electronic device 101 displays the augmented content 720a.

[0134] In some embodiments, electronic device 101 animates the transition between the user interface shown in Figure 7I and the user interface shown in Figure 7J. The progression of the animation is, optionally, proportional to the duration the user's gaze is on word 718a, the duration the user holds their head forward, or the distance and / or speed the user moves their head forward.

[0135] If the user looks at a different word or image of the content item while moving their head forward, but before displaying the augmented content for word 718a, electronic device 101 optionally displays augmented content associated with the different word or image, including first displaying a visual indication that augmented content is about to be displayed (e.g., by bolding the other word), and then displaying the augmented content if the head movement continues or is maintained and the gaze is held. For example, as shown in FIG. 7K, in response to detecting the user's gaze 716n and head movement toward word 718b, electronic device 101 displays augmented content 720b associated with word 718b.

[0136] In some embodiments, the electronic device 101 does not display enhanced content related to a portion of a content item if the user simply looks at the portion of the content item without performing a head movement. Additionally, the electronic device 101 also ceases displaying an indication that the electronic device 101 is about to display enhanced content (e.g., bolding a word or portion of content) unless a head movement is detected while the user is looking at the portion of the content. For example, referring to FIG. 7C , in response to detecting a user's gaze 716a on a word 718a without detecting a head movement, the electronic device 101 continues to display the user interface of FIG. 7C instead of displaying the user interface of FIG. 7I.

[0137] Also, in some embodiments, if the user looks at a different portion of the content item while the electronic device 101 is displaying the user interface of Figure 7I, the electronic device 101 ceases displaying the augmented content associated with word 718a. Instead, in some embodiments, the electronic device 101 first displays an indication that augmented content associated with another portion of the content item the user is looking at (e.g., bolding another portion of the content item) is about to be displayed, and then, if the user continues to look at that portion of the content item while maintaining or continuing their head movement, the electronic device 101 displays the augmented content associated with the other portion of the content item.

[0138] 7I , in response to detecting that the user is looking at word 718a while moving their head forward, electronic device 101 optionally detects that the user is looking at word 718b. In response to detecting that the user is looking at word 718b, electronic device 101 optionally stops displaying word 718a in bold and instead displays word 718b in bold. Then, in this example, if the user continues to look at word 718b and continues or maintains their head movement forward, electronic device 101 displays augmented content 720b associated with word 718b, as shown in FIG. 7K.

[0139] In some embodiments, as shown in Figures 7J and 7K, the electronic device displays augmented content 720 in the augmented content area 714. In some embodiments, the augmented content 720 is displayed overlaid on or near the word to which the augmented content 720 corresponds. While Figures 7J and 7K show the augmented content 720a displayed in the augmented content area 714 without being overlaid on the content text or image(s), in some embodiments, the augmented content 714 is overlaid on the content item displayed in the augmented content area 714.

[0140] However, in some embodiments, once augmented content corresponding to an individual word is displayed, viewing a different word does not cause electronic device 101 to display other words in bold typeface and / or augmented content for other words. For example, referring to FIG. 7J , if a user views word 718b while augmented content 720a associated with word 718a is displayed, electronic device 101 does not update the user interface to display word 718b in bold typeface. In some embodiments, a user can change the portion of the content in which the augmented content is displayed before the augmented content is displayed, but not afterward.

[0141] 7I, if electronic device 101 detects that the user moves their head back while electronic device 101 is displaying word 718a in bold to indicate that electronic device 101 is about to display augmented content associated with word 718a, electronic device 101 stops the process of displaying the augmented content. In response to detecting the backward head movement, electronic device 101 stops displaying word 718a in bold and returns the user interface to the user interface shown in FIG. 7C.

[0142] 7J , in some embodiments, while the electronic device 101 is displaying the augmented content 720a, the electronic device 101 optionally detects one or more criteria for stopping the display of the augmented content 720a. For example, the electronic device 101 optionally stops displaying the augmented content 720a in response to detecting a user's gaze 716l on the augmented content, and subsequently detects the user's gaze at a different location within the user interface (e.g., location 716j). As another example, if the electronic device 101 detects that the user is looking at the augmented content 720a and then resumes reading the content item text 704a or 710a, the electronic device 101 stops displaying the augmented content 720a. If the user is not looking at the augmented content 720a but is looking at a different location in the user interface, such as looking at location 716j on the image 702a, the electronic device 101 optionally continues displaying the augmented content 720a, allowing the user to view it at a later time.

[0143] However, in some embodiments, if the electronic device 101 detects that the user is reading content (e.g., text 704a or 710a), regardless of whether the user looks at the augmented content 720a, the electronic device 101 stops displaying the augmented content 720a. Also, in some embodiments, the electronic device 101 stops displaying the augmented content 720a in response to detecting that the user has closed their eyes for a predetermined period of time (e.g., 0.5 seconds, 1 second, 2 seconds, etc.) that is longer than a typical blink. The electronic device 101 optionally stops displaying the augmented content 720a by animating the augmented content 720a fading or fading away. In some embodiments, if the user looks at the augmented content 720a as it fades or fades away but before it is completely hidden, the electronic device 101 restores the user interface to display the augmented content 720a, as shown in FIG. 7J.

[0144] Additional explanation regarding FIGS. 7A-7K is provided below with reference to methods 800-1100 described with respect to FIGS. 8-11 below.

[0145] 8A-8G are flowcharts illustrating a method 800 for enhancing an item of content according to detection of a user's gaze, according to some embodiments. In some embodiments, method 800 is performed on a computer system (e.g., computer system 101 of FIG. 1). The display generation component (e.g., display generation component 120 of FIGS. 1, 3, and 4) includes a display (e.g., a heads-up display, a display, a touchscreen, a projector, etc.) and one or more cameras (e.g., cameras (e.g., color sensors, infrared sensors, and other depth-sensing cameras)) facing forward from a user's hand or a user's head. In some embodiments, method 800 is performed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of computer system 101 (e.g., control unit 110 of FIG. 1A). Some operations of method 800 are optionally combined and / or the order of some operations is optionally changed.

[0146] In method 800, in some embodiments, an electronic device (e.g., computer system 101 of FIG. 1 ) in communication with a display generation component and an eye-tracking device (e.g., 314) (e.g., a mobile device (e.g., tablet, smartphone, media player), computer (e.g., desktop computer, laptop computer), or wearable device (e.g., watch, head-worn device) displays (802), via the display generation component (e.g., 120), a user interface including a first content area (e.g., 700), the first content area including a first portion (e.g., 704a) of a content item, as shown in FIG. 7A (e.g., the content item includes textual content and the first portion of the content item is the first portion of the textual content).

[0147] In some embodiments, while displaying the user interface, the electronic device (e.g., 101) determines (804) via an eye-tracking device (e.g., 314) that a user of the electronic device's gaze (e.g., 716p) is directed toward a first portion (e.g., 704a) of the content item, as in FIG. 7B (e.g., the eye-tracking device optionally captures one or more images of the user's eyes and detects pupils and glints in the captured one or more images to track the user's gaze, as described in more detail above with reference to FIG. 6).

[0148] In some embodiments, in response to determining (806) that a user's gaze (e.g., 716p) is directed toward a first portion (e.g., 704a) of a content item, such as in FIG. 7B , and in response to determining (808) that movement of the user's gaze (e.g., 716p) indicates that the user is reading the first portion (e.g., 704a) of the content item (e.g., 704a) (e.g., the electronic device can determine whether the user is simply looking at the first portion of the content item or whether the user is actually reading the first portion of the content item based on the detected movement of the user's eyes), the electronic device (e.g., 101) updates (810) the user interface (e.g., expands the area of ​​the user interface in which the content item is displayed) to include a second content area (e.g., 714), such as in FIG. 7C , the second content area including the first portion (e.g., 704a) of the content item and a second portion (e.g., 710a) of the content item that was not displayed in the first content area (e.g., 700).

[0149] In some embodiments, following a determination (812) that the user's gaze (e.g., 716p) is directed toward the first portion of the content item (e.g., 704a) without moving in a manner that indicates the user is reading the first portion of the content item (e.g., 704a) shown in FIG. 7B (e.g., the electronic device determines, based on one or more images of the user's eyes, that the user's eye movements are not consistent with reading), the electronic device (e.g., 101) ceases updating the user interface to include the second content region (e.g., 714) shown in FIG. 7C (e.g., the electronic device continues to display the user interface with the first content region).

[0150] In some embodiments, the display generation component is a display integrated with the electronic device (optionally a touchscreen) and / or an external display such as a monitor, projector, television, etc. In some embodiments, the gaze tracking device includes one or more camera(s) (e.g., IR camera, NIR camera, visible light camera, etc.).

[0151] In some embodiments, the user interface includes portions of multiple content items. For example, the user interface is a digital newspaper user interface that includes a portion of a digital newspaper article headline and / or one or more articles. As another example, the user interface is an email inbox user interface, and the content item is an email preview that includes an indication of the sender of the email, an indication of the subject of the email, and / or a text preview of the email message. In some embodiments, the content item includes non-text content, such as one or more images.

[0152] The electronic device optionally compares one or more captured images of the user's eyes to determine whether the user's eye movements are consistent with movements consistent with the readings.

[0153] For example, in response to detecting that a user is reading a first portion of the text of an article, the electronic device begins displaying a second portion of the text of the article. In some embodiments, the first and second portions of the content item are displayed overlaid on a user interface including the first content area. In some embodiments, in response to detecting that a user is reading the first portion of the content before displaying the second content area, the electronic device begins animating the gradual display of the second content area. For example, the electronic device begins displaying a portion of the second portion of the content or displays the second portion of the content initially overlaid on the user interface with a transparent appearance that gradually becomes opaque. In response to detecting that a user moves their eyes away from the first content item while animating the gradual display of the second content area, the electronic device optionally stops displaying the second portion of the content and resumes displaying the first content area. In some embodiments, in response to detecting that a user continues to read the content item, the electronic device fully transitions to displaying the second content area including the first and second portions of the content item, such as by displaying an entire portion of the second portion of the content item or by displaying a fully opaque second content area. In some embodiments, updating of the user interface is performed in response to data collected by the gaze tracking component without receiving additional input from another input device in communication with the electronic device.

[0154] The above method of displaying a second content area along with the first and second portions of a content item in response to detecting that a user is reading the first portion of the content item allows the electronic device to expand the content item without requiring any user input beyond tracking the user's gaze, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by only expanding content when the user is actually reading it), allowing users to use the electronic device more quickly and efficiently by reducing user error, etc., thereby further reducing power usage and improving the battery life of the electronic device.

[0155] In some embodiments, such as in FIG. 7B , the user interface further includes a third content area, where the third content area includes a first portion (e.g., 704b) of a second content item (816) (e.g., in some embodiments, the second content item includes text content). The second content item is, optionally, different from the first content item. For example, the first content item and the second content item are articles in a digital newspaper.

[0156] In some embodiments, while displaying the user interface, the electronic device determines (818) via an eye-tracking device (e.g., 314) that the gaze of a user of the electronic device is directed toward a first portion (e.g., 704b) of the second content item (e.g., the eye-tracking device optionally captures one or more images of the user's eyes and detects pupils and glints in the captured one or more images to track the user's gaze, as described in more detail with reference to FIG. 6).

[0157] In some embodiments, in response to determining that the user's gaze is directed toward the first portion (e.g., 704b) of the second content item (820), and in response to determining that the user's gaze movement indicates that the user is reading the first portion (e.g., 704b) of the second content item (822) (e.g., the electronic device may determine that the user is reading the first portion of the second content item based on the user's eye movement(s) while the user is looking at the first portion of the second content item), the electronic device (e.g., 101) updates the user interface (824) to include a fourth content area that includes the first portion (e.g., 704b) of the second content item and the second portion of the second content item that was not displayed in the third content area, without displaying the second content area in a manner similar to the way the electronic device (e.g., 101) displays the content area (e.g., 714), as shown in FIG.

[0158] In some embodiments, in response to detecting that the user is reading a first portion of the second content item displayed in the third content area, the electronic device displays an additional portion of the second content item. Thus, while the electronic device is displaying multiple content items (e.g., the first content item and the second content item), the electronic device detects that the user has started reading one of the content items and, in response, expands the content item that the user is reading without expanding the content item that the user is not reading. In some embodiments, pursuant to determining that the user is not viewing the first content item or the second content item (or any other content item displayed in the first content area), the electronic device ceases updating the user interface to include separate areas having the additional portion(s) of the respective content items.

[0159] The above method of displaying a second portion of a second content item in response to detecting that a user is reading a second content item, and displaying a second portion of a first content item in response to detecting that a user is reading a first content item, enables an electronic device to provide an efficient way of presenting additional portions of a content item that a user is reading, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by accepting detection of user reading as input to expand the content item instead of requiring additional input to expand the content), and enabling the user to use the electronic device more quickly and efficiently, thereby further reducing power usage and improving battery life of the electronic device.

[0160] In some embodiments, before determining that the user's gaze (e.g., 716p) of the electronic device (e.g., 101) is directed toward the first portion (e.g., 704a) of the content item, the electronic device (e.g., 101), via the display generation component 120, displays (826) one or more other content items with visual characteristics having a first value, as in FIG. 7B (e.g., the electronic device displays multiple portions of the content item while displaying the first portion of the first content item in the first content area).

[0161] In some embodiments, in response to determining 828 that movement of the user's gaze (e.g., 716p) indicates that the user is reading a first portion (e.g., 704a) of the content item, and in response to determining 830 that the user's gaze (e.g., 716p) is directed toward the first portion (e.g., 704b) of the content item, as in Figure 7C, the electronic device (e.g., 101) updates 832 the user interface to display one or more other content items with visual characteristics having a second value different from the first value, as in Figure 7C. In some embodiments, while displaying the first content item within the second content area, the electronic device displays other (e.g., non-enhanced) content items with a reduced size, reduced opacity, darker or less contrast or saturated colors, and / or reduced clarity (e.g., more blur).

[0162] The electronic device optionally displays the plurality of content items with full size, full opacity, full color, and / or full clarity before expanding the content item that the user is reading. The first content item in the second content area is optionally displayed with full opacity, full color, and / or full clarity. In some embodiments, pursuant to a determination that the user is not reading the first portion of the content items and / or that the user is not looking at the first portion of the content items, the electronic device continues to display the one or more content items with the visual characteristic having the first value.

[0163] The above method of modifying the values ​​of visual characteristics at which one or more other content items are displayed in response to detecting that a user is reading a content item enables the electronic device to reduce visual clutter while the user is reading a content item in the second content area, reducing distractions to read the enhanced content item, thereby simplifying interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient, and allowing the user to use the electronic device more quickly and efficiently, thereby further reducing power usage and improving battery life of the electronic device.

[0164] 7B-7C, the visual characteristic is transparency (e.g., opacity and / or clarity), and the first value corresponds to a lower transparency than the second value (834). In some embodiments, while the electronic device displays the first content item in the second content area, the opacity and / or clarity of other content items is reduced (e.g., blurred).

[0165] The above-described method of increasing the transparency of one or more other content items in response to detecting that a user is reading a content item allows the electronic device to continue displaying the other content items in a manner that does not distract the user from the content item being read, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by allowing the user to view the other content items without discarding the second content area), and allowing the user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0166] 7B-7C, the visual characteristic is size, and the first value corresponds to a size that is larger than the second value (836). In some embodiments, the electronic device reduces the size of other content items while the electronic device displays the first content item in the second content area. In some embodiments, the content in the second content area is displayed at the same size as the original content in the first content area.

[0167] The above-described method of reducing the size of other content items in response to detecting that a user is reading a content item allows the electronic device to continue to display other content items in a manner that is less distracting to the user while the user is reading the content item, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by allowing content to be read more quickly and for the user to view other content items without navigating away from the second content area), and allowing the user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0168] In some embodiments, the user interface further includes a third content area, as shown in Figure 7B, where the third content area includes a first portion (e.g., 704b) of the second content item (838) (e.g., the third content area is displayed simultaneously with the first content area). The second content item optionally includes text content.

[0169] 7C , updating the user interface to include the second content region (e.g., 714) includes displaying the second content region (e.g., 714) overlaid on the third content region (840). In some embodiments, before displaying the second content region, the electronic device displays a content browsing user interface including portions of multiple content items including a first portion of a first content item and a first portion of a second content item. The electronic device optionally displays the second content region including the first and second portions of the first content item overlaid on the content browsing user interface.

[0170] The above-described method of displaying the second content area superimposed on the third content area allows the electronic device to continue to display the third content area in a manner that is less visually distracting to the user while the user is reading the content item, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by allowing the user to read the content item more quickly while continuing to look at the third content area without navigating away from the second content area), thereby allowing the user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0171] In some embodiments, before determining that the user's eye movement indicates that the user of the electronic device is reading the first portion (e.g., 704a) of the content item, the first content region (e.g., 700) and the third content region are displayed at a first z-height in the user interface (e.g., a top height such that no other content overlays the first and third content regions), as in FIG. 7B , and updating the user interface to include the second content region (e.g., 714) includes displaying the second content region (e.g., 714) at the first z-height in the user interface and changing the z-height of the third content region to a second z-height in the user interface (842), as in FIG. 7C . In some embodiments, the second z-height is lower than the first z-height (e.g., further back in the user interface). The electronic device optionally displays the second content region at the same z-height at which the first content region was displayed, overlaid on the third content region, in response to detecting that the user is reading the content. In some embodiments, the electronic device also displays a second content region superimposed on the first content region and moves the first content region to a lower z-height.

[0172] The above-described method of displaying the first content region at the same z-level as the third content region and the second content region at a higher z-level than the third content region in response to detecting that the user is reading a content item allows the electronic device to continue to display the third content region in a manner that is less visually distracting to the user while the user is reading the content item, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by allowing the user to read the content item more quickly while continuing to look at the third content region without navigating away from the second content region), thereby allowing the user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0173] In some embodiments, such as in FIG. 7C , updating the user interface to include the second content region (e.g., 714) includes expanding the size of the first content region (844) to become the second content region (e.g., 714). In some embodiments, the content region expands in one or more dimensions. For example, the second content region is longer than the first content region, allowing the electronic device to display more of the first content item in the second content region than in the first content region. In some embodiments, the height and width of the content region are expanded. In some embodiments, the second content region is displayed at the same z-level in the user interface as the first content region.

[0174] The above-described method of expanding a first content area to become a second content area enables the electronic device to display additional portions of a content item in the second content area while maintaining visual continuity, thereby simplifying interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by providing clear navigation continuity between the first content area and the second content area), thereby enabling the user to use the electronic device more quickly and efficiently, further reducing power usage and improving battery life of the electronic device.

[0175] In some embodiments, updating the user interface to include the second content area (e.g., 714) includes gradually animating the first content area to change into the second content area (e.g., 714) while the user continues to read the first portion of the content item (e.g., 704a). In some embodiments, the progression of the animation is proportional to the amount of time the user is continuously reading the first content item. For example, the user begins reading the first portion of the first content item, the animation begins, and the animation continues to progress as the user continues to read. In some embodiments, the animation has a predetermined duration after starting. Animating the first content area to change into the second content area optionally includes animating the progression of one or more differences between the first content area and the second content area. For example, the size of the first content area gradually increases so that the second content area is larger than the first content area. In some embodiments, the animation stops or reverses when the user stops reading the content item.

[0176] The above method of gradually animating a first content area changing into a second content area as the user continues to read the content item allows the electronic device to provide feedback to the user that the first content area is changing into the second content area as they read a first portion of the content item, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by giving the user an opportunity to stop the transition, e.g., by stopping reading the content item and / or looking away from the content item), and allowing the user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0177] In some embodiments, during or after updating the user interface to include the second content area (e.g., 714), the electronic device (e.g., 101) determines via its eye-tracking device (e.g., 314) that the gaze (e.g., 716c) of the user of the electronic device is no longer directed at the content item (848), as in FIG. 7C (e.g., the electronic device detects that the user is looking at a portion of the user interface that does not include the content item (e.g., an area other than the second content area) or that the user is looking away from the user interface). For example, the user is looking at a different content item with the first content item overlaid on it.

[0178] In some embodiments, in response to determining that the user's gaze (e.g., 716c) of the electronic device (e.g., 101) is no longer directed at the content item, as in FIG. 7C, the electronic device (e.g., 101) displays (850) a user interface including the first content area (e.g., 700) without displaying the second content area (e.g., 714), as in FIG. 7B.

[0179] For example, a first content area is displayed in a content browsing user interface that includes a first portion of the content item as well as portions of other content items, and in response to detecting the user looking away from the content item, the electronic device re-displays the content browsing user interface at its initial size, z-level, and / or appearance. Thus, the electronic device optionally displays a portion of the content item in the first content area (e.g., the first portion of the content item or the portion of the content item that the user last read before looking away from the content item) instead of displaying most of the content item in the second area.

[0180] The above method of returning the user interface to include the first content area without displaying the second content area in response to detecting that the user is no longer looking at the content item allows the electronic device to return to the first content area without additional input in response to the user's gaze, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by reducing the number of inputs required by the user to request to re-view the first content area), thereby allowing the user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0181] In some embodiments, during or after updating the user interface to include the second content area (e.g., 314), the electronic device (e.g., 101) determines via the eye tracking device (e.g., 314) that the user's eye movements no longer indicate that the user of the electronic device is reading the content item (852), as in FIG. 7C (e.g., the user continues to look at the content item, the user looks away from the content item, such as looking at a different area of ​​the user interface (e.g., an area other than the second content area), or looking away from the user interface).

[0182] In some embodiments, in response to determining that the user's eye movements no longer indicate that the user of the electronic device (e.g., 101) is reading the content item, the electronic device (e.g., 101) restores (854) the user interface to include the first content area (e.g., 700) without displaying the second content area, as in FIG. 7B . For example, the first content area is displayed in a content browsing user interface that includes portions of other content items in addition to the first portion of the content item, and in response to detecting that the user is no longer reading the content item, the electronic device redisplays the content browsing user interface with its initial size, z-level, and / or appearance. Thus, the electronic device optionally displays a portion of the content item in the first content area (e.g., the first portion of the content item or the portion of the content item that the user last read before stopping reading the content item) instead of displaying most of the content item in the second area.

[0183] The above method of returning the user interface to include the first content area without displaying the second content area in response to detecting that the user is no longer reading the content item allows the electronic device to return to the first content area without additional input in response to the user's gaze, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by reducing the number of inputs required by the user to request to re-view the first content area), thereby allowing the user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0184] 7B , determining that the movement of the user's gaze (e.g., 716p) indicates that the user is reading the first portion (e.g., 704a) of the content item is based on detecting, using an eye-tracking device (e.g., 314), the user's eye movements corresponding to reading text (856). In some embodiments, the electronic device detects that the user is reading the content by detecting that the user's gaze moves along a line of text in a direction that reads the language of the text (e.g., from left to right for English, right to left for Hebrew, etc.) and then moves quickly in the opposite direction (e.g., faster than the direction of the language of the text) along the line the user just read before looking down to the next line. Optionally, the electronic device detects that the user is reading even if the user does not directly look at every word or every character in the line of text, as people optionally read without directly looking at each character or even each word in a line of text. In some embodiments, the electronic device can detect that the user has finished reading a line of text by detecting the user's gaze movement in the opposite direction along the line of text that the user was just reading.

[0185] The above-described method of detecting that a user is reading content based on the user's eye movements allows the electronic device to use the determination that the user is reading to navigate a user interface without other input, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by reducing the inputs required to navigate the user interface), thereby allowing the user to use the electronic device more quickly and efficiently, further reducing power usage and improving battery life of the electronic device.

[0186] In some embodiments, the electronic device (e.g., 101) detects (858) an indication (e.g., 708) of a notification event (e.g., an alert generated by an application accessible to the electronic device) at the electronic device (e.g., 101), as shown in Figure 7A. For example, receiving a message from a user account other than the user account associated with the electronic device is one type of notification event, as is receiving a phone or video call, etc.

[0187] In some embodiments, such as in FIG. 7A , in response to detecting (860) an indication (e.g., 708) of the notification event and in accordance with determining that the user's eye movements indicate that the user is not reading the content item in the user interface (e.g., the electronic device uses an eye tracking device to determine that the user is not reading any of the content presented by the display generation component; in some embodiments, the user is simply looking at the content in the user interface without reading the content), the electronic device (e.g., 101) generates (862) an indication (e.g., 706) (e.g., audio, visual, etc.) of the notification event (e.g., the indication includes text, sound, and / or an image including haptic feedback associated with the notification event) at the electronic device (e.g., 101).

[0188] 7C , in accordance with a determination that the user's eye movement (e.g., 716b) indicates that the user is reading a content item within the user interface, the electronic device (e.g., 101) ceases generating indications of notification events (864). In some embodiments, no indications of notification events are presented while the user is reading. In response to detecting (e.g., via an eye-tracking component) that the user has stopped reading, the electronic device optionally presents indications of notification events detected while the user was reading (e.g., all indications previously suppressed, the last indication previously suppressed, etc.).

[0189] The above method of ceasing to present notification indications while a user is reading content so that the electronic device provides content without distraction simplifies the interaction between the user and the electronic device, improves usability of the electronic device, makes the user device interface more efficient (e.g., by allowing the user to read content in a shorter amount of time), and enables the user to use the electronic device more quickly and efficiently, thereby further reducing power usage and improving the battery life of the electronic device.

[0190] In some embodiments, in response to determining (866) that the user's gaze is directed (e.g., via an eye-tracking device) toward a first portion of the content item (e.g., 704a), and in response to determining (868) that movement of the user's gaze (e.g., 716p), as in FIG. 7B (e.g., via an eye-tracking device), indicates that the user is reading the first portion of the content item, the electronic device (e.g., 101) moves (870) a predetermined portion of the second content area from a first position in the user interface to a second position in the user interface as the user continues to read the content item.

[0191] In some embodiments, the electronic device moves the second content area (e.g., the vertical and / or horizontal center of the second content area) to the center (e.g., the vertical and / or horizontal center) of the display area of ​​the display generation component. The predetermined portion of the second content area optionally includes text, images, or other content of the content item. Thus, in some embodiments, the first content area is displayed at a position other than the second position within the user interface (e.g., displayed at the first position within the user interface), the second content area is displayed via an extension of the first content area at the first position within the user interface, and as the user continues to read the content item, the device shifts the user interface / display such that the second content area moves to the second position within the user interface. In some embodiments, pursuant to a determination that the user is viewing the first portion of the content item without reading the first portion of the content item, the electronic device ceases moving the second content area.

[0192] The above-described method of moving a predetermined portion of the second content area to a second position within the user interface as the user continues to read the content item enables the electronic device to display the content item in an ergonomic position within the user interface, making it easier for the user to read the content, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by allowing the user to read the content faster), thereby further reducing power usage and improving the battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently.

[0193] In some embodiments, moving the second content area from a first position in the user interface to a second position in the user interface (872) includes: moving the second content area a first portion of a movement from the first position to the second position (874) in response to detecting that the user has read a first portion of the content item (e.g., initially moving the second content area a first amount in response to detecting that the user has read a first portion of the first portion of the content item); and, after moving the second content area a first portion of a movement from the first position to the second position, moving the second content area a second portion of a movement from the first position to the second position in response to detecting that the user has read a second portion of the content item (e.g., moving the second content area a second amount in response to detecting that the user has read a second portion of the first portion of the content item) (876).

[0194] In some embodiments, the electronic device moves the second content region by an amount proportional to the amount of the first content item read by the user or the amount of time the user spent reading the first portion of the first content item. In some embodiments, moving the content region includes moving a boundary of the content region. In some embodiments, moving the content region includes moving content within the content region without moving a boundary of the content region itself.

[0195] The above-described method of moving the second content area to a second position within the user interface by an amount proportional to the amount of content read by the user allows the electronic device to efficiently move the second content area with visual continuity, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, and makes the user device interface more efficient (e.g., by reducing input, the user can attempt to cause the second content area to move to the second position within the user interface), thereby further reducing power usage and improving battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently.

[0196] In some embodiments, while the user's gaze is directed at the first portion of the content item and before the second content area is displayed, the first portion of the content item is displayed at a first position within the user interface, and updating the user interface to include the second content area includes maintaining the display of the first portion of the content item in the second content area at the first position within the user interface (878). In some embodiments, the position of the second content area is selected to give the appearance that the first portion of the content item is not moving. The first portion of the content item is optionally displayed at the same position in the display generation component, or optionally displayed to follow the user's head movement or gaze.

[0197] The above-described method of displaying a first portion of a content item at the same position in the user interface in the first content area and the second content area allows the electronic device to display the first portion of the content item without moving the first portion of the content item, thus maintaining visual continuity and making the content item easier to read for the user, which simplifies the interaction between the user and the electronic device, improves the usability of the electronic device, makes the user device interface more efficient (e.g., by allowing the user to read the content item faster), and further reduces power usage and improves the battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently.

[0198] In some embodiments, a first portion of a content item in a first content area (e.g., 700), such as in FIG. 7B, occupies an area of ​​the user interface having a first size, and a first portion of a content item in a second content area (e.g., 714), such as in FIG. 7C, occupies an area of ​​the user interface having a first size (880). In some embodiments, the size of the content remains the same when transitioning from displaying in the first content area to displaying in the second content area. In some embodiments, the width of the content remains the same, but the length changes (e.g., to display more content vertically). For example, the second portion of the content is displayed below the first portion of the content, making the content longer. In some embodiments, the width and length of the content change when transitioning from being displayed in the first content area to being displayed in the second content area.

[0199] The above-described method of displaying a first portion of a first content item at the same size in a first content area and a second content area allows the electronic device to continue displaying the first portion of the content item in a manner that is easy for the user to read while transitioning the user interface from including the first content area to displaying the second content area, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by allowing the user to read content faster), and further reducing power usage and improving battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently.

[0200] 9A-9L are flowcharts illustrating a method 900 for scrolling text of a content item pursuant to a determination that a user is reading the content item, according to some embodiments. In some embodiments, method 900 is performed on a computer system (e.g., computer system 101 of FIG. 1). The display generation component (e.g., display generation component 120 of FIGS. 1, 3, and 4) includes (e.g., a heads-up display, a display, a touchscreen, a projector, etc.) and one or more cameras (e.g., cameras (e.g., color sensors, infrared sensors, and other depth-sensing cameras)) facing forward from a user's hand or a user's head. In some embodiments, method 900 is performed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of computer system 101 (e.g., control unit 110 of FIG. 1A). Some operations of method 900 are optionally combined and / or the order of some operations is optionally changed.

[0201] In method 900, in some embodiments, an electronic device (e.g., 101) (e.g., computing system (e.g., 101)) in communication with a display generation component (e.g., 120) and an eye-tracking device (e.g., 314) (e.g., a mobile device (e.g., tablet, smartphone, media player), computer (e.g., desktop computer, laptop computer), or wearable device (e.g., watch, head-worn device)) may, via the display generation component (e.g., 120), generate a visual image, as shown in FIG. 7C , A user interface is displayed (902) including a first portion of text of the content item (e.g., 724a) and a second portion of text of the content item (e.g., 724b), where the first portion of text (e.g., 724a) is displayed in a first position (e.g., 722a) of the user interface and the second portion of text (e.g., 724b) is displayed in a second position (e.g., 722b) of the user interface (e.g., the first portion of text is a first line of text and the second portion of text is a second line of text below the first line of text).

[0202] In some embodiments, while displaying the user interface, the electronic device determines, via an eye-tracking device, that the user's eye movements indicate that the user is reading a first portion of text (e.g., the eye-tracking device optionally captures one or more images of the user's eyes and detects pupils and glints in the captured image or images to determine the position of the user's gaze, as described in more detail above with reference to FIG. 6 ) (904).

[0203] In some embodiments, while movement of the user's gaze (e.g., 716b) indicates that the user is reading a first portion of text (e.g., 724a) (906) (e.g., while the electronic device continues to detect eye movements consistent with reading the first portion of text), pursuant to a determination that movement of the user's gaze (e.g., 716b) indicates that the user has finished reading the first portion of text (e.g., 724a), as in FIG. 7C (e.g., detecting that the user has read to the end of the first portion of text and is beginning to look at a second portion of text), the electronic device (e.g., 101) scrolls the text (910) such that the second portion of text (e.g., 724b) moves in a first direction (e.g., toward the location where the user's gaze is directed, toward the location previously occupied by the first portion of text, or toward a predetermined portion of the user interface, such as the center of the user interface) (e.g., scrolling the second portion of text to become the current scroll position within the user interface), as in FIG. 7D.

[0204] In some embodiments, following a determination that the user's eye movement indicates that the user has not finished reading the first portion of the text (e.g., 724a) of FIG. 7C (e.g., the user is still reading the first portion of the text or has looked away from the first portion of the text without finishing it), the electronic device (e.g., 101) maintains (914) the first portion of the text (e.g., 724a) at the first position (e.g., 722a) as in FIG. 7C (e.g., stops scrolling the user interface).

[0205] In some embodiments, the display generation component is a display integrated with the electronic device (optionally a touchscreen) and / or an external display such as a monitor, projector, television, etc. In some embodiments, the gaze tracking device includes one or more camera(s) (e.g., IR camera, NIR camera, visible light camera, etc.).

[0206] For example, the text is vertically scrollable. In some embodiments, the first portion of the text is a first section of the line of text, the second portion of the text is a second section of the line of text that appears to the right (e.g., for languages ​​that read left-to-right) or left (e.g., for languages ​​that read right-to-left) of the first section of the line of text, and the text is horizontally scrollable. The current scroll position within the user interface is, optionally, a fixed position within the user interface relative to the rest of the user interface (e.g., a fixed position within a window of the user interface). In some embodiments, the current scroll position is selected for ergonomic convenience, such as a position that the user can see by looking straight ahead (e.g., while allowing the eyes to look straight ahead at a neutral height without bending the neck). In some embodiments, the electronic device repositions the current scroll position in response to detecting movement of the user's head relative to the user interface. For example, if a user is reading content on a monitor coupled to a desktop computer and moves their head (or moves their torso or body to move their head) relative to the monitor, the current scroll position optionally moves with the user's head movement. In some embodiments, the location of the first portion of text is selected to be ergonomically friendly for the user based on detecting the position and orientation of the user's head, e.g., the first portion of text is displayed in a position that the user can see while their head and / or eyes are in a neutral position.

[0207] Furthermore, in some embodiments, the electronic device may detect, based on the detected movements of the user's eyes, whether the user is simply looking at the first portion of the text or whether the user is reading the first portion of the text item. The electronic device optionally compares one or more captured images of the user's eyes to determine whether the movements of the user's eyes are consistent with movements consistent with reading.

[0208] For example, detecting that a user is reading an entire line of text and then moving their eyes to look at the beginning of a second line of text.

[0209] For example, the electronic device scrolls the text vertically to display the second portion of the text at the height at which the first portion of the text was previously displayed. As another example, the electronic device scrolls the text horizontally to display the second portion of the text at the horizontal position at which the first portion of the text was previously displayed. Updating the second scroll position of the second portion of the text optionally includes updating the position of the first portion of the text (e.g., moving the first portion of the text vertically or horizontally to make space for the second portion of the text) or ceasing display of the first portion of the text. In some embodiments, updating the user scroll position is performed in response to data collected by the eye-tracking component without receiving additional input from another input device in communication with the electronic device.

[0210] In some embodiments, the electronic device continues to display the first and second portions of text in their respective positions within the user interface.

[0211] The above method of updating the scroll position of a content item in response to detecting that a user has finished reading a first portion of text allows the electronic device to scroll at the rate at which the user is reading, without the user receiving input from input devices other than the eye-tracking device, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by reducing user errors in scrolling too much or too little and avoiding the need for input from other input devices), thereby further reducing power usage and improving battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently and / or by reducing user input errors.

[0212] In some embodiments, while displaying the user interface, the electronic device (e.g., 101) determines (916) via the eye tracking device (e.g., 314) that the user's gaze is directed at the first portion of the text (e.g., 724a) or the second portion of the text (e.g., 724b) shown in FIG. 7C, but the user's eye movements indicate that the user is not reading the first portion of the text or the second portion of the text (e.g., the user's eye movements do not correspond to eye movements indicative of reading).

[0213] In some embodiments, in response to determining that the user's gaze is directed at the first portion of the text (e.g., 724a) or the second portion of the text (e.g., 724b) but the user's gaze movement indicates that the user is not reading the first portion of the text or the second portion of the text shown in Figure 7C, the electronic device (e.g., 101) stops scrolling the text (918). In some embodiments, in response to detecting that the user is reading the text, the electronic device scrolls the text, and in response to detecting that the user is looking at the text without reading it, the electronic device stops scrolling the text (e.g., the text remains at its current position in the user interface).

[0214] The above-described method of ceasing scrolling of text in response to detecting that a user is looking at text without reading it enables the electronic device to avoid unnecessary scrolling, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, and makes the user device interface more efficient (e.g., by reducing the need for input to undo unnecessary scrolling), thereby allowing the user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0215] In some embodiments, while displaying the user interface, the electronic device (e.g., 101) determines (920) via the eye-tracking device (e.g., 314) that the user's gaze is no longer directed at the first portion (e.g., 724a) of the text (e.g., 724a) shown in FIG. 7C (e.g., the user begins looking at a second portion of the text, or looks at any other portion of the user interface, or looks away from the user interface, or looks at a different portion of the user interface).

[0216] 7C (e.g., 724a) pursuant to a determination detected based on the user's gaze movement over the first portion of the text (e.g., 724a) that the user is skimming the first portion of the text (e.g., 724a) without reading the first portion of the text (e.g., 724a), the electronic device (e.g., 101) stops scrolling the text (924) (e.g., continues to display the first and second portions of the text in the same location within the user interface). In some embodiments, while the user is skimming (or scanning) the text, the user's eye movements do not correspond to movements indicative of reading the text because people typically skim text differently than they read text.

[0217] 7C , the electronic device (e.g., 101) scrolls the text (926) such that a second portion of the text (e.g., 724b) moves in a first direction in accordance with a determination detected based on movement of the user's gaze (e.g., 716b) over a first portion of the text (e.g., 724a) that the user has read the first portion of the text (e.g., 724a). In some embodiments, the electronic device scrolls the text in response to detecting that the user is reading the text and ceases scrolling the text in response to detecting that the user is skimming or scanning the text.

[0218] The above-described method of ceasing scrolling of text in response to detecting that a user is skimming the text enables the electronic device to allow the user to control the scrolling speed of the text, which may be erratic while the user is skimming the text, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by providing an intuitive way to scroll text in different situations to reduce scrolling errors and the input required to correct errors), and further reducing power usage and improving the battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently.

[0219] In some embodiments, the electronic device (e.g., 101) determines (928) via the eye tracking device (e.g., 314) that the user is skimming the first portion of the text (e.g., 724a) based on the user's eye movement over the detected first portion of the text (e.g., 724a) shown in FIG. 7C (e.g., while the user is skimming the text, the user's eyes move across the text but do not coincide with an eye movement pattern indicative of reading).

[0220] In some embodiments, in response to determining that the user has finished skimming the first portion of the text (e.g., 724a), the electronic device (e.g., 101) scrolls (930) the text such that the second portion of the text (e.g., 724b) moves in the first direction, as in FIGS. 7C-7D. In some embodiments, the electronic device scrolls the text in response to detecting that the user is either reading or skimming the text. In some embodiments, in response to determining that the user has not yet skimmed the first portion of the text, the electronic device ceases scrolling the text.

[0221] The above-described method of scrolling text in response to reading or scanning allows an electronic device to efficiently scroll text as a user is scanning without requiring any user input other than gaze position, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by reducing the input required to scroll text), and further reducing power usage and improving battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently.

[0222] In some embodiments, scrolling the text so that the second portion of the text (e.g., 724b) shown in FIG. 7C moves in the first direction is in response to detecting that the user has reached the end of the first portion of the text (e.g., 724a) and is moving their eyes in a direction toward the beginning of the second portion of the text (e.g., 724b) (932).

[0223] For example, the beginning of the second portion of text is the first word of a (e.g., next) line of text. In some embodiments, the electronic device detects eye movements corresponding to the user reading the text. The eye movements corresponding to reading optionally include eye movements in the direction of the text (e.g., from left to right in English) where the user looks at words or parts of words as they read the line of text, followed by eye movements in the opposite direction of the text (e.g., from right to left in English) along the line the user just read, and then down to the beginning of the next line. In some embodiments, scrolling is performed in response to detecting the user's eye movements in the opposite direction of the text along the line the user just read, and / or in response to detecting the user's eye movements down to the beginning of the next line of text.

[0224] The above-described method of scrolling text in response to detecting eye movement towards the beginning of a second portion of the text allows the electronic device to scroll the text at a time while the user is not actively reading one or more words of the text, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by scrolling in a way that does not disorient the user or cause the user to lose their place in the content), thereby allowing the user to use the electronic device more quickly and efficiently, such as by allowing the user to read content more quickly, thereby further reducing power usage and improving the battery life of the electronic device.

[0225] In some embodiments, scrolling the text such that the second portion of the text (e.g., 724b) shown in FIG. 7C moves in the first direction occurs while the user moves their eyes in a direction toward the beginning of the second portion of the text (e.g., 724b) (934).

[0226] For example, the beginning of the second portion of text is the first word of the line of text. In some embodiments, the electronic device detects eye movements corresponding to the user reading the text. The eye movements corresponding to reading optionally include eye movements in the direction of the text (e.g., from left to right in English) where the user is looking at words or parts of words as they read the line of text, followed by eye movements in the opposite direction of the text (e.g., from right to left in English) along the line the user just read. In some embodiments, scrolling is performed in response to detecting the user's eye movements along the line the user just read in the opposite direction to the standard reading direction of the text, and while the user is moving their eyes along the line the user just read in the opposite direction of the text.

[0227] The above method of scrolling text while the user's eyes are moving towards the beginning of a second portion of text allows the electronic device to scroll the text in one go so that by the time the user's eyes return to the beginning of the line of text, the beginning of the next line of text is displayed in that position, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by scrolling in a way that does not confuse the user or cause the user to lose their place in the content), thereby allowing the user to use the electronic device more quickly and efficiently, such as by allowing the user to read content more quickly, thereby further reducing power usage and improving the battery life of the electronic device.

[0228] In some embodiments, by scrolling the text such that the second portion of the text (e.g., 724b) shown in FIG. 7C moves in the first direction, the second portion of the text (e.g., 724b) is displayed in a position within the user interface where the user would view it while moving their eyes in a direction toward the beginning of the second portion of the text (e.g., 724b) (936).

[0229] For example, the beginning of the second portion of text is the first word of the line of text. In some embodiments, the electronic device detects eye movements corresponding to the user reading the text. The eye movements corresponding to reading optionally include eye movements in the direction of the text (e.g., from left to right for English or from right to left for Arabic) where the user is looking at a word or portion of a word as they read the line of text, followed by eye movements in the opposite direction of the text (e.g., from right to left for English or from left to right for Arabic) along the line the user just read. In some embodiments, scrolling is performed such that the second portion of the text moves to the position where the first portion of the text was previously displayed while the user is moving their eyes backward along the first portion of the text. For example, scrolling is timed so that the second portion of the text moves to the position where the user is looking. For example, while the user is looking at a position in the user interface next to the beginning of the first portion of the text, the electronic device scrolls the text and displays the second portion of the text that the user is looking at. Thus, the second portion of the text "meets" the user's gaze when the user finishes looking backward along the first portion of the text.

[0230] The above-described method of scrolling text so that the second portion of the text appears at the location where the user is looking as the user moves their eyes toward the beginning of the second portion of the text allows the electronic device to scroll the text so that the text finishes scrolling when the user is likely to be reading the second portion of the text, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by allowing the user to read the text faster), thereby further reducing power usage and improving the battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently.

[0231] In some embodiments, the user interface includes a first portion of text (e.g., 724a) at a first location (e.g., 722a) in the user interface, a second portion of text (e.g., 724b) at a second location (e.g., 722b) in the user interface, and a third portion of text of the content item (e.g., 710a) at a third location in the user interface (e.g., the first, second, and third portions of text are optionally lines of text) (938), as in FIG. 7C . For example, the second portion of text is the line of text immediately following the first portion of text that is also a line of text, and the third portion of text is the line of text after the second line of text. In some embodiments, the third portion of text immediately follows the second portion of text, and in some embodiments, there are one or more other lines of text between the third portion of text and the second portion of text.

[0232] In some embodiments, as in FIG. 7C , while movement of the user's gaze (e.g., 716b) indicates that the user is reading a first portion of text (e.g., 724a), pursuant to a determination that movement of the user's gaze (e.g., 716b) indicates that the user has finished reading the first portion of text (e.g., 724a) and is beginning to read a second portion of text (e.g., 724b), the electronic device (e.g., 101) scrolls the text such that the second portion of text (e.g., 724b) moves a first amount in a first direction (e.g., the second portion of text moves to a position in the user interface where the first portion of text was displayed while the user was reading the first portion of text) (940).

[0233] In some embodiments, in response to a determination that the user's eye movement indicates that the user has finished reading a first portion of the text (e.g., 724a) shown in FIG. 7C and is about to begin reading a third portion of the text (e.g., 710a), the electronic device 10 scrolls (942) the text such that the third portion of the text (e.g., 710a) moves in the first direction by a second amount that is greater than the first amount.

[0234] In some embodiments, the second portion of text is a line of text immediately below the first portion of text, with the second portion of text progressing upward by one line height. In some embodiments, pursuant to a determination that the user has not yet finished reading the first portion of text, the electronic device ceases scrolling the text.

[0235] In some embodiments, the third portion of the text is a line of text that is farther from the first portion of the text than the distance between the second portion of the text and the first portion of the text. In response to detecting that the user is reading the third portion of the text, the electronic device optionally scrolls the third portion of the text to a position in the user interface where the first portion of the text was displayed while the user was reading the first portion of the text. Thus, in some embodiments, the third portion of the text is scrolled a greater distance than the second portion of the text is scrolled to display the portion of the text the user is reading at the position in the user interface where the first portion of the text was displayed. In some embodiments, the third portion of the text is not immediately scrolled to the position where the first portion of the text was displayed, but is scrolled partway to that position, and as the user reads more text, eventually, the portion of the text the user is currently reading is scrolled to match the position where the first portion of the text was displayed. In some embodiments, the text is scrolled more if the user skips multiple lines of text than if the user simply moves to the next line of text, so that the currently read portion of text can eventually catch up to the original position of the first portion of the text in the user interface.

[0236] The above-described method of scrolling a second portion of text by a first amount in response to detecting that a user is reading a second portion of text, and scrolling a third portion of text by a third amount in response to detecting that a user is reading a third portion of text, enables an electronic device to operate to display the portion of text that the user is reading at a fixed position within the user interface, thereby simplifying interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by allowing a user to read content more quickly), thereby further reducing power usage and improving battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently.

[0237] In some embodiments, the image (e.g., 712a) is displayed in a third region of the user interface (e.g., in some embodiments, the image is part of the content item and is positioned after the second portion of the text within the content item) by scrolling the text such that the second portion of the text (e.g., 724b) shown in Figure 7C moves in the first direction (944). For example, the third region of the user interface is below the second portion of the text and, optionally, has other portions of the text between the second portion of the text and the image.

[0238] In some embodiments, after scrolling the text (946) such that the second portion of the text (e.g., 724b) moves in the first direction, the electronic device (e.g., 101) determines (948) via the eye tracking device (e.g., 314) that the user's gaze (e.g., 716d) is directed toward the image (e.g., 712a), as in FIG. 7D, and in response to determining that the user's gaze (e.g., 716d) is directed toward the image (e.g., 712a), the electronic device (e.g., 101) scrolls (950) the text and the image (e.g., 712a) in the first direction to display the image (e.g., 712a) at a first position (e.g., 722a) within the user interface, as in FIG. 7E.

[0239] In some embodiments, the image is displayed in a position within the user interface where the first portion of the text was displayed while the user was reading the first portion of the text. Scrolling the text and image optionally includes revealing additional text displayed after the image. In some embodiments, pursuant to a determination that the user's gaze is not directed at the image, the electronic device ceases scrolling the text and image in the first direction to display the image in the first position within the user interface.

[0240] The above-described method of displaying an image at a first position within a user interface in response to detecting that a user is looking at the image enables the electronic device to move the image to a viewing position that is comfortable for the user without requiring any input other than gaze, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by reducing the number of inputs required to display the image), and enabling the user to use the electronic device more quickly and efficiently, further reducing power usage and improving battery life of the electronic device.

[0241] In some embodiments, while the image (e.g., 712a) is displayed in a first position (e.g., 722a) within the user interface, a third portion (e.g., 710b) of the content item's text (e.g., lines of text) is displayed below the image (952).

[0242] In some embodiments, while displaying the image (e.g., 712a) in a first position (e.g., 722a) of the user interface and displaying a third portion of the text (e.g., 710b) below the image (e.g., 712a), the electronic device (e.g., 101) determines (954) that one or more criteria are met, as in FIG. 7E (e.g., the one or more criteria include a criterion that is met when the user begins to read the third portion of the text (e.g., the user has read a predetermined portion of the text (e.g., one word, two words, three words) or has read the text for a predetermined amount of time (e.g., one second, two seconds, etc.) and the user looks at the third portion of the text, and / or a criterion that is met when the user looks away from the image).

[0243] In some embodiments, in response to determining that one or more criteria are met, the electronic device (e.g., 101) updates (956) the user interface to move the third portion of the text (e.g., 710b) so that it is superimposed on an image (e.g., 712a), such as in FIG. 7F.

[0244] In some embodiments, the content is scrolled so that the third portion of the text is displayed at the first position within the user interface. The image optionally extends above the first position within the user interface and optionally has a width greater than the third portion of the text. In some embodiments, the text is displayed on a background that contrasts highly with the text. For example, the third portion of the text is displayed as black text overlaid on a white rectangle that is overlaid on the image. In some embodiments, there is no background overlaid on the image; instead, the text color and / or transparency of the image is changed to create contrast between the text and the image. In some embodiments, an additional portion of the text is also overlaid on the image. For example, the third portion of the text is a line of text, and the additional line of text is displayed overlaid on the image. In some embodiments, following a determination that one or more criteria are not met, the electronic device ceases updating the user interface to move the third portion of the text so that it is overlaid on the image.

[0245] The above-described method of displaying a third portion of text superimposed on an image enables the electronic device to quickly and efficiently bring the current text of interest to the current reading position while maintaining visual continuity of the image, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by reducing the time it takes for the user to view the image and read the content together), thereby allowing the user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0246] In some embodiments, while displaying a third portion of text (e.g., 710b) overlaid on the image (e.g., 712a), as in FIG. 7F , the electronic device (e.g., 101) detects (958) via the eye tracking device (e.g., 314) that the user's eye movements indicate that the user is reading the third portion of text (e.g., 710a) (e.g., the electronic device detects the user's eye movements consistent with reading).

[0247] In some embodiments, in response to determining that the user's eye movement indicates that the user is reading the third portion of the text (e.g., 710b) shown in FIG. 7F, the electronic device (e.g., 101) scrolls (960) the image (e.g., 712a) in a first direction so that the third portion of the text (e.g., 710b) is no longer superimposed on the image (e.g., 712a).

[0248] In some embodiments, the third portion of the text continues to be displayed in the first position within the user interface, and the image moves to be displayed in a position within the user interface that does not overlap with the first position. For example, the image is displayed above the third portion of the text. In some embodiments, pursuant to a determination that the user is not reading the third portion of the text, the electronic device ceases scrolling the image in the first direction such that the third portion of the text no longer overlaps the image.

[0249] The above method of scrolling the image so that the third portion of the text no longer overlies the image in response to detecting that the user is reading the third portion of the content enables the electronic device to reduce visual clutter around the third portion of the text while the user is reading the third portion of the content, thereby simplifying interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by allowing the user to read the content quickly), thereby further reducing power usage and improving battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently.

[0250] In some embodiments, while displaying a user interface including a content item, the electronic device detects horizontal movement of the user's head (e.g., head movement along the horizontal direction of the user interface (e.g., left to right)) (962) via an eye-tracking device (e.g., 314).

[0251] In some embodiments, in response to detecting horizontal movement of the user's head, the electronic device (e.g., 101) updates the user interface to display the content item in a position central to the user's field of view (964).

[0252] For example, in response to detecting that the user has moved their head to the right, the electronic device moves the content to the right. Thus, in some embodiments, the electronic device displays the content in the center (e.g., horizontally and / or vertically) of the user's field of view, even after the user has moved their head. In some embodiments, if the electronic device does not detect horizontal movement of the user's head, the electronic device does not update the user interface.

[0253] The above-described method of moving a content item in response to detecting horizontal movement of a user's head enables an electronic device to display the content item in a position where it can be read efficiently by the user, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by allowing the user to read content more quickly), and further reducing power usage and improving battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently.

[0254] In some embodiments, the first direction is a direction that is vertical in a user interface (966), such as in FIGS. 7C-7D . Optionally, the content includes vertically scrollable text, such as multiple lines of text. In some embodiments, the electronic device scrolls the content as the user reads each line of text. The electronic device optionally scrolls the content downward in response to input received at an input device other than the eye-tracking device (e.g., the user provides input via a different input device corresponding to a request to scroll the content up), and the user, optionally, is not able to scroll the content downward in response to providing only gaze-based input.

[0255] The above-described method of scrolling text vertically enables an electronic device to automatically display text in a vertical position within the user interface that the user is already viewing, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by allowing the user to read content more quickly), thereby allowing the user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0256] In some embodiments, the first direction is a direction that is horizontal in the user interface (968). In some embodiments, the text is a (e.g., single) line of text that is horizontally scrollable. For example, if the text is in English or another language that reads left to right, the electronic device scrolls the text to the left as the user reads the text. The electronic device can optionally scroll the text backward in response to receiving input via an input device other than the eye-tracking component (e.g., the user provides input via a different input device that corresponds to a request to scroll the content backward), and the user, optionally, cannot scroll the content in the opposite direction in response to providing only eye-gaze-based input.

[0257] The above-described method of scrolling text horizontally enables an electronic device to automatically display text at a horizontal position within the user interface that the user is already viewing, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by allowing the user to read content more quickly), thereby allowing the user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0258] In some embodiments, the first direction has a vertical component and a horizontal component in the user interface (970). In some embodiments, the content is arranged in a grid or another two-dimensional array. For example, the content is a multi-panel comic. In some embodiments, after a user finishes reading the comic content of a panel, the electronic device displays the next panel in the first location in the user interface, thus allowing the user to continue reading the comic from panel to panel in a single location in the user interface.

[0259] The above-described methods for scrolling text horizontally and vertically enable an electronic device to automatically display text at a location in the user interface where the user is already looking, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by allowing the user to read content more quickly), thereby allowing the user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0260] In some embodiments, the electronic device (e.g., 101) detects (972) that the user has finished reading the first portion of the text (e.g., 724a) based on the movement of the user's gaze (e.g., 716b) over the detected first portion of the text (e.g., 724a), as in FIG. 7C.

[0261] In some embodiments, in response to detecting that the user has finished reading the first portion of the text (e.g., 724a), the electronic device (e.g., 101) displays (974) within the user interface a third portion of the text of the content item that was not displayed before determining that the user had finished reading the first portion of the text (e.g., 724a).

[0262] In some embodiments, the electronic device begins displaying a subsequent portion of the content when the user finishes reading the displayed portion of the content (e.g., if a user displays a second portion of text after finishing reading a first portion of text, and the second portion of text was visible / displayed before the user finished reading the first portion of text, then the second portion of text is visible / displayed in the user interface before the user finished reading the first portion of text). In some embodiments, following a determination that the user has not read the first portion of text, the electronic device ceases displaying a third portion of text of the content item.

[0263] The above-described method of starting to display a subsequent portion of content when a user finishes reading a displayed portion of content allows an electronic device to present the portion of content that the user is currently reading without visual distraction, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by allowing the user to read the content more quickly), and further reducing power usage and improving the battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently.

[0264] In some embodiments, as in FIG. 7C , in response to determining that the movement of the user's gaze (e.g., 716b) indicates that the user is reading a first portion of text (e.g., 724a), the electronic device (e.g., 101) generates (976) first audio associated with the first portion of text (e.g., the first audio optionally includes music, sound effects, or audio of the text of the first portion of text (e.g., audio of the first portion of text being read aloud)).

[0265] In some embodiments, after scrolling the text such that the second portion of the text (e.g., 724b) shown in FIG. 7D moves in the first direction, the electronic device (e.g., 101) determines via the eye tracking device (e.g., 314) that the user's eye movements indicate that the user is reading the second portion of the text (e.g., 724b) (978) (e.g., the electronic device detects the user's eye movements consistent with reading while the user is looking at the second portion of the text).

[0266] In some embodiments, in response to determining that the user's eye movement indicates that the user is reading the second portion of the text (e.g., 724b) shown in FIG. 7D , the electronic device (e.g., 101) determines (980) second audio associated with the second portion of the text. For example, the second audio optionally includes music, sound effects, or text audio for the second portion of the text (e.g., audio for the second portion of the text being read aloud). In some embodiments, when the user stops reading the text, the electronic device stops presenting one or more portions of audio associated with the text.

[0267] The above-described method of generating audio corresponding to a portion of text that a user is currently reading while the user is reading enables the electronic device to play the audio without receiving any input other than detection of the user's gaze, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by reducing the input required by the user to cause the electronic device to play audio coordinated with the content item), thereby allowing the user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0268] In some embodiments, the progression of the first audio associated with the first portion of text (e.g., 724a) shown in FIG. 7C corresponds to the rate at which the user is reading the first portion of text (e.g., 724a), and the progression of the second audio associated with the second portion of text (e.g., 724b) corresponds to the rate at which the user is reading the second portion of text (982).

[0269] In some embodiments, the faster a user reads the text, the faster the corresponding audio content is played. Similarly, in some embodiments, the slower a user reads the text, the slower the corresponding audio content is played. For example, the audio content may be a recording of the text being spoken, and the audio may be played at a speed such that the phrases and / or words being spoken are synchronized with the phrase / word currently being read by the user.

[0270] The above method of generating audio at a rate corresponding to the rate at which a user is reading text enables an electronic device to play audio that is adjusted to a user's reading rate without receiving any input other than detection of the user's gaze, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by reducing the input required by the user to cause the electronic device to play audio that is adjusted to the user's reading rate), thereby allowing the user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0271] In some embodiments, while the user's eye movements indicate that the user is reading each content item, the electronic device associates each content item with a respective time that the user was reading each content item (e.g., while the user is reading a content item, the electronic device stores an indication of the content item (or an indication of a particular portion of the content item) in association with a timestamp of when the user is reading the content item (984). For example, while the user is reading a news article, the electronic device stores an indication of the news article and the time that the user was reading the news article. As another example, while the user is reading individual chapters of an e-book, the electronic device stores indications of the individual chapters, pages, lines, and / or words in association with a timestamp of when the user is reading the chapter, page, line, and / or word).

[0272] In some embodiments, after the user's eye movements indicate that the user has finished reading the respective content item (986), the electronic device (e.g., 101) receives a request via one or more input devices to present information about the content item the user was reading at the first discrete time (988) (e.g., the user inputs input requesting the electronic device to present an indication of the content item the user was reading at the first discrete time). For example, the user asks the electronic device's virtual assistant, "What were you reading yesterday at 3:00 PM?" and, in response, the electronic device presents an indication of the content item (e.g., chapter, page, line, and / or word) the user was reading at 3:00 PM the previous day. In some embodiments, other forms of input are possible, such as providing text input via a keyboard or soft keyboard.

[0273] In some embodiments, in response to receiving (990) a request to display information about content items that a user was reading at a first discrete time, and in accordance with a determination that the first discrete time is a first time, the electronic device (e.g., 101) presents (992) information about the first discrete content item (e.g., if the user requests information about content that the user was reading at 10:00 AM the previous day, the electronic device presents an indication of the content items or portions of content items that the user was reading at 10:00 AM the previous day).

[0274] In some embodiments, following a determination that the first discrete time is a second time, the electronic device (e.g., 101) presents (994) information about the second discrete content item. For example, if the user requests information about content items the user was reading at 2:00 PM on Monday, the electronic device presents an indication of the content items or portions of content items the user was reading at 2:00 PM on Monday. In some embodiments, the user requests information about what the user was reading when the electronic device did not have a discrete read content item stored. In response to the request, the electronic device optionally indicates that the user was not reading anything at the indicated time and / or presents an indication of the content item the user was reading at the time closest to the requested time for which an associated content item was stored. In some embodiments, following a determination that the user was not reading a content item at the discrete time, the electronic device presents an indication that the user was not reading at the discrete time.

[0275] The above-described method of storing an indication of a content item being read by a user in relation to a current time and presenting an indication of a content item being read at a particular time in response to a request to present information about the content item being read at a particular time allows an electronic device to efficiently navigate to a content item without being provided with titles or other identifying information about the content item, thereby simplifying the interaction between the user and the electronic device, improving the usability of the electronic device, making the user device interface more efficient (e.g., by allowing a user to quickly recall and view previously viewed content), and enabling a user to use the electronic device more quickly and efficiently, thereby further reducing power usage and improving the battery life of the electronic device.

[0276] In some embodiments, while displaying a content item, the electronic device (e.g., 101) receives (996) a request via one or more input devices to stop displaying the content item, and in response to receiving the request to stop displaying the content item, stops displaying the content item (e.g., in response to detecting a user looking at a display area other than the display area in which the content item is being displayed while performing a head movement that meets one or more criteria, the electronic device stops displaying the content item or displays only a predetermined portion of the content item in a content viewing user interface according to one or more steps of method 1000, described in more detail below). The electronic device optionally stops displaying the content item in response to user input detected with an input device other than the eye-tracking device. In some embodiments, the electronic device stops displaying the content item in response to user input corresponding to a request to display a different content item or a user interface of an application other than the application presenting the content item.

[0277] In some embodiments, after ceasing to display the content item, the electronic device (e.g., 101) receives a request via one or more input devices to display the content item (e.g., the request to display the content item is optionally received after a relatively short period of time (e.g., 5 minutes to 1 hour) has passed, or after a longer period of time (e.g., one day or more) has passed) (998).

[0278] In some embodiments, in response to receiving a request to display a content item (998-2), the electronic device displays the individual content item (e.g., in some embodiments, the content item is displayed superimposed on a content viewing user interface according to one or more steps of process 800 described above) (998-4).

[0279] In some embodiments, in accordance with a determination that the user last read the first portion of the text of the content item before display of the content item was stopped, the electronic device (e.g., 101) provides an indication of the first portion of the text of the content item (e.g., displaying the indication of the first portion of the text of the content item includes displaying the first portion of the text of the content item together with a visual indication) (998-6).

[0280] In some embodiments, in accordance with a determination that the user last read the second portion of the text of the content item before the display of the content item was stopped, the electronic device (e.g., 101) provides an indication of the second portion of the text of the content item (998-8).

[0281] For example, the content item is scrolled to and displayed as a first portion of the content item even if the first portion of the text of the content item is not the beginning of the content item. In some embodiments, the beginning of the content item is displayed simultaneously with an indication of the first portion of the content item even if it does not include the first portion of the content item. In response to detecting a selection of the indication of the first portion of the content item, the electronic device optionally scrolls the content item to the first portion of the content item. In some embodiments, the indication is one or more of an audio or visual indication.

[0282] In some embodiments, displaying the indication of the second portion of the text of the content item includes displaying the second portion of the text of the content item together with a visual indication. For example, the content item is scrolled to and displayed to the second portion of the content item even if the second portion of the text of the content item is not at the beginning of the content item. In some embodiments, the beginning of the content item is displayed simultaneously with the indication of the second portion of the content item even if it does not include the second portion of the content item. In response to detecting a selection of the indication of the second portion of the content item, the electronic device optionally scrolls the content item to the second portion of the content item. In some embodiments, the indication is one or more of an audio or video indication.

[0283] The above-described method of displaying an indication of the portion of a content item that the user last read before stopping display of the content item allows the electronic device to quickly present the portion of the content item that the user was previously reading, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by reducing the time it takes for the user to resume reading content after navigating away from it), thereby allowing the user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0284] In some embodiments, while displaying the user interface, the electronic device (e.g., 101) determines, via the eye-tracking device (e.g., 314), that the user has blinked, including determining that the user's eyes have closed and then opened at predetermined timing criteria (e.g., the user's eyes are closed for longer than a time threshold (e.g., 0.05, 0.1, 0.2, 0.5 seconds, 1 second, 2 seconds, etc.), and then reopened) (998-10). In some embodiments, the threshold is longer than the user typically blinks while viewing the user interface or reading content.

[0285] In some embodiments, in response to determining that the user has blinked, the electronic device (e.g., 101) scrolls (998-12) the text in a second direction different from the first direction to display a previous portion of the content item at a first location (e.g., 722a) in the user interface. For example, the electronic device scrolls text down as the user reads, and in response to detecting a blink of the user that lasts longer than a predetermined threshold amount of time, the electronic device scrolls the content up. In some embodiments, the first portion of the content item is the beginning of the content item. In some embodiments, a long blink causes the content item to scroll back to the beginning of the content item. In some embodiments, a long blink causes the content item to scroll backward incrementally (each blink causes an increment of scrolling in the reverse direction). In some embodiments, pursuant to determining that the user's eyes have not closed and then opened within a predetermined timing criterion, the electronic device ceases scrolling the text in the second direction.

[0286] The above-described method of scrolling backward through a content item in response to detecting a blink lasting longer than a predetermined threshold time enables the electronic device to efficiently navigate to an earlier portion of the content item, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by reducing the time required to start reading the content item from an earlier portion), and thereby enabling the user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0287] In some embodiments, while displaying the user interface, the electronic device (e.g., 101) determines via an eye-tracking device (e.g., 314) that the user is scanning the content item (e.g., eye movements associated with scanning text are different from eye movements associated with reading text) (998-14). The electronic device can optionally determine via the eye-tracking device that the user is scanning the content.

[0288] In some embodiments, in response to determining that a user is skimming the content item, the electronic device (e.g., 101) presents one or more first words of the content item with a visual characteristic having a first value and presents one or more second words of the content item that are different from the first words of the content item with a visual characteristic having a second value that is different from the first value (998-16).

[0289] The visual characteristic is optionally one or more of color, opacity, size, highlighting, or text style (e.g., bold, underline, italic). In some embodiments, the electronic device optionally presents one or more predetermined words deemed important to the content item with an easy-to-read appearance (e.g., higher contrast color, higher opacity, larger size, highlighted, or presented in a bold, italic, or underlined typeface) and the remaining words of the content item with an easy-to-skim appearance (e.g., lower contrast color, lower opacity, smaller size, not highlighted, or presented in the text style in which the other words are presented). For example, the electronic device displays the content item on a white background, with keywords displayed in black text and other words displayed in gray text. In some embodiments, in response to determining that the user is not skimming the content item, the electronic device ceases presenting one or more first words of the content item with a visual characteristic having a first value and ceases presenting one or more second words of the content item that are different from the first words of the content item with a visual characteristic having a second value that is different from the first value.

[0290] By displaying the first word and the second word with different visual characteristics in response to detecting that the user is scanning the content, the electronic device can display the content in a manner that is easy for scanning, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, makes the user device interface more efficient (e.g., by reducing the time it takes a user to scan text), and further reduces power usage and improves battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently.

[0291] In some embodiments, as in FIG. 7C , after scrolling the text so that a second portion of the text (e.g., 724b) moves in a first direction in response to a determination that movement of the user's gaze (e.g., 716b) indicates that the user has finished reading a first portion of the text (e.g., 724a), the electronic device (e.g., 101) receives input (998-18) corresponding to a request to scroll the text in a second direction opposite the first direction via one or more input devices (e.g., keyboard, mouse, touchscreen, touch-sensitive surface, trackpad, voice command, etc.) in communication with the electronic device (e.g., 101) other than the eye tracking device (e.g., 314).

[0292] In some embodiments, in response to receiving one or more inputs, the electronic device (e.g., 101) scrolls text in a second direction (998-20). In some embodiments, in response to a request to scroll content backward received via an input device other than an eye-tracking device, the electronic device scrolls the content backward. For example, the electronic device automatically scrolls content down as a user reads the content, and in response to an input received via a trackpad corresponding to a request to scroll the content up, the electronic device scrolls the content up. In some embodiments, the electronic device does not scroll the content backward in response to an input detected via an eye-tracking device. For example, the electronic device scrolls content down as a user reads the content and detects that the user is looking at an area of ​​the display above a discrete portion of the content that the user is reading. In this example, the electronic device does not scroll the content up simply in response to detecting the user's gaze.

[0293] The above-described method of scrolling content in a first direction in response to gaze and in a second direction in response to an input device other than an eye-tracking device allows the electronic device to cease scrolling content in the second direction in response to gaze, thus allowing the user to freely look around the user interface without accidentally scrolling the text back, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, makes the user device interface more efficient (e.g., by reducing the likelihood of user error and reducing inputs required to correct user error), and further reduces power usage and improves battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently.

[0294] 10A-10G are flowcharts illustrating a method 1000 for navigating a user interface according to detection of a user's head movement and detection of a user's gaze, according to some embodiments. In some embodiments, method 1000 is performed on a computer system (e.g., computer system 101 of FIG. 1). The display generation component (e.g., display generation component 120 of FIGS. 1, 3, and 4) includes a display (e.g., a heads-up display, a display, a touchscreen, a projector, etc.) and one or more cameras (e.g., cameras (e.g., color sensors, infrared sensors, and other depth-sensing cameras)) facing forward from the user's hands or the user's head. In some embodiments, method 1000 is performed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of computer system 101 (e.g., control unit 110 of FIG. 1A). Some operations of method 1000 are optionally combined and / or the order of some operations is optionally changed.

[0295] In method 1000, in some embodiments, an electronic device (e.g., a computer system (e.g., 101)) (e.g., a mobile device (e.g., a tablet, a smartphone, a media player), a computer (e.g., a desktop computer, a laptop computer), or a wearable device (e.g., a watch, a head-worn device) in communication with a display generation component (e.g., 120) and an eye-tracking device (e.g., 314) detects (1002) via the display generation component (e.g., 120) a request to navigate from a first user interface (e.g., 700) to a second user interface (e.g., 714) (e.g., the first user interface may be a navigation user interface including one or more options that, when selected, cause the electronic device to navigate to a different user interface) while displaying the first user interface (e.g., 700) on a display as shown in FIG. 7B .

[0296] In some embodiments, in response to a request to navigate from a first user interface (e.g., 700) to a second user interface (e.g., 714), the electronic device (e.g., 101) navigates (1004) to the second user interface (e.g., the electronic device displays the second user interface, such as a content user interface including an item of content, in response to detecting the request to navigate to the second user interface).

[0297] In some embodiments, as shown in FIG. 7C , while displaying a second user interface (e.g., 714) via a display generation component (e.g., 120), the electronic device (e.g., 101) detects a user's head movement (e.g., the head movement meets one or more criteria indicating that the user's head movement is consistent with a predetermined head movement) (1006).

[0298] In some embodiments, in response to detecting a user's head movement (1008), in accordance with a determination that the head movement meets a first criterion (e.g., a sideways movement, then a forward movement) and the user's gaze (e.g., 716c) is directed toward a distinct portion of the user interface (e.g., 700) (e.g., a portion that is outside the content area and, optionally, in a first direction from the content area, such as to the left of the content area), the electronic device (e.g., 101) navigates from the second user interface (e.g., 714) back to the first user interface (e.g., 700) (e.g., the electronic device optionally stops displaying the second user interface and displays the first user interface) (1010), as shown in FIG. 7B.

[0299] In some embodiments, following a determination that the head movement does not meet the first criterion, the electronic device (e.g., 101) abandons navigation from the second user interface (e.g., 714) back to the first user interface (e.g., 700) (e.g., the electronic device optionally continues to display the second user interface) (1012).

[0300] In some embodiments, in accordance with a determination that the head movement meets a first criterion and the user's gaze is not directed toward a distinct portion of the user interface (e.g., 700), the electronic device (e.g., 101) abandons (1014) navigation from the second user interface (e.g., 714) back to the first user interface (e.g., 700).

[0301] In some embodiments, the display generation component is a display integrated with the electronic device (optionally a touchscreen) and / or an external display such as a monitor, projector, television, etc. In some embodiments, the gaze tracking device includes one or more camera(s) (e.g., IR camera, NIR camera, visible light camera, etc.).

[0302] For example, the first user interface is a content browsing user interface that includes one or more representations of the items of content. In response to detecting a request to view any of the items of content, such as according to one or more steps of method 800, the electronic device optionally displays the requested items of content in a second user interface. Thus, in some embodiments, the second user interface is a content viewing user interface that includes items of content (e.g., text content, photo content, and / or video content).

[0303] In some embodiments, such as according to one or more steps of method 800, the second user interface is displayed overlaid on the first user interface, and portions of the first user interface remain visible while the electronic device displays the second user interface.

[0304] For example, the head movement may include one or more of a forward movement (e.g., toward the display generating component and / or the eye tracking device and / or otherwise toward a user interface in the virtual environment) and / or a movement or rotation to the side of a user interface (e.g., in the virtual environment) in which a portion of a first user interface is displayed. For example, if a portion of a first user interface is visible to the left of a second user interface, the head movement may include detecting a movement and / or rotation toward the left.

[0305] In some embodiments in which the second user interface is overlaid on the first user interface, the electronic device detects that the user is looking at a portion of the first user interface.

[0306] In some embodiments, a head movement that satisfies the first criterion includes a head movement toward a side of the user interface where a portion of the first user interface is displayed. For example, if a portion of the first user interface is visible to the right of the second user interface, a head movement that satisfies the first criterion includes a head movement to the right. In some embodiments, the position of the user's gaze is detected using an eye-tracking device. The eye-tracking device optionally captures one or more images of the user's eyes and detects pupils and glints in the captured image or images to track the user's gaze, as described in more detail above with reference to FIG. 6 . In some embodiments, the first user interface object is partially overlaid on the second user interface object, and a portion of the second user interface object is visible. The electronic device optionally detects that the user is looking at a visible portion of the second user interface object.

[0307] In some embodiments, head movement does not meet the first criterion if one or more of the direction, magnitude, or duration of the head movement is outside a predetermined range or does not match a predetermined pattern.

[0308] In some embodiments, the electronic device does not navigate back to the first user interface unless it determines that the head movement meets the first criterion and that the user's gaze is directed toward a respective portion of the user interface. The electronic device optionally continues to display the second user interface pursuant to a determination that the head movement meets the first criterion and that the user's gaze is not directed toward a respective portion of the user interface.

[0309] The above method of navigating back to a first user interface in response to detecting a user's gaze and head movements enables the electronic device to navigate between user interface objects without user input beyond gaze and head movements, but also avoids erroneous reverse navigation due to the dual head and gaze requirement, thereby simplifying the interaction between the user and the electronic device, improving the usability of the electronic device, making the user device interface more efficient (e.g., by reducing the time it takes the user to enter navigation inputs and reducing erroneous navigation results), and enabling the user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0310] In some embodiments, as in FIG. 7C , a second user interface (e.g., 714) is displayed while at least a portion of a first user interface (e.g., 700) is displayed (e.g., the second user interface is overlaid on the first user interface, with portions of the first user interface visible next to the border of the second user interface) (1016). For example, the first user interface is a content browsing user interface, and the second user interface includes items of content to which the user navigated from the first user interface.

[0311] In some embodiments, the electronic device (e.g., 101) displays a second user interface (e.g., 714) and a first user interface (e.g., 700), wherein the first user interface (e.g., 700) is visually de-emphasized relative to the second user interface (e.g., 714), and while displaying at least a portion of the first user interface, the electronic device (e.g., 101) tracks, via an eye-tracking device (e.g., 314), the user's gaze (e.g., 716c) from the second user interface (e.g., 714) to the first user interface (e.g., 700), as in FIG. 7C . The first user interface is optionally displayed with a higher transparency than the second user interface, the first user interface is displayed with a higher blur than the second user interface, the first user interface is displayed with a lower brightness or contrast than the second user interface, the first user interface is displayed at a reduced size and the second user interface is displayed at full size while the user is viewing the second user interface, etc.).

[0312] In some embodiments, as shown in FIG. 7C , in response to determining that the user's gaze (e.g., 716c) has moved from the second user interface (e.g., 714) to the first user interface (e.g., 700), the electronic device (e.g., 101) decreases (1020) the amount of visual de-emphasis of the first user interface (e.g., 714) relative to the second user interface (e.g., 700) (e.g., displaying the second user interface with a visual characteristic having a third value different from the first value (e.g., increased transparency, increased blur, decreased brightness or contrast, reduced size, etc.), and / or displaying at least a portion of the first user interface with a visual characteristic having a fourth value different from the second value (e.g., decreased transparency, decreased blur, increased brightness or contrast, increased size, etc.), as shown in FIG. 7G .

[0313] In some embodiments, while the user is viewing the second user interface, the second user interface is displayed with full color, contrast, or brightness, full clarity, and transparency, and the first user interface is displayed with reduced size, muted colors, low contrast and / or brightness, reduced clarity, and / or greater transparency. In response to detecting the user viewing the first user interface, the electronic device optionally decreases the size, color, contrast, brightness, and / or clarity, and / or increases the transparency of the second user interface, and increases the size, contrast, brightness, and / or clarity, lightens the color, and / or decreases the transparency of the first user interface.

[0314] The above-described method of altering the visual characteristics of the first and second user interfaces in response to detecting a user's gaze moving from the second user interface to the first user interface allows the electronic device to efficiently visually enhance the user interface that the user is currently viewing, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by allowing the user to read content more quickly), and thereby allowing the user to use the electronic device more quickly and efficiently, further reducing power usage and improving battery life of the electronic device.

[0315] In some embodiments, as shown in FIG. 7G , while a user's gaze is directed at a first user interface (e.g., 700) and while displaying the first user interface (e.g., 700) and the second user interface (e.g., 714) with a reduced amount of visual de-emphasis of the first user interface (e.g., 700) relative to the second user interface (e.g., 714), the electronic device (e.g., 101) tracks, via the eye-tracking device (e.g., 314), whether the user's gaze (e.g., 716f) is directed at the first user interface (e.g., 716f). 700) to a second user interface (e.g., 714) (e.g., 1022) (e.g., the first user interface is displayed with increased size, contrast, clarity, and / or brightness, bright colors, and / or decreased transparency while the user is viewing the first user interface, and the second user interface is displayed with decreased size, contrast, clarity, and / or brightness, dull colors, and / or increased transparency compared to the presentation of the first user interface while the user is viewing the first user interface).

[0316] In some embodiments, as shown in FIG. 7G, in response to determining (1024) that a user's gaze (e.g., 716f) has moved from a first user interface (e.g., 700) to a second user interface (e.g., 714), the electronic device (e.g., 101) displays (1026) the first user interface (e.g., 700) and the second user interface (e.g., 714) with the first user interface (e.g., 700) visually de-emphasized relative to the second user interface (e.g., 714) as shown in FIG. 7C (e.g., restoring the level of visual de-emphasis to be the same as the amount of de-emphasis before detecting the user looking at the first user interface).

[0317] In some embodiments, the second user interface is displayed with reduced transparency, reduced blur, increased brightness or contrast, increased size, etc., and the first user interface is displayed with increased transparency, increased blur, decreased brightness or contrast, decreased size, etc. In some embodiments, in response to detecting the user viewing the second user interface, the electronic device displays the second user interface with increased size, contrast, clarity, and / or brightness, brighter colors, and / or reduced transparency, and displays the first user interface with reduced size, contrast, clarity, and / or brightness, duller colors, and / or increased transparency, compared to the presentation of the first and second user interfaces while the user is viewing the second user interface.

[0318] The above-described method of altering visual characteristics of the first and second user interfaces in response to detecting a user viewing the second user interface enables the electronic device to efficiently display the user interface viewed by the user with reduced visual clutter, thereby simplifying interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by allowing the user to more quickly render the user interface viewed by the user), thereby further reducing power usage and improving battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently.

[0319] In some embodiments, in response to detecting 1028 a user's head movement, following a determination that the head movement meets a first criterion (e.g., a sideways movement, then a forward movement) and the user's gaze is directed toward a distinct portion of the user interface (e.g., a portion outside the content area and, optionally, in a first direction from the content area, such as to the left of the content area), the electronic query displays 1030 an animated transition between the second user interface and the first user interface while navigating from the second user interface back to the first user interface, as in Figures 7G-7H. In some embodiments in which the second user interface is overlaid on the first user interface, the electronic device detects that the user is looking at a portion of the first user interface.

[0320] In some embodiments, pursuant to a determination that the head movement does not meet the first criterion and the user's gaze is directed at a discrete portion of the user interface, the electronic device (e.g., 101) displays (1032) a first portion of an animated transition between the second user interface and the first user interface, such as in FIG. 7G, without displaying the complete animated transition between the second user interface and the first user interface.

[0321] In some embodiments, the electronic device displays an animation of visual characteristics of the first and second user interfaces that change depending on which of the user interfaces the user is looking at. For example, in response to detecting that the user has moved their gaze from the second user interface to the first user interface, the electronic device displays an animation in which the first user interface becomes larger and sharper and the second user interface becomes smaller and less sharp. In some embodiments, the progression of the animation is proportional to how much the user's head moves. In some embodiments, the animation progresses by a greater amount when the user moves their head a greater amount.

[0322] The above-described method of animating changes in visual characteristics of a user interface as a user moves their head enables the electronic device to provide the user with efficient visual feedback indicating that head movements cause navigation between user interfaces, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by allowing the user to more quickly navigate between user interfaces with head movements), thereby allowing the user to use the electronic device more quickly and efficiently, further reducing power usage and improving battery life of the electronic device.

[0323] In some embodiments, the magnitude of the first portion of the animated transition depends on the amount of head movement 1034. For example, if the user moves their head a first amount, the animation progresses by the first amount, and if the user moves their head a greater amount, the animation progresses by a greater amount.

[0324] The above-described method of animating changes in the visual characteristics of a user interface by an amount that corresponds to the amount the user moves their head enables the electronic device to provide the user with efficient visual feedback indicating that their head movements cause navigation between the user interface, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by allowing the user to more quickly navigate between the user interface with head movements), and thereby allowing the user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0325] In some embodiments, the first portion of the animated transition progresses at a speed and / or direction determined based on the speed and / or direction of head movement (including forward and / or backward movement) (1036). In some embodiments, the electronic device progresses the animation at a relatively slow speed in response to detecting relatively slow movement of the user's head, and progresses the animation at a relatively fast speed in response to detecting relatively fast movement of the user's head. The electronic device optionally progresses the animation forward in response to detecting movement of the user's head in a first direction, and optionally progresses the animation backward in response to detecting movement of the user's head in a second direction opposite the first direction.

[0326] The above-described method of animating changes in visual characteristics of a user interface with a speed or direction based on the speed or direction of head movement enables the electronic device to provide the user with efficient visual feedback indicating that head movement causes navigation between the user interface, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by allowing the user to more quickly navigate between the user interface with head movements), thereby further reducing power usage and improving the battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently.

[0327] In some embodiments, in response to detecting an end of the head movement (e.g., an end of a head gesture or an end of gaze) without the head movement meeting a first criterion, the electronic device (e.g., 101) displays (1038) an animated transition back to the first user interface (e.g., 700) visually de-emphasized relative to the second user interface (e.g., 714) (e.g., reversing the first portion of the animated transition), as shown in Figure 7C. In some embodiments, the end of the head movement corresponds to detecting that the user has stopped moving their head or has reversed their head movement such that the head is positioned as before the start of the head movement.

[0328] The animated transition back to the first user interface visually de-emphasized relative to the second user interface is optionally an animated transition that reduces the amount of de-emphasis of the first user interface relative to the second user interface played in reverse. In some embodiments, the electronic device displays the animated transition back to the first user interface visually de-emphasized relative to the second user interface in response to detecting the user looking away from the first user interface. The above-described method of displaying the animated transition back to the first user interface visually de-emphasized relative to the second user interface enables the electronic device to provide efficient visual feedback to the user indicating that head movements cause navigation between user interfaces, thereby simplifying interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by allowing the user to navigate between user interfaces more quickly via head movements), thereby further reducing power usage and improving battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently.

[0329] In some embodiments, the first criterion includes a criterion requiring a movement of the user's head tilting to the side (1040). In some embodiments, the criterion is met if the user tilts their head left or right, the criterion is met if the user tilts their head left, or the criterion is met if the user tilts their head right. The electronic device optionally displays a second user interface superimposed on the first user interface, with portions of the first user interface visible. For example, the second user interface is displayed superimposed on the left side of the first user interface, with portions of the first user interface remaining displayed on the right side of the second user interface, and in response to detecting at least a movement of the user's head to the right, the electronic device navigates back to the first user interface.

[0330] The above-described method of navigating back to the first user interface in response to detecting at least a lateral movement of the user's head enables the electronic device to provide an efficient and intuitive way of navigating from the second user interface to the first user interface, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, and makes the user device interface more efficient (e.g., by reducing the time it takes the user to navigate between user interfaces), thereby enabling the user to use the electronic device more quickly and efficiently, further reducing power usage and improving battery life of the electronic device.

[0331] In some embodiments, the first criterion includes a criterion requiring forward movement of the user (e.g., relative to the user's body) 1042. In some embodiments, in response to detecting at least a movement of the user's head in a forward direction relative to the user's body, the electronic device navigates from the second user interface to the first user interface.

[0332] The above-described method of navigating back to the first user interface in response to detecting at least forward movement of the user's head enables the electronic device to provide an efficient and intuitive way of navigating from the second user interface to the first user interface, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, and makes the user device interface more efficient (e.g., by reducing the time it takes the user to navigate between user interfaces), thereby enabling the user to use the electronic device more quickly and efficiently, further reducing power usage and improving battery life of the electronic device.

[0333] In some embodiments, as shown in FIG. 7C , navigating from a first user interface (e.g., 700) to a second user interface (e.g., 714) includes displaying the second user interface (e.g., 714) superimposed on the first user interface (e.g., 700) (1044). In some embodiments, the second user interface is superimposed on a portion of the first user interface such that portions of the first user interface are visible while the second user interface is displayed. For example, the second user interface is superimposed on a left portion of the first user interface and a right portion of the first user interface is visible to the right of the second user interface.

[0334] The above-described method of displaying a second user interface superimposed on a first user interface allows the electronic device to simultaneously display the second user interface and portions of the first user interface, which simplifies interaction between the user and the electronic device, improves usability of the electronic device, and makes the user device interface more efficient (e.g., by allowing the user to see portions of the first user interface, the user can know whether or not they want to return to the first user interface before entering a request to navigate back to the first user interface), thereby further reducing power usage and improving battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently.

[0335] In some embodiments, navigating from the first user interface to the second user interface includes de-highlighting the first user interface and, according to a determination that the user's gaze is directed toward the second user interface, highlighting the second user interface (1046). In some embodiments, de-highlighting the first user interface includes one or more of decreasing a size, contrast, brightness, opacity, and / or clarity of the first user interface, and highlighting the second user interface includes one or more of increasing a size, contrast, brightness, opacity, and / or clarity of the second user interface.

[0336] The above-described method of de-highlighting a first user interface and highlighting a second user interface as part of navigating from a first user interface to a second user interface allows the electronic device to efficiently display the user interfaces with reduced visual clutter while the user is looking at the second user interface, which simplifies the interaction between the user and the electronic device, improves usability of the electronic device, and makes the user device interface more efficient (e.g., by allowing the user to more quickly understand the second user interface and by providing an efficient method of navigation from the second user interface back to the first user interface), thereby further reducing power usage and improving battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently.

[0337] In some embodiments, while displaying the second user interface (e.g., 714) (e.g., while navigating to the second user interface), while a characteristic (e.g., length of time) of the user's gaze does not meet the second criterion, or while the user's head movement does not meet the first criterion (e.g., length of time of head movement, distance of head movement, and / or velocity of head movement does not meet one or more criteria, and / or gaze position, duration, and / or other characteristics do not meet one or more criteria), the electronic device (e.g., 101) may display the second user interface (e.g., 714) based on the user's gaze or the user's head movement, as in FIG. 7G . and continuing to display the second user interface (e.g., 714) while changing (1048) the appearance (e.g., or the appearance of the first user interface) of the user interface (e.g., 714) (e.g., comparing the appearance of the first and second user interfaces and displaying the first user interface with increased contrast, brightness, size, opacity, and / or clarity and displaying the second user interface with decreased contrast, brightness, size, opacity, and / or clarity before detecting the user's gaze on the first user interface and the user's head movement).

[0338] In some embodiments, following a determination that the user's gaze characteristics meet the second criterion and the user's head movement meets the first criterion, the electronic device (e.g., 101) navigates (1050) from the second user interface (e.g., 714) back to the first user interface (e.g., 700), as in FIG. 7B.

[0339] In some embodiments, continuing to navigate to the second user interface includes continuing to display the second user interface (e.g., overlaid on the first user interface). Further, if the user looks away from both the first and second user interfaces (e.g., looks at a system user interface or system user interface element of the electronic device or looks away from the display area of ​​a display generation component) while the electronic device is navigating to the second user interface, the electronic device reverts to displaying the first and second user interfaces in the manner they were displayed while the user was looking at the second user interface.

[0340] In some embodiments, navigating from the second user interface back to the first user interface includes ceasing display of the second user interface. Further, once the electronic device returns to the first user interface, the electronic device continues to display the first user interface without displaying the second user interface if the user looks away from the first user interface, as described above.

[0341] The above method of returning to the first user interface in response to detecting that the user's gaze and the user's head movement meet predetermined criteria, but changing the appearance of both the first and second user interfaces based on head / gaze input that is not sufficient to return to the first user interface, enables the electronic device to efficiently provide feedback to the user to navigate back to the first user interface by continuing to perform head movements and look at the first user interface, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by allowing the user to more quickly return to the first user interface when the user intends to do so), thereby further reducing power usage and improving battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently.

[0342] In some embodiments, while displaying a second user interface (e.g., 714) in which the appearance of the second user interface (e.g., 714) has been changed, as in FIG. 7G (e.g., while navigating to the second user interface and displaying at least a portion of the first user interface), the electronic device (e.g., 101) determines (1052) via the eye tracking device (e.g., 314) that the user's gaze (e.g., 716f) is directed toward the second user interface (e.g., 714) (e.g., or the first user interface).

[0343] In some embodiments, in response to determining that the user's gaze (e.g., 716f) is directed toward the second user interface (e.g., 714), as shown in FIG. 7G, the electronic device (e.g., 101) returns the appearance of the second user interface (e.g., 714) to its initial appearance (1054), as shown in FIG. 7C.

[0344] In some embodiments, the changed appearance comprises one or more of reduced size, contrast, clarity, and / or opacity compared to the initial appearance of the second user interface while the user was looking at the first user interface and viewing the second user interface before performing the head movement.

[0345] In some embodiments, returning the appearance of the second user interface to the initial appearance includes abandoning navigating back to the first user interface. The electronic device also, optionally, returns the first user interface to the initial appearance with reduced size, contrast, clarity, and / or opacity compared to the appearance of the first user interface in response to detecting the user looking at the first user interface while performing a head movement.

[0346] The above-described method of returning the appearance of the second user interface to the initial appearance in response to detecting that the user is viewing the second user interface enables the electronic device to stop the process of navigating back to the first user interface and provide the user with an efficient way to continue viewing the second user interface, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by providing a quick and efficient way to resume displaying the second user interface having the initial appearance), thereby further reducing power usage and improving battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently.

[0347] In some embodiments, the electronic device (e.g., 101) displays the text content in the user interface (e.g., the text content is overlaid on the first and second user interfaces or displayed in the first or second user interface) (1056).

[0348] In some embodiments, while displaying the text content, the electronic device (e.g., 101) detects (1058) a request to stop displaying the text content (e.g., user input received at one of the input devices in communication with the electronic device other than the gaze detection device).

[0349] In some embodiments, in response to detecting a request to stop displaying the textual content (1060), the electronic device (e.g., 101) stops displaying (1062) the textual content (e.g., the one or more criteria are associated with a predetermined notification event, such as an important alert, a notification generated by an application indicated as important by a user or the system, or text indicated as important by the text author) in accordance with a determination that the textual content does not meet one or more criteria.

[0350] In some embodiments, following a determination 1064 that the textual content meets one or more criteria (e.g., the one or more criteria are associated with a predetermined notification event, such as an important alert, a notification generated by an application indicated as important by the user or the system, or text indicated as important by the text author), and following a determination (e.g., via an eye-tracking device) that the user has read the textual content, the electronic device (e.g., 101) stops displaying the textual content (e.g., the electronic device determines 1066 that the user's eye movements are consistent with reading the textual content based on data captured by the eye-tracking component).

[0351] In some embodiments, following a determination (e.g., via an eye-tracking device) that the user is not reading the textual content, the electronic device (e.g., 101) maintains display of the textual content (1068).

[0352] The textual content optionally includes text associated with an indication of a notification event. For example, the textual content is a message preview of a message received at the electronic device, the message preview being displayed in a notification indication that indicates receipt of the message. In some embodiments, the text is terms and conditions text associated with a software product or electronic device service.

[0353] In some embodiments, the input is an eye gaze input, as described above, for example to navigate away from the second user interface.

[0354] For example, the input may be one of pressing a key on a keyboard, selecting an option to stop displaying text via a touchscreen, trackpad, mouse, etc., or a voice command.

[0355] Following a determination that the text does not meet the criteria, the electronic device stops displaying the text in response to a user request (e.g., in an example where the text content is included in a second user interface, navigate away from the second user interface and cease displaying the second user interface).

[0356] In response to detecting input corresponding to a request to stop displaying the text, the electronic device stops displaying the text in accordance with a determination that the user has read the text (e.g., in an embodiment in which the text content is included in a second user interface, navigates away from the second user interface and ceases its display).

[0357] In some embodiments, if the text meets one or more criteria, the user must read the text in order to discard it in response to a user input corresponding to a request to stop displaying the text.

[0358] The above method of discarding text that meets one or more criteria only in response to detecting that the user has read the text and in response to an input requesting that the display of the text be stopped allows the electronic device to efficiently continue to display important text until it is read by the user, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by reducing the input required by the user to access the text in order to read it), and allowing the user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0359] In some embodiments, the electronic device (e.g., 101) displays (1070) in a user interface a notification (e.g., 706) associated with a particular application (e.g., the notification includes text or an image indicating a notification event detected on the electronic device). For example, the notification event is the receipt of a message, and the notification includes the text of the message and an indication of the user account from which the message was received. The message notification is optionally associated with a messaging application.

[0360] In some embodiments, while displaying a notification (e.g., 706) associated with a particular application, the electronic device (e.g., 101) determines (1072) via an eye-tracking device that the user has read the notification (e.g., 706) (e.g., the user's eye movements while viewing the notification are consistent with reading). In some embodiments, the electronic device determines that the user has read all of the text contained in the notification.

[0361] In some embodiments, in response to determining that the user has read the notification (e.g., 706) (e.g., determining that the user has finished reading the notification), the electronic device (e.g., 101) displays (1074) a user interface of the respective application via the display generation component (e.g., 120). For example, the notification is a message notification, and in response to detecting that the user has read the notification, the electronic device displays a messaging application. In some embodiments, in response to determining that the user has not finished reading the notification, the electronic device ceases displaying the user interface of the respective application.

[0362] The above-described method of displaying a user interface of an application associated with a notification that a user has read enables the electronic device to present additional information associated with the notification in an efficient manner, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by reducing the input required to open an application associated with the notification that the user has read), thereby allowing the user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0363] In some embodiments, while determining (1076) via the eye tracking device (e.g., 314) that the user's gaze is directed toward the second user interface (e.g., 714), in accordance with a determination that the user's head is in a first position, the electronic device (e.g., 101) displays (1078) the text content of the second user interface (e.g., 714) in a first text style (e.g., the first style including font, text size, and / or text styling such as bold, italics, or underline).

[0364] In some embodiments, in accordance with a determination that the user's head is in a second position different from the first position, the electronic device (e.g., 101) displays (1080) the text content of the second user interface (e.g., 714) in a second text style different from the first text style. For example, when the user moves their head forward relative to the rest of their body and / or relative to an initial position of their head, the electronic device optionally displays text in a larger size or a different font (e.g., a font with a higher resolution (DPI) but the same size, or a font that is optionally more readable from a shorter distance than the font of the original display, which is more readable from a longer distance).

[0365] The above-described method of altering text style in response to detecting user head movement enables an electronic device to provide an efficient way of altering text style to optimize text display at different head / eye distances, such as the size or level of text decoration, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by reducing the input required to make text more readable or aesthetically pleasing), and enabling the user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0366] 11 is a flowchart illustrating a method 1100 for displaying augmented content related to a portion of content according to detection of a user's head movement and detection of a user's gaze, according to some embodiments. In some embodiments, method 1100 is performed on a computer system (e.g., computer system 101 of FIG. 1). The display generation component (e.g., display generation component 120 of FIGS. 1, 3, and 4) includes a display (e.g., a heads-up display, a display, a touchscreen, a projector, etc.) and one or more cameras (e.g., cameras (e.g., color sensors, infrared sensors, and other depth-sensing cameras)) facing forward from a user's hand or a user's head. In some embodiments, method 1100 is performed by instructions stored on a non-transitory computer-readable storage medium and executed by one or more processors of the computer system, such as one or more processors 202 of computer system 101 (e.g., control unit 110 of FIG. 1A). Some operations of method 1100 are optionally combined and / or the order of some operations is optionally changed.

[0367] In method 1100, in some embodiments, an electronic device (e.g., a computing system (e.g., 101)) in communication with a display generation component (e.g., 120) and an eye-tracking device (e.g., 314) (e.g., a mobile device (e.g., tablet, smartphone, media player), computer (e.g., desktop computer, laptop computer), or wearable device (e.g., watch, head-worn device)) displays (1102) a user interface including first content, via the display generation component (e.g., 120), the first content including a first portion (e.g., 718a) associated with first augmented content and a second portion (e.g., 718b) associated with second augmented content (e.g., text content), as in FIG. 7I. In some embodiments, the content item includes text and / or images).

[0368] In some embodiments, while displaying the user interface, the electronic device (e.g., 101) detects (1104) a movement of the user's head (e.g., a forward movement of the user's head relative to the user's body).

[0369] In some embodiments, in response to detecting 1106 a movement of the user's head, a determination is made that the user's gaze (e.g., 716i) is directed toward a first portion (e.g., 718a) of a first content item (e.g., an eye-tracking device optionally captures one or more images of the user's eyes and detects pupils and glints in the one or more captured images to track the user's gaze, as described in more detail above with reference to FIG. 6 ), and a determination is made that the user's head movement while the user's gaze (e.g., 716i) is directed toward the first portion of the first content item, such as in FIG. 7I . In accordance with a determination that a first criterion is met (e.g., a forward movement of the head (e.g., toward a distinct portion within the virtual environment)), the electronic device (e.g., 101) updates the user interface to include first augmented content (e.g., 720a) associated with the first portion of the first content (e.g., 718a) without updating the user interface to include second augmented content associated with the second portion of the first content (e.g., the augmented content includes one or more of additional text and / or images), such as in FIG. 7J (1108).

[0370] In some embodiments, following a determination that the user's head movement while the user's gaze (e.g., 716n) is directed toward the second portion of the first content item satisfies a first criterion, the electronic device (e.g., 101) updates (1110) the user interface to include second augmented content (e.g., 720b) ​​associated with the second portion of the first content (e.g., 718b) (e.g., 718b) without updating the user interface to include first augmented content associated with the first portion of the first content, such as in FIG. 7K (e.g., the augmented content displayed in response to detecting that the user is looking at a distinct portion of the first content item while performing a head movement that satisfies one or more criteria varies depending on which portion of the first content item the user is looking at).

[0371] In some embodiments, in accordance with a determination that the user's head movement did not meet the first criterion, the electronic device (e.g., 101) discontinues updating the user interface to include the first augmented content and discontinues updating the user interface to include the second augmented content (1112).

[0372] In some embodiments, the display generation component is a display integrated with the electronic device (optionally a touchscreen) and / or an external display such as a monitor, projector, television, etc. In some embodiments, the gaze tracking device includes one or more camera(s) (e.g., IR camera, NIR camera, visible light camera, etc.).

[0373] For example, the content item may be an article, a message, an e-book, or other content that includes text. In some embodiments, the first portion is a first word of the text content and the second portion is a second word of the text content.

[0374] In some embodiments, detecting that the user is moving their head forward comprises detecting that the user is moving their head in the direction they are looking and / or in the direction of the display generating component and / or the eye tracking device. The one or more criteria optionally further include time and distance based criteria, such as performing the head movement for at least a predetermined amount of time, moving the head at least a predetermined distance, and / or moving the head at a velocity within a predetermined range.

[0375] For example, the first portion of the first content item may be a word, and the augmented content may be a definition of the word. As another example, the first portion of the first content item may be a hyperlink to a website, and the first augmented content may be a preview of the website. As another example, the augmented content may be an enlarged version of the first portion of the first content item. For example, the first portion of the first content item may be a portion of an image, and the augmented content may be the complete image. As another example, the augmented content may be the same as the first portion of the first content item, but displayed at a larger size. In some embodiments, the augmented content relates to a first portion of the first content item that supplements the first content item. The augmented content is optionally not included in the first content item. In some embodiments, the display of the augmented content is performed in response to detecting a user's gaze and head movement without receiving additional input from another input device in communication with the electronic device.

[0376] In some embodiments, the type of content of the augmented content changes depending on which portion of the first content item the user is looking at. For example, if the first portion of the first content item is a word, the first augmented content is a definition; if the second portion of the first content item is a website link, the second augmented content is a website preview; if the second portion of the augmented content is an image, the second augmented content is another image. In some embodiments, the display of the augmented content is performed in response to detecting the user's gaze and head movement without receiving additional input from another input device in communication with the electronic device.

[0377] In some embodiments, determining that the user has not performed a head movement that meets one or more criteria includes determining that the user is not moving their head or is moving their head in a manner that does not meet one or more criteria. For example, if the user performs an incorrect head movement (e.g., moving right or left or turning instead of moving forward), moves in the wrong direction (e.g., moving backward instead of forward), moves in a manner that does not meet the time or distance-based criteria described above, or does not move their head at all while viewing a particular portion of content, the electronic device refrains from updating the user interface to include augmented content that corresponds to that particular portion of content.

[0378] The above-described method of displaying augmented content in response to a user's gaze and head movements enables an electronic device to quickly display augmented content related to the portion of a content item that the user is viewing in response to only the user's gaze and head movements, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, and making the user device interface more efficient (e.g., by reducing the time it takes for a user to provide input requesting augmented content and reducing erroneous display of augmented content), thereby allowing a user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0379] In some embodiments, the first augmented content (e.g., 720a) as in FIG. 7J is one of a definition of the first portion of the first content (e.g., 718a), a website preview of the first portion of the first content (e.g., 718a), or an enlarged version of the first portion of the first content (e.g., 718a) and the second augmented content (e.g., 720b) ​​as in FIG. 7K, and is one of a definition of the second portion of the first content (e.g., 718b), a website preview of the second portion of the first content (e.g., 718b), or an enlarged version of the second portion of the first content (e.g., 718b) (1114).

[0380] For example, a separate piece of content may be a word and the augmented content may be the definition of the word. As another example, a separate piece of content may be an image (or a portion of an image) and the augmented content may be a larger version of the image (or the entire image). As another example, a separate piece of content may be a hyperlink and the augmented content may be a preview of the hyperlink's content.

[0381] The above-described methods of displaying definitions, images, or website previews enable an electronic device to quickly and efficiently provide additional context to portions of content while continuing to display the content item, thereby simplifying the interaction between a user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by reducing the time it takes a user to view definitions, images, or website previews associated with individual portions of a content item), and further reducing power usage and improving battery life of the electronic device by allowing a user to use the electronic device more quickly and efficiently.

[0382] In some embodiments, before displaying (1116) the first augmented content (e.g., 720a) shown in FIG. 7J or the second augmented content (e.g., 720b) ​​shown in FIG. 7K, and before detecting a user's head movement, the electronic device (e.g., 101) displays (1118) a first portion of the first content (e.g., 718a) and a second portion of the first content (e.g., 718b) with visual characteristics having a first value, as in FIG. 7C (e.g., the visual characteristics are one or more of size, color, thickness, or highlighting (or lack thereof)).

[0383] In some embodiments, in response to detecting (1120) at least a portion of the user's head movement (e.g., the electronic device displays the augmented content in response to detecting the user's head movement that exceeds a threshold amount of time, distance, or some other criterion), determining that the user's gaze (e.g., 716i) is directed toward a first portion (e.g., 718a) of the first content, and requiring that at least a portion of the user's head movement while the user's gaze (e.g., 716i) is directed toward the first portion (e.g., 718a) of the content be in a direction that satisfies a first criterion. In accordance with a determination that one or more second criteria are met, including criteria that include one or more of the first content being in a different visual characteristic than the first value, the electronic device (e.g., 101) updates (1122) the user interface to display the first portion of the first content (e.g., 718b) with visual characteristics having a second value different from the first value, while maintaining the second portion of the first content (e.g., 718a) with visual characteristics having the first value, as in FIG. 7I (e.g., the one or more second criteria are optionally similar to the one or more first criteria, but it is possible to meet the one or more second criteria without meeting the one or more first criteria).

[0384] In some embodiments, no portion of the user's head movement exceeds a threshold time or distance or meets other criteria for displaying augmented content.

[0385] For example, the one or more first criteria are met if the user moves their head a predetermined amount (e.g., 2 cm, 3 cm, 4 cm, etc.) and continues the movement for a predetermined amount of time (e.g., 1 second), and the one or more second criteria are met if the user moves their head a smaller amount and / or for a shorter period of time. Displaying the first portion of the content with visual characteristics having the second values ​​optionally includes displaying the first portion of the content at a larger size, highlighted, with an additional outline, or with additional text styling (e.g., bold, underlined, italicized) compared to presenting the first portion of the content with visual characteristics having the first values. Meanwhile, in some embodiments, the second portion of the content item optionally continues to be displayed with the same visual appearance as before detecting the user's head movement. In some embodiments, following a determination that the one or more first criteria are not met, the electronic device ceases updating the user interface to display the first portion of the first content with visual characteristics having the second values.

[0386] In some embodiments, in accordance with a determination that the user's gaze (e.g., 716n) is directed toward the second portion of the first content (e.g., 718b) and a determination that one or more second criteria are met while the user's gaze (e.g., 716n) is directed toward the second portion of the content (e.g., 718b), the electronic device (e.g., 101) updates (1124) the user interface to display the second portion of the first content (e.g., 718a) with visual characteristics having a second value while maintaining the first portion of the first content (e.g., 718b) with visual characteristics having a first value, as in FIG. 7K.

[0387] Displaying the first portion of the content with visual characteristics having second values ​​optionally includes displaying the second portion of the content at a larger size, highlighted, with an additional outline, or with additional text styling (e.g., bold, underlined, italicized) compared to presenting the second portion of the content with visual characteristics having second values, while in some embodiments the first portion of the content item optionally continues to be displayed with the same visual appearance as before detecting the user's head movement. In some embodiments, following a determination that the one or more second criteria are not satisfied, the electronic device ceases updating the user interface to display the second portion of the first content with visual characteristics having second values.

[0388] The above-described method of updating the visual appearance of the portion of the content item that the user is viewing while performing head movements that satisfy one or more second criteria allows the electronic device to efficiently provide feedback to the user as to which portion of the content is selected for display of the augmented content before displaying the augmented content, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by allowing the user to precisely request augmented content for a desired portion of the content), and enabling the user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0389] In some embodiments, the one or more second criteria include a criterion requiring the user to view the first portion of the first content (e.g., 718a) or the second portion of the first content (e.g., 718b) shown in FIG. 7I for longer than a time threshold (e.g., 0.2 seconds, 0.3 seconds, 0.4 seconds, 0.5 seconds, etc.) (1126).

[0390] In some embodiments, the one or more first criteria also include a criterion that is met when the user views the respective portion of the content for longer than a time threshold that is greater than a time threshold associated with the second criterion. For example, the time threshold associated with the one or more first criteria is, optionally, 0.5 seconds, 0.6 seconds, 0.7 seconds, 0.8 seconds, 0.9 seconds, or 1 second.

[0391] The above method of updating the appearance of individual portions of a content item that a user is viewing in response to determining that the user has viewed the individual portion for at least a threshold amount of time allows the electronic device to display content in a consistent manner as the user is viewing the content without attempting to view the augmented content, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by reducing visual clutter and thereby allowing the user to view the content more quickly), and thereby further reducing power usage and improving the battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently.

[0392] In some embodiments, in response to detecting at least a portion of the user's head movement (1128), in accordance with a determination that at least a portion of the user's head movement is not in a direction that satisfies a first criterion, the electronic device (e.g., 101) maintains (1130) the display of the first portion of the first content (e.g., 718a) and the second portion of the first content (e.g., 718b) with visual characteristics having a first value, as in FIG. 7C .

[0393] In some embodiments, if the user moves their head in a direction different from the movement direction that satisfies one or more first criteria, or if the user performs additional head movements while moving their head in a direction that satisfies the first criteria, the electronic device ceases updating the visual characteristics of the respective portions of the content. For example, a head movement associated with the first criteria is a forward head movement relative to the user's body. In this example, a backward head movement without forward movement, or a left or right head movement, does not cause the electronic device to update the visual characteristics of the portions of the content item. Also in this example, a forward head movement that includes a head up / down or sideways head rotation does not cause the electronic device to update the display of the respective portions of the content item if the first criteria excludes additional movements. In some embodiments, the first / second portions of the first content are not displayed with updated visual characteristics even if the head movement criteria are not met and the time threshold of the gaze criteria is met.

[0394] The above method of ceasing to update the visual characteristics of individual portions of a content item in response to detecting that the head movement is not progressing in a direction that meets the first criterion enables the electronic device to display content with reduced visual clutter when the user moves their head in a manner that differs from the movement associated with the first criterion, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by allowing the user to read the content more quickly even when the head is not stationary), and enabling the user to use the electronic device more quickly and efficiently, further reducing power usage and improving the battery life of the electronic device.

[0395] In some embodiments, before displaying the first augmented content (e.g., 720a) shown in FIG. 7J or the second augmented content (e.g., 720b) ​​shown in FIG. 7K, a first portion of the first content (e.g., 718a) is displayed with visual characteristics having a second value and a visual characteristic having a first value while one or more second criteria are met (1132) (e.g., the user is looking at the first or second portion of the first content and performs a head movement that meets one or more second criteria but does not meet one or more first criteria), as in FIG. 7I. While displaying the second portion of the first content (e.g., 718b) with the visual characteristic having a second value, the electronic device (e.g., 101) detects (1134) that the user's gaze (e.g., 716n) changes from being directed at the first portion of the first content (e.g., 718a) to being directed at the second portion of the first content (e.g., 718b), as in FIG. 7K (e.g., the first portion of the first content is displayed with visual characteristics having a second value in response to detecting that the user is looking at the first portion of the first content).

[0396] In some embodiments, in response to detecting a change in the user's gaze (e.g., 716n) from being directed at a first portion of the first content (e.g., 718a) to being directed at a second portion of the first content (e.g., 718b) (1136), the electronic device (e.g., 101) updates the user interface (1138) to display the first portion of the first content (e.g., 718a) with visual characteristics having a first value and display the second portion of the first content (e.g., 718b) with visual characteristics having a second value, as shown in FIG. 7K.

[0397] Similarly, if the user is viewing a second portion of the content while the second criterion is met, the electronic device optionally displays the second portion of the content with visual characteristics having a second value and displays the first portion of the content with visual characteristics having the first value.

[0398] In some embodiments, if the user views the second portion of the content while the one or more second criteria are met, and then views the first portion of the content, the electronic device displays the first portion of the content with visual characteristics having the second value and the second portion of the content with visual characteristics having the first value as the user's gaze moves from the second portion of the content to the first portion. In some embodiments, if the electronic device does not detect the user's gaze changing from being directed at the first portion of the content to being directed at the second portion of the first content, the electronic device optionally ceases updating the user interface to display the first portion of the first content with visual characteristics having the first value and the second portion of the first content with visual characteristics having the second value.

[0399] The above-described method of updating visual characteristics of the first and second portions of content in response to detecting a user's gaze movement from the first portion to the second portion of the content enables the electronic device to quickly and efficiently provide the user with feedback regarding which portion of the content the displayed augmented content corresponds to and the ability to change which portion of the content the augmented content corresponds to, thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by reducing the time and inputs it takes for the user to accurately select the portion of the content where the augmented content will be displayed), and further reducing power usage and improving the battery life of the electronic device by allowing the user to use the electronic device more quickly and efficiently.

[0400] In some embodiments, while displaying the second augmented content (e.g., 720b) ​​shown in FIG. 7K (e.g., augmented content associated with the second portion of the first content), while displaying the first portion of the first content (e.g., 718a) with visual characteristics having a first value and the second portion of the first content (e.g., 718b) with visual characteristics having a second value, as in FIG. 7K, the electronic device (e.g., 101) detects (1140) that the user's gaze changes from being directed toward the second portion of the first content (e.g., 718b) to being directed toward the first portion of the first content (e.g., 718a).

[0401] In some embodiments, as shown in FIG. 7K, in response to detecting that the user's gaze moves from being directed at the second portion of the first content (e.g., 718b) to being directed at the first portion of the first content (e.g., 718a), the electronic device (e.g., 101) continues to display (1142) the first portion of the first content (e.g., 718a) with visual characteristics having a first value and the second portion of the first content (e.g., 718b) with visual characteristics having a second value.

[0402] In some embodiments, when the augmented content is displayed, moving the user's gaze from one portion of the first content to another does not change the portion of the first content with which the augmented content is associated. The second augmented content optionally continues to be displayed in response to detecting a user's gaze movement from a second portion of the first content to the first portion of the first content. In some embodiments, before displaying the second augmented content, the electronic device displays the first portion of the content with visual characteristics having a second value, and in response to detecting a user's head movement that meets a criterion while displaying the second portion of the content with visual characteristics having the first value, the electronic device displays the augmented content corresponding to the first portion. Optionally, in response to detecting that the user is looking at the second portion of the content item while displaying the first augmented content corresponding to the first portion of the content item, the electronic device continues to display the first augmented content, displaying the first portion of the content item with visual characteristics having the second value and the second portion of the second content item with visual characteristics having the first value.

[0403] The above-described method of continuing to display the second portion of the first content with visual characteristics having a second value after detecting the user's gaze moving from the second portion of the first content to the first portion of the first content while displaying the second augmented content allows the electronic device to avoid unnecessary user interface changes (e.g., the visual characteristics of the first and second portions of the content item changing), thereby simplifying the interaction between the user and the electronic device, improving usability of the electronic device, making the user device interface more efficient (e.g., by allowing the user to review the augmented content more quickly), and allowing the user to use the electronic device more quickly and efficiently, thereby further reducing power usage and improving the battery life of the electronic device.

[0404] In some embodiments, while displaying a first portion of the first content (e.g., 718a) with visual characteristics having a second value, as in FIG. 7I (1144), before displaying the first augmented content (e.g., 720a) shown in FIG. 7J or the second augmented content (e.g., 720b) ​​shown in FIG. 7K, the electronic device (e.g., 101) detects a second movement of the user's head that satisfies one or more third criteria, including a criterion requiring the second movement of the user's head to be a progression away from satisfying the first criterion (e.g., a forward movement of the user's head relative to the user's body satisfies the first and second criteria, and a backward movement of the user's head (e.g., to a ne...

Claims

1. 1. A method comprising:

1. An electronic device in communication with a display generation component and an eye-tracking device, comprising: displaying, via the display generation component, a user interface including a first content area, the first content area including a first portion of a content item; determining, via the eye-tracking device, that a gaze of a user of the electronic device is directed toward the first portion of the content item while displaying the user interface; In response to determining that the user's gaze is directed toward the first portion of the content item, In response to a determination that the eye movement of the user indicates that the user is reading the first portion of the content item, scrolling the user interface, where scrolling the user interface includes updating the user interface to include a second content area, the second content area including the first portion of the content item and a second portion of the content item that was not displayed in the first content area; and responsive to a determination that the user's gaze is directed toward the first portion of the content item without movement in a manner indicative of the user reading the first portion of the content item; ceasing to update the user interface to include the second content region.

2. The user interface further includes a third content area, the third content area including a first portion of a second content item, and the method further includes: determining, via the eye-tracking device, that the gaze of a user of the electronic device is directed toward the first portion of the second content item while displaying the user interface; In response to determining that the user's gaze is directed toward the first portion of the second content item, In response to a determination that the movement of the user's gaze indicates that the user is reading the first portion of the second content item, updating the user interface to not display the second content area and to include a fourth content area, the fourth content area including the first portion of the second content item and a second portion of the second content item that was not displayed in the third content area; The method of claim 1 further comprising:

3. displaying, via the display generation component, one or more other content items with visual characteristics having a first value before determining that the gaze of the user of the electronic device is directed toward the first portion of the content item; In response to determining that the user's gaze is directed toward the first portion of the content item, In response to the determination that the movement of the user's gaze indicates that the user is reading the first portion of the content item, updating the user interface to display the one or more other content items with the visual characteristic having a second value different from the first value; The method of claim 1 or 2, further comprising:

4. The method of claim 3 , wherein the visual property is transparency, and the first value corresponds to a lower transparency than the second value.

5. The method of claim 3 or 4, wherein the visual property is size, and the first value corresponds to a larger size than the second value.

6. the user interface further includes a third content area, the third content area including a first portion of a second content item; 6. The method of claim 1, wherein updating the user interface to include the second content region comprises displaying the second content region superimposed on the third content region.

7. 7. The method of claim 6, wherein before determining that the movement of the user's gaze indicates that the user of the electronic device is reading the first portion of the content item, the first content area and the third content area are displayed at a first z-height within the user interface, and updating the user interface to include the second content area comprises displaying the second content area at the first z-height within the user interface and changing the z-height of the third content area to a second z-height within the user interface.

8. 8. The method of claim 1, wherein updating the user interface to include the second content area comprises expanding a size of the first content area to become the second content area.

9. 9. The method of claim 1, wherein updating the user interface to include the second content area comprises gradually animating the first content area into the second content area while the user continues to read the first portion of the content item.

10. determining, via the eye-tracking device, during or after updating the user interface to include the second content region, that the gaze of the user of the electronic device is no longer directed at the content item; and displaying the user interface including the first content area without displaying the second content area in response to determining that the gaze of the user of the electronic device is no longer directed at the content item; 10. The method of claim 1, further comprising:

11. determining, via the eye-tracking device, during or after updating the user interface to include the second content area, that the movement of the user's eye gaze no longer indicates that the user of the electronic device is reading the content item; and In response to determining that the movement of the user's gaze no longer indicates that the user of the electronic device is reading the content item, restoring the user interface to include the first content area without displaying the second content area; 10. The method of claim 1, further comprising:

12. 12. The method of claim 1, wherein the determining that the movement of the user's gaze indicates that the user is reading the first portion of the content item is based on detecting, using the gaze tracking device, eye movements of the user corresponding to reading text.

13. detecting, at the electronic device, an indication of a notification event; In response to detecting the indication of the notification event, generating, at the electronic device, an indication of the notification event in accordance with a determination that the movement of the user's gaze indicates that the user is not reading a content item within the user interface; and ceasing to generate the indication of the notification event in accordance with a determination that the movement of the user's gaze indicates that the user is reading a content item within the user interface; and 13. The method of claim 1, further comprising:

14. In response to determining that the user's gaze is directed toward the first portion of the content item, In response to the determination that the movement of the user's gaze indicates that the user is reading the first portion of the content item, moving a predetermined portion of the second content area from a first position within the user interface to a second position within the user interface as the user continues to read the content item; 14. The method of any one of claims 1 to 13, further comprising:

15. Moving the second content region from the first position in the user interface to the second position in the user interface includes: In response to detecting that the user has read a first portion of the content item, moving the second content area a first portion of the movement from the first location to the second location; 15. The method of claim 14, further comprising: after moving the second content region during the first portion of the movement from the first location to the second location, in response to detecting that the user has read a second portion of the content item, moving the second content region during the second portion of the movement from the first location to the second location.

16. 16. The method of claim 1, wherein while the user's gaze is directed at the first portion of the content item and before the second content area is displayed, the first portion of the content item is displayed at a first position within the user interface, and updating the user interface to include the second content area comprises maintaining the display of the first portion of the content item in the second content area at the first position within the user interface.

17. 17. The method of claim 1, wherein the first portion of the content item in the first content area occupies an area of ​​the user interface having a first size, and the first portion of the content item in the second content area occupies an area of ​​the user interface having the first size.

18. 1. An electronic device comprising: one or more processors; Memory and and one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing the method of any one of claims 1 to 17. Electronic devices.

19. A program comprising instructions which, when executed by one or more processors of an electronic device, cause said electronic device to perform the method of any one of claims 1 to 17.

Images