Gaze target for activating user interface element

Abstract

The invention relates to a gaze target for activating a user interface element. A method includes displaying a user interface (UI) including a first UI element associated with a first UI operation and a second UI element associated with a second UI operation. The method includes, in response to detecting a first gaze input directed to one of the first UI element and the second UI element, displaying a visual indication at a fixed location separate from the first UI element and the second UI element. The method includes, in response to detecting a second gaze input directed to the visual indication via an eye tracker, performing the first UI operation in response to the first gaze input directed to the first UI element, and performing the second UI operation in response to the first gaze input directed to the second UI element.

Description

Technical Field

[0001] The present disclosure generally relates to gaze targets for activating user interface elements. Background Art

[0002] Some devices include a display that presents content. The content may include a user interface having various user interface elements that can be selected by a user of the device. The user may select one of the user interface elements by providing user input. The user may provide user input through various input devices (such as a touch screen, a mouse, a keyboard) or via staring. The device may include a camera that captures an image of the user's eyes, and the device may determine the direction of staring based on the image of the user's eyes. Stare tracking may sometimes be resource intensive and / or inaccurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] So that the present disclosure can be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0004] Figures 1A to 1S is an illustration of an example operating environment according to some specific implementations.

[0005] Figure 2 is an illustration of an eye tracking system according to some specific implementations.

[0006] Figure 3 is a flowchart representation of a method for selecting a user interface element based on gaze according to some specific implementations.

[0007] Figure 4 is a block diagram of a device for gaze-based selection of user interface elements according to some specific implementations.

[0008] As is common practice, the various features illustrated in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. Summary of the invention

[0009] Various embodiments disclosed herein include devices, systems, and methods for selecting user interface elements based on gaze. In some embodiments, a method is performed at an electronic device including a non-volatile memory, one or more processors, a display, and an eye tracker. In some embodiments, the method includes: displaying a user interface (UI) on a display, the user interface including a first UI element associated with a first UI operation and a second UI element associated with a second UI operation. In some embodiments, the method includes: in response to detecting a first gaze input pointing to one of the first UI element and the second UI element via an eye tracker, displaying a visual indicator at a fixed position separated from the first UI element and the second UI element. In some embodiments, the method includes: in response to detecting a second gaze input pointing to the visual indicator via an eye tracker, in response to the first gaze input pointing to the first UI element, performing a first UI operation. In some embodiments, the method includes: in response to detecting a second gaze input pointing to the visual indicator via an eye tracker, in response to the first gaze input pointing to the second UI element, performing a second UI operation.

[0010] According to some specific implementations, the device includes one or more processors, multiple sensors, non-transitory memory, and one or more programs. In some specific implementations, one or more programs are stored in non-transitory memory and executed by one or more processors. In some specific implementations, one or more programs include instructions for executing or causing the execution of any method described herein. According to some specific implementations, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to execute or cause the execution of any of the methods described herein. According to some specific implementations, a device includes one or more processors, non-transitory memory, and a component for executing or causing the execution of any of the methods described herein. DETAILED DESCRIPTION

[0011] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail so as not to obscure more relevant aspects of the example implementations described herein.

[0012] Some devices utilize staring as an input method. The device may include a user-facing camera that captures an image of the user's eyes. The device utilizes the captured image to determine the display location that the user is staring at, the direction that the user is staring at, and / or the duration that the user has stared at the display location. Some devices register a user selection of a user interface (UI) element after detecting that the user's stare is directed at the UI element for a certain amount of time. Stare tracking may sometimes be inaccurate. For example, a user may inadvertently select a UI element by staring at the UI element.

[0013] The present disclosure provides a method, system and / or device for selecting a UI element via staring. The device detects that a user is staring at a first UI element from a set of UI elements. After detecting that the user is staring at the first UI element, the device displays a visual indicator at a fixed position. The visual indicator is not displayed before the user stares at the first UI element. Staring at the visual indicator is used as a confirmation that the user intends to select the first UI element. In this way, if the user stares at the visual indicator after staring at the first UI element, the first UI element is activated and the device performs a first UI operation associated with the first UI element. For example, if the first UI element is a first icon for launching a first application, when the user stares at the first icon and then stares at the visual indicator, the device starts the first application. If the user initially stares at the second UI element, the device still displays the visual indicator at the same fixed position. However, this time staring at the visual indicator is used as a confirmation that the user intends to select the second UI element. In this way, if the user stares at the visual indicator after staring at the second UI element, the second UI element is activated and the device performs a second UI operation associated with the second UI element. For example, if the second UI element is a second icon for launching a second application, when the user stares at the second icon and then stares at the visual indication, the device launches the second application.

[0014] Using visual indicators as a common auxiliary staring target for confirming the selection of various UI elements often saves space in UIs with limited spacing. For example, some UIs may include UI elements that are densely packed together. In such densely packed UIs, there may not be enough space to display a dedicated auxiliary staring target for each UI element. In this way, the device saves space by using a visual indicator as a common auxiliary staring target shared by various UI elements. Using a dedicated auxiliary staring target for each UI element may also prevent the device from displaying the UI element at the display position where the dedicated auxiliary staring target should be displayed, thereby leaving less space available for the UI element. In this way, using a visual indicator as a common auxiliary staring target allows the device to display additional UI elements, thereby using the display space more efficiently. Regardless of which UI element the user initially stares at, the device displays a visual indicator at a fixed position, so that the visual indicator appears predictably at the same position, and the user does not have to search for the visual indicator after staring at a specific UI element.

[0015] The device may display a visual indicator with a specific visual appearance based on the visual characteristics of the UI element that the user stares at to trigger the display of the visual indicator. The device may use color to associate the visual indicator with the UI element that the user stares at just before displaying the visual indicator. For example, in response to the user staring at a blue button, the device displays the visual indicator as a blue circle to indicate that staring at the visual indicator confirms the selection of the blue button. For another example, in response to the user staring at a red button, the device displays the visual indicator as a red circle to indicate that staring at the visual indicator confirms the selection of the red button. Additionally or alternatively, the device may change the color of the UI element that the user stares at so that the color of the UI element matches the color of the visual indicator. For example, if all UI elements are displayed with the same background color (e.g., blue), the device may change the background color of the UI element that the user stares at to a different color (e.g., red) to match the color of the visual indicator.

[0016] Some UI elements are associated with multiple actions. For example, tapping a mail icon launches the mail application to a default state (e.g., the inbox), and long pressing a mail icon triggers display of a menu with more specific actions (such as composing a new message, performing a search, viewing "VIP" messages, etc.). The device can simultaneously display multiple visual indicators that trigger different actions associated with the previously stared UI element. For example, staring at a mail icon causes the device to simultaneously display a first visual indicator that, when stared at, launches the mail application to a default state and a second visual indicator that, when stared at, opens a message composition window.

[0017] The device provides the user with enough time for his / her gaze to travel from the UI element to the visual indicator in order to confirm the selection of the UI element. The path from the position of the UI element initially stared at to the position of the visual indicator is designated as a safe area, which allows the user to stare along the path without triggering the selection of other UI elements that coincide with the path. In this way, if the gaze is moving from the first button to the visual indicator used as an auxiliary target for the first button, the gaze passing the second button will not activate the second button. However, the gaze on the second button may cause the safe area to dissipate and trigger the selection of the second button. If the gaze stays in the safe area without moving toward the visual indicator, the safe area dissipates and the first button no longer remains selected.

[0018] The device can specify a specific display position as a trigger for displaying the home screen (e.g., a grid of icons for launching the corresponding application). When the device detects that the gaze is directed to the specific display position, the device displays the home screen. The display position for triggering the display of the home screen can be a corner (e.g., the lower left corner of the screen). The display position for triggering the display of the home screen can be referred to as a "triggering angle". As an example, a user stares at a specific UI element. In this example, staring at a specific UI element triggers the display of a visual indicator, which is used as an auxiliary staring target for confirming the selection of the specific UI element that the user stared at. Continuing with this example, if the user stares at the triggering angle instead of the visual indicator, the device cancels the selection of the specific UI element and displays the home screen.

[0019] Figure 1A 1 is a diagram of an illustrative example physical environment 10 according to some specific implementations. Although relevant features are shown, those of ordinary skill in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and in order not to obscure more relevant aspects of the example specific implementations disclosed herein. To this end, as a non-limiting example, the physical environment 10 includes an electronic device 20 and a user 12 of the electronic device 20. The user 12 has eyes 14 (e.g., a left eye 14a and a right eye 14b). In some specific implementations, the electronic device 20 includes a handheld computing device that can be held by the user 12. For example, the electronic device 20 may include a smart phone, a tablet computer, a media player, a laptop computer, etc. that can be held by the user 12. Alternatively, the electronic device 20 includes a wearable computing device that can be worn by the user 12. For example, the electronic device 20 may include an HMD or an electronic watch that can be worn by the user 12.

[0020] In various implementations, the electronic device 20 includes a display 22, an image sensor 24 (e.g., a camera), a flash source 26 (e.g., a set of one or more light emitting diodes (LEDs)), and an eye tracking system 200 (hereinafter referred to as "system 200" for brevity). In various implementations, the display 22 displays a user interface (UI) 30 including various UI elements 40 (e.g., a first UI element 40a, a second UI element 40b, a third UI element 40c, ..., and a twelfth UI element 40l). In some implementations, the UI 30 is referred to as a graphical user interface (GUI), and the UI elements 40 are referred to as GUI elements. In some implementations, the UI 30 is part of an extended reality (XR) environment, and the UI elements 40 are virtual objects displayed within the XR environment. In various implementations, UI element 40 includes a user-selectable affordance, such as an icon, a button, a drop-down box, a text field for receiving text input, an expandable information panel, a link (e.g., a hyperlink for a web page or a uniform resource identifier (URI) for an application screen), playable multimedia content (e.g., a thumbnail representing an audio content item (such as an album or song) or a video content item (such as a music video, a TV show, or a movie), etc. In some implementations, UI element 40 displays text, graphics, images, and / or videos. In some implementations, UI element 40 is associated with a corresponding visual characteristic value (e.g., a corresponding background color, a corresponding size, a corresponding font color, a corresponding font style, etc.).

[0021] In some implementations, user activation of a specific one of the UI elements 40 triggers the electronic device 20 to perform a corresponding UI operation. For example, user activation of the first UI element 40a triggers the electronic device 20 to perform a first UI operation, user activation of the second UI element 40b triggers the electronic device 20 to perform a second UI operation, ..., and user activation of the twelfth UI element 40l triggers the electronic device 20 to perform a twelfth UI operation. In some implementations, the UI 30 represents a home screen, and the UI elements 40 represent corresponding icons for launching corresponding applications installed on the electronic device 20. For example, the first UI element 40a represents a news application icon for launching a news application installed on the electronic device 20, the second UI element 40b represents an instant messaging icon for launching an instant messaging application, the third UI element 40c represents a calendar application icon for launching a calendar application, the fourth UI element 40d represents a music application icon for launching a music application, the fifth UI element 40e represents a camera application icon for launching a camera application, and the sixth UI element 40f represents a mail application icon for launching a mail application installed on the electronic device 20.

[0022] In various implementations, system 200 obtains image data 50 from image sensor 24. Image data 50 includes a set of one or more images of eye 14 of user 12. Figure 1B , system 200 uses image data 50 to detect gaze input 52. Gaze input 52 indicates that user 12 is gazing at first UI element 40a. Figure 1C In response to detecting a gaze input 52 at a position corresponding to the first UI element 40a, the electronic device 20 displays a visual indicator 54. The visual indicator 54 is used as an auxiliary gaze target for confirming the user selection of the first UI element 40a. The gaze input 52 is used as a preliminary selection of the first UI element 40a. However, before the user 12 stares at the visual indicator 54 to confirm the preliminary selection of the first UI element 40a, the electronic device 20 does not activate the first UI element 40a. In this way, the gaze input 52 that coincides with the first UI element 40a will not immediately trigger the electronic device to perform the UI operation associated with the first UI element 40a. For example, if the first UI element 40a is a news icon for a news application, then simply staring at the news icon will not cause the electronic device 20 to launch the news application. In this example, in order to launch the news application, the user 12 must stare at the news icon and then stare at the visual indicator 54 to confirm his / her intention to launch the news application, thereby reducing the possibility of accidentally launching the news application.

[0023] like Figure 1C As shown in FIG. 4 , the electronic device 20 displays a visual indication mark 54 at a position separated from the first UI element 40a. Figure 1C In the example of , the visual indicator 54 is displayed at a location that does not overlap with any of the UI elements 40. In some implementations, the electronic device 20 displays the visual indicator 54 at a location where user-selectable UI elements are not typically displayed. In some implementations, the electronic device 20 displays the visual indicator 54 at a predetermined location (e.g., a pre-specified location). In some implementations, the visual indicator 54 is displayed at a location reserved for confirming a previous gaze selection. Figure 1C In the example of FIG. 5 , the visual indicator 54 is displayed near the top edge of the display 22. However, the visual indicator 54 may be displayed in other portions of the display 22 (e.g., near the bottom edge of the display 22, at the center of the display 22, or near a corner of the display 22).

[0024] exist Figure 1CIn the example of , electronic device 20 has indicated a user selection of first UI element 40a by changing the visual appearance of first UI element 40a while maintaining the corresponding visual appearance of the remaining UI elements 40. For example, electronic device 20 displays the text included in first UI element 40a in bold, without bolding the text included in the remaining UI elements 40. As another example, electronic device 20 displays the border of first UI element 40a with a thicker outline, without bolding the borders of the remaining UI elements 40. Changing the visual appearance of first UI element 40a indicates that visual indicator 54 is associated with first UI element 40a and staring at visual indicator 54 will trigger activation of first UI element 40a, thereby causing electronic device 20 to perform a first UI operation associated with first UI element 40a.

[0025] refer to Figure 1D and Figure 1E , the electronic device 20 detects a gaze input 56 at a location corresponding to the visual indicator 54. The gaze input 56 serves as a confirmation of the user 12's intention to activate the first UI element 40a and perform a UI operation associated with the first UI element 40a. In response to detecting the gaze input 56 directed to the visual indicator 54, the electronic device 20 displays a news main page 60 having UI elements 62 representing news reports (e.g., UI element 62a representing a first news report, UI element 62b representing a second news report, ..., and UI element 62f representing a sixth news report). Similar to Figures 1A to 1D , UI element 62 may be selected via gaze and may be referred to as a gaze-selectable element.

[0026] refer to Figure 1F and 1G , the electronic device 20 detects the gaze input 70 directed to the sixth UI element 40f. Figure 1G , in response to detecting a gaze input 70 directed to sixth UI element 40f, electronic device 20 displays visual indicator 54. Electronic device 20 changes the visual appearance of sixth UI element 40f so that sixth UI element 40f appears different from the remaining UI elements 40. For example, the text of sixth UI element 40f is shown in bold, and the border of sixth UI element 40f is shown in a thicker line. Changing the visual appearance of sixth UI element 40f indicates that staring at visual indicator 54 will confirm the selection of sixth UI element 40f, thereby triggering electronic device 20 to perform a sixth UI operation associated with sixth UI element 40f.

[0027] refer to Figure 1H, electronic device 20 detects gaze input 71 directed to visual indicator 54. Gaze input 71 confirms the user selection of sixth UI element 40f. In response to detecting gaze input 71 confirming the user selection of sixth UI element 40f, electronic device 20 performs a sixth UI operation associated with sixth UI element 40f. Fig. 1I As shown in FIG. 1 , the electronic device 20 displays a mail application home page 72. In some implementations, the sixth UI element 40f is a mail application icon that triggers the electronic device 20 to launch a mail application and display the mail application home page 72. The mail application home page 72 includes a search bar 74 for searching for messages and messages 76 received by the electronic device 20.

[0028] Figure 1J A gaze input 80 is illustrated directed to UI element 62d representing a fourth news story displayed by the news application within the news home page 60. Figure 1K As shown in FIG, in response to detecting gaze input 80, electronic device 20 displays a visual indicator 82 for confirming user selection of UI element 62d. Electronic device 20 also changes the corresponding visual appearance of UI element 62. Figure 1K In the example of FIG. 1 , electronic device 20 displays UI element 62d in bold and with a thicker border, and electronic device 20 displays the remaining UI elements 62 in gray and with a narrower border. Changing the visual appearance of UI element 62d relative to the remaining UI elements 62 indicates that UI element 62d has been preliminarily selected and user 12 needs to stare at visual indicator 82 in order to confirm the selection of UI element 62d.

[0029] Figure 1K An expected gaze travel path 84 from UI element 62d to visual indicator 82 is illustrated. In some implementations, when the gaze of user 12 is traveling from UI element 62d to visual indicator 82 and the gaze remains within the boundaries defining expected gaze travel path 84, electronic device 20 maintains the initial selection of UI element 62d and continues to display visual indicator 82. If electronic device 20 detects that the gaze is not traveling along expected gaze travel path 84 or the gaze breaks (e.g., exceeds) the boundaries defining expected gaze travel path 84, electronic device 20 stops displaying visual indicator 82 and cancels the selection of UI element 62d (e.g., cancels the initial selection thereof).

[0030] Figure 1LThe actual gaze path of travel 86 is illustrated as remaining within the boundaries defining the expected gaze path of travel 84. As the gaze is traveling along the actual gaze path of travel 86, since the actual gaze path of travel 86 does not include gaze positions outside the boundaries defining the expected gaze path of travel 84, the electronic device 20 maintains the initial selection of the UI element 62d and continues to display the visual indicator 82.

[0031] Figure 1M An actual gaze travel path 88 that breaks through the boundaries defining the expected gaze travel path 84 is illustrated. Figure 1M In the example of , actual gaze path of travel 88 goes from UI element 62d to UI element 62e. Because actual gaze path of travel 88 includes a gaze position outside the boundary defining expected gaze path of travel 84, electronic device 20 stops maintaining the initial selection of UI element 62d (e.g., as indicated by the blackening of text included in the remaining UI elements 62) and displays the gradual disappearance of visual indicator 82 (e.g., as indicated by the graying of visual indicator 82). When actual gaze path of travel 88 goes outside the boundary defined by expected gaze path of travel 84, expected gaze path of travel 84 dissipates (e.g., as indicated by the graying of the boundary defining expected gaze path of travel 84) and visual indicator 82 gradually disappears (e.g., as indicated by the graying of visual indicator 82).

[0032] Figure 1N and Fig.1O A continuous event of gaze resting on UI element 62b coinciding with expected gaze travel path 84 is illustrated. Figure 1N An actual gaze travel path 90 ending at gaze location 92 is illustrated. While gaze is moving along actual gaze travel path 90, gaze remains stationary at gaze location 92 for a duration 94 that exceeds a threshold duration 96. Since gaze remains stationary at gaze location 92 corresponding to the location of UI element 62b, electronic device 20 determines that user 12 may have intended to select UI element 62b. As such, electronic device 20 cancels the selection of UI element 62d, and electronic device 20 preliminarily selects UI element 62b. Figure 1N and Fig.1O , deselection of UI element 62d is shown as a change in text color from black to gray, and initial selection of UI element 62b is shown as a change in text color from gray to black.

[0033] exist Fig.1O , the electronic device 20 displays the dissipation of the expected gaze path of travel 84 by displaying the expected gaze path of travel 84 in gray. Fig.1O In the embodiment, the electronic device 20 maintains the display of the visual indicator 82. However, in Fig.1O, visual indicator 82 serves as confirmation for selecting UI element 62b instead of UI element 62d, as indicated by UI element 62b being shown in black and UI element 62d being shown in gray. In some implementations, staring remaining stationary at staring location 92 refers to microsaccades of staring at or near staring location 92. Therefore, smaller movements of staring near staring location 92 are still considered to be that user 12 is staring at UI element 62b. More generally, detecting microsaccades of staring when staring travels along expected staring travel path 84 can trigger electronic device 20 to cancel the selection of the preliminarily selected UI element and preliminarily select another UI element for which microsaccades are occurring.

[0034] Figure 1P 4. The example shows that after the first UI element 40a has been preliminarily selected via gaze (e.g., after the electronic device 20 has detected Figure 1B 30 after the gaze input 52 shown in FIG. Figure 1C , the electronic device 20 displays a visual indicator 54 to confirm the selection of the first UI element 40a. However, the electronic device 20 also displays a second visual indicator 100. The first UI element 40a can be associated with multiple UI operations. For example, the first UI element 40a corresponds to a news application icon, and the first UI operation associated with the news application icon includes launching the news application and displaying the following: Figure 1E , and a second UI operation associated with the news application icon includes launching the news application and displaying a "My Sports" page (e.g., showing sports-related news and the latest scores of favorite sports teams) Figure 1R ).

[0035] exist Figure 1P In the example of FIG. 5 , visual indicator 54 is associated with a first UI operation (e.g., opening the news application home page), and second visual indicator 100 is associated with a second UI operation (e.g., opening the “My Sports” page). Thus, staring at visual indicator 54 serves as a confirmation for opening the news application home page of the news application (e.g., Figure 1E ), and staring at the second visual indicator 100 serves as confirmation for opening the "My Sports" page of the news application (as shown in Figure 1R ).

[0036] refer to Figure 1Q, the electronic device 20 detects a gaze input 102 directed to the second visual indicator 100. The gaze input 102 corresponds to a request to perform a second UI operation associated with the first UI element 40a. For example, the gaze input 102 corresponds to a request to open a "My Sports" page of the news application instead of opening the news application main page of the news application. Thus, in response to detecting the gaze input 102 at a position corresponding to the second visual indicator 100, the electronic device 20 opens the "My Sports" page 60' of the news application, which includes UI elements 62g, 62h, and 62i representing sports-related news.

[0037] refer to Figure 1S , the sixth UI element 40f (e.g., a mail application icon) may be associated with four UI operations: a first UI operation of opening the mail application to a default state (e.g., an inbox), a second UI operation of opening a new message composition window generated by the mail application, a third UI operation of opening a VIP message folder having a message subgroup that has been marked as "VIP", and a fourth UI operation of opening a search bar for searching messages. After the electronic device 20 detects the gaze input 70 directed to the sixth UI element 40f, the electronic device 20 displays four separate buttons for triggering a corresponding one of the four UI operations associated with the mail application icon.

[0038] exist Figure 1S In the example of FIG. 1 , the electronic device 20 displays an open button 110 near the top edge of the display 22. The operation of the open button 110 is similar to Figure 1G Thus, staring at the open button 110 triggers the electronic device 20 to perform the first UI operation and open the mail application to the default state, such as Fig. 1I . In addition to displaying the open button 110, the electronic device 20 also displays a new message button 112 near the bottom edge of the display 22. The new message button 112 is associated with a second UI operation of opening a new message composition window. In this way, staring at the new message button 112 triggers the electronic device 20 to perform the second UI operation and open a new message composition window of the mail application, thereby reducing the need for the user 12 to provide additional user input corresponding to the request to open a new message composition window after launching the mail application.

[0039] In addition to displaying the open button 110 and the new message button 112, the electronic device 20 also displays a VIP message button 114 near the left edge of the display 22. The VIP message button 114 is associated with a third UI operation of opening a VIP message folder. In this way, staring at the VIP message button 114 triggers the electronic device 20 to perform the third UI operation and open the VIP message folder of the mail application, thereby reducing the need for the user 12 to provide additional user input corresponding to a request to open the VIP message folder after launching the mail application.

[0040] In addition to displaying the open button 110, the new message button 112, and the VIP message button 114, the electronic device 20 also displays a search button 116 near the right edge of the display 22. The search button 116 is associated with a fourth UI operation of opening a search bar. Thus, staring at the search button 116 triggers the electronic device 20 to perform the fourth UI operation: opening the search bar of the mail application and placing a cursor in the search bar to indicate that the electronic device 20 is ready to accept a search string, thereby reducing the need for the user 12 to provide additional user input corresponding to selecting the search bar after launching the mail application.

[0041] Figure 1S Illustrated Figure 1C , Figure 1D , Figure 1G and Figure 1H The visual indicator 54 shown in FIG. 5 can be replaced with a button including text. The text can provide an indication of the UI operation that the electronic device 20 will perform when the user 12 stares at the visual indicator 54. Return to reference Figure 1C In some implementations, visual indicator 54 may include the same text as first UI element 40a to indicate that staring at visual indicator 54 serves as a confirmation for selecting first UI element 40a and performing a UI operation associated with first UI element 40a.

[0042] Figure 2 2 is a block diagram of a system 200 according to some implementations. In various implementations, the system 200 includes a data acquirer 210, a selection determiner 220, and a UI 232 (e.g., Figure 1A In various implementations, the data acquirer 210 obtains image data 212 (e.g., Figure 1A In various implementations, the data acquirer 210 generates a three-dimensional (3D) gaze vector 214 (hereinafter referred to as "gaze vector 214") based on the image data 212. For example, referring to Figure 1A and Figure 1B, the data acquirer 210 obtains the image data 50 from the image sensor 24, and the gaze vector 214 indicates the gaze input 52. In some implementations, the data acquirer 210 performs pupil detection and glint detection to generate the gaze vector 214. In some implementations, the gaze vector 214 indicates the origin of the user's gaze and the direction of the gaze. In some implementations, the gaze vector 214 indicates the gaze position, the gaze direction, the gaze intensity, and / or the gaze shape.

[0043] In some implementations, gaze vector 214 includes a first eye (e.g., Figure 1A 14a) and a first gaze vector of a second eye (eg, Figure 1A 14b). In some implementations, gaze vector 214 represents an average of the first gaze vector of the first eye and the second gaze vector of the second eye. For example, the gaze direction indicated by gaze vector 214 may be an average of the first gaze direction of the first eye and the second gaze direction of the second eye. In some implementations, the first gaze vector of the first eye is associated with a first confidence score, and the second gaze vector of the second eye is associated with a second confidence score. In such implementations, gaze vector 214 includes one of the first gaze vector and the second gaze vector based on their corresponding confidence scores (e.g., data acquirer 210 selects the gaze vector with a higher confidence score).

[0044] In various implementations, the selection determiner 220 determines which of the UI elements 224 has been selected via the gaze input indicated by the 3D gaze vector 214. In some implementations, the selection determiner 220 obtains a UI map 222 for the UI 232. The UI map 222 indicates the UI elements 224, the corresponding positions 226 of the UI elements 224, and the corresponding sizes 228 of the UI elements 224. The selection determiner 220 uses the UI map 222 to determine that the gaze position indicated by the 3D gaze vector 214 corresponds to the position of (e.g., overlaps with) a specific one of the UI elements 224. The specific one of the UI elements 224 to which the gaze is directed is referred to as the selected UI element 224a. The selection determiner 220 provides an indication of the selected UI element 224a to the UI renderer 230.

[0045] In various implementations, the UI renderer 230 renders a UI 232 having UI elements 224 (e.g., Figure 1A In some implementations, the UI renderer 230 presents a selection indicator 234 to indicate the selected UI element 224a. The selection indicator 234 indicates that the user has preliminarily selected the selected UI element 224a. For example, referring to Figure 1C , the UI renderer 230 displays the first UI element 40a with different visual characteristics from the remaining UI elements 40 (eg, by displaying the text of the first UI element 40a in bold while not displaying the text of the remaining UI elements 40 in bold).

[0046] In various implementations, the UI presenter 230 presents a visual indicator 236 to allow the user to confirm the selection of the selected UI element 224a. Figure 1C After the UI presenter 230 presents the visual indicator 236, the data acquirer 210 obtains the gaze input (eg, Figure 1D In response to detecting the gaze input directed to the visual indicator 236, the UI renderer 230 displays the execution of the UI operation associated with the selected UI element 224a (e.g., the UI renderer 230 displays the execution of the UI operation associated with the selected UI element 224a). Figure 1E The news main page 60 shown in FIG.

[0047] In some implementations, after presenting the visual indicator 236, the UI presenter 230 determines an expected gaze travel path from the selected UI element 224a to the visual indicator 236 (eg, Figure 1K In some implementations, when the user's gaze is traveling along the expected gaze travel path, the UI renderer 230 makes the subgroup of UI elements 224 that overlap the expected gaze travel path unselectable. Figure 1L As shown in , movement of gaze along actual gaze travel path 86 does not result in selection of UI element 62b.

[0048] Figure 3 is a flowchart representation of a method 300 for selecting a user interface element based on gaze. In various specific implementations, the method 300 comprises Figures 1A to 2 The system 200 shown is executed. In some specific implementations, the method 300 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, the method 300 is performed by a processor executing a code stored in a non-transitory computer readable medium (e.g., a memory).

[0049] As represented by block 310, in various implementations, method 300 includes: displaying a user interface (UI) on a display, the user interface including a first UI element associated with a first UI operation and a second UI element associated with a second UI operation. Figure 1A As shown in , the electronic device 20 displays a UI element 40 associated with a corresponding UI operation.

[0050] As represented by block 310a, in some implementations, the UI includes multiple elements having a spacing less than a threshold (e.g., a distance between UI elements less than a threshold). Figure 1A As shown in FIG. 4 , the UI 30 includes UI elements 40 displayed in a grid, without sufficient spacing to display a dedicated virtual indicator for each UI element 40 .

[0051] As represented by block 310b, in some implementations, the first UI element includes a first icon, a first button, a first link, or a first multimedia content item, and the second UI element includes a second icon, a second button, a second link, or a second multimedia content item. Figure 1A As shown in FIG. 4 , UI element 40 represents an icon for launching a corresponding application.

[0052] As represented by block 320, in some implementations, in response to detecting a first gaze input directed to one of the first UI element and the second UI element via the eye tracker, a visual indicator is displayed at a fixed location separate from the first UI element and the second UI element. Figure 1B to Figure 1C As shown in FIG. 5 , in response to detecting a gaze input 52 directed toward the first UI element 40a, the electronic device 20 displays a visual indicator 54, and as shown in FIG. Figure 1F to Figure 1G As shown in FIG. 4 , in response to detecting a gaze input 70 directed toward the sixth UI element 40 f , the electronic device 20 displays a visual indication 54 .

[0053] As represented by frame 320a, in some specific implementations, the fixed position is equidistant from the first UI element and the second UI element. In some specific implementations, no matter which UI element is stared at, a visual indicator is displayed at the fixed position, so that the user does not have to search for the visual indicator to confirm the selection of the UI element stared at by the user.

[0054] As represented by block 320b, in some implementations, displaying the visual indicator includes, in response to the first gaze input pointing to the first UI element, displaying the visual indicator with a first visual appearance that matches a first visual characteristic of the first UI element. In some implementations, the first visual characteristic of the first UI element includes a first color of the first UI element, and displaying the visual indicator with the first visual appearance includes displaying the visual indicator in the first color. For example, in Figure 1C, electronic device 20 may display visual indicator 54 in a first color that matches a background color of first UI element 40a. In some implementations, in response to the first gaze input being directed to a second UI element, the visual indicator is displayed in a second visual appearance that matches a second visual characteristic of the second UI element. In some implementations, the second visual characteristic of the second UI element includes a second color of the second UI element, and displaying the visual indicator in the second visual appearance includes displaying the visual indicator in the second color. For example, in Figure 1G In some embodiments, the method 300 includes: displaying the visual indicator in a predetermined color, and after detecting a first gaze input directed to the first UI element, changing the background color or font color of the first UI element to a predetermined color.

[0055] In some specific implementations, the first visual characteristic of the first UI element includes a first shape of the first UI element, and displaying the visual indicator with the first visual appearance includes displaying the visual indicator with the first shape. For example, if the first UI element is rectangular, the visual indicator is displayed as a rectangle. For another example, if the first UI element is rectangular with rounded corners, the visual indicator is displayed as a rectangle. In some specific implementations, the second visual characteristic of the second UI element includes a second shape of the second UI element, and displaying the visual indicator with the second visual appearance includes displaying the visual indicator with the second shape. For example, if the second UI element is elliptical, the visual indicator is displayed as an ellipse. For another example, if the second UI element is circular, the visual indicator is displayed as a circle. In some specific implementations, method 300 includes: after detecting a first gaze input directed to the first UI element, changing the shape of the first UI element (e.g., changing the shape of the first UI element from a rectangle to an ellipse).

[0056] As represented by block 320c, in some implementations, method 300 includes, in response to the first gaze input being directed to the first UI element, associating a visual indicator with a first UI operation such that staring at the visual indicator confirms user selection of the first UI element and triggers execution of the first UI operation associated with the first UI element. Figure 1KAs shown in , the association between visual indicator 82 and UI element 62d is indicated by displaying UI element 62d in black while graying out the remaining UI elements 62a, 62b, 62c, 62e, and 62f. In some implementations, method 300 includes: in response to the first gaze input being directed at the second UI element, associating the visual indicator with the second UI operation, so that staring at the visual indicator confirms the user selection of the second UI element and triggers the execution of the second UI operation associated with the second UI element. For example, Fig.1O As shown in FIG. 8 , the association between visual indicator 82 and UI element 62b is indicated by displaying UI element 62b in black and graying out the remaining UI elements 62a , 62c , 62d , 62e , and 62f .

[0057] As represented by block 320d, in some implementations, method 300 includes, when displaying the visual indicator at the fixed position, displaying a second visual indicator at a second fixed position separate from the first UI element and the second UI element. Figure 1P As shown in , the electronic device 20 simultaneously displays the visual indicator 54 at a fixed position (e.g., near the top of the display) and displays the second visual indicator 100 at a second fixed position (e.g., near the bottom of the display). In some embodiments, when the first gaze input is directed to the first UI element, staring at the second visual indicator triggers the execution of a third UI operation associated with the first UI element. In some embodiments, the third UI operation includes a first quick action related to the first application represented by the first UI element. For example, Figure 1Q and Figure 1R As shown in , staring at the second visual indicator 100 triggers the electronic device 20 to display the "My Sports" page of the news application. In some specific implementations, when the first gaze input is directed to the second UI element, staring at the second visual indicator triggers the execution of a fourth UI operation associated with the second UI element. In some specific implementations, the fourth UI operation includes a second quick action related to the second application represented by the second UI element. For example, Figure 1S As shown in , the electronic device 20 displays a new message button 112 for opening a new message composition window.

[0058] As represented by block 330, in various implementations, method 300 includes: in response to detecting a second gaze input directed to a visual indicator via an eye tracker, in response to the first gaze input directed to a first UI element, performing a first UI operation. Figures 1B to 1EAs shown in , when the news application icon is the last icon that the user 12 stares at before staring at the visual indicator 54, the electronic device 20 opens the news application home page. In some specific implementations, the method 300 includes: in response to detecting a second staring input directed to the visual indicator via the eye tracker, in response to the first staring input directed to the second UI element, performing a second UI operation. For example, Figure 1F to Figure 1I As shown in , when the mail application icon is the last icon that the user 12 stares at before staring at the visual indicator 54, the electronic device 20 opens the mail application main page.

[0059] As represented by block 330a, in some implementations, method 300 includes: when gaze is traveling along a first path from the first UI element to the visual indicator, in response to a first gaze input pointing to the first UI element, maintaining selection of the first UI element, and when gaze is traveling along a second path from the second UI element to the visual indicator, in response to the first gaze input pointing to the second UI element, maintaining selection of the second UI element. Figure 1L As shown in , as gaze travels along an actual gaze travel path 86 that is within the boundaries defining the expected gaze travel path 84 , the electronic device 20 maintains selection of the UI element 62 d .

[0060] In some implementations, method 300 includes: when gaze is maintained on a third UI element on a first path from the first UI element to the visual indicator, deselecting the first UI element, and when gaze is maintained on a fourth UI element on a second path from the second UI element to the visual indicator, deselecting the second UI element. Figure 1N and Fig.1O As shown in , when a microsaccade (e.g., a gaze with a small eye movement within a tolerance threshold) is detected at a location corresponding to UI element 62b on expected gaze travel path 84, electronic device 20 deselects UI element 62d.

[0061] In some implementations, method 300 includes: when gaze does not move along a first path from the first UI element toward the visual indicator, deselecting the first UI element, and when gaze does not move along a second path from the second UI element toward the visual indicator, deselecting the second UI element. Figure 1M As shown in , upon detecting an actual gaze path of travel 88 that deviates from the expected gaze path of travel 84 , the electronic device 20 deselects the UI element 62 d .

[0062] As represented by block 330b, in some implementations, the user interface includes a home screen, the first UI element includes a first icon for launching a first application, the second UI element includes a second icon for launching a second application, performing a first UI operation includes launching the first application in response to the first gaze input being directed to the first icon and the second gaze input being directed to the visual indicator, and performing a second UI operation includes launching the second application in response to the first gaze input being directed to the second icon and the second gaze input being directed to the visual indicator. For example, FIG. 1A to FIG. 1I As shown in FIG. 4 , the UI 30 includes a grid of icons, a first UI element 40 a includes a news application icon for launching a news application, and a sixth UI element 40 f includes a mail application icon for launching a mail application.

[0063] In some implementations, the first UI element includes a first user-selectable link for opening a first page, the second UI element includes a second user-selectable link for opening a second page, performing the first UI operation includes opening the first page in response to the first gaze input being directed to the first user-selectable link and the second gaze input being directed to the visual indicator, and performing the second UI operation includes opening the second page in response to the first gaze input being directed to the second user-selectable link and the second gaze input being directed to the visual indicator. For example, Figure 1E As shown in FIG. 6 , UI element 62 represents a news story, and user 12 may select UI element 62 to view an expanded version of the news story.

[0064] In some implementations, method 300 includes: detecting a first gaze input at a first time and detecting a second gaze input at a second time, the second time being within a threshold time period of the first time. In some implementations, in response to detecting the first gaze input, the device starts a timer. In such implementations, the device waits for the second gaze input until the timer expires. If the device detects the second gaze input before the timer expires, the device performs a UI operation associated with the UI element pointed to by the first gaze input. However, if the device does not detect the second gaze input before the timer expires, the device stops displaying the visual indicator and the device cancels the selection of the UI element selected by the first gaze input. In this way, after staring at the UI element, a limited amount of time is provided to the user to stare at the visual indicator in order to confirm the selection of the UI element stared at by the user. If the user does not stare at the visual indicator to confirm the selection of the UI element after staring at the UI element, the UI element is not selected and the visual indicator is no longer displayed. In some implementations, the duration of the timer is a function of the distance between the selected UI element and the visual indicator (e.g., proportional to the distance).

[0065] Figure 4 is a block diagram of a device 400 according to some implementations. In some implementations, the device 400 implements Figure 1A The electronic device 20 and / or Figure 1A and Figure 2 . Although some specific features are illustrated, those of ordinary skill in the art will recognize from this disclosure that various other features are not illustrated for the sake of brevity and so as not to obscure more relevant aspects of the specific implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the device 400 includes one or more processing units (CPUs) 401, a network interface 402, a programming interface 403, a memory 404, one or more input / output (I / O) devices 408, and one or more communication buses 405 for interconnecting these and various other components.

[0066] In some implementations, a network interface 402 is provided to, among other purposes, establish and maintain a metadata tunnel between a cloud-hosted network management system and at least one private network including one or more compatible devices. In some implementations, one or more communication buses 405 include circuits for interconnecting and controlling communications between system components. Memory 404 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 404 optionally includes one or more storage devices located remotely from one or more CPUs 401. Memory 404 includes non-transitory computer-readable storage media.

[0067] In some implementations, one or more I / O devices 408 include a display (e.g., Figure 1A In some implementations, the display comprises an XR display. In some implementations, the display comprises an opaque display. Alternatively, in some implementations, the display comprises an optically transparent display. In some implementations, one or more I / O devices 408 comprise an eye tracker (e.g., Figure 1A and Figure 2 In some implementations, one or more I / O devices 408 include an image sensor (e.g., Figure 1A In some implementations, one or more I / O devices 408 include a flash source (e.g., Figure 1A The flash source 26 shown in FIG.

[0068] In some implementations, the memory 404 or a non-transitory computer-readable storage medium of the memory 404 stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 406, a data acquirer 210, a selection determiner 220, and a UI presenter 230. In various implementations, the data acquirer 210 includes a device for acquiring image data indicating gaze input (e.g., Figure 1A The image data 50 shown in FIG. Figure 1B The gaze input 52 and Figure 1D In various implementations, the selection determiner 220 includes instructions 210a for identifying a particular UI element that the user selected via gaze (e.g., for identifying a Figure 1C In various implementations, the UI presenter 230 includes instructions 220a for presenting a visual indication (e.g., Figure 1C 54) as shown in the instructions 230a and heuristics and metadata 230b for the visual indicator 54) shown in FIG. 5, which serves as a common auxiliary gaze input for confirming the selection of the selected UI element.

[0069] It should be understood that Figure 4 Serves as a functional description of various features that may be present in a particular implementation, as opposed to a structural schematic diagram of the implementation described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 4 Some functional blocks shown separately in the figure may be implemented as a single block, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of blocks and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a specific specific implementation.

[0070] Although various aspects of specific implementations within the scope of the appended claims are described above, it should be apparent that the various features of the above-mentioned specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement a device and / or a method can be practiced. In addition, in addition to or different from one or more aspects set forth herein, other structures and / or functions can be used to implement such a device and / or such a method can be practiced.

Claims

1. A method, comprising: At an electronic device including a non-transitory memory, one or more processors, a display, and an eye tracker: displaying a user interface (UI) on the display, the user interface comprising a first UI element associated with a first UI operation and a second UI element associated with a second UI operation; In response to detecting a first gaze input directed to one of the first UI element and the second UI element via the eye tracker, displaying a visual indicator at a fixed location separate from the first UI element and the second UI element; as well as In response to detecting, via the eye tracker, a second gaze input directed toward the visual indicator: In response to the first gaze input pointing to the first UI element, performing the first UI operation; as well as In response to the first gaze input being directed to the second UI element, performing the second UI operation. 2 . The method of claim 1 , wherein the first UI element and the second UI element are part of a plurality of UI elements having a spacing less than a threshold. 3 . The method of claim 1 , wherein the fixed position is equidistant from the first UI element and the second UI element.

4. The method according to any one of claims 1 to 3, wherein displaying the visual indication comprises: In response to the first gaze input being directed to the first UI element, displaying the visual indicator with a first visual appearance, the first visual appearance matching a first visual characteristic of the first UI element; as well as In response to the first gaze input being directed to the second UI element, the visual indicator is displayed with a second visual appearance that matches a second visual characteristic of the second UI element.

5. The method of claim 4, wherein the first visual characteristic of the first UI element comprises a first color of the first UI element, and displaying the visual indicator in the first visual appearance comprises displaying the visual indicator in the first color; and The second visual characteristic of the second UI element includes a second color of the second UI element, and displaying the visual indicator in the second visual appearance includes displaying the visual indicator in the second color.

6. The method of any one of claims 4 and 5, wherein the first visual characteristic of the first UI element comprises a first shape of the first UI element, and displaying the visual indicator in the first visual appearance comprises displaying the visual indicator in the first shape; and The second visual characteristic of the second UI element includes a second shape of the second UI element, and displaying the visual indicator in the second visual appearance includes displaying the visual indicator in the second shape.

7. The method according to any one of claims 1 to 6, further comprising: In response to the first gaze input being directed to the first UI element, associating the visual indicator with the first UI operation such that staring at the visual indicator confirms user selection of the first UI element and triggers performance of the first UI operation associated with the first UI element; as well as In response to the first gaze input being directed to the second UI element, the visual indicator is associated with the second UI operation such that staring at the visual indicator confirms user selection of the second UI element and triggers performance of the second UI operation associated with the second UI element.

8. The method according to any one of claims 1 to 7, further comprising: When the visual indication is displayed at the fixed position, a second visual indication is displayed at a second fixed position separate from the first UI element and the second UI element.

9. The method of claim 8, wherein when the first gaze input is directed at the first UI element, staring at the second visual indicator triggers execution of a third UI operation associated with the first UI element; and When the first gaze input points to the second UI element, staring at the second visual indication mark triggers execution of a fourth UI operation associated with the second UI element.

10. The method according to any one of claims 1 to 9, further comprising: while gaze is traveling along a first path from the first UI element to the visual indicator, in response to the first gaze input being directed to the first UI element, maintaining selection of the first UI element; as well as While the gaze is traveling along a second path from the second UI element to the visual indicator, in response to the first gaze input being directed to the second UI element, maintaining selection of the second UI element.

11. The method according to claim 10, further comprising: When the gaze remains fixed on a third UI element on the first path from the first UI element to the visual indication, canceling the selection of the first UI element; as well as While the gaze remains fixed on a fourth UI element that is on the second path from the second UI element to the visual indication, the selection of the second UI element is cancelled.

12. The method according to any one of claims 10 and 11, further comprising: When the gaze does not move along the first path from the first UI element toward the visual indicator, canceling the selection of the first UI element; as well as When the gaze does not move along the second path from the second UI element toward the visual indication, the selection of the second UI element is cancelled.

13. The method according to any one of claims 1 to 12, wherein: The user interface includes a home screen; The first UI element includes a first icon for launching a first application; The second UI element includes a second icon for launching a second application; Executing the first UI operation includes launching the first application in response to the first gaze input being directed to the first icon and the second gaze input being directed to the visual indicator; as well as Executing the second UI operation includes launching the second application in response to the first gaze input being directed to the second icon and the second gaze input being directed to the visual indication.

14. The method according to any one of claims 1 to 12, wherein: The first UI element includes a first user-selectable link for opening a first page; the second UI element comprising a second user-selectable link for opening a second page; Executing the first UI operation includes opening the first page in response to the first gaze input being directed to the first user-selectable link and the second gaze input being directed to the visual indication; as well as Performing the second UI operation includes opening the second page in response to the first gaze input being directed to the second user-selectable link and the second gaze input being directed to the visual indication.

15. The method according to any one of claims 1 to 14, wherein: The first UI element includes a first icon, a first button, a first link or a first multimedia content item; and The second UI element includes a second icon, a second button, a second link or a second multimedia content item.

16. A device, comprising: monitor; Eye tracker; one or more processors; non-transitory memory; as well as one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to: displaying a user interface (UI) on the display, the user interface comprising a first UI element associated with a first UI operation and a second UI element associated with a second UI operation; In response to detecting a first gaze input directed to one of the first UI element and the second UI element via the eye tracker, displaying a visual indicator at a fixed location separate from the first UI element and the second UI element; as well as In response to detecting, via the eye tracker, a second gaze input directed toward the visual indicator: In response to the first gaze input pointing to the first UI element, performing the first UI operation; as well as In response to the first gaze input being directed to the second UI element, performing the second UI operation. 17 . The device of claim 16 , wherein the first UI element and the second UI element are part of a plurality of UI elements having a spacing less than a threshold.

18. The device of any one of claims 16 and 17, wherein the fixed position is equidistant from the first UI element and the second UI element.

19. The apparatus of any one of claims 16 to 18, wherein displaying the visual indicator comprises: In response to the first gaze input being directed to the first UI element, displaying the visual indicator with a first visual appearance, the first visual appearance matching a first visual characteristic of the first UI element; as well as In response to the first gaze input being directed to the second UI element, the visual indicator is displayed with a second visual appearance that matches a second visual characteristic of the second UI element.

20. The device of claim 19, wherein the first visual characteristic of the first UI element comprises a first color of the first UI element, and displaying the visual indicator in the first visual appearance comprises displaying the visual indicator in the first color; and The second visual characteristic of the second UI element includes a second color of the second UI element, and displaying the visual indicator in the second visual appearance includes displaying the visual indicator in the second color.

21. The device of any one of claims 19 and 20, wherein the first visual characteristic of the first UI element comprises a first shape of the first UI element, and displaying the visual indicator in the first visual appearance comprises displaying the visual indicator in the first shape; and The second visual characteristic of the second UI element includes a second shape of the second UI element, and displaying the visual indicator in the second visual appearance includes displaying the visual indicator in the second shape.

22. The apparatus of any one of claims 16 to 21, wherein the one or more programs further cause the apparatus to: In response to the first gaze input being directed to the first UI element, associating the visual indicator with the first UI operation such that staring at the visual indicator confirms user selection of the first UI element and triggers performance of the first UI operation associated with the first UI element; and In response to the first gaze input being directed to the second UI element, the visual indicator is associated with the second UI operation such that staring at the visual indicator confirms user selection of the second UI element and triggers performance of the second UI operation associated with the second UI element.

23. The apparatus of any one of claims 16 to 22, wherein the one or more programs further cause the apparatus to: When the visual indication is displayed at the fixed position, a second visual indication is displayed at a second fixed position separate from the first UI element and the second UI element.

24. The device of claim 23, wherein when the first gaze input is directed at the first UI element, staring at the second visual indicator triggers performance of a third UI operation associated with the first UI element; and When the first gaze input points to the second UI element, staring at the second visual indication mark triggers execution of a fourth UI operation associated with the second UI element.

25. The apparatus of any one of claims 16 to 24, wherein the one or more programs further cause the apparatus to: while gaze is traveling along a first path from the first UI element to the visual indicator, in response to the first gaze input being directed to the first UI element, maintaining selection of the first UI element; and While the gaze is traveling along a second path from the second UI element to the visual indicator, in response to the first gaze input being directed to the second UI element, maintaining selection of the second UI element.

26. The device of claim 25, wherein the one or more programs further cause the device to: When the gaze remains fixed on a third UI element on the first path from the first UI element to the visual indicator, canceling the selection of the first UI element; and When the gaze remains fixed on a fourth UI element on the second path from the second UI element to the visual indication, the selection of the second UI element is cancelled.

27. The apparatus according to any one of claims 25 and 26, wherein the one or more programs further cause the apparatus to: When the gaze does not move along the first path from the first UI element toward the visual indicator, canceling the selection of the first UI element; and When the gaze does not move along the second path from the second UI element toward the visual indication, the selection of the second UI element is cancelled.

28. Apparatus according to any one of claims 16 to 27, wherein: The user interface includes a home screen; The first UI element includes a first icon for launching a first application; The second UI element includes a second icon for launching a second application; Executing the first UI operation includes launching the first application in response to the first gaze input being directed to the first icon and the second gaze input being directed to the visual indicator; as well as Executing the second UI operation includes launching the second application in response to the first gaze input being directed to the second icon and the second gaze input being directed to the visual indication.

29. Apparatus according to any one of claims 16 to 27, wherein: The first UI element includes a first user-selectable link for opening a first page; The second UI element includes a second user-selectable link for opening a second page; Executing the first UI operation includes opening the first page in response to the first gaze input being directed to the first user-selectable link and the second gaze input being directed to the visual indication; as well as Performing the second UI operation includes opening the second page in response to the first gaze input being directed to the second user-selectable link and the second gaze input being directed to the visual indication.

30. Apparatus according to any one of claims 16 to 29, wherein: The first UI element includes a first icon, a first button, a first link or a first multimedia content item; and The second UI element includes a second icon, a second button, a second link or a second multimedia content item.

31. A non-transitory memory storing one or more programs which, when executed by one or more processors of a device having an eye tracker and a display, cause the device to perform any of the methods of claims 1 to 15.

32. A device, comprising: monitor; Eye tracker; one or more processors; non-transitory memory; as well as A component, the component being configured to cause the device to perform any one of the methods according to claims 1 to 15.

Images