Gaze-based control

Abstract

Aspects of the subject technology relate to gaze-based control of electronic devices. The gaze-based control can include implementing an option for providing a user authorization upon determining that the user has viewed and / or read text associated with a request for authorization of the user. The gaze-based control can also include modifying a user interface or user interface element based on the user viewing and / or reading. The gaze-based control can be based on determining whether a user has viewed and / or read an electronic document and / or a physical document.

Description

[0001] This application is a divisional application of patent application No. 202180028325.9, filed on March 22, 2021, entitled "Gaze-Based Control". Cross-references to related applications

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 011,975, filed April 17, 2020, entitled “Gaze-Based Control,” the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This specification relates in its entirety to electronic devices, and more specifically, but not exclusively, to gaze-based control for electronic devices. Background Technology

[0004] Portable electronic devices, such as laptops, cell phones, and tablets, are typically equipped with displays for showing visual information. These portable devices usually require the user to authorize actions before the device can perform any action. Attached Figure Description

[0005] Some features of this subject matter are shown in the appended claims. However, for illustrative purposes, several embodiments of this subject matter are illustrated in the following figures.

[0006] Figure 1 A perspective view of an exemplary electronic device according to various aspects of the subject matter is shown.

[0007] Figure 2 This illustrates the various aspects of the technology described in this subject, as viewed by the user at the gaze position. Figure 1 A perspective view of an electronic device.

[0008] Figure 3 Another specific implementation example of an electronic device according to various aspects of the subject matter is shown.

[0009] Figure 4 Examples of user authorization requests based on various aspects of the technology in this subject are shown.

[0010] Figure 5 This illustrates various aspects of the technology in this subject matter that can be viewed and / or read by the user. Figure 4 User authorization request.

[0011] Figure 6 A flowchart illustrating an exemplary process for gaze-based control for user authorization, according to various aspects of the subject matter, is shown.

[0012] Figure 7 Examples of gaze positions that can be obtained over time according to various aspects of the techniques in this subject matter are shown.

[0013] Figure 8 The gaze positions corresponding to eye movements during reading are shown according to various aspects of the technology in this subject matter.

[0014] Figure 9 Examples of highlighting reading text based on gaze position, according to various aspects of the techniques in this subject, are shown.

[0015] Figure 10 An exemplary user interface for viewing by a user is shown, according to various aspects of the technology of this subject.

[0016] Figure 11 This illustrates various aspects of the technology according to this subject matter. Figure 10 An example of a user interface where the interface has been modified based on the user's gaze position.

[0017] Figure 12 This illustrates various aspects of the technology according to this subject matter. Figure 10 Another example of a user interface where the interface has been modified based on the user's gaze position.

[0018] Figure 13 A flowchart illustrating an exemplary process for gaze-based control of a user interface, according to various aspects of the subject matter, is shown.

[0019] Figure 14 A flowchart illustrating an exemplary process for gaze-based control of a user interface element that displays time-dependent information, according to various aspects of the subject matter, is shown.

[0020] Figure 15 An exemplary electronic device configured to determine whether a physical document has been viewed and / or read, according to various aspects of the subject matter, is shown.

[0021] Figure 16 Exemplary electronic devices with transparent or semi-transparent displays and configured to determine whether a physical document has been viewed and / or read, according to various aspects of the subject matter, are shown.

[0022] Figure 17 An exemplary electronic device with an opaque display and configured to determine whether a physical document has been viewed and / or read, according to various aspects of the subject matter, is shown.

[0023] Figure 18A flowchart illustrating an exemplary process for gaze-based control of electronic devices based on physical documents, according to various aspects of the art in this subject matter, is shown.

[0024] Figure 19 An electronic system that can implement one or more specific embodiments of the technology of this subject is shown. Detailed Implementation

[0025] The specific embodiments shown below are intended to describe various configurations of the subject matter and are not intended to represent the only configuration in which the subject matter can be practiced. The accompanying drawings are incorporated herein and form part of the detailed description. The detailed description includes specific details intended to provide a thorough understanding of the subject matter. However, it will be clear and apparent to those skilled in the art that the subject matter is not limited to the specific details shown herein and can be practiced without such specific details. In some cases, well-known structures and components are shown in block diagram form in order to avoid obscuring the concepts of the subject matter.

[0026] This disclosure provides systems and methods for controlling electronic devices in part based on whether a user of the device has viewed and / or read text information (such as text information displayed by the device (or another device) or included on a physical document viewed by the user of the device). The device may use eye-tracking data from one or more eye trackers on the device and / or another device to determine whether a user has viewed and / or read text information. The eye-tracking data may include gaze information of one or both of the user's eyes, such as gaze location and / or gaze direction. The gaze information may indicate where and when the user is viewing various locations (such as locations on the device's display), and / or how the user's eyes move over time.

[0027] This gaze information can be used to control various aspects of electronic devices. In one example, gaze information can be used to confirm that the user has viewed and / or read text presented to them before the user can authorize the device to perform an action. For example, to prevent the user from accidentally deleting data from the device's memory, the device can present a dialog box including text such as "Are you sure you want to delete this file?".

[0028] In conventional devices, this type of text is typically accompanied by a "Yes" button for authorizing deletion and a "No" button for opting out of deletion. However, users who might inadvertently instruct the device to delete files and are busy working on the device, or users who are otherwise distracted, might select the "Yes" button without reading the dialog box text. In these cases, the deleted data may be lost, which can lead to user frustration, delays and / or inefficiency, as well as wasted device processing and / or memory resources, such as resources used for deleting data and / or for recreating unintentionally deleted data.

[0029] The systems and methods disclosed herein can be used to confirm that the user has viewed and / or read the text in the dialog box before allowing the user to select an option for authorized deletion.

[0030] The systems and methods disclosed herein can also be applied to determine whether a user has viewed and / or read other information before allowing a user to authorize various device actions. As an example, an electronic device may determine, before allowing a user to authorize associated actions, whether the user of the device has viewed and / or read text associated with permission to access personal data, text associated with permission to access one or more components of the device, text associated with permission to install software on the device, and / or text corresponding to one or more terms and / or conditions.

[0031] The systems and methods disclosed herein can also be applied to modify device operation based on determining whether and / or when a user has viewed and / or read text information. For example, after a user has viewed and / or read text information a predetermined number of times, text forming part of the user interface or other device display can be removed from the user interface or display screen. As another example, user interface elements (such as widgets) displaying time-related information can be updated at a frequency that depends on how frequently the user views the user interface elements.

[0032] exist Figure 1 The illustration shows an exemplary electronic device that can implement the techniques described herein. Figure 1 In the example, device 100 has been implemented in the form of a portable computer. For example... Figure 1 As shown, the electronic device 100 may include a keyboard 102 with keys 110, a display 104, a housing 106, and a touchpad such as a touchpad 112. Figure 1 As shown, the display 104 can be mounted on the front of the housing 106.

[0033] The outer casing 106, sometimes referred to as a housing, may be formed of plastic, glass, ceramic, fiber composite material, metal (e.g., stainless steel, aluminum, etc.), other suitable materials, or any combination of two or more of these materials. The outer casing 106 may be formed in an integral configuration in which some or all of the outer casing 106 are machined or molded into a single structure, or it may be formed using multiple structures (e.g., an internal frame structure, one or more structures forming the surface of the outer casing, etc.). Figure 1 As shown, housing 106 may have multiple portions. For example, housing 106 may have an upper portion 106A and a lower portion 106B. The upper portion 106A may be coupled to the lower portion 106B using a hinge that allows the upper portion 106A to rotate relative to the lower portion 106B about a rotation axis 108. In some embodiments, a keyboard 102 and a touchpad 112 may be mounted in the lower portion 106B.

[0034] Display 104 may be a touchscreen incorporating capacitive touch electrodes or other touch sensor components, or it may be a non-touch-sensitive display. Display 104 may include display pixels formed from: light-emitting diodes (LEDs), organic light-emitting diodes (OLEDs), plasma units, electrophoretic display elements, electrowetting display elements, liquid crystal display (LCD) components, or other suitable display pixel structures. Display 104 may have openings (e.g., openings in the passive or active portions of display 104), such as openings for accommodating buttons or one or more cameras (such as camera 105). Camera 105 and / or other cameras disposed around or behind the display may be operated by electronic device 100 to capture images of one or more features of a user's face, one or both of the user's eyes, and / or one or both of the user's eyes.

[0035] Images captured by camera 105 and / or other cameras (e.g., cameras located at or near a corner of display 104) may be processed by processing circuitry within electronic device 100 to detect the user's eye and determine the position and / or orientation of the user's gaze relative to the camera, relative to display 104 and / or away from display 104, and / or relative to electronic device 100. For example, the processing circuitry may determine the position of the user's gaze relative to text in an electronic document displayed on the display, relative to one or more portions of a user interface or user interface element displayed on the display, and / or relative to text in a physical document detached from electronic device 100.

[0036] Camera 105 and associated processing circuitry can form an eye tracker for electronic device 100, which can be configured to track the gaze position and / or movement of a user's eyes at various times and / or over time. The eye tracker may include a single camera (e.g., an infrared camera or a visible light camera) whose field of view is configured to include the user's eye position, multiple such cameras, and / or one or more additional sensors such as head-mounted sensors, near-infrared projectors, etc.

[0037] Figure 1 The configuration of the electronic device 100 in the example is merely illustrative. In other specific embodiments, the electronic device 100 may be implemented in the form of a computer, such as a computer integrated into a display (such as a computer monitor), a tablet computer, a smartphone, a smaller portable device (such as a wristwatch, a wristband device, or other wearable or micro-device), a media player, a gaming device, a navigation device, a computer monitor, a television, or other electronic device. In another example discussed in further detail below (see example...), Figure 3 The electronic device 100 can be implemented as a head-mountable device. In various configurations, the electronic device 100 can be communicatively coupled to one or more cameras and / or other eye-tracking sensors that are separate from the electronic device.

[0038] like Figure 1 As shown, the display content 120 can be displayed by the display 104 (e.g., by manipulating the display pixel array according to the corresponding digital pixel values ​​in the display frame). Figure 1 In the example, the displayed content 120 includes the application display window 114, the dialog box 116, and the user interface element 118.

[0039] For example, application display window 114 can be a word processor, compiler, rendering editor, spreadsheet, image viewer, file manager, web browser, video player, audio player, browser, or generally any user interface window. Application display window 114 can display electronic documents for the user of electronic device 100 to view and / or edit. In one example, application display window 114 displays an electronic document edited by the user. In another example, application display window 114 displays a terms and conditions document that will be read by the user.

[0040] Dialog box 116 may be a dialog box for requesting user authorization for an action to be performed by electronic device 100 (e.g., by the processing circuitry of the electronic device). User authorization may be for actions related to the application display window 114 (e.g., actions such as closing a document or other editable data structure without saving changes, accepting the terms and conditions described in an electronic document, deleting data from the memory of electronic device 100, granting access to data stored by electronic device 100, granting access to one or more components of electronic device 100, granting access to the user's personal information), authorizing e-commerce and / or NFC transactions, or generally any action. Dialog box 116 may be a pop-up window that appears in response to a user request for an action on electronic device 100 and / or in response to an automated action (e.g., a software update, an attempt by the application to access information, etc.). Although dialog box 116 is in Figure 1 In this example, the dialog box 116 is displayed separately from the application display window 114, but it should be understood that in other examples, the dialog box 116 may be generated on, within, and / or as part of the application display window 114.

[0041] User interface element 118 can be a widget that displays time-related information, such as a clock widget, calendar widget, news feed widget, weather widget, stock quote widget, etc.

[0042] According to aspects of this disclosure, electronic device 100 controls one or more actions associated with an application based on one or more gaze positions of one or both of the user's eyes, such as... Figure 2 As shown.

[0043] like Figure 2 As shown, the gaze direction 203 of a user (such as user 200) can be determined by electronic device 100 to intersect with a physical or virtual object (such as display 104) at gaze position 202. Images of user 200 captured by camera 105 and / or other sensors (e.g., depth sensors that determine the distance between the user's eyes and camera 105) (e.g., infrared and / or visible light images of one or more of the user's eyes) can be used to determine gaze position 202. If gaze position 202 is a position on display 104, gaze position 202 can be determined based on the coordinates of pixels or pixel groups 204 of the display. Based on the determined gaze position 202, electronic device 100 can control one or more aspects of application display windows 114, dialog boxes 116, and / or user interface elements 118. Because electronic device 100 controls the position where display content 120 is displayed, gaze position 202 in pixel coordinates can be determined relative to the displayed content.

[0044] exist Figure 2In the example, the gaze position 202 is within the boundary of the application display window 114. In some cases, the operation of the application associated with the application display window 114 and / or the appearance of the application display window 114 can be controlled based on the gaze position 202. For example, the application display window 114 can be visually highlighted or activated from an inactive state when the gaze position 202 is within and / or remains within the boundary of the application display window 114 for a predetermined period of time (e.g., a few milliseconds, tens of milliseconds, or hundreds of milliseconds). In some cases, one or more sub-sections of the application display window 114 can be modified and / or otherwise controlled based on the gaze position 202.

[0045] A dialog box 116 may be displayed to request user authorization for one or more actions on the electronic device 100. For example, a user may not have saved edits to an electronic document in the application display window 114 and may enter a request to close the document. To prevent a user from losing desired edits to a document by closing it without saving, the electronic device 100 may provide a dialog box 116, such as “Are you sure you want to close without saving?”. To help reduce the likelihood of a user unintentionally agreeing to close a document without saving (e.g., by clicking the optional option to close without saving without viewing or reading the text in dialog box 116), eye-tracking data (such as one or more gaze positions 202) may be used to determine whether the user has viewed and / or read the text in dialog box 116.

[0046] As mentioned above Figure 1 The discussed electronic device 100 is implemented as a laptop computer. Figure 1 and Figure 2 The examples are merely illustrative. For example, Figure 3 Another exemplary embodiment of the electronic device 100 is shown. For example... Figure 3 As shown in the example, one or more sensors (such as sensor 152) may be positioned at various locations on or within the body 300 of the electronic device 100. The body 300 may be arranged to anchor the electronic device 100 to a user (e.g., to the user's head), such that the display 104 is positioned between the user's eyes and a view of the environment of the electronic device 100.

[0047] In this example, display 104 may be an opaque display that prevents a user from directly viewing the environment surrounding electronic device 100, or it may be a transparent or semi-transparent display that allows a user to directly view the environment surrounding electronic device 100 through the display. In a specific implementation where display 104 is an opaque display that blocks direct viewing of the environment, one or more cameras (such as camera 150) may provide a through-view video view to display 104 in some operational scenarios. In a specific implementation where display 104 is a transparent or semi-transparent display, display 104 may overlay displayed content on a direct view of a portion of the physical environment.

[0048] Transparent or semi-transparent displays may have a medium through which light representing an image is directed to the human eye. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light sources, or any combination of these technologies. The medium may be an optical waveguide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, a transparent or semi-transparent display may be configured to selectively become opaque. Projection-based systems may employ retinal projection techniques that project graphic images onto the human retina. Projection systems may also be configured to project virtual objects onto a physical environment, such as as holograms or on a physical surface.

[0049] As shown in the figure, electronic device 100 may include one or more sensors (such as sensor 152). Sensor 152 may include, but is not limited to, cameras, image sensors, touch sensors, microphones, inertial measurement units (IMUs), heart rate sensors, temperature sensors, LiDAR sensors, radar sensors, sonar sensors, GPS sensors, Wi-Fi sensors, near-field communication sensors, etc. Sensors 152 and / or one or more cameras (such as camera 150), each having a field of view, may be positioned to capture images and / or three-dimensional information (e.g., depth information) of the environment surrounding electronic device 100.

[0050] Processing circuitry 306 (e.g., one or more processors and memories) and / or local processing circuitry including eye tracker 310 with camera 105 can acquire eye-tracking information and determine the gaze position, orientation, and / or movement of one or more of the user's eyes. One or more internal sensors 304 (such as IMUs) and / or one or more power sources (such as batteries 308) may also be located in electronics 100 and electrically and / or communicatively coupled to processing circuitry 306.

[0051] The processing circuit 306 can control various aspects of the operation of the electronic device 100 based on eye-tracking data from the eye tracker 310. For example, Figure 4A dialog box 116 is shown, displayed on display 104 and including text 400 of a user authorization request. As an example, the user authorization request in text 400 could be a request for user authorization to delete data from electronic device 100, a request for user authorization to access data stored by electronic device 100 (e.g., contact data, calendar data, passwords, photos, videos, etc.), a request for user authorization to access components (e.g., camera, sensors, etc.) and / or otherwise change the security settings of electronic device 100, a request for user authorization to accept the terms and conditions in a document displayed by electronic device, and / or a request for user authorization to perform e-commerce and / or NFC transactions. Figure 4 As shown in the example, dialog box 116 may include a scroll bar (e.g., for scrolling text 400 if it is too long to be displayed on a single screen). In some implementations, electronic device 100 may prevent the user from scrolling the document if it is determined that the current gaze position 202 is not within a text portion of the document (e.g., the use of scroll bar 410 may be disabled if the gaze position 202 is not within the boundaries of dialog box 116, on text 400, and / or indicates reading movement of the user's eyes). As another example, the user authorization request in text 400 may be a request for user authorization to install software on electronic device 100, restore factory settings of electronic device 100, couple a remote device to electronic device 100, etc.

[0052] like Figure 4 As shown in the example, a request for user authorization (e.g., as provided in dialog box 116) may include an optional option 404 for denying authorization. The optional option 404 for denying authorization may be provided independently of the user's gaze position 202. Figure 4 As shown in the example, a request for user authorization can be set with a deactivation option 406 for providing user authorization.

[0053] When the gaze position 202 is outside the boundary of the dialog box 116 rather than above the text 400 and / or (e.g., along with the attached previous gaze position) does not indicate that the text 400 has been read, a deactivation option 406 may be provided to grant user authorization. The deactivation option 406 may be set with an initial request for user authorization before determining whether the text 400 has been viewed and / or read.

[0054] like Figure 4As shown, after determining that gaze position 202 indicates that the user has viewed and / or read text 400, electronic device 100 can provide selectable options 500 for granting user authorization. If gaze position 202 is within the boundaries of dialog box 116 (e.g., momentarily or for a predetermined time period) and / or on text 400 (e.g., momentarily or for a predetermined time period), gaze position 202 can indicate that text 400 has been viewed. If multiple gaze positions 202 over time indicate movement of the gaze position along a line of text 400 and / or reading movement of the user's eyes, gaze position 202 can indicate that text 400 has been read. Reading movements that can be indicated by gaze position 202 include saccades and fixation.

[0055] exist Figure 4 and Figure 5 In this example, deactivation option 406 is enabled or activated to provide optional option 500 for granting user authorization. However, this is merely illustrative. In other examples, dialog box 116 may provide text 400 requesting user authorization and optional option 404 for denying user authorization, without providing a deactivation option for granting user authorization. In this example, optional option 500 may be added to dialog box 116 when it is determined that the user has viewed and / or read text 400.

[0056] Figure 6 A flowchart illustrating an exemplary process 600 for gaze-based control for user authorization, according to various aspects of the subject matter, is provided. For purposes of explanation, reference is made to... Figure 1 and / or Figure 3 The components are described Figure 6 An exemplary process. Further for illustrative purposes, Figure 6 The exemplary process outlined herein is described as occurring sequentially or linearly. However, Figure 6 Multiple frames of the exemplary process can occur in parallel. Furthermore, Figure 6 The boxes of the exemplary process do not need to be executed in the order shown and / or Figure 6 One or more of the exemplary processes in the box need not be executed.

[0057] In the illustrated example flowchart, at box 602, an electronic device (such as electronic device 100) displays a request for user authorization from the user. Displaying the request for user authorization may include displaying a dialog box 116 with text 400, as in conjunction with... Figure 4 As described.

[0058] At box 604, when a display request is made, the electronic device can identify at least one gaze position (such as gaze position 202) of at least one of the user's eyes.

[0059] At box 606, the electronic device may determine whether a user has viewed the displayed authorization request based on at least one gaze position. Determining whether the displayed authorization request has been viewed by the user based on at least one gaze position may include determining whether at least one gaze position 202 has been within the boundaries of the displayed request (e.g., within the boundaries of dialog box 116 or within the boundaries of the text portion of dialog box 116) for at least a predetermined period of time.

[0060] At box 608, if it is determined that the user has viewed the displayed request for authorization, the electronic device can provide selectable options for granting the user authorization, such as... Figure 5 The optional option 500. Providing optional options for granting user authorization may include activating / deactivating option 406, or adding optional options to the displayed request after confirmation. The process 600 may also include identifying the dwelling of at least one of the user's eyes on the user interface of the inactive application based on multiple gaze positions 202; and activating the inactive application based on the identified dwelling.

[0061] As mentioned above Figure 3 and Figure 4 The requested authorization to the user discussed may include text 400 associated with the authorization. Determining whether the user has viewed the displayed request for authorization based on at least one gaze position may include determining whether at least one gaze position indicates that the user has read the text (e.g., based on multiple gaze positions that indicate the movement of the user's eyes in conjunction with reading the text over time).

[0062] Figure 7 and Figure 8 This illustrates how multiple gaze positions 202 obtained over time can indicate whether a user has read text 400. (Example) Figure 7 As shown, when user 200's eyes rotate from a previous gaze direction 203P to the current gaze direction 203 during reading text (such as text 400), as indicated by arrow 702, multiple gaze positions 700 acquired over a certain period of time prior to the current gaze position 202 can fall along a substantially straight line. If the substantially straight line of movement of the gaze position 700 over time corresponds to a line of text 400 (e.g., falling along the text line, the position of which is known because the device itself displays the text, or determined by the device using an image of the text in a physical document, or displayed by another device), then electronic device 100 can determine that the text line has been read. However, Figure 7 It is also indicated that the movement of the gaze position 700 over time, as indicated by arrow 702, is not constant. This may be because during reading, the human eye tends to move in saccadic and fixed characteristic patterns.

[0063] Figure 8 It is shown how group 802 of gaze position 700 can indicate relatively slow movements of the user's eyes (such as during fixation), and group 804 of gaze position 700 can indicate rapid movements of the user's eyes (such as during saccades). Electronic device 100 can correspond to text 800 to be read for authorization (e.g., Figure 4 and Figure 5 The electronic device 100 detects saccade and fixation feature patterns at the location of the text (400) and determines that the text at the corresponding location has been read. In this example, the electronic device 100 determines at least one reading location that indicates that the user has read the text by determining at least one gaze position, which includes multiple temporally separated gaze positions corresponding to reading eye movements (e.g., including at least one saccade and at least one fixation).

[0064] like Figure 9 As shown, in operational scenarios where the electronic device 100 determines that text has been read (e.g., when multiple temporally separated gaze positions move across a portion of the text and / or based on detected reading eye movement at a location corresponding to the displayed text), the electronic device 100 may also display a highlight 900 on the portion of the text 800 that has been read. As indicated by arrow 902, the highlight 900 may extend over the text 800 along with the reading movement of the current gaze position 202.

[0065] In some operational scenarios, electronic device 100 can recognize sounds associated with words (e.g., animal sounds associated with animal names or descriptions of animal sounds, or the sound of words spoken in a specific language). This is also relevant in scenarios where the user's gaze position indicates which word they are reading (e.g., as...). Figure 8 and Figure 9 In the example, electronic device 100 can determine that a user is reading a word associated with a recognized sound, and can output the recognized sound while the user is reading the word (e.g., using one or more speakers or other audio output components). This can be helpful when children are learning to read, when users have difficulty reading words, and / or when reading stories to children.

[0066] and combination Figures 4 to 9 Compared to the aspects described, the processing circuit 306 can control other aspects of the operation of the electronic device 100, including the display 104. For example, Figure 10 An example is shown where the application display window 114 displays the application's user interface 1000, and the processing circuitry controls the user interface based on the user's eye gaze position. Figure 10In this example, user interface 1000 is the user interface of a media player application and includes selectable control options, including a play button 1002 and a stop button 1003. In this example, play button 1002 includes a play icon 1004 and text 1006 indicating to the user that button 1002 and play icon 1004 are used to initiate “playback” of media (e.g., audio, video, or other media) accessible by the application corresponding to the displayed user interface. In this example, stop button 1003 includes a stop icon 1005 and text 1007 indicating to the user that stop button 1003 and stop icon 1005 are operable to “stop” media playback.

[0067] However, over time, the user will know that the play button 1002 and play icon 1004 are used to play media, which can make the text 1006 redundant and potentially inefficient in using system resources for display. In some implementations, the electronic device 100 determines that the user's gaze position 202 has fallen within the boundary of the play button 1002 and / or the user's gaze position 202 indicates that the user has read the text 1006 a predetermined number of times.

[0068] Figure 11 This illustration shows how the user interface 1000 in the application display window 114 can be modified to remove text 1006 in response to determining that the user's gaze position 202 has fallen within the boundary of the play button 1002 and / or that the user's gaze position 202 indicates that the user has read text 1006 a predetermined number of times. In this example, text 1007 remains part of the user interface 1000, and the size of the play icon 1004 is increased to utilize the display area available due to the removal of text 1006. However, in other specific implementations, instead of increasing the size of the icon associated with the removed text, other parts of the user interface can be expanded or modified to utilize the newly available space in the user interface.

[0069] For example, Figure 12 This illustrates that texts 1006 and 1007 have been viewed and / or read a sufficient number of times to be removed, thereby allowing other parts of the user interface 1000 to extend into the additional space 1200 available by removing texts 1006 and 1007.

[0070] It should be understood that in some scenarios, text 1006 and / or 1007 can be removed based on the gaze position 202 indicating that the user has stopped looking at the text when operating buttons 1002 and 1003 (e.g., and only looks at icons 1004 and 1005), rather than based on the number of times the text is viewed and / or read. It should also be understood that, although... Figures 10 to 12Examples of user interface 1000 including buttons 1002 and 1003 and icons 1004 and 1005 are depicted, but the systems and methods described herein may be applied to remove substantially any text from the user interface after determining that the text has been viewed and / or read a sufficient number (e.g., a predetermined number) of times.

[0071] Figure 13 A flowchart illustrating an exemplary process 1300 for gaze-based control of a user interface according to various aspects of the subject matter is provided. For illustrative purposes, reference is made to... Figure 1 and / or Figure 3 The components are described Figure 13 An exemplary process. Further for illustrative purposes, Figure 13 The exemplary process outlined herein is described as occurring sequentially or linearly. However, Figure 13 Multiple frames of the exemplary process can occur in parallel. Furthermore, Figure 13 The boxes of the exemplary process do not need to be executed in the order shown and / or Figure 13 One or more of the exemplary processes in the box need not be executed.

[0072] In the illustrated exemplary flowchart, at box 1302, an electronic device (such as electronic device 100) displays a user interface such as the user interface 1000 of an application, which includes text such as text 1006 or 1007.

[0073] At frame 1304, the device obtains multiple gaze positions of at least one of the user's eyes (e.g., using eye tracking, such as an eye tracker 310 formed by one or more cameras 105 and associated processing circuitry).

[0074] At box 1306, the electronic device may determine that a user has viewed text based on multiple gaze positions, such as the user's eyes remaining on the text for at least a predetermined time period, and / or each of the words and / or characters in the text that the user has read. Determining that a user has viewed text may include determining that the user has viewed the text at least a predetermined number of times.

[0075] At box 1308, the electronic device modifies the user interface to remove text based on a confirmation. (See also...) Figures 10 to 12 The example description suggests that the user interface may include icons associated with text (e.g., one or more icons such as icons 1004 and 1005), and modifying the user interface to remove text may include providing icons without providing text, such as... Figure 11 and Figure 12 As shown in the example.

[0076] In some operational scenarios, a user interface or user interface element (e.g., a widget) (e.g., via the display 104 of electronic device 100) may be displayed to provide the user with time-related information. Processing circuitry 306 can control the user interface or user interface element by adjusting the frequency of updating time-based information based on how frequently the user views the user interface or user interface element.

[0077] Figure 14 A flowchart illustrating an exemplary process 1400 for gaze-based control of a user interface or user interface element, displaying time-related information, according to various aspects of the subject matter, is provided. For illustrative purposes, reference is made to... Figure 1 and / or Figure 3 The components are described Figure 14 An exemplary process. Further for illustrative purposes, Figure 14 The exemplary process outlined herein is described as occurring sequentially or linearly. However, Figure 14 Multiple frames of the exemplary process can occur in parallel. Furthermore, Figure 14 The boxes of the exemplary process do not need to be executed in the order shown and / or Figure 14 One or more of the exemplary processes in the box need not be executed.

[0078] In the illustrated exemplary flowchart, at box 1402, an electronic device (such as electronic device 100) displays a widget (e.g., Figure 1 and Figure 2 The widget (user interface element 118) displays time-related information updated at a specified frequency. Time-related information may include one or more of the following: weather information, financial information, news supply information, calendar information, time information, or any other information that changes over time. Updates may initially be performed at the default update frequency and may include new information obtained from data locally stored on the device and / or from one or more remote devices or servers via a network such as the Internet. Obtaining updated information can consume power, processing time, and network bandwidth. The goal is to provide updates before the user views the widget, without obtaining more updates than the user would normally see.

[0079] At box 1404, the electronic device can acquire multiple gaze positions 202 of at least one of the user's eyes. These multiple gaze positions can be acquired over a period of time and can include gaze positions within the boundaries of a widget that appears when the user looks at the widget (e.g., within the boundaries of user interface element 118).

[0080] At box 1406, the electronic device can determine the frequency with which a user views a displayed widget based on multiple gaze positions. This viewing frequency can be determined, for example, by dividing the number of times the widget is viewed by the amount of time the view is recognized.

[0081] At box 1408, the electronic device can modify the update frequency of the widget based on the viewing frequency. For example, the update frequency can be set to a frequency that is a predetermined amount greater than the viewing frequency. In some operational scenarios, the user interface or user interface element can be activated from an inactive state based on the gaze position within the boundary of the user interface or user interface element, which indicates the dwell time of at least one of the user's eyes (e.g., a static gaze position over a predetermined amount of time).

[0082] In the examples described herein, the gaze position is primarily described as the intersection of the user's gaze direction with a portion of the display 104 of the electronic device 100. However, it should also be understood that the systems and methods described herein can be applied to scenarios where a user is viewing a physical document.

[0083] For example, Figure 15 An example is shown where the user's gaze position 202 intersects with text 1502 on the physical document 1500. In this example, the camera 150 of the electronic device 100 captures an image of the physical document 1500. The processing circuitry 306 can then determine the position of the physical document 1500 relative to the electronic device 100 based on the image and / or depth information captured by one or more sensors of the device. In various examples, the processing circuitry can also determine that text 1502 is present on the physical document 1500, determine the position of the text 1502, and / or recognize the text 1502 based on the image.

[0084] An eye tracker (e.g., including a camera 105) of an electronic device 100 can determine the gaze direction 203 of a user 200's eyes, and determine the gaze position 202 as a position on the physical document 1500 by using the determined position and / or distance of the physical document to determine the intersection of the gaze direction 203 and the physical document 1500.

[0085] Over time, the electronic device 100 can determine multiple gaze positions 202, and the multiple gaze positions 202 can indicate that the user is reading or has already read text 1502 (e.g., using...). Figures 7 to 9 (Some or all of the same techniques described in the examples combined with electronic text). Figure 15 In the example, user 200 views the physical document 1500 by looking over monitor 104. However, as shown below... Figure 16 and Figure 17 As indicated, user 200 can view through a transparent or semi-transparent display. Figure 16 () or by viewing the electronic version 1700 of the physical document 1500 displayed on an opaque monitor (e.g., using one or more images captured by camera 150, Figure 17To view the physical document 1500. The electronic version 1700 displayed by the display 104 may include electronic text 1701 corresponding to text 1502 of the physical document 1500.

[0086] like Figure 17 As shown, in some scenarios, electronic device 100 can detect (e.g., based on one or more gaze positions and / or other information) text in electronic or physical documents that the user has difficulty reading. Based on this determination, electronic device 100 can provide visual or audio enhancements to assist the user in reading the text. Figure 16 In the example, the enhancement is a visual enhancement 1702 representing a magnified view of text 1701. Determining that a user has difficulty reading text may include determining that the gaze position has stopped across a period of time that spans longer than a fixed time, and / or determining that the user is squinting (e.g., based on one or more images and / or depth information from one or more other sensors from camera 105 and / or electronic device 100).

[0087] Figure 18 A flowchart illustrating an exemplary process 1800 for gaze-based control of an electronic device in conjunction with a physical document, according to various aspects of the art of this subject, is provided. For illustrative purposes, reference is made to... Figure 1 and / or Figure 3 The components are described Figure 18 An exemplary process. Further for illustrative purposes, Figure 18 The exemplary process outlined herein is described as occurring sequentially or linearly. However, Figure 18 Multiple frames of the exemplary process can occur in parallel. Furthermore, Figure 18 The boxes of the exemplary process do not need to be executed in the order shown and / or Figure 18 One or more of the exemplary processes in the box need not be executed.

[0088] In the exemplary flowchart depicted, at block 1802, at least one image of a physical document (such as physical document 1500) can be obtained by the camera of an electronic device (such as electronic device 100) (such as camera 150).

[0089] At box 1804, the electronic device may determine the position of the physical document (e.g., relative to the electronic device) based on at least one image (e.g., and / or based on additional information such as depth information from a sensor (such as one of sensor 152). The electronic device may also identify text in the physical document based on at least one image.

[0090] At box 1806, the electronic device can obtain multiple gaze positions 202 of at least one eye of the user 200 of the electronic device via the electronic device's eye tracker (e.g., including camera 105 and associated circuitry).

[0091] At box 1808, the electronic device can determine that the user is reading a physical document based on multiple gaze positions relative to the document's location. The electronic device can also display a highlight (such as highlight 900) over a view of the identified text portion already read by the user, via the electronic device's display (such as display 104) and based on the multiple gaze positions. In some examples, the view of the identified text portion can be as follows: Figure 16 The image depicts a direct view through a semi-transparent or transparent display, and displaying a highlight may include displaying the highlight through a transparent or semi-transparent display. In other examples, the electronic device may, as... Figure 17 The depiction includes displaying at least one image via a display of an electronic device, and the display highlight may include displaying a highlight overlaid on the displayed at least one image.

[0092] As described above, one aspect of this technology is the collection and use of data obtained from specific and legitimate sources to determine whether a user has viewed and / or read text information. This disclosure envisions that, in some instances, the collected data may include personal information data that uniquely identifies or can be used to identify a specific person. Such personal information data may include video data, three-dimensional geometric data, demographic data, location-based data, online identifiers, telephone numbers, email addresses, home addresses, data or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, eye position or movement data, gaze tracking data, or any other personal information.

[0093] This disclosure recognizes that the use of such personal information data in the techniques of this invention can be used to benefit users. For example, personal information data can be used to determine whether a user has viewed and / or read text information.

[0094] This disclosure assumes that entities responsible for collecting, analyzing, disclosing, transmitting, storing, or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, it is expected that such entities will implement and consistently apply privacy practices generally recognized as meeting or exceeding industry or governmental requirements for protecting user privacy. Such information regarding the use of personal data should be prominently displayed and easily accessible to the user, and should be updated as the collection and / or use of data changes. Users' personal information should be collected only for lawful use. Furthermore, such collection / sharing should only occur after receiving user consent or other lawful grounds provided for in applicable law. In addition, such entities should consider taking any necessary steps to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and processes. Additionally, such entities may be subject to third-party evaluations to demonstrate their compliance with widely accepted privacy policies and practices. Furthermore, policies and practices should be tailored to the specific types of personal information data collected and / or accessed, and made applicable to applicable laws and standards, including specific jurisdictional considerations that may be used to impose higher standards. For example, in the United States, the collection or access to certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA); while health data in other countries may be subject to other regulations and policies and should be handled accordingly.

[0095] Regardless of the foregoing, this disclosure also anticipates implementation schemes for users to selectively block the use or access to personal information data. That is, this disclosure anticipates providing hardware and / or software components to prevent or block access to such personal information data. For example, in determining whether a user views and / or reads text information, the inventive technology can be configured to allow the user to opt-in or opt-out at any time during or after registering for the service to participate in the collection and / or sharing of personal information data. In addition to providing opt-in and opt-out options, this disclosure envisions providing notifications related to access to or use of personal information. For example, users may be notified when downloading an application that their personal information data will be accessed, and then reminded again just before the application accesses the personal information data.

[0096] Furthermore, the purpose of this disclosure is to manage and process personal information data to minimize the risk of unintentional or unauthorized access or use. Once data is no longer needed, this risk can be minimized by limiting data collection and deleting data. Additionally, and where applicable, including in certain health-related applications, data deidentification can be used to protect user privacy. Deidentification can be facilitated, where appropriate, by removing identifiers, controlling the amount or specificity of stored data (e.g., collecting location data at the city level rather than the address level or at a level insufficient for facial recognition), controlling how data is stored (e.g., aggregating data among users), and / or other methods such as differentiated privacy.

[0097] Therefore, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, it is also contemplated that various embodiments can be implemented without access to such personal information data. That is, various embodiments of the present invention will not be unable to function properly due to the absence of all or part of such personal information data.

[0098] Figure 19 An electronic system 1900 is shown that can be used to implement one or more specific embodiments of the subject matter. The electronic system 1900 may be... Figure 1 and / or Figure 3 The illustrated electronic device 100, and / or may be part of it. The electronic system 1900 may include various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 1900 includes a bus 1908, one or more processing units 1912 (e.g., including…). Figure 3 The specific implementation of the processing circuit 306), system memory 1904 (and / or buffer), ROM 1910, permanent storage device 1902, input device interface 1914, output device interface 1906 and one or more network interfaces 1916, or subsets and variations thereof.

[0099] Bus 1908 generally represents all system, peripheral, and chipset buses that communicatively connect numerous internal devices of electronic system 1900. In one or more embodiments, bus 1908 communicatively connects one or more processing units 1912 to ROM 1910, system memory 1904, and permanent storage device 1902. One or more processing units 1912 retrieve instructions to be executed and data to be processed from these various memory units in order to perform the processes disclosed in this subject matter. In different embodiments, one or more processing units 1912 may be a single processor or a multi-core processor.

[0100] ROM 1910 stores static data and instructions required by one or more processing units 1912 and other modules of electronic system 1900. On the other hand, persistent storage device 1902 can be a read-write memory device. Persistent storage device 1902 can be a non-volatile memory cell that stores instructions and data even when electronic system 1900 is powered off. In one or more embodiments, mass storage devices (such as magnetic disks or optical disks and their corresponding disk drives) can be used as persistent storage device 1902.

[0101] In one or more embodiments, a removable storage device (such as a floppy disk, flash drive, and its corresponding disk drive) may be used as persistent storage device 1902. Like persistent storage device 1902, system memory 1904 may be a read-write memory device. However, unlike persistent storage device 1902, system memory 1904 may be volatile read-write memory, such as random access memory. System memory 1904 may store any instructions and data that one or more processing units 1912 may need during operation. In one or more embodiments, the processes disclosed in this subject matter are stored in system memory 1904, persistent storage device 1902, and / or ROM 1910. One or more processing units 1912 retrieve instructions to be executed and data to be processed from these various memory units to execute the processes of one or more embodiments.

[0102] Bus 1908 is also connected to input device interface 1914 and output device interface 1906. Input device interface 1914 enables the user to transmit information and select commands to electronic system 1900. Input devices that can be used with input device interface 1914 may include, for example, an alphanumeric keypad and a pointing device (also known as a "cursor control device"). Output device interface 1906 enables, for example, the display of images generated by electronic system 1900. For example, output devices that can be used with output device interface 1906 may include a display (such as...) Figure 1 The display 104). One or more embodiments may include a device that acts as both an input device and an output device, such as a touch screen. In these embodiments, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input.

[0103] Finally, as Figure 19 As shown, bus 1908 also couples electronic system 1900 to one or more networks and / or to one or more network nodes via one or more network interfaces 1916, such as Figure 1The electronic device 100 is shown. In this way, the electronic system 1900 may be part of a computer network (such as a LAN, wide area network (“WAN”), or intranet), or may be part of a network of networks (such as the Internet). Any or all components of the electronic system 1900 may be used in conjunction with the subject matter disclosed herein.

[0104] According to various aspects of this disclosure, a method is provided, the method comprising: displaying a request for authorization from a user by an electronic device; identifying at least one gaze position of at least one eye of the user while displaying the request; determining, based on the at least one gaze position, whether the user has viewed the displayed request for authorization; and if it is determined that the user has viewed the displayed request for authorization, providing an optional option for granting authorization to the user.

[0105] According to other aspects of this disclosure, a method is provided, the method comprising: providing a user interface for an application by an electronic device, the user interface including text; obtaining multiple gaze positions of at least one eye of a user; determining, based on the multiple gaze positions, that the user has viewed the text; and modifying the user interface based on the determination to remove the text.

[0106] According to other aspects of this disclosure, a method is provided, the method comprising: displaying a widget or user interface element by an electronic device, the user interface element presenting time-related information updated by an update frequency; obtaining multiple gaze positions of at least one eye of a user; determining a viewing frequency of the displayed widget or user interface element based on the multiple gaze positions; and modifying the update frequency of the widget or user interface element based on the viewing frequency.

[0107] According to other aspects of this disclosure, a method is provided, the method comprising: acquiring at least one image of a physical document via a camera of an electronic device; determining the location of the physical document based on the at least one image; acquiring multiple gaze positions of at least one eye of a user of the electronic device via an eye tracker of the electronic device; and determining, via the electronic device, that the user is reading the physical document based on the multiple gaze positions relative to the location of the physical document.

[0108] According to other aspects disclosed in this subject matter, an electronic device is provided, the electronic device comprising: a display; an eye tracker configured to provide eye tracking data for at least one eye of a user; and processing circuitry configured to: provide a request for authorization from the user for display by the display; identify at least one gaze position based on the eye tracking data from the eye tracker when displaying the request; determine, based on the at least one gaze position, whether the user has viewed the displayed request for authorization; and if it is determined that the user has viewed the displayed request for authorization, provide an optional option for granting authorization to the user.

[0109] The various functions described above can be implemented in digital electronic circuits, computer software, firmware, or hardware. This technology can be implemented using one or more computer program products. Programmable processors and computers can be included in or packaged as mobile devices. The process and logic flow can be executed by one or more programmable processors and one or more programmable logic circuits. General-purpose and special-purpose computing devices, as well as storage devices, can be interconnected via communication networks.

[0110] Some specific implementations include storing computer program instructions in machine-readable or computer-readable media (or computer-readable storage media, machine-readable media, or machine-readable storage media), such as microprocessors, storage devices, and memories. Examples of such computer-readable media include RAM, ROM, read-only optical discs (CD-ROM), recordable optical discs (CD-R), rewritable optical discs (CD-RW), read-only digital versatile optical discs (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and / or solid-state hard disk drives, high-density optical discs, any other optical or magnetic media, and floppy disks. Computer-readable media can store computer programs that can be executed by at least one processing unit and include a set of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as machine code generated by a compiler, and files that include higher-level code that can be executed by a computer, electronic components, or microprocessor using an interpreter.

[0111] While the above discussion primarily concerns microprocessors or multi-core processors that execute software, some implementations are performed by one or more integrated circuits such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some implementations, such integrated circuits execute instructions stored on the circuit itself.

[0112] As used in this specification and any claim of this patent application, the terms "computer," "processor," and "memory" refer to electronic or other technical devices. These terms exclude persons or groups of persons. For the purposes of this specification, the terms "display" or "being displayed" mean display on an electronic device. As used in this specification and any claim of this patent application, the terms "computer-readable medium" and "computer-readable media" are entirely limited to tangible, touchable objects that store information in a form readable by a computer. These terms do not include any wireless signals, wired download signals, or any other transient signals.

[0113] To provide interaction with the user, specific implementations of the subject matter described herein can be implemented on a computer having a display device for displaying information to the user, as well as keyboards and pointing devices that the user can use to provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0114] Many of the features and applications described above can be implemented as software processes that specify a set of instructions to be recorded on a computer-readable storage medium (also referred to as a computer-readable medium). When these instructions are executed by one or more processing units (e.g., one or more processors, processor cores, or other processing units), the instructions cause the one or more processing units to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard disk drives, EPROMs, etc. Computer-readable media do not include carrier waves and electrical signals transmitted wirelessly or via wired connections.

[0115] In this specification, the term "software" is intended to include firmware residing in read-only memory or applications stored in magnetic storage devices, which can be read into memory for processing by a processor. Similarly, in some embodiments, multiple software aspects disclosed herein may be implemented as sub-parts of a larger program while retaining the different software aspects disclosed herein. In some embodiments, multiple software aspects may also be implemented as independent programs. Finally, any combination of independent programs that collectively implement the software aspects described herein is within the scope of this disclosure. In some embodiments, when installed to run on one or more electronic systems, a software program defines one or more specific machine implementations that execute and perform the operations of the software program.

[0116] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, objects, or other units suitable for use in a computing environment. Computer programs may, but do not necessarily, correspond to files in a file system. A program may be stored as a part of a file containing other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in its description, or in multiple coordinating files (e.g., a file storing one or more modules, subroutines, or code sections). Computer programs can be deployed to execute on a single computer or on multiple computers located at the same site or distributed across multiple sites and interconnected via a communication network.

[0117] It should be understood that the specific order or hierarchical structure of the blocks in the process disclosed in this invention is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchical structure of the blocks in the process may be rearranged or all illustrated blocks may be executed. Some of these blocks may be executed simultaneously. For example, in some cases, multitasking and parallel processing may be advantageous. Furthermore, the division of various system components in the above embodiments should not be construed as requiring such division in all embodiments, and it should be understood that the program components and system may generally be integrated together in a single software product or packaged into multiple software products.

[0118] The preceding descriptions are provided to enable those skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, this claim is not intended to be limited to the aspects shown herein, but rather to be consistent with the language of the claim, wherein references to elements in singular values ​​are not intended to mean “one and only one,” but rather “one or more,” unless specifically indicated. Unless otherwise specifically stated, the term “some” means one or more. Male pronouns (e.g., his) include female and neutral (e.g., her and its), and vice versa. Titles and subtitles (if any) are used for convenience only and do not limit the disclosure of this subject matter.

[0119] The predicates “configured to,” “capable of operating,” and “programmed to” do not imply any specific tangible or intangible modification to a particular subject but are intended to be used interchangeably. For example, a component or a processor configured to monitor and control operations may also mean that the processor is programmed to monitor and control operations or that the processor is operable to monitor and control operations. Similarly, a processor configured to execute code can be interpreted as either programmed to execute code or operable to execute code.

[0120] The phrase "aspect" does not imply that this aspect is essential to the present subject matter or that this aspect applies to all configurations of the present subject matter. Disclosures relating to an aspect may apply to all configurations, or one or more configurations. The phrase "aspect" may refer to one or more aspects, and vice versa. The phrase "configuration" does not imply that this configuration is essential to the present subject matter or that this configuration applies to all configurations of the present subject matter. Disclosures relating to a configuration may apply to all configurations, or one or more configurations. The phrase "configuration" may refer to one or more configurations, and vice versa.

[0121] The word “example” is used in this document to mean “used as an example or illustration.” Any aspect or design described in this document as an “example” is not necessarily to be construed as superior or advantageous to any other aspect or design.

[0122] All structural and functional equivalents of elements throughout the various aspects described herein that are known or later become apparent to those skilled in the art are expressly incorporated herein by reference and are intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be made public, regardless of whether such disclosure is expressly stated in the claims. No claim element should be interpreted in accordance with 35 U.S.SC §112(f) unless the element is expressly stated using the phrase “means for…” or, in the case of a method claim, using the phrase “steps for…”. Furthermore, terms such as “comprising,” “having,” etc., are used to a certain extent in the specification or claims, and such terms are intended to be included in a manner similar to how the term “comprising” is interpreted when used as a transitional word in a claim.

Claims

1. A method comprising: An electronic device provides the user interface for an application, the user interface including text; Obtain multiple gaze positions of at least one of the user's eyes; Based on the multiple gaze positions, it is determined that the user has viewed the text; as well as Based on the determination, the user interface is modified to remove the text.

2. The method according to claim 1, wherein the determination includes: It is determined that the user has viewed the text at least a predetermined number of times.

3. The method of claim 2, wherein the user interface includes an icon associated with the text, and wherein modifying the user interface to remove the text includes: The icon is provided, but the text is not.

4. A method comprising: The electronic device displays user interface elements that present time-related information updated at a specific frequency. Obtain multiple gaze positions of at least one of the user's eyes; The viewing frequency of the user interface elements displayed is determined based on the multiple gaze positions; as well as The update frequency of the time-related information is modified based on the viewing frequency.

5. The method according to claim 4, wherein the time-related information includes one or more of weather information, financial information, news supply information, or calendar information.

6. The method according to claim 4, further comprising: The user's at least one eye is identified on the user interface of an inactive application based on the multiple gaze positions; as well as The inactive application is activated based on the identified dwell time.

7. The method of claim 1, wherein determining that the user has viewed the text includes detecting a fixed gaze on the text lasting at least a threshold dwell time.

8. The method of claim 1, wherein determining that the user has viewed the text includes detecting a gaze movement pattern corresponding to the movement of the reading eye.

9. The method of claim 1, wherein modifying the user interface includes enlarging the icon located near the removed text.

10. The method of claim 1, wherein modifying the user interface includes expanding the control elements into the space freed up by the removed text.

11. The method of claim 1, wherein modifying the user interface includes suppressing the reappearance of the removed text unless triggered by user accessibility settings.

12. The method of claim 1 further includes recording a count of text viewing events for future UI adaptation.

13. The method of claim 1, further comprising determining that the user has stopped looking at the text while operating a control associated with the text.

14. The method of claim 4, wherein determining the viewing frequency includes identifying discrete viewing events separated by at least a predetermined minimum interval.

15. The method of claim 4, wherein modifying the update frequency includes reducing the update frequency when the viewing frequency is below a threshold.

16. The method of claim 4, wherein modifying the update frequency includes increasing the update frequency when the viewing frequency exceeds a threshold.

17. The method of claim 4, wherein modifying the update frequency includes ensuring that the update frequency remains above a predetermined minimum.

18. The method of claim 4, wherein determining the viewing frequency includes counting gaze positions that fall within the boundaries of the user interface element within a sliding time window.

19. The method of claim 4, wherein the user interface elements include widgets presented on the home screen or lock screen environment.

20. The method of claim 4, wherein modifying the update frequency includes temporarily pausing the update in response to no user eye being detected on the user interface element within a predetermined time period.

21. The method of claim 1, further comprising highlighting a portion of the text as the user's gaze moves across the text.

22. The method of claim 4, further comprising generating a notification when the change in viewing frequency exceeds a threshold.

23. An electronic device, comprising: monitor; An eye tracker configured to provide gaze position data for at least one of a user's eyes; as well as One or more processors, the one or more processors being configured to perform the method as described in any one of claims 1 to 22.

24. The electronic device of claim 23, wherein the electronic device includes a head-mounted device, the head-mounted device including a transparent or translucent display.

25. A computer-readable storage medium storing instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 22.

Images