Gaze-based control

Abstract

Aspects of the subject technology relate to gaze-based control of electronic devices. The gaze-based control can include implementing an option for providing a user authorization upon determining that the user has viewed and / or read text associated with a request for authorization of the user. The gaze-based control can also include modifying a user interface or user interface element based on the user viewing and / or reading. The gaze-based control can be based on determining whether a user has viewed and / or read an electronic document and / or a physical document.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 011,975, filed April 17, 2020, entitled “Gaze-Based Control,” the disclosure of which is hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0003] This specification generally relates to electronic devices, and more specifically but not exclusively to gaze-based control for electronic devices. BACKGROUND

[0004] Portable electronic devices, such as laptop computers, cellular telephones, and tablet computers, are often provided with displays for displaying visual information. These portable devices often require a user to authorize an action before the device takes the action. BRIEF DESCRIPTION OF DRAWINGS

[0005] Some features of the subject technology are set forth in the appended claims. However, for purpose of explanation, several implementations of the subject technology are set forth in the following figures.

[0006] Figure 1 A perspective view of an example electronic device is shown in accordance with various aspects of the subject technology.

[0007] Figure 2 A perspective view of an electronic device viewed by a user at a gaze location is shown in accordance with various aspects of the subject technology. Figure 1

[0008] Figure 3 An example of another implementation of an electronic device is shown in accordance with various aspects of the subject technology.

[0009] Figure 4 An example of a user authorization request is shown in accordance with various aspects of the subject technology.

[0010] Figure 5 A user authorization request viewed and / or read by a user is shown in accordance with various aspects of the subject technology. Figure 4

[0011] Figure 6 A flowchart of an example process for gaze-based control for user authorization is shown in accordance with various aspects of the subject technology.

[0012] Figure 7 An example of gaze locations that can be obtained over time is shown in accordance with various aspects of the subject technology. ​​

[0013] Figure 8 Gaze positions corresponding to reading eye movements are shown in accordance with various aspects of the subject technology.

[0014] Figure 9 Examples of highlighting reading text based on gaze position are shown in accordance with various aspects of the subject technology.

[0015] Figure 10 An example user interface viewed by a user is shown in accordance with various aspects of the subject technology.

[0016] Figure 11 An example of a user interface of Figure 10 where the user interface has been modified based on the user gaze position in accordance with various aspects of the subject technology.

[0017] Figure 12 Another example of a user interface of Figure 10 where the user interface has been modified based on the user gaze position in accordance with various aspects of the subject technology.

[0018] Figure 13 A flowchart of an example process for gaze-based control of a user interface in accordance with various aspects of the subject technology is shown.

[0019] Figure 14 A flowchart of an example process for gaze-based control of a user interface element that displays time-dependent information in accordance with various aspects of the subject technology is shown.

[0020] Figure 15 An example electronic device configured to determine whether a physical document has been viewed and / or read in accordance with various aspects of the subject technology is shown.

[0021] Figure 16 An example electronic device having a transparent or semi-transparent display and configured to determine whether a physical document has been viewed and / or read in accordance with various aspects of the subject technology is shown.

[0022] Figure 17 An example electronic device having an opaque display and configured to determine whether a physical document has been viewed and / or read in accordance with various aspects of the subject technology is shown.

[0023] Figure 18 A flowchart of an example process for gaze-based control of an electronic device based on a physical document in accordance with various aspects of the subject technology is shown.

[0024] Figure 19 An electronic system in which one or more implementations of the subject technology can be implemented is shown. DETAILED DESCRIPTION

[0025] The detailed description set forth below is intended as a description of various configurations of the subject technology and is not intended to represent the only configurations in which the subject technology can be practiced. The appended drawings are incorporated herein and constitute a part of the detailed description. The detailed description includes specific details for the purpose of providing a thorough understanding of the subject technology. However, it will be clear and apparent to those skilled in the art that the subject technology is not limited to the specific details set forth herein and can be practiced without some of the specific details. In some instances, well-known structures and components are shown in block diagram form in order to avoid obscuring the concepts of the subject technology.

[0026] The present disclosure provides systems and methods for controlling an electronic device based in part on whether a user of the device has viewed and / or read textual information, such as textual information displayed by the device (or another device) or textual information included on a physical file viewed by the user of the device. The device can use eye tracking data from one or more eye trackers on the device and / or another device to determine whether the user has viewed and / or read the textual information. The eye tracking data can include gaze information for one or both of the user's eyes, such as gaze position and / or gaze direction. The gaze information can indicate where and when the user is looking at various locations, such as locations on a display of the device, and / or how the user's eyes move over time.

[0027] Various aspects of the electronic device can be controlled based on the gaze information. In one example, the gaze information can be used to confirm that the user has viewed and / or read text that has been presented to the user before the user can authorize an action by the device. For example, to prevent a user from inadvertently deleting data from a memory of the device, the device can present a dialog box that includes text, such as "Are you sure you want to delete this file?"

[0028] In conventional devices, this type of text is typically provided with a "Yes" button for authorizing the deletion, and a "No" button that can be selected to reject the deletion. However, a user who is busy working on the device, or otherwise distracted, can inadvertently instruct the device to delete the file and select the "Yes" button without reading the text of the dialog box. In these cases, the deleted data can be lost, which can result in frustration, delay, and / or inefficiency for the user, as well as waste processing and / or memory resources of the device, such as resources used to delete the data and / or resources used to recreate the data that was inadvertently deleted.

[0029] The systems and methods disclosed herein can be applied to confirm that a user has viewed and / or read text in a dialog box before allowing the user to select an option for authorizing deletion.

[0030] The systems and methods disclosed herein can also be applied to determine whether a user has viewed and / or read other information before allowing the user to authorize various device actions. As an example, an electronic device can determine whether a user of the device has viewed and / or read text associated with allowing access to personal data, text associated with allowing access to one or more components of the device, text associated with allowing installation of software on the device, and / or text corresponding to one or more terms and / or conditions before allowing the user to authorize the associated action.

[0031] The systems and methods disclosed herein can also be applied to modify operation of a device based on determining whether and / or when a user has viewed and / or read textual information. For example, text that forms part of a user interface or other device display screen can be removed from the user interface or display screen after the user has viewed and / or read the textual information a predetermined number of times. As another example, a user interface element (such as a widget) that displays time-dependent information can be updated at a frequency that depends on how frequently the user views the user interface element.

[0032] An illustrative electronic device in which the technology described herein can be implemented is shown in Figure 1 An illustrative electronic device in which the technology described herein can be implemented is shown in Figure 1 In the example of FIG. 1, the device 100 has been implemented in the form of a portable computer. As shown in Figure 1 The electronic device 100 can include a keyboard 102 having keys 110, a display 104, a housing 106, and a touchpad such as the touchpad 112. As shown in Figure 1 The display 104 can be mounted on a front portion of the housing 106.

[0033] The housing 106, which can sometimes be referred to as a case, can be formed from plastic, glass, ceramic, fiber composites, metal (e.g., stainless steel, aluminum, etc.), other suitable materials, or a combination of any two or more of these materials. The housing 106 can be formed using a unibody configuration in which some or all of the housing 106 is machined or molded into a single structure, or can be formed using multiple structures (e.g., an internal frame structure, one or more structures that form the exterior housing surfaces, etc.). As shown in Figure 1 The housing 106 can have multiple portions. For example, the housing 106 can have an upper portion 106A and a lower portion 106B. The upper portion 106A can be coupled to the lower portion 106B using a hinge that allows the upper portion 106A to rotate relative to the lower portion 106B about an axis of rotation 108. In some embodiments, the keyboard 102 and the touchpad 112 can be mounted in the lower portion 106B.

[0034] Display 104 can be a touch screen incorporating capacitive touch electrodes or other touch sensor components, or can be a non-touch sensitive display. Display 104 can include display pixels formed from light emitting diodes (LEDs), organic light emitting diodes (OLEDs), plasma cells, electrophoretic display elements, electro wetting display elements, liquid crystal display (LCD) components, or other suitable display pixel structures. Display 104 can have openings (e.g., openings in a passive portion or an active portion of display 104), such as openings to accommodate buttons or one or more cameras, such as camera 105. Camera 105 and / or other cameras disposed around or behind the display can be operated by electronic device 100 to capture images of a user’s face, one or both of the user’s eyes, and / or one or more features of one or both of the user’s eyes.

[0035] Images captured by camera 105 and / or other cameras (e.g., cameras located at or near the corners of display 104) can be processed by processing circuitry within electronic device 100 to detect the user’s eyes and determine the position of the user’s gaze relative to the camera, relative to display 104 and / or a location away from display 104, and / or relative to the electronic device 100 and / or a direction. For example, the processing circuitry can determine the position of the user’s gaze relative to text in an electronic document displayed by the display, relative to one or more portions of a user interface or user interface elements displayed by the display, and / or relative to text in a physical document separate from electronic device 100.

[0036] Camera 105 and associated processing circuitry can form an eye tracker of electronic device 100, which can be configured to track the gaze position and / or motion of the user’s eyes at various times and / or over time. The eye tracker can include a single camera (e.g., an infrared camera or a visible light camera) whose field of view is configured to include the user’s eye position, multiple such cameras, and / or one or more additional sensors such as a head-mounted sensor, a near-infrared projector, etc.

[0037] Figure 1 The configuration of electronic device 100 in the example is merely illustrative. In other implementations, electronic device 100 can be implemented in the form of a computer, such as a computer integrated into a display such as a computer monitor, a tablet computer, a smartphone, a slightly smaller portable device such as a wristwatch device, a pendant device, or other wearable device or a micro device, a media player, a gaming device, a navigation device, a computer monitor, a television, or other electronic device. In another example discussed in further detail below (see, e.g., FIG. 2), electronic device 100 can be implemented as a computer monitor or other display device that is integrated with a camera and / or other sensors. Figure 3The electronic device 100 can be implemented as a head-mountable device. In various configurations, the electronic device 100 can be communicatively coupled to one or more cameras and / or other eye-tracking sensors that are separate from the electronic device.

[0038] like Figure 1 As shown, the display content 120 can be displayed by the display 104 (e.g., by manipulating the display pixel array according to the corresponding digital pixel values ​​in the display frame). Figure 1 In the example, the displayed content 120 includes the application display window 114, the dialog box 116, and the user interface element 118.

[0039] For example, application display window 114 can be a word processor, compiler, rendering editor, spreadsheet, image viewer, file manager, web browser, video player, audio player, browser, or generally any user interface window. Application display window 114 can display electronic documents for the user of electronic device 100 to view and / or edit. In one example, application display window 114 displays an electronic document edited by the user. In another example, application display window 114 displays a terms and conditions document that will be read by the user.

[0040] Dialog box 116 may be a dialog box for requesting user authorization for an action to be performed by electronic device 100 (e.g., by the processing circuitry of the electronic device). User authorization may be for actions associated with application display window 114 (e.g., actions such as closing a document or other editable data structure without saving changes, accepting the terms and conditions described in an electronic document, deleting data from the memory of electronic device 100, granting access to data stored by electronic device 100, granting access to one or more components of electronic device 100, granting access to the user's personal information), authorizing e-commerce and / or NFC transactions, or generally any action. Dialog box 116 may be a pop-up window that appears in response to a user request for an action on electronic device 100 and / or in response to an automated action (e.g., a software update, an attempt by the application to access information, etc.). Although dialog box 116 is in Figure 1 In this example, the dialog box 116 is displayed separately from the application display window 114, but it should be understood that in other examples, the dialog box 116 may be generated on, within, and / or as part of the application display window 114.

[0041] User interface element 118 can be a widget that displays time-related information, such as a clock widget, calendar widget, news feed widget, weather widget, stock quote widget, etc.

[0042] According to aspects of the disclosure, the electronic device 100 controls one or more actions associated with an application based on one or more gaze locations of one or both of the user’s eyes, as shown in Figure 2

[0043] As shown in Figure 2 , a gaze direction 203 of a user, such as the user 200, can be determined by the electronic device 100 to intersect with a physical or virtual object at a gaze location 202, such as the display 104. An image of the user 200 (e.g., infrared and / or visible light images of one or more of the user’s eyes) captured by the camera 105 and / or other sensors (e.g., a depth sensor that determines a distance between the user’s eyes and the camera 105) can be used to determine the gaze location 202. In cases where the gaze location 202 is a location on the display 104, the gaze location 202 can be determined in terms of coordinates of a pixel or group of pixels 204 of the display. Based on the determined gaze location 202, the electronic device 100 can control one or more aspects of the application display window 114, the dialog box 116, and / or the user interface element 118. Because the electronic device 100 controls the location at which the display content 120 is displayed, the gaze location 202 in pixel coordinates can be determined relative to the displayed content.

[0044] In the example of Figure 2 , the gaze location 202 is within the bounds of the application display window 114. In some cases, operations of the application associated with the application display window 114 and / or the appearance of the application display window 114 can be controlled based on the gaze location 202. For example, when the gaze location 202 is within the bounds of the application display window 114 and / or dwells within the bounds of the application display window 114 for a predetermined period of time (e.g., a few milliseconds, tens of milliseconds, or hundreds of milliseconds), the application display window 114 can be visually highlighted or activated from an inactive state. In some cases, one or more sub-portions of the application display window 114 can be modified and / or otherwise controlled based on the gaze location 202.

[0045] ​A dialog box 116 can be displayed to request user authorization for one or more actions of the electronic device 100. For example, the user can not have saved edits to an electronic document in the application display window 114 and can enter a request to close the document. To prevent the user from losing desired edits to the document by closing the document without saving the document, the electronic device 100 can provide the dialog box 116 such as "Are you sure you want to close without saving?" To help reduce the likelihood of the user inadvertently agreeing to close the document without saving (e.g., by clicking on an authorizable selectable option to close without saving without viewing or reading the text in the dialog box 116), eye tracking data such as one or more gaze locations 202 can be used to determine whether the user has viewed and / or read the text in the dialog box 116.

[0046] As discussed above in connection with Figure 1 the example in which the electronic device 100 is implemented as a laptop computer Figure 1 and Figure 2 the example shown is merely illustrative. For example, Figure 3 another example implementation of the electronic device 100 is shown. As shown in the example of Figure 3 One or more sensors such as the sensor 152 can be positioned at various locations on or within the body 300 of the electronic device 100. The body 300 can be arranged to secure the electronic device 100 to a user (e.g., to the head of the user) such that the display 104 is positioned between the eyes of the user and a view of the environment of the electronic device 100.

[0047] In this example, the display 104 can be an opaque display that prevents the user from directly viewing some or all of the environment surrounding the electronic device 100, or can be a transparent or semi-transparent display that allows the user to directly view portions of the environment surrounding the electronic device 100 through the display. In implementations in which the display 104 is an opaque display that blocks direct viewing of the environment, one or more cameras such as the camera 150 can provide a see-through video view to the display 104 in some operational scenarios. In implementations in which the display 104 is a transparent or semi-transparent display, the display 104 can overlay display content on a direct view of a portion of the physical environment.

[0048] A transparent or semi-transparent display can have a medium through which light representing an image is directed to a person's eye. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, the transparent or semi-transparent display can be configured to selectively become opaque. Projection-based systems can employ retinal projection technology that projects graphical images onto a person's retina. The projection system can also be configured to project virtual objects into the physical environment, for example, as a hologram or on a physical surface.

[0049] As shown, the electronic device 100 can include one or more sensors, such as the sensor 152. The sensor 152 can include, but is not limited to, a camera, an image sensor, a touch sensor, a microphone, an inertial measurement unit (IMU), a heart rate sensor, a temperature sensor, a Lidar sensor, a radar sensor, a sonar sensor, a GPS sensor, a Wi-Fi sensor, a near-field communication sensor, and the like. The sensor 152 and / or one or more cameras, such as the camera 150, each having a field of view, can be positioned such that images and / or three-dimensional information (e.g., depth information) of the environment surrounding the electronic device 100 can be captured.

[0050] The processing circuitry 306 (e.g., one or more processors and memory) and / or local processing circuitry of the eye tracker 310 including the camera 105 can obtain eye tracking information and determine one or more gaze locations, directions, and / or movements of one or both eyes of a user. One or more internal sensors 304, such as an IMU, and / or one or more power sources, such as the battery 308, can also be disposed in the electronic device 100 and electrically and / or communicatively coupled to the processing circuitry 306.

[0051] The processing circuitry 306 can control various aspects of the operation of the electronic device 100 based on eye tracking data from the eye tracker 310. For example, Figure 4 A dialog box 116 is shown displaying text 400 by the display 104 and including a user authorization request. As an example, the user authorization request in the text 400 can be a request for user authorization to delete data from the electronic device 100, a request for user authorization to allow access to data (e.g., contact data, calendar data, passwords, photos, videos, etc.) stored by the electronic device 100, a request for user authorization to allow access to components (e.g., cameras, sensors, etc.) and / or otherwise change security settings of the electronic device 100, a request for user authorization to accept terms and conditions in a document displayed by the electronic device, and / or a request for user authorization to perform an e-commerce and / or NFC transaction. As shown, the text 400 includes a request for user authorization to delete data from the electronic device 100. Figure 4As shown in the example of FIG. 4B, the dialog box 116 can include a scroll bar (e.g., to scroll the text 400 if the text 400 is too long to display on a single screen). In some embodiments, the electronic device 100 can prevent the user from scrolling the document if it is determined that the current gaze location 202 is not within a text portion of the document (e.g., the use of the scroll bar 410 can be disabled if the gaze location 202 is not within the bounds of the dialog box 116, on the text 400, and / or indicates a reading movement of the user's eyes). As other examples, the request for user authorization in the text 400 can be a request for user authorization to install software on the electronic device 100, restore factory settings of the electronic device 100, couple a remote device to the electronic device 100, etc.

[0052] As shown in the example of FIG. 4B, the dialog box 116 can include a scroll bar (e.g., to scroll the text 400 if the text 400 is too long to display on a single screen). In some embodiments, the electronic device 100 can prevent the user from scrolling the document if it is determined that the current gaze location 202 is not within a text portion of the document (e.g., the use of the scroll bar 410 can be disabled if the gaze location 202 is not within the bounds of the dialog box 116, on the text 400, and / or indicates a reading movement of the user's eyes). As other examples, the request for user authorization in the text 400 can be a request for user authorization to install software on the electronic device 100, restore factory settings of the electronic device 100, couple a remote device to the electronic device 100, etc. Figure 4 Figure 4 As shown in the example of FIG. 4B, the dialog box 116 can include a scroll bar (e.g., to scroll the text 400 if the text 400 is too long to display on a single screen). In some embodiments, the electronic device 100 can prevent the user from scrolling the document if it is determined that the current gaze location 202 is not within a text portion of the document (e.g., the use of the scroll bar 410 can be disabled if the gaze location 202 is not within the bounds of the dialog box 116, on the text 400, and / or indicates a reading movement of the user's eyes). As other examples, the request for user authorization in the text 400 can be a request for user authorization to install software on the electronic device 100, restore factory settings of the electronic device 100, couple a remote device to the electronic device 100, etc.

[0053] As shown in the example of FIG. 4B, the dialog box 116 can include a scroll bar (e.g., to scroll the text 400 if the text 400 is too long to display on a single screen). In some embodiments, the electronic device 100 can prevent the user from scrolling the document if it is determined that the current gaze location 202 is not within a text portion of the document (e.g., the use of the scroll bar 410 can be disabled if the gaze location 202 is not within the bounds of the dialog box 116, on the text 400, and / or indicates a reading movement of the user's eyes). As other examples, the request for user authorization in the text 400 can be a request for user authorization to install software on the electronic device 100, restore factory settings of the electronic device 100, couple a remote device to the electronic device 100, etc.

[0054] As shown in the example of FIG. 4B, the dialog box 116 can include a scroll bar (e.g., to scroll the text 400 if the text 400 is too long to display on a single screen). In some embodiments, the electronic device 100 can prevent the user from scrolling the document if it is determined that the current gaze location 202 is not within a text portion of the document (e.g., the use of the scroll bar 410 can be disabled if the gaze location 202 is not within the bounds of the dialog box 116, on the text 400, and / or indicates a reading movement of the user's eyes). As other examples, the request for user authorization in the text 400 can be a request for user authorization to install software on the electronic device 100, restore factory settings of the electronic device 100, couple a remote device to the electronic device 100, etc. Figure 4 As shown in the example of FIG. 4B, the dialog box 116 can include a scroll bar (e.g., to scroll the text 400 if the text 400 is too long to display on a single screen). In some embodiments, the electronic device 100 can prevent the user from scrolling the document if it is determined that the current gaze location 202 is not within a text portion of the document (e.g., the use of the scroll bar 410 can be disabled if the gaze location 202 is not within the bounds of the dialog box 116, on the text 400, and / or indicates a reading movement of the user's eyes). As other examples, the request for user authorization in the text 400 can be a request for user authorization to install software on the electronic device 100, restore factory settings of the electronic device 100, couple a remote device to the electronic device 100, etc.

[0055] Figure 4 As shown in the example of FIG. 4B, the dialog box 116 can include a scroll bar (e.g., to scroll the text 400 if the text 400 is too long to display on a single screen). In some embodiments, the electronic device 100 can prevent the user from scrolling the document if it is determined that the current gaze location 202 is not within a text portion of the document (e.g., the use of the scroll bar 410 can be disabled if the gaze location 202 is not within the bounds of the dialog box 116, on the text 400, and / or indicates a reading movement of the user's eyes). As other examples, the request for user authorization in the text 400 can be a request for user authorization to install software on the electronic device 100, restore factory settings of the electronic device 100, couple a remote device to the electronic device 100, etc. Figure 5 ​​In this example, deactivation option 406 is enabled or activated to provide optional option 500 for granting user authorization. However, this is merely illustrative. In other examples, dialog box 116 may provide text 400 requesting user authorization and optional option 404 for denying user authorization, without providing a deactivation option for granting user authorization. In this example, optional option 500 may be added to dialog box 116 when it is determined that the user has viewed and / or read text 400.

[0056] Figure 6 A flowchart illustrating an exemplary process 600 for gaze-based control for user authorization, according to various aspects of the subject matter, is provided. For purposes of explanation, reference is made to... Figure 1 and / or Figure 3 The components are described Figure 6 An exemplary process. Further for illustrative purposes, Figure 6 The exemplary process outlined herein is described as occurring sequentially or linearly. However, Figure 6 Multiple frames of the exemplary process can occur in parallel. Furthermore, Figure 6 The boxes of the exemplary process do not need to be executed in the order shown and / or Figure 6 One or more of the exemplary processes in the box need not be executed.

[0057] In the illustrated example flowchart, at box 602, an electronic device (such as electronic device 100) displays a request for user authorization from the user. Displaying the request for user authorization may include displaying a dialog box 116 with text 400, as in combination with... Figure 4 As described.

[0058] At box 604, when a display request is made, the electronic device can identify at least one gaze position (such as gaze position 202) of at least one of the user's eyes.

[0059] At box 606, the electronic device may determine whether a user has viewed the displayed authorization request based on at least one gaze position. Determining whether the displayed authorization request has been viewed by the user based on at least one gaze position may include determining whether at least one gaze position 202 has been within the boundaries of the displayed request (e.g., within the boundaries of dialog box 116 or within the boundaries of the text portion of dialog box 116) for at least a predetermined period of time.

[0060] At box 608, if it is determined that the user has viewed the displayed request for authorization, the electronic device can provide selectable options for granting the user authorization, such as... Figure 5of the selectable options 500. Providing a selectable option for providing user authorization can include activating the deactivation option 406, or adding the selectable option to the displayed request after determining. The process 600 can also include identifying a dwell of at least one eye of the user on a user interface of an inactive application based on the plurality of gaze locations 202; and activating the inactive application based on the identified dwell.

[0061] As discussed above in connection with Figure 9 and Figure 8 The displayed request for user authorization can include text 400 associated with the authorization. Determining whether the user has viewed the displayed request for authorization based on the at least one gaze location can include determining whether the at least one gaze location indicates that the user has read the text (e.g., based on a plurality of gaze locations over time that indicate reading movements of the user’s eyes in conjunction with the text).

[0062] Figure 9 and Figure 4 to Figure 9 The plurality of gaze locations 202 obtained over time are shown as to how they can indicate whether the user has read the text 400. As shown in Figure 10 When the eyes of the user 200 rotate from a previous gaze direction 203P to a current gaze direction 203 during reading of the text, such as the text 400, a plurality of gaze locations 700 obtained over a period of time prior to the current gaze location 202 can fall along a substantially straight line as indicated by the arrow 702. If the substantially straight line of movement of the gaze locations 700 over time corresponds to a line of the text 400 (e.g., falls along a line of the text, the location of which is known due to the device itself displaying the text, or determined by the device using an image of the text in a physical document, or displayed by another device), the electronic device 100 can determine that the line of text has been read. However, Figure 10 It is also indicated that the movement of the gaze locations 700 over time as indicated by the arrow 702 is not constant. This can be because the human eye tends to move in a saccadic and fixed characteristic pattern during reading.

[0063] Figure 11 The sets of gaze locations 802 are shown as to how they can indicate relatively slow movement of the user’s eyes, such as during fixation, and the sets of gaze locations 804 can indicate fast movement of the user’s eyes, such as during saccades. The electronic device 100 can identify a dwell of at least one eye of the user on the text 800 to be read for authorization (e.g., the text 400) based on the plurality of gaze locations 202. The electronic device 100 can determine whether the user has read the text 800 based on the identified dwell. Figure 12 and Figure 10 to Figure 12saccades and fixation patterns at locations of text 400) and determines that the text at the corresponding locations has been read. In this example, electronic device 100 determines at least one reading location indicating that the user has read the text by determining that the at least one gaze location includes multiple time-separated gaze locations corresponding to reading eye movements (e.g., including at least one saccade and at least one fixation).

[0064] As Figure 13 shown, in an operating scenario in which electronic device 100 determines that text has been read (e.g., when multiple time-separated gaze locations move across portions of text and / or based on detected reading eye movements at locations corresponding to displayed text), electronic device 100 can also display a highlight 900 over portions of text 800 that have been read. As shown by arrow 902, highlight 900 can expand over text 800 along with reading movements of current gaze location 202.

[0065] In some operating scenarios, electronic device 100 can identify a sound associated with a word (e.g., an animal sound associated with an animal name or text describing an animal sound, or a sound of text spoken in a particular language sound). In scenarios in which a gaze location indicates which word a user is reading (e.g., as in the examples of Figure 1 and Figure 3 ), electronic device 100 can determine that the user is reading a word associated with the identified sound, and can output the identified sound (e.g., using one or more speakers or other audio output components) while the user is reading the word. This can be helpful when a child is learning to read, when a user is having difficulty reading a word, and / or when a story is being read to a child.

[0066] In addition to aspects described in connection with Figure 13 , processing circuitry 306 can control other aspects of operation of electronic device 100, including display 104. For example, Figure 13 shows an example in which application display window 114 displays a user interface 1000 of an application, and processing circuitry controls the user interface based on gaze locations of a user’s eyes. In Figure 13In the example shown, user interface 1000 is a user interface of a media player application and includes selectable control options, including a play button 1002 and a stop button 1003. In this example, play button 1002 includes a play icon 1004 and text 1006 to indicate to the user that button 1002 and play icon 1004 are for initiating “play” of media (e.g., audio, video, or other media) accessible by the application corresponding to the displayed user interface. In this example, stop button 1003 includes a stop icon 1005 and text 1007 to indicate to the user that stop button 1003 and stop icon 1005 are operable to “stop” play of the media.

[0067] However, over time, the user can learn that play button 1002 and play icon 1004 are for playing media, which can make text 1006 redundant and possibly inefficient to display using system resources. In some embodiments, electronic device 100 determines that gaze location 202 of the user has fallen within the bounds of play button 1002 and / or that gaze location 202 of the user indicates that the user has read text 1006 a predetermined number of times.

[0068] Figure 13 The example shows how user interface 1000 in application display window 114 can be modified to remove text 1006 in response to determining that gaze location 202 of the user has fallen within the bounds of play button 1002 and / or that gaze location 202 of the user indicates that the user has read text 1006 a predetermined number of times. In this example, text 1007 remains part of user interface 1000, and the size of play icon 1004 is increased to take advantage of the display area available due to the removal of text 1006. However, in other implementations, instead of increasing the size of the icon associated with the removed text, other portions of the user interface can expand or be modified to take advantage of the newly available space in the user interface.

[0069] For example, Figure 13 The example shows a scenario in which both text 1006 and text 1007 have been viewed and / or read a sufficient number of times to remove the text, allowing other portions of user interface 1000 to expand into the additional space 1200 available by removing text 1006 and 1007.

[0070] It will be appreciated that in some scenarios, text 1006 and / or 1007 can be removed based on gaze location 202 indicating that the user has stopped looking at the text (e.g., and is only looking at icons 1004 and 1005) while operating buttons 1002 and 1003, rather than based on the number of times the text has been viewed and / or read. It will also be appreciated that although the examples shown in FIGS. 10A-10C show text 1006 and 1007 being removed in response to the user’s gaze location 202 falling within the bounds of the buttons 1002 and 1003, in other implementations, text 1006 and 1007 can be removed in response to the user’s gaze location 202 falling within the bounds of the icons 1004 and 1005. Figure 10 to Figure 12Examples are depicted in which the user interface 1000 includes buttons 1002 and 1003 and icons 1004 and 1005, but the systems and methods described herein can be applied to remove substantially any text from a user interface after determining that the text has been viewed and / or read a sufficient number of times (e.g., a predetermined number of times).

[0071] Figure 11 A flowchart of an example process 1300 for gaze-based control of a user interface in accordance with various aspects of the subject technology is described. For purposes of explanation, the example process is described herein with reference to the components of the system 1000. Figure 12 and / or Figure 14 The blocks of the example process 1300 are described herein as occurring sequentially or linearly. However, Figure 1 Multiple blocks of the example process 1300 can occur in parallel. Further, Figure 3 The blocks of the example process 1300 do not have to be performed in the order shown and / or Figure 14 One or more of the blocks of the example process 1300 do not have to be performed. Figure 14 Figure 14 In the depicted example flowchart, at block 1302, the electronic device (such as the electronic device 100) displays a user interface, such as the user interface 1000 of an application, that includes text, such as the text 1006 or 1007.

[0072] At block 1304, the device obtains a plurality of gaze locations of at least one eye of the user (e.g., using eye tracking, such as the eye tracker 310 formed by the one or more cameras 105 and associated processing circuitry).

[0073] At block 1306, the electronic device can determine, based on the plurality of gaze locations, that the user has viewed the text, e.g., that the user’s eye dwelled on the text for at least a predetermined period of time, and / or that the user has read each of the words and / or characters of the text. Determining that the user has viewed the text can include determining that the user has viewed the text at least a predetermined number of times.

[0074] At block 1308, the electronic device modifies the user interface to remove the text based on the determination. As described in connection with the examples of The user interface can include icons (e.g., one or more icons such as the icons 1004 and 1005) associated with the text, and modifying the user interface to remove the text can include providing the icons without providing the text, as shown in the examples of

[0075] and Figure 14 Figure 14 Figure 1

[0076] ​​​In some operational scenarios, a user interface or user interface element (e.g., a widget) (e.g., via the display 104 of electronic device 100) may be displayed to provide the user with time-related information. Processing circuitry 306 can control the user interface or user interface element by adjusting the frequency of updating time-based information based on how frequently the user views the user interface or user interface element.

[0077] Figure 2 A flowchart illustrating an exemplary process 1400 for gaze-based control of a user interface or user interface element, displaying time-related information, according to various aspects of the subject matter, is provided. For illustrative purposes, reference is made to... Figure 15 and / or Figure 7 to Figure 9 The components are described Figure 15 An exemplary process. Further for illustrative purposes, Figure 16 The exemplary process outlined herein is described as occurring sequentially or linearly. However, Figure 17 Multiple frames of the exemplary process can occur in parallel. Furthermore, Figure 16 The boxes of the exemplary process do not need to be executed in the order shown and / or Figure 17 One or more of the exemplary procedures in the box do not need to be executed.

[0078] In the illustrated exemplary flowchart, at box 1402, an electronic device (such as electronic device 100) displays a widget (e.g., Figure 17 and Figure 16 The widget (118) is a user interface element that displays time-related information updated at a specified frequency. Time-related information may include one or more of the following: weather information, financial information, news supply information, calendar information, time information, or any other information that changes over time. Updates may initially be performed at the default update frequency and may include new information obtained from data locally stored on the device and / or from one or more remote devices or servers via a network such as the Internet. Obtaining updated information can consume power, processing time, and network bandwidth. The goal is to provide updates before the user views the widget, without obtaining more updates than the user would normally see.

[0079] At box 1404, the electronic device can acquire multiple gaze positions 202 of at least one of the user's eyes. These multiple gaze positions can be acquired over a period of time and can include gaze positions within the boundaries of a widget that appears when the user looks at the widget (e.g., within the boundaries of user interface element 118).

[0080] At box 1406, the electronic device can determine the frequency with which a user views a displayed widget based on multiple gaze positions. This viewing frequency can be determined, for example, by dividing the number of times the widget is viewed by the amount of time the view is recognized.

[0081] At box 1408, the electronic device can modify the update frequency of the widget based on the viewing frequency. For example, the update frequency can be set to a frequency that is a predetermined amount greater than the viewing frequency. In some operational scenarios, the user interface or user interface element can be activated from an inactive state based on the gaze position within the boundary of the user interface or user interface element, which indicates the dwell time of at least one of the user's eyes (e.g., a static gaze position over a predetermined amount of time).

[0082] In the examples described herein, the gaze position is primarily described as the intersection of the user's gaze direction with a portion of the display 104 of the electronic device 100. However, it should also be understood that the systems and methods described herein can be applied to scenarios where a user is viewing a physical document.

[0083] For example, Figure 18 An example is shown where the user's gaze position 202 intersects with text 1502 on the physical document 1500. In this example, the camera 150 of the electronic device 100 captures an image of the physical document 1500. The processing circuitry 306 can then determine the position of the physical document 1500 relative to the electronic device 100 based on the image and / or depth information captured by one or more sensors of the device. In various examples, the processing circuitry can also determine that text 1502 is present on the physical document 1500, determine the position of the text 1502, and / or recognize the text 1502 based on the image.

[0084] An eye tracker (e.g., including a camera 105) of an electronic device 100 can determine the gaze direction 203 of a user 200's eyes, and determine the gaze position 202 as a position on the physical document 1500 by using the determined position and / or distance of the physical document to determine the intersection of the gaze direction 203 and the physical document 1500.

[0085] Over time, the electronic device 100 can determine multiple gaze positions 202, and the multiple gaze positions 202 can indicate that the user is reading or has already read text 1502 (e.g., using...). Figure 1 (Some or all of the same techniques described in the examples combined with electronic text). Figure 3 In the example, user 200 views the physical document 1500 by looking over monitor 104. However, as shown below... Figure 18 and Figure 18 As indicated, user 200 can view through a transparent or semi-transparent display. Figure 18 () or by viewing the electronic version 1700 of the physical document 1500 displayed on an opaque monitor (e.g., using one or more images captured by camera 150, Figure 18The electronic version 1700 displayed by the display 104 can include electronic text 1701 that corresponds to the text 1502 of the physical document 1500.

[0086] As shown in Figure 18 some scenarios, the electronic device 100 can detect (e.g., based on one or more gaze locations and / or other information) that a user is having difficulty reading text of an electronic or physical document. Based on this determination, the electronic device 100 can provide a visual or audio enhancement to assist the user in reading the text. In the example of FIG. 17, Figure 16 the enhancement is a visual enhancement 1702 that represents a zoomed-in view of the text 1701. Determining that the user is having difficulty reading the text can include determining that the gaze location has stopped moving across the text for a period of time that is longer than a fixed time, and / or determining that the user is squinting (e.g., based on one or more images and / or depth information from the camera 105 and / or one or more other sensors of the electronic device 100).

[0087] Figure 17 A flow diagram of an example process 1800 for gaze-based control of an electronic device in conjunction with a physical document, in accordance with various aspects of the subject technology, is depicted. For purposes of explanation, the example process of Figure 19 and / or Figure 1 components of the system 100 are described herein. Further for purposes of explanation, the blocks of the example process of Figure 3 are described herein as occurring in sequence or linearly. However, Figure 3 multiple blocks of the example process of Figure 1 may occur in parallel. Moreover, Figure 19 the blocks of the example process of Figure 1 do not have to be performed in the order shown and / or one or more of the blocks of the example process of do not have to be performed.

[0088] In the depicted example flow diagram, at block 1802, at least one image of a physical document (such as the physical document 1500) can be obtained by a camera (such as the camera 150) of an electronic device (such as the electronic device 100).

[0089] At block 1804, the electronic device can determine a location of the physical document (e.g., relative to the electronic device) based on the at least one image (e.g., and / or based on additional information such as depth information from a sensor such as one of the sensors 152). The electronic device can also identify text in the physical document based on the at least one image.

[0090] At block 1806, the electronic device can obtain a plurality of gaze locations 202 of at least one eye of a user 200 of the electronic device by an eye tracker (e.g., including the camera 105 and associated circuitry) of the electronic device.

[0091] At block 1808, the electronic device can determine, based on the plurality of gaze locations relative to the location of the physical document, that the user is reading the physical document. The electronic device can also display, through a display of the electronic device, such as display 104 and based on the plurality of gaze locations, a highlight, such as highlight 900, overlaid on a view of the portion of the recognized text that has been read by the user. In some examples, the view of the portion of the recognized text can be a direct view through a semi- transparent or transparent display as depicted in FIG. 9A, and displaying the highlight can include displaying the highlight through the transparent or semi-transparent display. In other examples, the electronic device can display, through a display of the electronic device, at least one image as depicted in FIG. 9B, and displaying the highlight can include displaying the highlight overlaid on the displayed at least one image. ​ ​

[0092] As described above, one aspect of the present technology is the collection and use of data from specific and lawful sources to determine whether a user has viewed and / or read textual information. The present disclosure contemplates that in some instances, the collected data can include personal information data that uniquely identifies or can be used to identify a specific person. Such personal information data can include video data, three-dimensional geometry data, demographic data, location-based data, online identifiers, telephone numbers, email addresses, home addresses, data or records relating to a user’s health or fitness level (e.g., vital signs measurements, medication information, exercise information), date of birth, eye position or movement data, gaze tracking data, or any other personal information.

[0093] The present disclosure recognizes that the use of such personal information data, in the present technology, can be used to the benefit of users. For example, personal information data can be used to determine whether a user has viewed and / or read textual information.

[0094] ​​The present disclosure contemplates that the entities responsible for the collection, analysis, disclosure, transfer, storage, or other use of such personal information data will comply with well-established privacy policies and / or privacy practices. In particular, such entities should implement and consistently apply leader privacy practices generally recognized with respect to the maintenance and / or transfer of all personally identifiable information. Such practices will include, for example, employing appropriate data protection technologies and / or procedures for the purposes of protecting the confidentiality, integrity, and availability of all personally identifiable information. In addition, such entities should maintain, at least until the personal information data is destroyed, some manner of which personally identifiable information was collected as well as sources for such information. Such notification will generally be accomplished in compliance with applicable law and / or industry standards relevant to the maintenance and / or transfer of such information. In addition, such entities should be prepared to verify that the personal information data was collected, used, shared, and / or disposed of in accordance with applicable laws and / or industry standards. Further, such entities should be prepared to verify that other entities with access to such personal information data are bound to comply with the rules set forth herein as well as the same privacy practices and policies as the collecting entity.

[0095] Notwithstanding the foregoing, the present disclosure also contemplates embodiments in which users selectively block the use of, or access to, personal information data. That is, the present disclosure contemplates that hardware and / or software elements can be provided with mechanisms for preventing or blocking access to such personal information data. For example, in terms of determining whether a user views and / or reads textual information, an application can be configured such that the user has the ability to opt in or opt out of having personal information data collected and / or shared. In addition to providing “opt in” and “opt out” options, the present disclosure contemplates providing notifications relating to the access or use of personal information data. For instance, a user can be notified upon download of an application that their personal information data will be accessed and then reminded again just before the personal information data is accessed by the application.

[0096] Moreover, it is the intent of the present disclosure that personal information data should be managed and handled in a way to minimize risks of unintentional or unauthorized access or use of personal information data. Risk can be minimized by limiting the collection of data and deleting data once it is no longer needed. In addition, and when applicable, including in certain health related applications, data de-identification can be used to protect user privacy. De-identification can be facilitated, when appropriate, by removing identifiers, controlling the amount or

[0097] Thus, although the present disclosure broadly covers uses of personal information data to implement one or more various disclosed embodiments, the present disclosure also contemplates that the various embodiments can also be implemented without the need for accessing such personal information data. That is, the various embodiments of the present technology can be implemented in a way that does not require access to such personal information data.

[0098] ​ An electronic system 1900 is shown that can be used to implement one or more implementations of the subject technology. The electronic system 1900 can be ​ and / or ​ the electronic device 100 shown and / or can be a part of it. The electronic system 1900 can include various types of computer readable media and interfaces for various other types of computer readable media. The electronic system 1900 includes a bus 1908, one or more processing unit(s) 1912 (for example, including implementations of the processing circuitry 306), a system memory 1904 (and / or buffer(s)), a ROM 1910, the permanent storage device 1902, an input device interface 1914, an output device interface 1906, and one or more network interfaces 1916, or a subset or variation of these components. ​

[0099] The bus 1908 generally represents a communication device that communicatively couples various internal devices of the electronic system 1900. In one or more implementations, the bus 1908 communicatively connects the one or more processing unit(s) 1912 with the ROM 1910, the system memory 1904, and the permanent storage device 1902. The one or more processing unit(s) 1912 retrieve instructions from these various memory units and process data in order to execute the processes of the subject disclosure. In different implementations, the one or more processing unit(s) 1912 can be a single processor or multiple processors.

[0100] ​ROM 1910 stores static data and instructions required by one or more processing units 1912 and other modules of electronic system 1900. On the other hand, persistent storage device 1902 can be a read-write memory device. Persistent storage device 1902 can be a non-volatile memory cell that stores instructions and data even when electronic system 1900 is powered off. In one or more embodiments, mass storage devices (such as magnetic disks or optical disks and their corresponding disk drives) can be used as persistent storage device 1902.

[0101] In one or more embodiments, a removable storage device (such as a floppy disk, flash drive, and its corresponding disk drive) may be used as persistent storage device 1902. Like persistent storage device 1902, system memory 1904 may be a read-write memory device. However, unlike persistent storage device 1902, system memory 1904 may be volatile read-write memory, such as random access memory. System memory 1904 may store any instructions and data that one or more processing units 1912 may need during operation. In one or more embodiments, the processes disclosed in this subject matter are stored in system memory 1904, persistent storage device 1902, and / or ROM 1910. One or more processing units 1912 retrieve instructions to be executed and data to be processed from these various memory units in order to execute the processes of one or more embodiments.

[0102] Bus 1908 is also connected to input device interface 1914 and output device interface 1906. Input device interface 1914 enables the user to transmit information and select commands to electronic system 1900. Input devices that can be used with input device interface 1914 may include, for example, an alphanumeric keypad and a pointing device (also known as a "cursor control device"). Output device interface 1906 enables, for example, the display of images generated by electronic system 1900. For example, output devices that can be used with output device interface 1906 may include a display (such as...) ​ The display 104). One or more embodiments may include a device that acts as both an input device and an output device, such as a touch screen. In these embodiments, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input.

[0103] Finally, as ​ As shown, bus 1908 also couples electronic system 1900 to one or more networks and / or one or more network nodes via one or more network interfaces 1916, such as ​The electronic device 100 is shown. In this way, the electronic system 1900 can be part of a computer network such as a LAN, a wide area network ("WAN"), or an intranet, or can be part of a network of networks, such as the Internet. Any or all components of the electronic system 1900 can be used in conjunction with the subject disclosure.

[0104] According to various aspects of the disclosure, a method is provided that includes displaying, by an electronic device, a request from a user for user authorization; identifying, while displaying the request, at least one gaze location of at least one eye of the user; determining, based on the at least one gaze location, whether the user has viewed the displayed request for authorization; and providing, if it is determined that the user has viewed the displayed request for authorization, a selectable option for providing the user authorization.

[0105] According to other aspects of the disclosure, a method is provided that includes providing, by an electronic device, a user interface of an application, the user interface including text; obtaining a plurality of gaze locations of at least one eye of a user; determining, based on the plurality of gaze locations, that the user has viewed the text; and modifying the user interface to remove the text based on the determination.

[0106] According to other aspects of the disclosure, a method is provided that includes displaying, by an electronic device, a widget or user interface element, the user interface element presenting time-dependent information that is updated by an update frequency; obtaining a plurality of gaze locations of at least one eye of a user; determining, based on the plurality of gaze locations, a viewing frequency with which the user views the displayed widget or user interface element; and modifying the update frequency of the widget or user interface element based on the viewing frequency.

[0107] According to other aspects of the disclosure, a method is provided that includes obtaining, by a camera of an electronic device, at least one image of a physical document; determining, based on the at least one image, a location of the physical document; obtaining, by an eye tracker of the electronic device, a plurality of gaze locations of at least one eye of a user of the electronic device; and determining, by the electronic device, that the user is reading the physical document based on the plurality of gaze locations relative to the location of the physical document.

[0108] According to other aspects of the subject disclosure, there is provided an electronic device comprising: a display; an eye tracker configured to provide eye tracking data of at least one eye of a user; and processing circuitry configured to: provide, for display by the display, a request from the user for user authorization; identify, while displaying the request, at least one gaze location based on the eye tracking data from the eye tracker; determine, based on the at least one gaze location, whether the user has viewed the displayed request for authorization; and provide, if it is determined that the user has viewed the displayed request for authorization, a selectable option for providing the user authorization.

[0109] The various functions discussed above can be implemented in digital electronic circuitry, in computer software, firmware, or hardware. The techniques can be implemented using one or more computer program products. Programmable processors and computers can include one or more of the various types of programmable processing devices, such as microprocessors, microcontrollers, system on a chip (SoC), or the like. The program code can be stored in a memory, such as one or more of volatile or non-volatile memory. In some examples, the program code can be executed by a processor as a stand-alone program, by a plurality of separate programs, by a plurality of programs in a distributed computing environment, or any combination thereof. The various processes and functions described herein can be implemented using a plurality of separate programs in a distributed computing environment, or using one or more multi-threaded programs in a single computing device.

[0110] Some implementations include electronic components, such as microprocessors, storage and memory that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a variety of recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic and / or solid state hard drives, ultra-density optical discs, any other optical or magnetic media, and floppy disks. The computer-readable media can store a computer program that is executable by at least one processing unit and includes sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as is produced by a compiler, and files including higher-level code that is executed by a computer, an electronic component, or a microprocessor using an interpreter.

[0111] While the above discussion primarily refers to microprocessor or multi-core processors that execute software, some implementations are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some implementations, such integrated circuits execute instructions that are stored on the circuit itself.

[0112] As used in the description of the specification and in the claims of this patent application, the terms “computer”, “processor”, and “memory” all refer to electronic or other technological devices. These terms exclude humans or groups of humans. For purposes of this specification, the term “display” or “displaying” means displaying on an electronic device. As used in the description of the specification and in the claims of this patent application, the terms “computer readable medium” and “computer readable media” are entirely restricted to tangible, physical objects that store information in a form readable by a computer. These terms exclude any wireless signals, wired download signals, and any other ephemeral signals.

[0113] To provide for interaction with a user, the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user and a keyboard and a pointing device, such as a mouse, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0114] Many of the above-described features and applications can be implemented as software processes that are specified as a set of instructions recorded on a computer readable storage medium (also referred to as computer readable medium). When these instructions are executed by one or more processing unit(s) (e.g., one or more processors, cores of processors, or other processing units), they cause the processing unit(s) to perform the actions indicated in the instructions. Examples of computer readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc. Computer readable media does not include carrier waves and electronic signals over wire, fiber optic, or other communication

[0115] In this specification, the term “software” is meant to include firmware residing in read-only memory or applications stored in magnetic storage which can be read into memory for processing by a processor. Also, in some implementations, multiple software aspects of the subject disclosure can be implemented as sub-parts of a larger program while remaining within the scope of the subject disclosure. In some implementations, multiple software aspects can also be implemented in as separate programs. Finally, any combination of separate programs that together implement a software aspect described here is within the scope of the subject disclosure. In some implementations, the software programs, when installed to operate on one or more electronic systems, define one or more specific machine implementations that execute and perform the operations of the software programs.

[0116] A computer program, which can also be referred to or referred to as a program, software, a software application, an app, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or code portions). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and are interconnected by a communication network.

[0117] It is to be understood that the specific order or hierarchy of steps in the processes disclosed is an illustration. Based upon implementation requirements, the specific order or hierarchy of steps in the processes can be re-arranged, or omitted. Some of the steps can be performed simultaneously. The omission of such steps is not to be construed as a limitation of the processes. The steps as presented in this disclosure are in such a order for ease of explanation.

[0118] The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein, but is to be accorded the full scope consistent with the language claims, wherein reference to an element in the singular is not intended to mean "one and only one" unless specifically so stated, but rather "one or more." Unless specifically stated otherwise, the term "some" refers to one or more. Pronouns in the context of the word "a" or "an" shall not limit the significance of the phrase following the word to a single implement unless otherwise specifically stated herein. Headings of sections and titles of subsections are used for convenience only and do not necessarily affect the meaning of the scientific and technical content.

[0119] The predicate words "configured to", "operable to", and "programmed to" do not imply any particular tangible or intangible modification of a subject, but, rather, are intended to be synonymous with the phrase "caused to", with respect to any subject. For example, a processor configured to monitor and control operations is a processor that is caused to monitor and control operations. Likewise, a processor configured to execute code is a processor that is caused to execute code.

[0120] The phrase "aspect" does not mean that a particular aspect is essential to the subject technology or that the aspect applies across all configurations of the subject technology. A disclosure related to one aspect can apply to all configurations or one or more configurations. The phrase "aspect" can refer to one or more aspects, and vice versa. The phrase "configuration" does not mean that a particular configuration is essential to the subject technology or that the configuration applies across all configurations of the subject technology. A disclosure related to a configuration can apply to all configurations or one or more configurations. The phrase "configuration" can refer to one or more configurations and vice versa.

[0121] The word "example" is used herein to mean "serving as an example or illustration." Any aspect or design that is described herein as "example" is not necessarily to be construed as preferred or advantageous over other aspects or designs.

[0122] All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and intended to be encompassed by the claims. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether these disclosure is explicitly recited in the claims. No claim element is to be construed under the provisions of 35 U.S.C. § 112(f) unless the element is expressly recited in the claim as "means plus function." Additionally, no claim element is to be construed as a mere means for performing the recited function unless the element is expressly recited by the phrase "means for." Furthermore, the terms "comprise," "have" and like terms, when used in this document, are not meant to be limiting, but rather are intended to be interpreted as "including."

Claims

1. A method comprising: displaying, by an electronic device, a request from a user for a user authorization, wherein the displayed request includes text associated with the user authorization, and the request for a user authorization includes a request for a user authorization to perform an action for the electronic device, the action including at least one of a request for a user authorization to delete data from the electronic device, a request for a user authorization to allow access to data stored by the electronic device, or a request for a user authorization to allow access to a component of the electronic device; obtaining a plurality of time-separated gaze locations of at least one eye of the user while the request is displayed; determining, based on the plurality of time-separated gaze locations relative to a location of the text in the displayed request, whether the user has viewed the displayed request for authorization, wherein the determination is based at least in part on determining whether the plurality of gaze locations correspond to a reading eye movement; if it is determined that the user has viewed the displayed request for authorization, providing a selectable option to provide the user authorization; and performing the action in response to selecting the selectable option.

2. The method of claim 1, wherein determining, based on the at least one gaze location, whether the user has viewed the displayed request for authorization comprises: determining whether the at least one gaze location is within a boundary of the displayed request for at least a predetermined period of time.

3. The method of claim 1, wherein determining whether the at least one gaze location indicates that the user has read the text comprises: determining the at least one gaze location includes a plurality of time-separated gaze locations corresponding to the reading eye movement.

4. The method of claim 3, wherein the reading eye movement includes at least one saccade and at least one fixation.

5. The method of claim 3, further comprising highlighting a portion of the text as the plurality of time-separated gaze locations corresponding to the reading eye movement move through the portion of the text.

6. The method of claim 1, wherein providing the selectable option for providing the user authorization comprises: adding the selectable option to the displayed request after the determination.

7. The method of claim 1, wherein providing the selectable option to provide the user authorization includes: providing a deactivation option to provide the user authorization prior to the determination; and activating the selectable option after the determination.

8. The method of claim 1, wherein the request for a user authorization includes a selectable option to deny the authorization.

9. The method of claim 1, wherein the request for a user authorization includes a request for a user authorization to accept terms and conditions in a document displayed by the electronic device.

10. The method of claim 9, further comprising preventing the document from being scrolled by the user if it is determined that a current gaze location is not within a text portion of the document.

11. A method comprising: obtaining, by a camera of an electronic device, at least one image of a physical document; determining, based on the at least one image, a location of the physical document; obtaining, by an eye tracker of the electronic device, a plurality of time-separated gaze locations of at least one eye of a user of the electronic device; and ​ determining, by the electronic device, that the user is reading the physical document based on the plurality of time-separated gaze positions relative to the location of the physical document, wherein the determining is based at least in part on determining whether the plurality of gaze positions of the at least one eye of the user correspond to reading eye movements.

12. The method of claim 11, further comprising identifying text in the physical document based on the at least one image.

13. The method of claim 12, further comprising displaying, by a display of the electronic device and based on the plurality of gaze positions, a highlight overlaid on a view of a portion of the identified text that the user has read.

14. The method of claim 13, wherein the view of the portion of the recognized text is a direct view through a semi-transparent or transparent display, and wherein displaying the highlighting includes: displaying, by the transparent or semi-transparent display, the highlight.

15. The method of claim 13, further comprising: displaying, by the display of the electronic device, the at least one image, and wherein displaying the highlight includes displaying the highlight overlaid on the displayed at least one image.

16. The method of claim 12, the method further comprising: detecting, based on the plurality of gaze positions, that the user is having difficulty reading the text; and in response to the detecting, providing a visual or audio enhancement to assist the user in reading the text.

17. The method of claim 16, wherein detecting that the user is having difficulty comprises: detecting a strabismus of the at least one eye of the user.

18. The method of claim 12, the method further comprising: identifying a sound associated with a word; determining, based on the plurality of gaze positions, that the user is reading the word; and outputting the identified sound.

19. An electronic device, comprising: a display; an eye tracker configured to provide eye tracking data of at least one eye of a user; and processing circuitry configured to: provide, for display by the display, a request for a user authorization from the user, wherein the displayed request includes text associated with the user authorization, and the request for a user authorization includes a request for a user authorization for an electronic device to perform an action, the action including at least one of a request for a user authorization to delete data from the electronic device, a request for a user authorization to allow access to data stored by the electronic device, or a request for a user authorization to allow access to a component of the electronic device; obtain, while displaying the request, a plurality of time-separated gaze positions based on the eye tracking data from the eye tracker; determine, based on the plurality of time-separated gaze positions relative to a location of the text in the displayed request, whether the user has viewed the displayed request for authorization, wherein the determining is based at least in part on determining whether the plurality of gaze positions correspond to reading eye movements; if it is determined that the user has viewed the displayed request for authorization, provide a selectable option for providing the user authorization; and in response to selecting the selectable option, perform the action.

Images