Algorithmic adjustment of icon hitboxes based on pre-gaze and click information

Abstract

1. A method comprising: determining historical user data relating to events occurring on a wearable device; determining a probability of interacting with an object on a display of the wearable device based on the historical user data; scaling a hit box associated with the object to form a scaled hit box; detecting user input based on eye tracking within the scaled hit box; and in response to detecting the user input, initiating an action corresponding to the object.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of and claims priority to U.S. Non-Provisional Patent Application No. 17 / 662,175, entitled "ALGORITHMICALLY ADJUSTING THE HIT BOX OF ICONS BASED ON PRIOR GAZE AND CLICK INFORMATION," filed on May 5, 2022, the disclosure of which is incorporated by reference in its entirety herein.

[0002] Embodiments relate to a user input interface (e.g., a pointing device) in a wearable device that includes a display(s). [Background technology]

[0003] Head-mounted computing devices (e.g., smart glasses) may be configured with various sensors that enable augmented reality (AR), where virtual elements are presented along with real elements of the environment. The virtual elements may be presented on a head-up display. The virtual elements may or may not be displayed as if they are located in the real world (e.g., a system menu). The head-up display may be implemented with a device similar to glasses (i.e., AR glasses). Summary of the Invention

[0004] In a general aspect, a wearable device, system, non-transitory computer-readable medium (storing computer-executable program code that can be executed on a computer system), and / or method can perform a process in a manner that includes determining historical user data related to events occurring on the wearable device, determining a probability of interacting with an object on a display of the wearable device based on the historical user data, scaling a hit box associated with the object to form a scaled hit box, detecting user input based on eye tracking within the scaled hit box, and in response to detecting the user input, initiating an action corresponding to the object.

[0005] Implementations include one or more of the following features. For example, the historical user data may include a single frame of user interaction with an object on a display of the wearable device, and determining the probability of interacting with the object may be based on a histogram of the historical user data. The historical user data may include multiple frames of user interaction with an object on a display of the wearable device, and the historical user data may include a time history, and determining the probability of interacting with the object may be based on a prior distribution and the time history of the object interactions at each frame. The historical user data may include multiple frames of user interaction with an object on a display of the wearable device, and the historical user data may include a time history, and determining the probability of interacting with the object may be based on a joint prior distribution and the time history of the object interactions at each frame. The historical user data may include a focus tendency, and scaling of a hit box associated with the object may be based on the focus tendency, and the hit box may be non-uniformly scaled based on the focus tendency.

[0006] Determining the probability of interacting with the object on the display of the wearable device may include determining a first probability of selecting the first object. The method further includes determining a second probability of interacting with a second object on the display of the wearable device based on the historical user data and the selection of the first object, scaling a second hit box associated with the second object, detecting a second user input based on eye tracking within the scaled second hit box, and in response to detecting the user input, initiating a second action corresponding to the second object. The eye tracking may include determining Cartesian coordinates on the display of the wearable device, and the Cartesian coordinates may be temporally filtered. The method may further include storing historical user data associated with the selected object on the display of the wearable device.

[0007] Exemplary embodiments will become more fully understood from the detailed description given herein below and the accompanying drawings, in which like elements are represented with like reference numerals, and which are given by way of example only and therefore not as limitations of the exemplary embodiments. [Brief description of the drawings]

[0008] [Figure 1A] 1 illustrates a diagram of a head-up display according to an exemplary embodiment. [Figure 1B] 1 illustrates a diagram of a head-up display according to an exemplary embodiment. [Figure 1C] 1 illustrates a diagram of a head-up display according to an exemplary embodiment. [Figure 1D] 1 illustrates a diagram of a head-up display according to an exemplary embodiment. [Diagram 2] 1 illustrates a head-up display according to an exemplary embodiment. [Diagram 3]FIG. 1 illustrates a block diagram of successive layers in a head-up display in accordance with an exemplary implementation. [Figure 4] 1 illustrates a head-up display according to an exemplary embodiment. [Diagram 5] 1 illustrates a perspective view of AR glasses, according to an exemplary embodiment. [Figure 6] 1 illustrates a method for implementing a user interface on a head-mounted computing device, according to an exemplary implementation. [Figure 7] FIG. 1 illustrates a block diagram of a system compatible with a head-mounted computing device, according to an exemplary implementation. [Figure 8] 1 illustrates an example of a computing device and a mobile computing device in accordance with at least one exemplary embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] It should be noted that these figures are intended to illustrate general features of methods, structures, and / or materials utilized in certain exemplary embodiments and to supplement the written description provided below. However, these figures are not to scale, may not precisely reflect the precise structural or performance features of any given embodiment, and should not be construed as defining or limiting the range of values ​​or properties encompassed by the exemplary embodiments. For example, the relative thicknesses and positioning of molecules, layers, regions, and / or structural elements may be reduced or exaggerated for clarity. The use of similar or identical reference numbers in various figures is intended to indicate the presence of similar or identical elements or features.

[0010] The input routine of a wearable device can be enabled by an eye tracking (ET) module and a wristband (WB). This combination can operate similarly to a mouse interface, where gaze estimated by ET can act as pointing / scrolling, and finger pinch gestures indirectly detected by WB can act as clicks for a physical mouse. However, unlike a mouse, which is a physical interface, the ET+WB interaction framework can rely on perception and data-driven algorithms. Therefore, the ET+WB interaction framework can function with some degree of error. For example, the error of the ET module can be quantified using an angle deviation metric (e.g., the target is (x=30,y=40) degrees, and the ET module can predict (x=31,y=45)). For example, the error of the WB module can be quantified using a binary detection metric (e.g., precision-recall).

[0011] Exemplary implementations can use historical user data to predict likely user inputs and resize generated hit boxes (e.g., background hit detection elements associated with rendered icons, rendered buttons, rendered menu items, etc.) such that the hit box associated with a likely user input is larger than any other hit box(es). Exemplary implementations can use the described techniques to reduce errors associated with ET modules. A hit box may be an area of ​​virtual or real space associated with a (virtual) detection element that triggers an action. For example, if a user's line of sight is determined to intersect with the hit box, that action may be triggered. The hit box area may be defined by coordinates in real and / or virtual space.

[0012] 1A, 1B, 1C, and 1D show diagrams of a head-up display according to an exemplary embodiment. Referring to FIG. 1A, a plurality of icons 110 (hereinafter referred to as icons 110) can be rendered on a display 105. The display 105 can be associated with a wearable device. For example, the display 105 can be a head-up display associated with a head-mounted display (e.g., smart glasses). For example, the display 105 can be a head-up display associated with AR glasses 500 described below. Hereinafter, a device including the display 105 is referred to as a wearable device (which can include a head-mounted display, smart glasses, AR glasses, etc.).

[0013] The icons 110 may be user interfaces (UIs) that cause actions to be performed by the wearable device. The actions may, for example, render items including information, UI, home screen, previous item, menu, settings, etc. The actions may cause the wearable device to power on / off, play / pause / stop audio, play / pause / stop video, interact with an assistant, etc. In an exemplary implementation, the ET+WB interaction framework may be used by a user of the wearable device to interact with (e.g., select) the icons 110.

[0014] 1B, each icon 110 has an associated hit box 115. The hit box 115 is not visible to the user of the wearable device. In other words, the hit box 115 is not rendered on the display 105. Each hit box 115 can be associated with some pixels of the display 105. Each hit box 115 can be associated with a pixel region of the display 105. Each pixel of the display 105 and the pixel region of the display 105 can be associated with a Cartesian (x,y) coordinate of the display 105. In an exemplary implementation, a user of the wearable device focuses on the icon 110, and the ET module can be configured to determine (e.g., using eye tracking functionality) a Cartesian coordinate on the display 105 associated with the view of the user of the wearable device. If the Cartesian coordinate associated with the view of the user of the wearable device is within the pixel region of the display 105 associated with the hit box 115 of the icon 110, the icon 110 is interacted with (e.g., in response to a finger pinch gesture detected by the WB).

[0015] Referring to Fig. 1C, as one example, an incoming call may be received by the wearable device. The user may be notified of the incoming call by a rendered message 120 (e.g., incoming call). Referring to Fig. 1D, as one example, in response to the incoming call, an icon 125 and an icon 130 may be rendered on the display 105. By interacting with icon 125, the incoming call may be rejected, and by interacting with icon 130, the incoming call may be answered.

[0016] Hitbox 135 and hitbox 140 identify pixel regions of display 105 where the user is looking (e.g., using the eye tracking functionality of the ET module) to interact with the respective icon 125, 130. However, as noted above, due to errors associated with the ET module, interactions with the respective icon 125, 130 may not be detected by the wearable device. Thus, the action associated with the respective icon 125, 130 may not be initiated. In this example, the call is not answered or rejected.

[0017] In an exemplary implementation, the hit box 145 identifies a pixel region of the display 105 associated with the icon 130. The hit box 145 may be generated based on historical user data. For example, the historical user data may indicate that a user of the wearable device is likely to answer a call from, for example, Jane Doe. Thus, the hit box 145 is generated to include a larger region of pixels of the display 105 compared to the hit boxes 135, 140. In other words, the user of the wearable device is predicted to be more likely to answer an incoming call by interacting with the icon 130. Thus, the hit box 145 is generated to include a larger region of pixels of the display 105 such that errors associated with the ET module may be minimized, resulting in a higher likelihood of an interaction with the icon 130 causing a corresponding action (e.g., answering a call) to be initiated by the wearable device (or a companion device). As described above, the hit boxes 135, 140, 145 are invisible to the user of the wearable device. In other words, the hit boxes 135 , 140 , 145 are not rendered on the display 105 .

[0018] FIG. 2 illustrates a head-up display according to an exemplary implementation. FIG. 2 illustrates a pictorial representation of head-up displays 205-1, 205-2. Head-up displays 205-1, 205-2 include icons 210. Each icon 210 has an associated hit box 215. As noted above, the hit boxes 215 are not visible to a user of the wearable device. Head-up displays 205-1, 205-2 may include other rendered information. Head-up displays 205-1, 205-2 display the time, date, and weather. However, other displayable information is within the scope of this disclosure. Head-up displays 205-1, 205-2 may be in a first state (e.g., a home screen).

[0019] In this example, a user of the wearable device may want to switch the heads-up displays 205-1, 205-2 to a second state. For example, the user of the wearable device may want to view a calendar. Thus, the user of the wearable device may focus on a calendar icon (e.g., the leftmost icon 210), and the ET module may be configured to determine (e.g., using eye tracking functionality) a Cartesian coordinate 220 on the heads-up display 205-1 that is associated with the view of the user of the wearable device. In this example, the Cartesian coordinate 220 is shown as outside the pixel region of the heads-up display 205-1 that corresponds to the hit box 215 of the calendar icon (e.g., the leftmost icon 210). Thus, an action corresponding to the calendar icon (e.g., the leftmost icon 210) is not initiated, and the heads-up display 205-1 does not switch to the second state (e.g., open the calendar UI).

[0020] In an exemplary implementation, as shown on the heads-up display 205-2, a hit box 225 of a calendar icon (e.g., the leftmost icon 210) is generated to include a larger area of ​​pixels on the heads-up display 205-2 as compared to the hit box 215. Thus, when a user of the wearable device focuses on the calendar icon (e.g., the leftmost icon 210) and the ET module determines (e.g., using eye tracking functionality) a Cartesian coordinate 220 on the heads-up display 205-2 associated with the user's view of the wearable device, the Cartesian coordinate 220 is shown as being within a pixel area on the heads-up display 205-2 that corresponds to the hit box 225 of the calendar icon (e.g., the leftmost icon 210). Thus, an action corresponding to the calendar icon (e.g., the leftmost icon 210) is initiated and the heads-up display 205-2 switches to a second state (e.g., opening the calendar UI).

[0021] Exemplary implementations can use algorithmic techniques to adjust the hit boxes of the wearable device UI to mask errors (as described above) associated with ET+WB uncertainty. For example, using the example described above with respect to Figures 1C-1D, an incoming call can be received and icons 125, 130 can be rendered on the display 105. The rendered icons 125, 130 can give the user two options to accept or reject the incoming call. The hit boxes 135, 140 each have a specific number of pixels allocated to the "accept" and "reject" icons.

[0022] In this example, if the hit boxes 135, 140 both occupy 100 pixels, then for an ET+WB combo model with 2 sigma error around the center of each icon 125, 130, there is a 95% chance that the user will press the correct button by gazing and clicking (assuming the ET error function is Gaussian and using a statistical rule of thumb sometimes referred to as the 68-95-99.7 rule).

[0023] However, in an exemplary implementation, the user's past call history with a particular caller indicates that the user has, for example, an 80% chance of receiving that call (e.g., a work call or a call from a family member). This prior history is used directly in generating hit boxes 145 in the UI (with the "accept" hit box being slightly larger than the "reject" hit box) to increase the posterior probability that the user will hit the intended icon, which in this case may have a net accuracy of, for example, greater than 95%. As noted above, the exemplary implementation does not change the actual visuals of the UI, only the hit boxes 145. Accepting or rejecting a call is only one example, and the exemplary implementation can be applied to shrinking or expanding the wearable device's display hit boxes for general menu selections, the user's likely travel route, shopping, lenses, etc. Another extension is that this does not necessarily have to be "frame-based"; prior information could also be gleaned from the end-to-end (e2e) flow described below (e.g., when the glasses start up, the user first checks the calendar and then books a meeting, so we might scale the size of the hitbox of the icon describing these two interaction points). Thus, the exemplary implementation can improve net accuracy by incorporating prior information (e.g., histograms) of the individualized routines.

[0024] Hit boxes (e.g., hit boxes 115, 135, 140, 215) may be scaled to increase the size of the pixel region associated with the hit box associated with an icon associated with a user input that is likely to be expected. The amount of hit box scaling can be described or derived as follows: · Imagine a fixed sys UI screen with N options. For the i-th icon, oc iis the total count of users who select icon i at the time of selection, oa i is the default hitbox size of icon i, Define. next, p i =c i / sum(c i ) is the prior probability p(where, by normalization, sum(p i )=1), the scaled hitbox area could then be:

[0025] a i '=a i *(1+p i r ) where r>1 is a "damping factor" that can control the relative strength of expanding or contracting the hitbox size based on the prior distribution. For example, if r is relatively large, hitbox renormalization becomes ineffective. An exemplary choice for r can be around 5.

[0026] FIG. 3 illustrates a block diagram of successive layers in a head-up display according to an exemplary embodiment. FIG. 3 illustrates an exemplary embodiment in which the user flow of historical user data and / or the current user flow is flow-based, or end-to-end (e2e) flow. As shown in FIG. 3, a home layer 305, a layer 1 310, and a layer 2 315 can represent flow frames or display states in a user flow. A flow frame or display state can be a UI representation in which the home layer 305 is a home screen, the layer 1 310 is a UI state after a first user interaction, and the layer 2 315 is a UI state after a second user interaction. A UI state can be a rendering of a wearable device's UI on a wearable device's display.

[0027] The objects 320-1, 320-2 are associated with the home layer 305. The objects 320-1, 320-2 may represent objects (e.g., icons, menus, etc.) rendered on a display of the wearable device. The objects 325-1, 325-2 are associated with the layer 1 310. The objects 325-1, 325-2 may represent objects (e.g., icons, menus, etc.) rendered on a display of the wearable device. The objects 330-1, 330-2 are associated with the layer 2 315. The objects 330-1, 330-2 may represent objects (e.g., icons, menus, etc.) rendered on a display of the wearable device. As described above, the flow frame or display state may be a UI representation in which the home layer 305 is a home screen, the layer 1 310 is a UI state after a first user interaction, and the layer 2 315 is a UI state after a second user interaction. A UI state may be a rendering of the UI of the wearable device on the display of the wearable device. Thus, objects 320-1, 320-2 may represent objects rendered on the UI home screen, objects 325-1, 325-2 may represent objects rendered on the UI after a first user interaction, and objects 330-1, 330-2 may represent objects rendered on the UI after a second user interaction. Lines 335-1, 335-2, 335-3 may represent an interaction flow in which object 320-1 is interacted with (e.g., selected) on the home layer 305, object 325-2 is interacted with (e.g., selected) on layer 1 310, and object 330-1 is interacted with (e.g., selected) on layer 2 315.

[0028] In this exemplary implementation, the user flow may represent historical user flow. In this example, it may be useful to estimate a prior distribution that includes the time history as follows:

[0029] p(t1,t2) ∝ p(t2) * p(t1|t2) Where: p is a probability (e.g., the probability of an object being interacted with), t is a timestamp.

[0030] In the two-layer example, knowing the joint prior corresponds to simply using the chain rule of probability where the individual elements are histograms, plugged in to be estimated from the user's historical data. The joint prior is then calculated using the Naive Bayes prior (=p i ) instead of scaled hitboxes for a cleaner result i ' can be estimated.

[0031] FIG. 4 illustrates a head-up display according to an exemplary embodiment. In an exemplary embodiment, the hit box may be non-uniformly scaled. For example, as shown in FIG. 4, head-up displays 205-2, 205-3 show that hit box 405 is not uniformly scaled compared to hit box 225. For example, hit box 405 is shown as extending in one direction (e.g., toward the top of head-up display 205-3). As an example, an average user may focus on the top of an icon 210 (e.g., a calendar icon) to click on. In this example, detecting this focus tendency, learning the prior probability of the icon, and scaling the area uniformly across the xy plane may not be optimal. The focus tendency may be a location, direction, position, object, etc. that the user consistently looks at. For example, the focus tendency may be a location, direction, position, etc. relative to a displayed object (e.g., icon 210). Thus, an exemplary embodiment may take user gaze deviation information into account in the historical data and the scaling algorithm. Non-uniform scaling can reduce errors, resulting in improved hit success rates.

[0032] FIG. 5 is a perspective view of AR glasses according to a possible embodiment of the present disclosure. The AR glasses 500 may be a wearable device configured to be attached to a user's head and face. The AR glasses 500 include a right ear temple 501 and a left ear temple 502 that are supported by the user's ears. The AR glasses further include a bridge portion 503 that is supported by the user's nose such that a left lens 504 and a right lens 505 can be placed in front of the user's left eye and the user's right eye, respectively. Collectively, the portions of the AR glasses may be referred to as a frame of the AR glasses 500. The frame of the AR glasses 500 may include electronics that enable a function(s). For example, the frame may include a battery, a processor, memory (e.g., a non-transitory computer-readable medium), and electronics that support sensors (e.g., a camera, a depth sensor, etc.), and interface devices (e.g., a speaker, a display, a network adapter, etc.). For example, the functions may include hit box generation and scaling as described above.

[0033] The AR glasses 500 may include a FOV camera 510 (e.g., an RGB camera) that is oriented with a camera field of view that overlaps with the natural field of view of the user's eyes when the glasses are worn. In possible implementations, the AR glasses may further include a depth sensor 511 (e.g., a LIDAR camera, a structured light camera, a time-of-flight camera, a depth camera) that is oriented with a depth sensor field of view that overlaps with the natural field of view of the user's eyes when the glasses are worn. Using data from the depth sensor 511 and / or the FOV camera 510, the depth within the field of view (i.e., the area of ​​interest) of the user (i.e., the wearer) may be measured. In possible implementations, the camera field of view and the depth sensor field of view may be calibrated to allow the depth (i.e., range) of an object in an image from the FOV camera 510 to be identified, and the depth is measured between the object and the AR glasses.

[0034] The AR glasses 500 may further include a display 515. The display may present AR data (e.g., images, graphics, text, icons, etc.) on a portion of the lens(es) of the AR glasses such that the user can view the AR data when looking through the lenses of the AR glasses. In this manner, the AR data may be overlaid with the user's view of the environment.

[0035] The AR glasses 500 may further include an eye tracking sensor. The eye tracking sensor may include a right eye camera 520 and a left eye camera 521. The right eye camera 520 and the left eye camera 521 may be positioned in the lens portion of the frame such that the right FOV 522 of the right eye camera includes the right eye of the user and the left FOV 523 of the left eye camera includes the left eye of the user when the AR glasses are worn. The viewpoint (x, y) may be determined at the frequency of the video feed of the camera (e.g., the right eye camera 520, the left eye camera 521). For example, the gaze coordinate (x, y) may be measured at or below the frame rate of the camera (e.g., 15 frames / second). The ET model may be configured to use the eye tracking sensor and / or the gaze coordinate (x, y) to predict and / or use as Cartesian coordinates (e.g., Cartesian coordinates 220).

[0036] The AR glasses 500 can be communicatively coupled to a peripheral device 530. The peripheral device 530 can be configured to detect and / or sense gestures associated with a user's hand 535. The peripheral device 530 can be, include, or perform the WB functions described above. For example, the peripheral device 530 can be configured to detect or help detect a finger pinch gesture used to initiate the user interaction described above (e.g., to select an icon). Other gestures or techniques may be used to initiate a user interaction.

[0037] The AR glasses 500 may further include a left speaker 541 and a right speaker 542 configured to transmit sound to the user. Additionally or alternatively, transmitting sound to the user may include transmitting sound to a listening device (e.g., hearing aids, earphones, etc.) via a wireless communication link 545. For example, the AR glasses 500 may transmit sound to a left wireless earphone 546 and a right earphone 547.

[0038] FIG. 6 illustrates a method of implementing a user interface on a head-mounted computing device according to an exemplary embodiment. As shown in FIG. 6, in step S605, historical user data associated with an event occurring on the wearable device is identified. For example, the historical user data can be stored in a memory, e.g., in a database, associated with the wearable device (e.g., the AR glasses 500) and / or a companion device (e.g., a mobile phone communicatively coupled to the wearable device). The historical user data can be associated with interacting with (e.g., selecting) an object rendered on a display (e.g., display 105, 205) of the wearable device. The historical user data can be associated with an event occurring on the wearable device (e.g., on a display of the wearable device). The historical user data can be stored in response to interacting with an object in response to an event.

[0039] Thus, an event may occur at the wearable device (e.g., an incoming call is received) and historical user data related to the event may be read from the memory. Determining the historical user data may include determining characteristics related to the event (e.g., the incoming call and its originator) and filtering the historical user data based on the characteristics related to the event. The historical user data may include a single frame of user interaction with an object on a display of the wearable device. In other words, a single frame of user interaction(s) may include user interaction with an object displayed on the wearable within a single frame of a multi-frame video used to display content including the object. Thus, the historical data may be associated with a single frame(s) in which the user interacted with the object.

[0040] The historical user data may include multiple frames of user interactions with an object on a display of the wearable device. In other words, the multiple frame user interaction(s) may include user interactions with an object displayed on the wearable within two or more frames (e.g., consecutive frames in a time frame) of a multi-frame video used to display content including the object. Thus, the historical data may be associated with multiple frames (two or more) in which the user interacted with the object. The historical user data may include a time history. The time history may include multiple frames of user interactions with the object, where each frame is time stamped. Identifying the historical user data may include recalling the relevant frames ordered based on the time stamps. Additionally, in some implementations, the historical user data may include a total number of user interactions with a particular hit box or a number per unit of time (e.g., day, week, etc.).

[0041] In step S610, a probability of interacting with an object on the display of the wearable device is determined based on historical user data. For example, a histogram of the historical user data can be generated. The histogram can include historical objects that have been interacted with based on events. Once the probability of interacting with an object on the display of the wearable device is determined, it can be based on the columns of the histogram. The histogram with the most entries can be the object with the highest probability of being interacted. The determination of the probability of interacting with the object can be based on a prior distribution and a time history of object interactions at each frame. The determination of the probability of interacting with the object can be based on a joint prior distribution and a time history of object interactions at each frame.

[0042] In step S615, a hit box associated with the object is scaled. For example, an object with the highest probability of being interacted with may have an associated hit box that is scaled. Scaling the hit box may include increasing a size of the hit box. The hit box may be associated with a number of pixels of a display of the wearable device. The hit box may be associated with a pixel region of a display of the wearable device. Each pixel of the display and the pixel region of the display may be associated with a Cartesian (x,y) coordinate of the display. Scaling the hit box may include increasing a number of pixels of the pixel region of the display that corresponds to the hit box. The historical user data may include focus tendencies, and the scaling of the hit box associated with the object may be based on the focus tendencies. The hit box may be non-uniformly scaled based on the focus tendencies.

[0043] In step S620, user input is detected based on eye tracking within the scaled hit box. For example, the user input can be detected using an ET+WB interaction framework. The eye tracking can include determining Cartesian coordinates on a display of the wearable device (e.g., using an ET module). The Cartesian coordinates can be filtered in time (e.g., across multiple frames). The wearable device can be communicatively coupled to a peripheral device. The peripheral device can be configured to detect and / or sense gestures associated with the user's hand. The peripheral device can be, include, or perform the WB functions described above. For example, the peripheral device can be configured to detect or help detect finger pinch gestures used to initiate user interaction. The detected gestures can be combined with eye tracking to generate user input. Other gestures or techniques may be used to initiate user interaction.

[0044] One possible method of eye tracking involves using a camera to measure visual indicators and determine eye position. In one possible implementation, the pupil position relative to a light pattern (near infrared light) projected onto the eye can be measured by analyzing high-resolution images of the eye and the pattern. The eye position can then be applied to a machine learning model to determine the Cartesian coordinates of the gaze point. Variations of this method that do not use a projected pattern are also possible. For example, there are standard glint-based tracking techniques or convolutional neural net techniques that can convert a two-dimensional (2D) infrared image captured by a camera pointed at the eye (or an image of the eye reflected from a mirror) into coordinates (x,y) within the field of view of the AR glasses.

[0045] Identifying gaze can be difficult due to rapid eye movements (i.e., saccades). Therefore, the method further includes temporally filtering the gaze coordinates. For example, gaze coordinates obtained from real-time eye tracking can be low-pass filtered to generate a time-varying signal corresponding to gaze with less change over time. In a possible implementation, eye tracking coordinates can be measured and averaged over time to obtain an average eye tracking coordinate. When the average eye tracking coordinate meets a dwell time criterion, the user's gaze direction can be identified. For example, average eye tracking coordinates that stay within a range (e.g., area) for longer than a threshold time can indicate a stable gaze.

[0046] In step S625, in response to detecting a user input, an action corresponding to the object is initiated. For example, the object can be an icon and the action can be initiated in response to interacting with (e.g., selecting) the icon. For example, an incoming call can be accepted or rejected based on an interaction with the icon. For example, the object can be a menu and the action can be initiated in response to interacting with (e.g., selecting) an icon in the menu.

[0047] In some implementations (e.g., involving multiple frames of user interaction with an object), multiple objects are interacted for a short period of time. Below is an example where two objects are interacted. Determining the probability of selecting an object on the display of the wearable device can be determining a first probability of selecting a first object. Then, a second probability of interacting with a second object on the display of the wearable device can be determined based on the historical user data and the selection of the first object. A second hit box associated with the second object can be scaled. A second user input can be detected based on eye tracking within the scaled second hit box, and in response to detecting the user input, a second action corresponding to the second object can be initiated.

[0048] FIG. 7 illustrates a block diagram of a system that corresponds to and / or includes a wearable device (e.g., a wearable computing device, a head-worn display, smart glasses, AR glasses, a head-mounted display, etc.) according to an exemplary embodiment. In the example of FIG. 7, it should be understood that the system (e.g., an augmented reality system, a virtual reality system, a companion device, etc.) may include a computing system or at least one computing device, and represents virtually any computing device configured to implement the techniques described herein. Thus, the device may be understood to include various components that may be utilized to implement the techniques described herein, or different or future versions thereof. By way of example, the system may include a processor 705 and a memory 710 (e.g., a non-transitory computer-readable memory). The processor 705 and the memory 710 may be coupled (e.g., communicatively coupled) by a bus 715.

[0049] The processor 705 may be utilized to execute instructions stored in the at least one memory 710. As such, the processor 705 may perform various features and functions described herein or additional or alternative features and functions. The processor 705 and the at least one memory 710 may be utilized for a variety of other purposes. For example, the at least one memory 710 may represent examples of various types of memory and associated hardware and software that may be used to implement any one of the modules described herein.

[0050] The at least one memory 710 may be configured to store data and / or information related to the device. The at least one memory 710 may be a shared resource. Thus, the at least one memory 710 may be configured to store data and / or information related to other elements in a larger system (e.g., image / video processing or wired / wireless communication). The processor 705 and the at least one memory 710 may be utilized together to perform the techniques described herein. Thus, the techniques described herein may be implemented as code segments (e.g., software) stored on the memory 710 and executed by the processor 705. Thus, the memory 710 may include an ET module 720, a WB module 725, and a scaling module 730.

[0051] The ET module 720 can be configured to determine Cartesian or gaze coordinates associated with the line of sight of a user of the wearable device. In an exemplary implementation, the user of the wearable device focuses on an object (e.g., an icon), and the ET module 720 can be configured to determine (e.g., using eye tracking functionality) Cartesian coordinates on a display associated with the view of the user of the wearable device. The ET module 720 can be communicatively coupled to an eye tracking sensor. The eye tracking sensor can include a right-eye camera and a left-eye camera. The right-eye camera and the left-eye camera can be positioned in a lens portion of the frame such that, when the AR glasses are worn, the right FOV of the right-eye camera includes the user's right eye, and the left FOV of the left-eye camera includes the user's left eye. The ET module 720 can be configured to determine the Cartesian or gaze coordinates at the frequency of the camera (e.g., right-eye camera, left-eye camera) video feed. For example, the gaze coordinates (x, y) can be measured at or below the frame rate of the camera (e.g., 15 frames / second). The ET module 720 can be configured to use an eye tracking sensor and / or gaze coordinates (x,y) to make predictions and / or to use as Cartesian coordinates (eg, Cartesian coordinates 220).

[0052] The WB module 725 can be configured to receive a communication from a peripheral device and trigger an event based on the communication. For example, the event can be a "click" event associated with a pointing device. The peripheral device can be configured to detect and / or sense gestures associated with a user's hand. The WB module 725 in conjunction with the peripheral device can be configured to detect, or help detect, a finger pinch gesture used to initiate the above user interactions (e.g., selecting an icon). Other gestures or techniques may be used to initiate a user interaction.

[0053] The scaling module 730 can be configured to scale (e.g., increase) a size of a hit box associated with a user interface based on historical user data. For example, the hit box can be associated with a number of pixels of a display of the wearable device. The hit box can be associated with a pixel region of the display of the wearable device. Each pixel of the display and the pixel region of the display can be associated with a Cartesian (x,y) coordinate of the display. Scaling the hit box can include increasing the number of pixels of the pixel region of the display corresponding to the hit box.

[0054] 8 illustrates an example of a computing device 800 and a mobile computing device 850 that may be used with the described techniques. The computing device 800 includes a processor 802, a memory 804, a storage device 806, a high-speed interface 808 that connects to the memory 804 and a high-speed expansion port 810, and a low-speed interface 812 that connects to a low-speed bus 814 and the storage device 806. Each of the components 802, 804, 806, 808, 810, and 812 are interconnected using various buses and may be mounted on a common motherboard or in other manners as desired. The processor 802 can process instructions for execution within the computing device 800, including instructions stored in the memory 804 or the storage device 806, to display graphical information for a GUI on an external input / output device, such as a display 816 connected to the high-speed interface 808. In other implementations, multiple processors and / or multiple buses may be used as needed, along with multiple memories and memory types. Additionally, multiple computing devices 800 may be connected together (eg, as a bank of servers, a group of blade servers, or a multi-processor system) with each device providing a portion of the required operations.

[0055] The memory 804 stores information within the computing device 800. In one implementation, the memory 804 is a volatile memory unit(s). In another implementation, the memory 804 is a non-volatile memory unit(s). The memory 804 may also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0056] The storage device 806 can provide mass storage for the computing device 800. In one embodiment, the storage device 806 can be or include a computer-readable medium such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices including a storage area network or other configuration of devices. The computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 804, the storage device 806, or a memory on the processor 802.

[0057] The high-speed controller 808 manages bandwidth-intensive operations of the computing device 800, and the low-speed controller 812 manages lower bandwidth-intensive operations. Such an allocation of functions is merely an example. In one implementation, the high-speed controller 808 is coupled to the memory 804 (e.g., via a graphics processor or accelerator), the display 816, and is coupled to a high-speed expansion port 810 that may accept various expansion cards (not shown). In this implementation, the low-speed controller 812 is coupled to the storage device 806 and the low-speed expansion port 814. The low-speed expansion port may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet). The low-speed expansion port may be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, or a network device such as a switch or router, for example, via a network adapter.

[0058] The computing device 800 may be implemented in many different forms, as shown. For example, it may be implemented as a standard server 820, or multiple times in a group of such servers. It may also be implemented as part of a rack server system 824. Additionally, it may be implemented in a personal computer, such as a laptop computer 822. Alternatively, the components of the computing device 800 may be combined with other components in a mobile device (not shown), such as device 850. Each such device may include one or more of the computing devices 800, 850, and the entire system may be made up of multiple computing devices 800, 850 in communication with each other.

[0059] Computing device 850 includes, among other components, a processor 852, memory 864, input / output devices such as a display 854, a communications interface 866, and a transceiver 868. Device 850 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of components 850, 852, 864, 854, 866, and 868 are interconnected using various buses, and some of the components may be mounted on a common motherboard or in other manners as desired.

[0060] The processor 852 can execute instructions within the computing device 850, including instructions stored in the memory 864. The processor may be implemented as a chipset of chips including separate analog and digital processors. The processor may provide coordination for other components of the device 850, such as control of a user interface, applications run by the device 850, and wireless communications by the device 850.

[0061] The processor 852 may communicate with a user via a control interface 858 and a display interface 856 coupled to a display 854. The display 854 may be, for example, a thin film transistor liquid crystal display (TFT LCD) and a light emitting diode (LED) or organic light emitting diode (OLED) display, or other suitable display technology. The display interface 856 may include suitable circuitry for driving the display 854 to present graphical and other information to the user. The control interface 858 may receive commands from the user and translate the commands for submission to the processor 852. Additionally, an external interface 862 may be provided in communication with the processor 852 to enable short-range communication between the device 850 and other devices. The external interface 862 may provide, for example, wired communication in some implementations or wireless communication in other implementations, and multiple interfaces may be used.

[0062] The memory 864 stores information within the computing device 850. The memory 864 can be implemented as one or more of a computer-readable medium(s), a volatile memory unit(s), or a non-volatile memory unit(s). An expansion memory 874 may also be provided to connect to the device 850 via an expansion interface 872, which may include, for example, a SIMM (single in-line memory module) card interface. Such expansion memory 874 may provide additional storage space for the device 850 or may store applications or other information for the device 850. In particular, the expansion memory 874 may include instructions for performing or supplementing the above-mentioned processes, and may also include secure information. Thus, for example, the expansion memory 874 may be provided as a security module for the device 850 and may be programmed with instructions that enable secure use of the device 850. Furthermore, secure applications may be provided via a SIMM card along with additional information, such as placing identifying information on the SIMM card in an unhackable manner.

[0063] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one embodiment, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, examples of which include memory 864, expansion memory 874, or memory on processor 852. The information carrier may be received, for example, via transceiver 868 or external interface 862.

[0064] The device 850 may communicate wirelessly via a communication interface 866, which may include digital signal processing circuitry if necessary. The communication interface 866 may provide for communication in a variety of modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, among others. Such communication may occur, for example, via a radio frequency transceiver 868. Further, short-range communication may occur, such as using Bluetooth, Wi-Fi, or other such transceivers (not shown). Additionally, a GPS (Global Positioning System) receiver module 870 may provide additional navigation and location related wireless data to the device 850, which may be used as needed by applications executing on the device 850.

[0065] Device 850 may also perform voice communications using an audio codec 860, which may receive voice information from a user and convert it into usable digital information. Audio codec 860 may also generate sounds that are audible to the user, such as through a speaker (e.g., in a handset of device 850). Such sounds may include sounds from voice telephone calls, recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications running on device 850.

[0066] Computing device 850 may be implemented in many different forms, as shown in the figure, such as a mobile phone 880, or as part of a smartphone 882, personal digital assistant, or other similar mobile device.

[0067] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, which may be special-purpose or general-purpose, coupled to receive data and instructions from the storage system, and to transmit data and instructions to the storage system.

[0068] These computer programs (also known as programs, software, software applications, or code) contain machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic circuits (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0069] To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (such as a light emitting diode (LED) or organic LED (OLED) or liquid crystal display (LCD) monitor / screen) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices may also be used to provide interaction with a user, for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and input from the user may be received in any form, including acoustic, speech, or tactile input.

[0070] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., a data server), or that includes middleware components (e.g., an application server), or that includes a front-end component (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0071] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0072] In some implementations, the computing device shown in the figure may include sensors that interface with the AR headset / HMD device 890 to generate an augmented environment for viewing content inserted in a physical space. For example, one or more sensors included in the computing device 850 shown in the figure or other computing devices may provide input to the AR headset 890 or generally provide input to the AR space. The sensors may include, but are not limited to, a touch screen, an accelerometer, a gyroscope, a pressure sensor, a biometric sensor, a temperature sensor, a humidity sensor, and an ambient light sensor. The computing device 850 may use the sensors to determine the absolute position and / or detected rotation of the computing device in the AR space, which may then be used as input to the AR space. For example, the computing device 850 may be incorporated into the AR space as a virtual object, such as a controller, a laser pointer, a keyboard, a weapon, etc. When incorporated into the AR space, the positioning of the computing device / virtual object by the user allows the user to position the computing device to view the virtual object in a particular way in the AR space. For example, if the virtual object represents a laser pointer, the user may manipulate the computing device as if it were an actual laser pointer. A user can move a computing device left and right, up and down, in a circular direction, etc., to use the device in a manner similar to using a laser pointer. In some implementations, a user can use a virtual laser pointer to point to a target location.

[0073] In some implementations, one or more input devices included in or connected to the computing device 850 can be used as input to the AR space. The input devices can include, but are not limited to, a touch screen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, an earphone or earbuds with input capabilities, a game controller, or other connectable input devices. A user can interact with the input devices included in the computing device 850 when the computing device is embedded in the AR space to cause certain actions to occur in the AR space.

[0074] In some implementations, the touch screen of the computing device 850 can be rendered as a touch pad in the AR space. A user can interact with the touch screen of the computing device 850. The interaction is rendered, for example in the AR headset 890, as a movement on the rendered touch pad in the AR space. The rendered movement can control a virtual object in the AR space.

[0075] In some implementations, one or more output devices included in the computing device 850 can provide output and / or feedback to a user of the AR headset 890 within the AR space. The output and feedback can be visual, tactical, or audio. The output and / or feedback can include, but is not limited to, vibration, turning one or more lights or strobes on and off, or blinking and / or flashing, sounding an alarm, chiming, playing a song, and playing an audio file. The output devices can include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light emitting diodes (LEDs), strobes, and speakers.

[0076] In some implementations, the computing device 850 may be displayed as another object in the computer-generated 3D environment. A user's interaction with the computing device 850 (e.g., rotating, shaking, touching the touch screen, swiping a finger on the touch screen) may be interpreted as an interaction with an object in the AR space. In the example of a laser pointer in the AR space, the computing device 850 appears as a virtual laser pointer in the computer-generated 3D environment. As the user manipulates the computing device 850, the user in the AR space sees the movement of the laser pointer. The user receives feedback from the interaction with the computing device 850 in the AR environment on the computing device 850 or the AR headset 890. The user's interaction with the computing device may be translated into an interaction with a user interface generated in the AR environment for the controllable device.

[0077] In some implementations, the computing device 850 may include a touch screen. For example, a user may interact with the touch screen to interact with a user interface of the controllable device. For example, the touch screen may include user interface elements, such as sliders, that can control characteristics of the controllable device.

[0078] Computing device 800 is intended to represent various types of digital computers and devices, including, but not limited to, laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 850 is intended to represent various types of mobile devices, such as personal digital assistants, mobile phones, smart phones, and other similar computing devices. The components illustrated herein, their connections and relationships, and their functions are intended to be illustrative only and are not intended to limit the implementation of the invention described and / or claimed herein.

[0079] A number of embodiments have been described. Of course, it will be understood that various modifications can be made without departing from the spirit and scope of the present disclosure.

[0080] Moreover, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desired results. Moreover, other steps may be provided in or removed from the described flows, and other components may be added to or removed from the described systems. Accordingly, other embodiments are within the scope of the following claims.

[0081] In addition to the above, the system, program, or functionality described herein may provide controls to the user that allow the user to select both whether and when the system, program, or functionality described herein may enable collection of user information (e.g., information regarding the user's social network, social actions, or activities, occupation, user preferences, or the user's current location) and whether to send content or communications from the server to the user. Additionally, certain data may be processed in one or more ways such that personally identifiable information is removed before it is stored or used. For example, the user's identity may be processed such that the user's personally identifiable information cannot be determined, or if location information is obtained (such as to the city, zip code, or state level), the user's geographic location may be generalized such that the user's specific location cannot be determined. Thus, the user may control what information is collected about the user, how that information is used, and what information is provided to the user.

[0082] As described herein, while certain features of the described embodiments have been illustrated, numerous modifications, substitutions, changes, and equivalents will occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and changes that fall within the scope of the embodiments. They are presented only by way of example, not limitation, and it should be understood that various changes in form and details may be made. Any part of the apparatus and / or method described herein may be combined in any combination, except in mutually exclusive combinations. The embodiments described herein may include various combinations and / or subcombinations of the functions, components, and / or features of the different embodiments described.

[0083] While exemplary embodiments may include various modifications and alternative forms, embodiments thereof have been shown by way of example in the drawings and will be described in detail herein. It should be understood, however, that there is no intention to limit the exemplary embodiments to any particular form, but rather to cover all modifications, equivalents, and alternatives falling within the scope of the claims. Like numerals refer to like components throughout the description of the drawings.

[0084] Some of the above exemplary embodiments are described as a process or method that is depicted as a flowchart. Although the flowcharts describe operations as sequential processes, many of the operations may occur in parallel, simultaneously, or simultaneously. Also, the order of operations may be rearranged. A process may be terminated when its operations are completed, or may have additional steps not included in the drawings. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0085] The above methods, some of which are illustrated by flowcharts, may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine- or computer-readable medium, such as a storage medium. A processor(s) may perform the necessary tasks.

[0086] Specific structural and functional details disclosed herein are merely representative for purposes of describing example embodiments, however, example embodiments may be embodied in many alternative forms and should not be construed as being limited to only the embodiments set forth herein.

[0087] It is also understood that the terms "first", "second", etc. may be used herein to describe various elements, but these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a "first element" may be referred to as a "second element", and similarly, a "second element" may be referred to as a "first element" without departing from the scope of the exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0088] When an element is referred to as being "connected" or "coupled" to another element, it will be understood that the element may be directly connected or coupled to the other element, or there may be intervening elements. In contrast, when an element is referred to as being "directly connected" or "directly coupled" to another element, there are no intervening elements. Other words used to indicate the relationship between elements should be interpreted similarly (e.g., between versus directly between, adjacent versus directly adjacent, etc.).

[0089] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It is further understood that the terms "comprises," "comprising," "includes," and / or "including," when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0090] It should also be noted that in some alternative implementations, the functions / acts shown may occur out of the order noted in the figures. For example, two figures shown in succession may in fact be executed concurrently or may sometimes be executed in the reverse order, depending on the functions / acts involved.

[0091] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which the exemplary embodiments belong. Furthermore, it will be understood that terms (e.g., those defined in commonly used dictionaries) should be interpreted to have a meaning consistent with their meaning in the context of the relevant art, and not in an idealized or overly formal sense unless expressly defined herein.

[0092] Portions of the above exemplary embodiments and corresponding detailed description are presented in terms of software, or algorithms, and symbolic representations of operations on data bits within a computer memory. These descriptions and representations are the ones by which those skilled in the art effectively convey the substance of their work to others skilled in the art. An algorithm, as the term is used herein, and as it is used generally, is conceived to be a self-consistent sequence of steps leading to a desired result. These steps require physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of optical, electrical, or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It proves convenient at times, primarily for reasons of common usage, to refer to these symbols as "bits," "values," "elements," "symbols," "characters," "terms," ​​"numbers," or the like.

[0093] In the above exemplary embodiments, references to symbolic representations of acts and operations that may be implemented as program modules or functional processes (e.g., in the form of a flowchart) include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types, and may be described and / or implemented using existing hardware in existing structural elements. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application specific integrated circuits, field programmable gate array (FPGA) computers, etc.

[0094] It should be recognized, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise noted, or as will be apparent from the discussion, terms such as processing or calculating or computing or determining or displaying refer to the actions and processes of a computer system or similar electronic computing device that manipulate and convert data, which are represented as physical, electronic quantities in the computer system's registers and memory, into other data, which are also represented as physical quantities in the computer system's memory or registers, or other such information storage, transmission, or display device.

[0095] It should also be noted that the software-implemented aspects of the exemplary embodiments are typically encoded on some form of non-transitory program storage medium or implemented via some type of transmission medium. The program storage medium may be magnetic (e.g., floppy disk or hard drive) or optical (e.g., compact disk read-only memory, or CD ROM), and may be read-only or random access. Similarly, the transmission medium may be twisted wire pairs, coaxial cable, optical fiber, or some other suitable transmission medium known in the art. The exemplary embodiments are not limited by these aspects of any given implementation.

[0096] Finally, it should also be noted that although the appended claims set forth particular combinations of features described in this specification, the scope of the disclosure is broadened to encompass any combination of features or embodiments disclosed herein, and is not limited to the specific combinations claimed below, regardless of whether that particular combination is specifically recited in the appended claims at this point.

Claims

1. 1. A method comprising: determining historical user data relating to events occurring at the wearable device; determining a probability of interacting with an object on a display of the wearable device based on the historical user data; scaling a hit box associated with the object to form a scaled hit box; Detecting user input based on eye tracking within the scaled hit box; and in response to detecting the user input, initiating an action corresponding to the object.

2. the historical user data includes a single frame of user interaction with an object on the display of the wearable device; The method of claim 1 , wherein the determining the probability of interacting with the object is based on a histogram of the historical user data.

3. the historical user data includes a plurality of frames of user interactions with objects on the display of the wearable device; the historical user data includes a time history; The method of claim 1 or 2, wherein the determining the probability of interacting with the object is based on a prior distribution and the time history of object interactions at each frame.

4. the historical user data includes a plurality of frames of user interactions with objects on the display of the wearable device; the historical user data includes a time history; 2. The method of any preceding claim, wherein the determining the probability of interacting with the object is based on a joint prior distribution of object interactions at each frame and the time history.

5. the historical user data includes focus tendencies; the scaling of the hit box relative to the object is based on the focus tendency; 2. The method of any preceding claim, wherein the hit boxes are non-uniformly scaled based on the focus tendency.

6. determining the probability of interacting with the object on the display of the wearable device includes determining a first probability of selecting a first object; determining a second probability of interacting with a second object on a display of the wearable device based on the historical user data and the selection of the first object; Scaling a second hit box associated with the second object; and Detecting a second user input based on eye tracking within the scaled second hit box; The method of any of claims 1 to 5, further comprising initiating a second action corresponding to the second object in response to detecting the user input.

7. the eye tracking includes determining Cartesian coordinates on the display of the wearable device; The method according to any of claims 1 to 6, wherein the Cartesian coordinates are temporally filtered.

8. The method of any preceding claim, further comprising storing historical user data associated with a selected object on the display of the wearable device.

9. A wearable device, At least one processor; at least one memory containing computer program code; The at least one memory and the computer program code, together with the at least one processor, configure the wearable device to: determining historical user data relating to events occurring at the wearable device; determining a probability of interacting with an object on a display of the wearable device based on the historical user data; scaling a hit box associated with the object to form a scaled hit box; Detecting user input based on eye tracking within the scaled hit box; and in response to detecting the user input, initiating an action corresponding to the object.

10. the historical user data includes a single frame of user interaction with an object on the display of the wearable device; The wearable device of claim 9 , wherein the determining the probability of interacting with the object is based on a histogram of the historical user data.

11. the historical user data includes a plurality of frames of user interactions with objects on the display of the wearable device; the historical user data includes a time history; The wearable device of claim 9 or 10, wherein the determining the probability of interacting with the object is based on a prior distribution of object interactions at each frame and the time history.

12. the historical user data includes a plurality of frames of user interactions with objects on the display of the wearable device; the historical user data includes a time history; A wearable device according to any one of claims 9 to 11, wherein the determining the probability of interacting with the object is based on a joint prior distribution of object interactions at each frame and the time history.

13. the historical user data includes focus tendencies; the scaling of the hit box relative to the object is based on the focus tendency; A wearable device according to any one of claims 9 to 12, wherein the hit box is non-uniformly scaled based on the focus tendency.

14. The determining the probability of interacting with the object on the display of the wearable device includes determining a first probability of selecting a first object, and the computer program code further comprises: determining a second probability of interacting with a second object on a display of the wearable device based on the historical user data and the selection of the first object; Scaling a second hit box associated with the second object; and Detecting a second user input based on eye tracking within the scaled second hit box; The wearable device of claim 9 , further configured to: initiate a second action corresponding to the second object in response to detecting the user input.

15. The eye tracking includes determining Cartesian coordinates on the display of the wearable device; A wearable device according to any one of claims 9 to 14, wherein the Cartesian coordinates are temporally filtered.

16. The computer program code further comprises: A wearable device according to any one of claims 9 to 15, configured to store historical user data relating to selected objects on the display of the wearable device.

17. A non-transitory computer-readable storage medium including instructions that, when stored on the non-transitory computer-readable storage medium and executed by at least one processor, cause an apparatus to: determining historical user data relating to events occurring at the wearable device; determining a probability of interacting with an object on a display of the wearable device based on the historical user data; scaling a hit box associated with the object to form a scaled hit box; Detecting user input based on eye tracking within the scaled hit box; and in response to detecting the user input, initiating an action corresponding to the object.

18. the historical user data includes a single frame of user interaction with an object on the display of the wearable device; The non-transitory computer-readable storage medium of claim 17 , wherein the determining the probability of interacting with the object is based on a histogram of the historical user data.

19. the historical user data includes a plurality of frames of user interactions with objects on the display of the wearable device; the historical user data includes a time history; 19. The non-transitory computer-readable storage medium of claim 17 or 18, wherein the determining the probability of interacting with the object is based on a prior distribution and the time history of object interactions at each frame.

20. determining the probability of interacting with the object on the display of the wearable device includes determining a first probability of selecting a first object, the instructions comprising: determining a second probability of interacting with a second object on a display of the wearable device based on the historical user data and the selection of the first object; Scaling a second hit box associated with the second object; and Detecting a second user input based on eye tracking within the scaled second hit box; 20. The non-transitory computer-readable storage medium of claim 17, further configured to: initiate a second action corresponding to the second object in response to detecting the user input.

21. A non-transitory computer readable storage medium comprising instructions, the instructions being stored on the non-transitory computer readable storage medium and configured to, when executed by at least one processor, cause a computing system to perform a method according to any of claims 1 to 8.

22. Apparatus comprising means for carrying out the method according to any one of claims 1 to 8.

23. An apparatus comprising: At least one processor; at least one memory containing computer program code; Apparatus, wherein said at least one memory and said computer program code, together with said at least one processor, are configured to cause said apparatus to perform at least the method according to any of claims 1 to 8.

Images