Algorithmically adjusting hit boxes for icons based on priori gaze and click information
By combining the eye tracking module and the wristband module, the size of the hit box is adjusted using historical user data, the error problem of user input interface in the wearable device is solved, and the interaction accuracy and user experience are improved.
Patent Information
- Application Number
- CN202380037410.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-05-05
- Filing Date
- 2023-05-05
- Publication Date
- 2024-11-29
AI Technical Summary
In the augmented reality application of existing wearable devices, error problems in user input interfaces lead to inaccurate interaction of virtual elements, affecting the user experience.
By combining the eye tracking module and the wristband module, the historical user data is used to predict user input and adjust the size of the hit box, reducing errors and improving interaction accuracy.
It effectively reduces the errors related to the eye tracking module, improves the accuracy of user input and the interactive success rate of wearable devices, and enhances the user experience.
Smart Images

Figure CN119053940A_ABST
Abstract
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application is a continuation of and claims priority to U.S. non-provisional patent application No. 17 / 662,175, filed on May 5, 2022, entitled “ALGORITHMICALLY ADJUSTING THE HITBOX OF ICONS BASED ON PRIOR GAZE AND CLICK INFORMATION,” the disclosure of which is incorporated herein by reference in its entirety. Technical Field
[0002] Implementations relate to a user input interface (e.g., a pointing device) in a wearable device that includes a display. Background Art
[0003] Head-worn computing devices (e.g., smart glasses) can be configured with various sensors to enable augmented reality (AR), in which virtual elements are presented along with real elements of an environment. Virtual elements can be presented on a heads-up display. Virtual elements may appear as if they are in the real world, or may not appear (e.g., system menus) as if they are in the real world. Head-up displays can be implemented in glasses-like devices (i.e., AR glasses). Summary of the invention
[0004] In general aspects, a wearable device, a system, a non-transitory computer-readable medium (having a computer-executable program code stored thereon that can be executed on a computer system), and / or a method can use a method to perform a process, the method comprising: determining historical user data associated with an event occurring on a wearable device; determining a probability of interacting with an object on a display of the wearable device based on the historical user data; scaling a hit box associated with the object to form a scaled hit box; detecting user input within the scaled hit box based on eye tracking; and initiating an action corresponding to the object in response to detecting the user input.
[0005] Implementations may include one or more of the following features. For example, historical user data may include a single frame user interaction with an object on a display of a wearable device, and the determination of the probability of interacting with the object may be based on a histogram of the historical user data. The historical user data may include multiple frames of user interaction with an object on a display of a wearable device, the historical user data may include a time history, and the determination of the probability of interacting with the object may be based on a prior distribution of the object interaction on each frame and the time history. The historical user data may include multiple frames of user interaction with an object on a display of a wearable device, the historical user data may include a time history, and the determination of the probability of interacting with the object may be based on a joint prior of the object interaction on each frame and the time history. The historical user data may include a focus trend, the scaling of a hit box associated with an object may be based on the focus trend, and the hit box may be non-uniformly scaled based on the focus trend.
[0006] Determining the probability of interacting with an object on the display of the wearable device may include determining a first probability of selecting a first object. The method may also include: determining a second probability of interacting with a second object on the display of the wearable device based on historical user data and the selection of the first object; scaling a second hit box associated with the second object; detecting a second user input within the scaled second hit box based on eye tracking; and in response to detecting the user input, initiating a second action corresponding to the second object. Eye tracking may include determining Cartesian coordinates on the display of the wearable device, and the Cartesian coordinates may be filtered by time. The method may also include storing historical user data associated with the selected object on the display of the wearable device. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] Example implementations will be more fully understood from the detailed description given below and the accompanying drawings, in which like elements are represented by like reference numerals, which are given by way of illustration only and thus do not limit example implementations, and in which:
[0008] Figure 1A , Figure 1B , Figure 1C and Figure 1D A drawing of a heads-up display is shown according to an example implementation.
[0009] Figure 2 A heads-up display according to an example implementation is shown.
[0010] Figure 3 A block diagram of sequential layers in a heads-up display is shown according to an example implementation.
[0011] Figure 4 A heads-up display according to an example implementation is shown.
[0012] Figure 5 A perspective view of AR glasses according to an example implementation is shown.
[0013] Figure 6 A method of implementing a user interface in a head-worn computing device according to an example implementation is shown.
[0014] Figure 7 A block diagram of a system corresponding to a head worn computing device is shown according to an example implementation.
[0015] Figure 8 Examples of a computer device and a mobile computer device according to at least one example embodiment are shown.
[0016] It should be noted that these drawings are intended to illustrate the general characteristics of methods, structures and / or materials utilized in certain example implementations and to supplement the written description provided below. However, these drawings are not necessarily drawn to scale and may not accurately reflect the exact structure or performance characteristics of any given embodiment and should not be interpreted as defining or limiting the range of values or properties covered by the example implementations. For example, the relative thickness and positioning of molecules, layers, regions and / or structural elements may be reduced or enlarged for clarity. The use of similar or identical reference numerals in the various drawings is intended to indicate the presence of similar or identical elements or features. DETAILED DESCRIPTION
[0017] Input routines for wearable devices can be enabled by an eye tracking (ET) module plus a wristband (WB). This combination can operate similarly to a mouse interface, where the gaze estimated by ET can be used as pointing / scrolling, and the finger pinch gestures indirectly detected by the WB can be used as clicks on a physical mouse. However, unlike the mouse as a physical interface, the ET+WB interaction framework can rely on perception algorithms and data-driven algorithms. Therefore, the ET+WB interaction framework can operate with a certain amount of error. For example, in the ET module, the error can be quantified using an angular deviation metric (e.g., in degrees, the target is at (x=30, y=40), and the ET module can predict (x=31, y=45)). For example, in the WB module, the error can be quantified using a binary detection metric (e.g., precision-recall).
[0018] Example implementations can use historical user data to predict possible user inputs and modify the size of the generated hit boxes (e.g., background hit detection elements associated with rendered icons, rendered buttons, rendered menu items, etc.) so that the hit box associated with the possible user input is larger than any other hit box. Example implementations can use the described techniques to reduce errors associated with the ET module. The hit box can be an area of virtual or real space associated with a (virtual) detection element for invoking an action. For example, if it is determined that the user's gaze intersects the hit box, an action can be triggered. The hit box area can be defined by coordinates in real and / or virtual space.
[0019] Figure 1A , Figure 1B , Figure 1C and Figure 1D A drawing of a heads-up display according to an example implementation is shown. Figure 1A , multiple icons 110 (hereinafter referred to as icons 110) can be rendered on the display 105. The display 105 can be associated with a wearable device. For example, the display 105 can be a head-up display associated with a head-mounted display (e.g., smart glasses). For example, the display 105 can be a head-up display associated with the AR glasses 500 described below. Hereinafter, a device including the display 105 will be referred to as a wearable device (which may include a head-mounted display, smart glasses, AR glasses, etc.).
[0020] Icon 110 may be a user interface (UI) that enables an action to be performed by the wearable device. An action may be a rendered item that includes, for example, information, a UI, a home screen, a previous item, a menu, a setting, etc. An action may power on / off the wearable device, play / pause / stop audio, play / pause / stop video, interact with an assistant, etc. In an example implementation, the ET+WB interaction framework may be used by a user of the wearable device to interact with icon 110 (e.g., select an icon).
[0021] refer to Figure 1B, each icon 110 has an associated hit box 115. The hit box 115 is not visible to the user of the wearable device. In other words, the hit box 115 is not rendered on the display 105. Each hit box 115 can be associated with a number of pixels of the display 105. Each hit box 115 can be associated with a pixel area of the display 105. Each pixel of the display 105 and the pixel area of the display 105 can be associated with a Cartesian (x, y) coordinate of the display 105. In an example implementation, the user of the wearable device focuses on the icon 110, and the ET module can be configured to determine (e.g., using eye tracking functionality) Cartesian coordinates on the display 105 associated with the view of the user of the wearable device. If the Cartesian coordinates associated with the view of the user of the wearable device are within the pixel area of the display 105 associated with the hit box 115 of the icon 110, then (e.g., in response to a finger pinch gesture detected by the WB) interact with the icon 110.
[0022] refer to Figure 1C As an example, an incoming phone call may be received by the wearable device. The user may be notified of the incoming phone call by a rendered message 120 (eg, incoming call). Figure 1D As an example, in response to an incoming phone call, icon 125 and icon 130 may be rendered on display 105. Interacting with icon 125 may cause the incoming phone call to be rejected, and interacting with icon 130 may cause the incoming phone call to be answered.
[0023] Hit boxes 135 and hit boxes 140 (e.g., using the eye tracking functionality of the ET module) identify the pixel area of the display 105 that the user views to interact with the respective icons 125, 130. However, as discussed above, errors associated with the ET module may cause the wearable device to not detect the interaction with the respective icons 125, 130. As a result, the action associated with the respective icon 125, 130 may not be initiated. In this example, the phone call will not be answered or rejected.
[0024] In an example implementation, hit box 145 identifies a pixel area of display 105 associated with icon 130. Hit box 145 may be generated based on historical user data. For example, historical user data may indicate that the user of the wearable device is likely to answer a phone call from, for example, Jane Doe. Therefore, hit box 145 is generated to include a larger pixel area of display 105 than hit boxes 135, 140. In other words, the user of the wearable device is predicted to be likely to answer the incoming phone call by interacting with icon 130. Therefore, hit box 145 is generated to include a larger pixel area of display 105 so that errors associated with the ET module can be minimized, resulting in a likelihood that interaction with icon 130 causes the wearable device (or companion device) to initiate a corresponding action (e.g., to answer the phone call). As described above, hit boxes 135, 140, 145 are not visible to the user of the wearable device. In other words, hit boxes 135, 140, 145 are not rendered on display 105.
[0025] Figure 2 A heads-up display according to an example implementation is shown. Figure 2 A graphical representation of a heads-up display 205-1, 205-2 is shown. The heads-up display 205-1, 205-2 includes icons 210. Each icon 210 has an associated hit box 215. As described above, the hit box 215 is not visible to the user of the wearable device. The heads-up display 205-1, 205-2 may include other rendered information. The time, date, and weather are shown on the heads-up display 205-1, 205-2. However, other displayable information is within the scope of the present disclosure. The heads-up display 205-1, 205-2 may be in a first state (e.g., a home screen).
[0026] In this example, the user of the wearable device may want to switch the heads-up display 205-1, 205-2 to the second state. For example, the user of the wearable device may want to view a calendar. Therefore, the user of the wearable device may focus on a calendar icon (e.g., the leftmost icon 210), and the ET module may be configured to determine (e.g., using eye tracking functionality) Cartesian coordinates 220 on the heads-up display 205-1 associated with the viewing of the user of the wearable device. In this example, the Cartesian coordinates 220 are shown as being outside the pixel area of the heads-up display 205-1 corresponding to the hit box 215 of the calendar icon (e.g., the leftmost icon 210). Therefore, the action corresponding to the calendar icon (e.g., the leftmost icon 210) will not be initiated, and the heads-up display 205-1 will not be switched to the second state (e.g., opening the calendar UI).
[0027] In an example implementation, as shown on the heads-up display 205-2, the hit box 225 of the calendar icon (e.g., the leftmost icon 210) is generated to include a larger pixel area of the heads-up display 205-2 than the hit box 215. Therefore, when the user of the wearable device focuses on the calendar icon (e.g., the leftmost icon 210) and the ET module determines (e.g., using eye tracking functionality) Cartesian coordinates 220 on the heads-up display 205-2 associated with the viewing of the user of the wearable device, the Cartesian coordinates 220 are shown as being within the pixel area of the heads-up display 205-2 corresponding to the hit box 225 of the calendar icon (e.g., the leftmost icon 210). Therefore, the action corresponding to the calendar icon (e.g., the leftmost icon 210) will be initiated, and the heads-up display 205-2 will switch to the second state (e.g., opening the calendar UI).
[0028] Example implementations may use algorithmic techniques to adjust the hit box of the wearable device UI to mask the errors associated with ET+WB uncertainty (discussed above). Figure 1C and Figure 1D In the described example, an incoming call may be received and icons 125, 130 may be rendered on the display 105. The rendered icons 125, 130 may give the user two options to accept or decline the incoming call. The hit boxes 135, 140 will have a certain number of pixels allocated for the "accept" and "decline" icons, respectively.
[0029] In this example, if the hit boxes 135, 140 both occupy 100 pixels, then for the ET+WB combo model with two (2) sigma error around the center of each icon 125, 130, the user has a 95% accuracy rate of hitting the correct button by looking and clicking (assuming the ET error function is a Gaussian function and using a statistical rule of thumb, sometimes called the 68-95-99.7 rule).
[0030] However, in an example implementation, based on the user's previous call history with a particular call initiator, the user may accept the call, for example, 80% of the time (e.g., the call is a work call or from a family member). This a priori history can be used directly to generate a hit box 145 in the UI (the "accept" hit box will be slightly larger than the "decline" hit box) to increase the posterior probability that the user hits the expected icon, and achieve a net accuracy of, for example, over 95% in this case. As described above, the example implementation does not modify the actual visualization of the UI, only the hit box 145. Call acceptance-rejection is just one example; however, the example implementation can be applied to shrink or enlarge the wearable device display hit box for general menu selections, the user's possible travel routes, shopping, lenses, etc. Another extension is that this does not necessarily have to be "frame-based" - a priori information can be collected from the end-to-end (e2e) flow described below (e.g., when the glasses are started, the user first checks the calendar, then books a meeting, and therefore the icon describing those two interaction points can scale the hit box size). Thus, example implementations may result in a net accuracy improvement due to incorporating prior information (eg, histograms) into the personalization routine.
[0031] A hit box (e.g., hit box 115, 135, 140, 215) may be scaled to increase the size of a pixel region associated with the hit box that is associated with an icon associated with the predicted possible user input. The amount of hit box scaling may be described or derived as follows: Assume a fixed sys UI screen with N options For the i-th icon, define c i The total count of times the user selected icon i when selecting a i The default hitbox size for icon i but, describes the prior distribution probability p, where due to normalization, sum(p i ) = 1, and the scaled hit box area can be: Where r>1 is a "damping factor" that can control the relative strength of increasing or decreasing the hit box size based on a prior distribution. For example, if r is relatively high, the hit box renormalizes to become empty. An example choice of r can be about 5.
[0032] Figure 3A block diagram of sequential layers in a heads-up display is shown according to an example implementation. Figure 3 An example implementation is shown in which the historical user data user flow and / or the current user flow is based on a flow or an end-to-end (e2e) flow. Figure 3 As shown, the main layer 305, layer 1 310, and layer 2 315 can represent a flow frame or display state in a user flow. The flow frame or display state can be a UI representation, where the main layer 305 is the main screen, layer 1 310 is the UI state after the first user interaction, and layer 2 315 is the UI state after the second user interaction. The UI state can be a rendering of the UI of the wearable device on the display of the wearable device.
[0033] Objects 320-1, 320-2 are associated with the main layer 305. Objects 320-1, 320-2 may represent objects rendered on a display of a wearable device (e.g., icons, menus, etc.). Objects 325-1, 325-2 are associated with layer 1 310. Objects 325-1, 325-2 may represent objects rendered on a display of a wearable device (e.g., icons, menus, etc.). Objects 330-1, 330-2 are associated with layer 2 315. Objects 330-1, 330-2 may represent objects rendered on a display of a wearable device (e.g., icons, menus, etc.). As discussed above, a stream frame or display state may be a UI representation, where the main layer 305 is a home screen, layer 1 310 is a UI state after a first user interaction, and layer 2 315 is a UI state after a second user interaction. The UI state may be a rendering of a UI of a wearable device on a display of the wearable device. Thus, objects 320-1, 320-2 may represent objects rendered on the UI home screen, objects 325-1, 325-2 may represent objects rendered on the UI after a first user interaction, and objects 330-1, 330-2 may represent objects rendered on the UI after a second user interaction. Lines 335-1, 335-2, 335-3 may represent an interaction flow, where object 320-1 is interacted with on the main layer 305 (e.g., object 320-1 is selected), object 325-2 is interacted with on layer 1 310 (e.g., object 325-2 is selected), and object 330-1 is interacted with on layer 2 315 (e.g., object 330-1 is selected).
[0034] In this example implementation, the user flow may represent a historical user flow. In this example, it may be useful to estimate the prior distribution including the time history as: in, p is a probability (e.g., the probability of interacting with an object), and t is the timestamp.
[0035] For the two-layer example, knowing the joint prior is equivalent to simply using the probability chain rule, where the individual components can be the histogram and plugin estimated from the user's historical data. The joint prior can then replace the naive Bayes prior (=p i ), to estimate the scaled hit box a i ' to get more refined results.
[0036] Figure 4 A heads-up display according to an example implementation is shown. In an example implementation, the hit box can be non-uniformly scaled. For example, Figure 4 As shown, the heads-up display 205-2, 205-3 shows that the hit box 405 is not uniformly scaled compared to the hit box 225. For example, the hit box 405 is shown to extend in one direction (e.g., toward the top of the heads-up display 205-3). As an example, the user may focus on the top of the icon 210 (e.g., the calendar icon) on average to click. In this example, such a focus trend can be detected, and learning the icon prior probability and scaling its area as a whole in the xy plane on average may be suboptimal. The focus trend can be a place, direction, position, object, etc. that the user always (tends to) look at (focus on). For example, the focus trend can be a place, direction, position, etc. relative to the displayed object (e.g., icon 210). Therefore, the example implementation can take into account the information of the user's gaze deviation in the historical data and the scaling algorithm. Non-uniform scaling can reduce errors, resulting in an improved success rate.
[0037] Figure 5 is a perspective view of AR glasses according to possible implementations of the present disclosure. AR glasses 500 may be a wearable device configured to be worn on the head and face of a user. AR glasses 500 include a right temple 501 and a left temple 502 supported by the user's ears. The AR glasses also include a bridge portion 503, which is supported by the user's nose so that the left lens 504 and the right lens 505 can be located in front of the user's left eye and the user's right eye, respectively. The various parts of the AR glasses may be collectively referred to as a frame of the AR glasses 500. The frame of the AR glasses 500 may include electronic devices for implementing functions. For example, the frame may include a battery, a processor, a memory (e.g., a non-transitory computer-readable medium), and electronic devices for supporting sensors (e.g., cameras, depth sensors, etc.) and interface devices (e.g., speakers, displays, network adapters, etc.). For example, the function may include generating and scaling a hit box as described above.
[0038] The AR glasses 500 may include a FOV camera 510 (e.g., an RGB camera) pointing to a camera field of view that overlaps the natural field of view of the user's eyes when wearing the glasses. In a possible implementation, the AR glasses may also include a depth sensor 511 (e.g., a laser radar (LIDAR), structured light, time of flight, depth camera) pointing to a depth sensor field of view that overlaps the natural field of view of the user's eyes when wearing the glasses. Data from the depth sensor 511 and / or the FOV camera 510 may be used to measure the depth in the field of view (i.e., the area of interest) of the user (i.e., the wearer). In a possible implementation, the camera field of view and the depth sensor field of view may be calibrated so that the depth (i.e., range) of an object in an image from the FOV camera 510 may be determined, where the depth is measured between the object and the AR glasses.
[0039] The AR glasses 500 may also include a display 515. The display may present AR data (e.g., images, graphics, text, icons, etc.) on a portion of a lens (or lenses) of the AR glasses so that the user can view the AR data when the user looks through the lenses of the AR glasses. In this way, the AR data may overlap with the user's view of the environment.
[0040] The AR glasses 500 may also include an eye tracking sensor. The eye tracking sensor may include a right eye camera 520 and a left eye camera 521. The right eye camera 520 and the left eye camera 521 may be located in the lens portion of the frame so that when the AR glasses are worn, the right FOV 522 of the right eye camera includes the user's right eye, and the left FOV 523 of the left eye camera includes the user's left eye. The gaze point (x, y) may be determined at the frequency of the video feed of the camera (e.g., right eye camera 520, left eye camera 521). For example, the gaze point coordinates (x, y) may be measured at a frame rate of the camera (e.g., 15 frames per second) or lower. The ET model may be configured to use the eye tracking sensor and / or the gaze point coordinates (x, y) to predict and / or as Cartesian coordinates (e.g., Cartesian coordinates 220).
[0041] The AR glasses 500 may be communicatively coupled to a peripheral device 530. The peripheral device 530 may be configured to detect and / or sense gestures associated with a user's hand 535. The peripheral device 530 may be or may include or may perform the WB functions described above. For example, the peripheral device 530 may be configured to detect or help detect a finger pinch gesture for initiating the above-described user interaction (e.g., selecting an icon). Other gestures or techniques may be used to initiate user interactions.
[0042] The AR glasses 500 may also include a left speaker 541 and a right speaker 542 configured to transmit audio to the user. Additionally or alternatively, transmitting audio to the user may include transmitting the audio to a listening device (e.g., a hearing aid, earbuds, etc.) via a wireless communication link 545. For example, the AR glasses 500 may transmit audio to a left wireless earbud 546 and a right earbud 547.
[0043] Figure 6 A method of implementing a user interface in a head-worn computing device according to an example implementation is shown. Figure 6 As shown, in step S605, historical user data associated with an event occurring on the wearable device is determined. For example, the historical user data may be stored in a memory associated with the wearable device (e.g., AR glasses 500) and / or a matching device (e.g., a mobile phone communicatively coupled to the wearable device), such as in a database. The historical user data may be associated with an interaction with an object rendered on a display of the wearable device (e.g., display 105, 205) (e.g., selecting an object). The historical user data may be associated with an event occurring on the wearable device (e.g., on a display of the wearable device). The historical user data may be stored in response to an object interacted with in response to an event.
[0044] Thus, an event may occur on the wearable device (e.g., an incoming call is received), and historical user data associated with the event may be read from the memory. Determination of the historical user data may include determining characteristics about the event (e.g., an incoming call and who the call is from), and filtering the historical user data based on the characteristics about the event. The historical user data may include a single frame of user interaction with an object on the display of the wearable device. In other words, the single frame of user interaction may include user interaction with an object displayed on the wearable device within a single frame of a multi-frame video that is used to display content including the object. Thus, historical data may be associated with a single frame in which the user interacted with the object.
[0045] The historical user data may include multi-frame user interactions with an object on the display of the wearable device. In other words, the multi-frame user interaction may include a user interaction with an object displayed on the wearable device within two or more frames (e.g., in time frame order) of a multi-frame video for displaying content including the object. Therefore, the historical data may be associated with multiple (e.g., two or more) frames in which the user interacts with the object. The historical user data may include a time history. The time history may include a multi-frame interaction between the user and the object, so that each frame is timestamped. The determination of the historical user data may include recalling multiple associated frames sorted based on timestamps. In addition, in some implementations, the historical user data may include the total number of interactions of the user with a particular hit box or the number per unit time (e.g., day, week, etc.).
[0046] In step S610, based on the historical user data, a probability of interacting with an object on a display of the wearable device is determined. For example, a histogram of the historical user data may be generated. The histogram may include historical objects interacted with based on events. The probability of interacting with an object on a display of the wearable device may be determined based on columns in the histogram. The histogram with the most entries may be the object with the highest probability of interacting with it. The determination of the probability of interacting with an object may be based on a prior distribution of object interactions on each frame and a time history. The determination of the probability of interacting with an object may be based on a joint prior of object interactions on each frame and a time history.
[0047] In step S615, the hit box associated with the object is scaled. For example, the object with the highest probability of interaction therewith may have an associated hit box that is scaled. Scaling the hit box may include increasing the size of the hit box. The hit box may be associated with a number of pixels of a display of the wearable device. The hit box may be associated with a pixel area of the display of the wearable device. Each pixel of the display and the pixel area of the display may be associated with a Cartesian (x, y) coordinate of the display. Scaling the hit box may include increasing the number of pixels in the pixel area of the display corresponding to the hit box. The historical user data may include a focus trend, and the scaling of the hit box associated with the object may be based on the focus trend. The hit box may be non-uniformly scaled based on the focus trend.
[0048] In step S620, based on eye tracking within the scaled hit box, user input is detected. For example, the ET+WB interaction framework can be used to detect user input. Eye tracking can include determining Cartesian coordinates on a display of the wearable device (e.g., using an ET module). Cartesian coordinates can be filtered by time (e.g., over multiple frames). The wearable device can be communicatively coupled to a peripheral device. The peripheral device can be configured to detect and / or sense gestures associated with the user's hand. The peripheral device can be or can include or can perform a WB function as described above. For example, the peripheral device can be configured to detect or help detect a finger pinch gesture for initiating user interaction. The detected gesture can be combined with eye tracking to generate user input. Other gestures or techniques can be used to initiate user interaction.
[0049] A possible method for eye tracking includes measuring eye metrics using a camera to determine the position of the eye. In one possible implementation, the position of the pupil relative to the pattern of light (near infrared light) projected onto the eye can be measured by analyzing a high-resolution image of the eye and the pattern. The position of the eye can then be applied to a machine learning model to determine the Cartesian coordinates of the gaze point. Variations of this method that do not use projected patterns are possible. For example, there are standard flash-based tracking or convolutional neural network methods that can convert a two-dimensional (2D) infrared image captured by a camera pointing to the eye (or an eye image reflected from a mirror) into coordinates (x, y) in the field of view of the AR glasses.
[0050] Rapid eye movements (i.e., glances) may make it difficult to determine gaze. Therefore, the method may also include filtering the gaze coordinates by time. For example, the gaze coordinates from real-time eye tracking may be low-pass filtered to produce a time-varying signal that is less variable over time corresponding to the gaze. In a possible implementation, the eye tracking coordinates may be measured over time and averaged to obtain average eye tracking coordinates. When the average eye tracking coordinates meet the dwell time criterion, the user's gaze direction may be determined. For example, average eye tracking coordinates within a range (e.g., region) longer than a threshold period may indicate a stable gaze.
[0051] In step S625, in response to detecting the user input, an action corresponding to the object is initiated. For example, the object may be an icon, and the action may be initiated in response to interacting with the icon (e.g., selecting the icon). For example, an incoming call may be accepted or rejected based on interacting with the icon. For example, the object may be a menu, and the action may be initiated in response to interacting with an item in the menu (e.g., selecting the item).
[0052] In some implementations (e.g., with multi-frame user interaction with an object), more than one object is interacted with in a short period of time. The following is an example in which two objects are interacted with. The determination of the probability of selecting an object on the display of the wearable device can be to determine a first probability of selecting a first object. Then, based on the historical user data and the selection of the first object, a second probability of interacting with a second object on the display of the wearable device can be determined. A second hit box associated with the second object can be scaled. A second user input can be detected within the scaled second hit box based on eye tracking, and in response to detecting the user input, a second action corresponding to the second object can be initiated.
[0053] Figure 7 A block diagram of a system corresponding to and / or including a wearable device (e.g., a wearable computing device, a head-mounted display, smart glasses, AR glasses, a head-mounted display, etc.) according to an example implementation is shown. Figure 7 In the examples of, a system (e.g., an augmented reality system, a virtual reality system, an ancillary device, etc.) may include a computing system or at least one computing device, and should be understood to represent almost any computing device configured to perform the techniques described herein. Therefore, a device may be understood to include various components that can be used to implement the techniques described herein or different or future versions thereof. By way of example, a system may include a processor 705 and a memory 710 (e.g., a non-transitory computer-readable memory). The processor 705 and the memory 710 may be coupled (e.g., communicatively coupled) via a bus 715.
[0054] The processor 705 may be used to execute instructions stored on at least one memory 710. Thus, the processor 705 may implement various features and functions described herein or additional or alternative features and functions. The processor 705 and at least one memory 710 may be used for various other purposes. For example, the at least one memory 710 may represent examples of various types of memory and related hardware and software that may be used to implement any of the modules described herein.
[0055] At least one memory 710 may be configured to store data and / or information associated with the device. At least one memory 710 may be a shared resource. Therefore, at least one memory 710 may be configured to store data and / or information associated with other elements (e.g., image / video processing or wired / wireless communication) in a larger system. The processor 705 and at least one memory 710 may be used together to implement the technology described herein. Therefore, the technology described herein may be implemented as a code segment (e.g., software) stored on the memory 710 and executed by the processor 705. Therefore, the memory 710 may include an ET module 720, a WB module 725, and a scaling module 730.
[0056] The ET module 720 may be configured to determine Cartesian coordinates or gaze point coordinates associated with the gaze of the user of the wearable device. In an example implementation, the user of the wearable device may focus on an object (e.g., an icon), and the ET module 720 may be configured to determine Cartesian coordinates on a display associated with the viewing of the user of the wearable device (e.g., using an eye tracking function). The ET module 720 may be communicatively coupled to an eye tracking sensor. The eye tracking sensor may include a right eye camera and a left eye camera. The right eye camera and the left eye camera may be located in a lens portion of the frame so that when the AR glasses are worn, the right FOV of the right eye camera includes the right eye of the user, and the left FOV of the left eye camera includes the left eye of the user. The ET module 720 may be configured to determine Cartesian coordinates or gaze points at the frequency of the video feed of the camera (e.g., right eye camera, left eye camera). For example, the gaze point coordinates (x, y) may be measured at a frame rate of the camera (e.g., 15 frames per second) or lower. The ET module 720 may be configured to use the eye tracking sensor and / or gaze point coordinates (x, y) to predict and / or as Cartesian coordinates (eg, Cartesian coordinates 220).
[0057] The WB module 725 may be configured to receive communications from a peripheral device and trigger an event based on the communication. For example, the event may be a "click" event associated with a pointing device. The peripheral device may be configured to detect and / or sense gestures associated with a user's hand. The WB module 725, together with the peripheral device, may be configured to detect or help detect a finger pinch gesture for initiating the above-mentioned user interaction (e.g., selecting an icon). Other gestures or techniques may be used to initiate user interaction.
[0058] The scaling module 730 can be configured to scale (e.g., increase) the size of a hit box associated with a user interface based on historical user data. For example, the hit box can be associated with the number of pixels of a display of the wearable device. The hit box can be associated with a pixel area of the display of the wearable device. Each pixel of the display and the pixel area of the display can be associated with a Cartesian (x, y) coordinate of the display. Scaling the hit box can include increasing the number of pixels in the pixel area of the display corresponding to the hit box.
[0059] Figure 8An example of a computer device 800 and a mobile computer device 850 that can be used with the described techniques is shown. The computing device 800 includes a processor 802, a memory 804, a storage device 806, a high-speed interface 808 connected to the memory 804 and a high-speed expansion port 810, and a low-speed interface 812 connected to a low-speed bus 814 and the storage device 806. Each of the components 802, 804, 806, 808, 810, and 812 are interconnected using various buses and can be mounted on a common motherboard or otherwise as appropriate. The processor 802 can process instructions for execution within the computing device 800, including instructions stored in the memory 804 or on the storage device 806, to display graphical information of a GUI on an external input / output device (such as a display 816 coupled to the high-speed interface 808). In other implementations, multiple processors and / or multiple buses can be used as appropriate along with multiple memories and multiple memory types. Additionally, multiple computing devices 800 may be connected, with each device providing portions of the necessary operations (eg, as a server bank, a group of blade servers, or a multi-processor system).
[0060] The memory 804 stores information within the computing device 800. In one implementation, the memory 804 is one or more volatile memory units. In another implementation, the memory 804 is one or more non-volatile memory units. The memory 804 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.
[0061] The storage device 806 can provide mass storage for the computing device 800. In one implementation, the storage device 806 can be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a magnetic tape device, a flash memory or other similar solid-state memory device, or an array of devices (including devices in a storage area network or other configuration). A computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as the memory 804, the storage device 806, or a memory on the processor 802.
[0062] The high-speed controller 808 manages bandwidth-intensive operations of the computing device 800, while the low-speed controller 812 manages less bandwidth-intensive operations. This allocation of functions is used as an example only. In one implementation, the high-speed controller 808 is coupled to the memory 804, the display 816 (e.g., through a graphics processor or accelerator), and a high-speed expansion port 810 that can accept various expansion cards (not shown). In this implementation, the low-speed controller 812 is coupled to the storage device 806 and the low-speed expansion port 814. The low-speed expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a network device, such as a switch or a router, for example, through a network adapter.
[0063] As shown, computing device 800 can be implemented in many different forms. For example, it can be implemented as a standard server 820, or multiple times in a group of such servers. It can also be implemented as part of a rack server system 824. In addition, it can be implemented in a personal computer such as a laptop computer 822. Alternatively, components from computing device 800 can be combined with other components in a mobile device (not shown) such as device 850. Each of such devices can contain one or more of computing devices 800, 850, and the entire system can be composed of multiple computing devices 800, 850 that communicate with each other.
[0064] The computing device 850 includes a processor 852, a memory 864, an input / output device (such as a display 854), a communication interface 866, and a transceiver 868, among other components. The device 850 may also be provided with a storage device (such as a micro drive or other device) to provide additional storage. Each of the components 850, 852, 864, 854, 866, and 868 are interconnected using various buses, and several of the components may be mounted on a common motherboard or otherwise as appropriate.
[0065] Processor 852 can execute instructions within computing device 850, including instructions stored in memory 864. The processor can be implemented as a chipset including separate multiple analog and digital processor chips. The processor can provide, for example, coordination of other components of device 850, such as controlling a user interface, applications run by device 850, and wireless communications performed by device 850.
[0066] The processor 852 can communicate with the user through the control interface 858 and the display interface 856 coupled to the display 854. The display 854 can be, for example, a TFT LCD (thin film transistor liquid crystal display) and an LED (light emitting diode) or OLED (organic light emitting diode) display or other appropriate display technology. The display interface 856 may include an appropriate circuit system for driving the display 854 to present graphical information and other information to the user. The control interface 858 can receive commands from the user and transform them to submit to the processor 852. In addition, an external interface 862 for communication with the processor 852 can be provided to enable the device 850 to communicate with other devices in a near area. The external interface 862 can, for example, provide wired communication in some implementations, or provide wireless communication in other implementations, and multiple interfaces can also be used.
[0067] The memory 864 stores information within the computing device 850. The memory 864 may be implemented as one or more of one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. An expansion memory 874 may also be provided and connected to the device 850 via an expansion interface 872, which may include, for example, a SIMM (single in-line memory module) card interface. Such an expansion memory 874 may provide additional storage space for the device 850, or may also store applications or other information of the device 850. Specifically, the expansion memory 874 may include instructions for executing or supplementing the above-mentioned processes, and may also include security information. Therefore, for example, the expansion memory 874 may be provided as a security module for the device 850, and may be programmed with instructions that permit secure use of the device 850. In addition, security applications and additional information may be provided via a SIMM card, such as placing identification information on a SIMM card in an unbreakable manner.
[0068] The memory may include, for example, flash memory and / or NVRAM memory, as discussed below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or machine-readable medium, such as memory 864, expansion memory 874, or memory on processor 852, which may be received, for example, via transceiver 868 or external interface 862.
[0069] The device 850 can communicate wirelessly via a communication interface 866, which may include a digital signal processing circuit system when necessary. The communication interface 866 can provide communication under various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, etc. Such communication can occur, for example, via a radio frequency transceiver 868. In addition, short-range communication can occur, such as using Bluetooth, Wi-Fi or other such transceivers (not shown). In addition, a GPS (global positioning system) receiver module 870 can provide additional wireless data related to navigation and positioning to the device 850, which can be used as appropriate by applications running on the device 850.
[0070] Device 850 may also communicate audibly using audio codec 860, which may receive verbal information from a user and convert it into usable digital information. Audio codec 860 may also generate audible sounds for the user, such as through a speaker in a handset of device 850, for example. Such sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications operating on device 850.
[0071] As shown, computing device 850 can be implemented in many different forms. For example, it can be implemented as a cellular phone 880. It can also be implemented as part of a smart phone 882, a personal digital assistant, or other similar mobile device.
[0072] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor that can be either special purpose or general purpose and can be coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.
[0073] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages and / or in assembly / machine languages. As used herein, the terms "machine-readable medium," "computer-readable medium," refer to any computer program product, apparatus, and / or device (e.g., a disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0074] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (LED (light emitting diode), or OLED (organic LED), or LCD (liquid crystal display) monitor / screen) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) that the user can use to provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including sound, voice, or tactile input.
[0075] The systems and techniques described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a client can interact with an implementation of the systems and techniques described herein), or any combination of these back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.
[0076] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server is created by computer programs running on the respective computers and placing each other in a client-server relationship.
[0077] In some implementations, the computing device depicted in the figure may include sensors interfaced with the AR headset / HMD device 890, which are used to generate an augmented environment for viewing inserted content in the physical space. For example, one or more sensors included on the computing device 850 depicted in the figure or other computing devices may provide input to the AR headset 890, or generally speaking, to the AR space. Sensors may include, but are not limited to, touch screens, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. The computing device 850 may use sensors to determine the absolute position and / or detected rotation of the computing device in the AR space, which may then be used as input to the AR space. For example, the computing device 850 may be incorporated into the AR space as a virtual object (such as a controller, laser pointer, keyboard, weapon, etc.). Positioning of the computing device / virtual object by the user when incorporated into the AR space may allow the user to position the computing device so as to view the virtual object in some way in the AR space. For example, if the virtual object represents a laser pointer, the user may manipulate the computing device as if it were manipulating an actual laser pointer. The user can move the computing device left to right, up and down, in circles, etc., and use the device in a manner similar to using a laser pointer. In some implementations, the user can use a virtual laser pointer to aim at a target location.
[0078] In some implementations, one or more input devices included on or connected to the computing device 850 can be used as input to the AR space. The input device can include, but is not limited to, a touch screen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, headphones or earbuds with input functionality, a game controller, or other connectable input device. When the computing device 850 is incorporated into the AR space, user interaction with the input device included on the computing device can cause specific actions to occur in the AR space.
[0079] In some implementations, the touch screen of computing device 850 can be rendered as a touch pad in the AR space. A user can interact with the touch screen of computing device 850. For example, in AR headset 890, the interaction is rendered as movement on the rendered touch pad in the AR space. The rendered movement can control virtual objects in the AR space.
[0080] In some implementations, one or more output devices included on the computing device 850 may provide output and / or feedback to a user of the AR headset 890 in the AR space. The output and feedback may be visual, tactile, or auditory. The output and / or feedback may include, but are not limited to, vibration, turning one or more lights or flashes on and off or blinking and / or flashing, sounding an alarm, playing a chime, playing a song, and playing an audio file. The output device may include, but is not limited to, a vibration motor, a vibration coil, a piezoelectric device, an electrostatic device, a light emitting diode (LED), a flash, and a speaker.
[0081] In some implementations, computing device 850 may appear as another object in a computer-generated 3D environment. User interactions with computing device 850 (e.g., rotating, shaking, touching a touch screen, swiping a finger across a touch screen) may be interpreted as interactions with objects in the AR space. In the example of a laser pointer in the AR space, computing device 850 appears as a virtual laser pointer in the computer-generated 3D environment. As the user manipulates computing device 850, the user in the AR space sees the movement of the laser pointer. The user receives feedback from the interaction with computing device 850 in the AR environment on computing device 850 or AR headset 890. User interactions with the computing device may be converted into interactions with a user interface generated for a controllable device in the AR environment.
[0082] In some implementations, computing device 850 may include a touch screen. For example, a user may interact with the touch screen to interact with a user interface of a controllable device. For example, the touch screen may include user interface elements, such as sliders, that may control characteristics of the controllable device.
[0083] Computing device 800 is intended to represent various forms of digital computers and devices, including but not limited to laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 850 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions are intended to be examples only and are not intended to limit implementations of the inventions described and / or claimed in this document.
[0084] A number of embodiments have been described, however, it will be appreciated that various modifications can be made without departing from the spirit and scope of the specification.
[0085] In addition, the logic flows depicted in the figures do not require the particular order or sequential order shown to achieve the desired results. In addition, other steps can be provided, or steps can be deleted from the described processes, and other components can be added to or removed from the described systems. Therefore, other embodiments are within the scope of the appended claims.
[0086] In addition to the above, controls may be provided to allow the user to make a choice about whether and when the systems, programs, or features described herein may be able to collect user information (e.g., information about the user's social network, social behavior, or activities, occupation, user preferences, or the user's current location) and whether to send content or communications from a server to the user. In addition, before certain data is stored or used, it may be processed in one or more ways to remove personally identifiable information. For example, the user's identity may be processed so that the user's personally identifiable information cannot be determined, or the user's geographic location for which location information is obtained may be generalized (such as to a city, zip code, or state level) so that the user's specific location cannot be determined. Thus, the user can control what information is collected about the user, how the information is used, and what information is provided to the user.
[0087] Although certain features of the described implementations have been shown as described herein, those skilled in the art will now expect many modifications, substitutions, variations and equivalents. Therefore, it should be understood that the appended claims are intended to cover all these modifications and variations that fall within the scope of the implementations. It should be understood that they are presented only in an exemplary and non-restrictive manner, and various changes in form and detail may be made. Except for mutually exclusive combinations, any part of the devices and / or methods described herein may be combined in any combination. The implementations described herein may include various combinations and / or sub-combinations of the functions, components and / or features of the different implementations described.
[0088] Although the example embodiments may include various modifications and alternative forms, embodiments thereof are shown by way of example in the drawings and will be described in detail herein. However, it should be understood that it is not intended to limit the example embodiments to the particular forms disclosed, but rather, the example embodiments will cover all modifications, equivalents, and alternatives falling within the scope of the claims. The same numbers refer to the same elements in the description of the drawings.
[0089] Some of the above example embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the operations as sequential processes, many operations may be performed in parallel, concurrently, or simultaneously. In addition, the order of the operations may be rearranged. Processes may terminate when their operations are completed, but may also have additional steps not included in the figure. A process may correspond to a method, function, procedure, subroutine, subprogram, etc.
[0090] The methods discussed above (some of which are illustrated by flowcharts) may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented by software, firmware, middleware, or microcode, the program code or code segments for performing the necessary tasks may be stored in a machine or computer readable medium, such as a storage medium. One or more processors may perform the necessary tasks.
[0091] The specific structural and functional details disclosed herein are merely representative for describing example embodiments. Example embodiments may, however, be embodied in many alternative forms and should not be construed as limited to only the embodiments set forth herein.
[0092] It should be understood that although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the scope of the exemplary embodiments, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. As used herein, the terms and / or include any and all combinations of one or more of the associated listed items.
[0093] It should be understood that when an element is referred to as being connected or coupled to another element, the element may be directly connected or coupled to the other element, or there may be intervening elements. Conversely, when an element is referred to as being directly connected or coupled to another element, there are no intervening elements. Other words used to describe the relationship between elements should be interpreted in a similar manner (e.g., between versus directly between, adjacent versus directly adjacent, etc.).
[0094] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments. As used herein, unless the context clearly indicates otherwise, the singular forms one, a kind and the are also intended to include plural forms. It should be further understood that the terms include, contain, contain and / or have when used herein to specify the presence of stated features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups.
[0095] It should also be noted that in some alternative implementations, the functions / behaviors noted may not occur in the order noted in the figures. For example, two figures shown in succession may actually be executed simultaneously or may sometimes be executed in the reverse order, depending on the functionality / behaviors involved.
[0096] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the example embodiments belong. It should be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless explicitly defined as such herein.
[0097] Parts of the above exemplary embodiments and corresponding detailed descriptions are presented in terms of software or algorithms and symbolic representations of the operations of the data bits in the computer memory. These descriptions and representations are descriptions and representations that effectively convey the essence of their work to other persons of ordinary skill in the art. As a term used herein and as a commonly used term, an algorithm is considered to be a self-consistent sequence of steps leading to a desired result. A step is a step that requires physical manipulation of a physical quantity. Typically, although not necessarily, these quantities are in the form of optical, electrical or magnetic signals that can be stored, transmitted, combined, compared and otherwise manipulated. Mainly for general reasons, it has sometimes been proven to be convenient to refer to these signals as bits, values, elements, symbols, characters, items, numbers, etc.
[0098] In the above exemplary embodiments, references to symbolic representations of actions and operations that can be implemented as program modules or functional processes (e.g., in the form of flowcharts) include routines, programs, objects, components, data structures, etc., which perform specific tasks or implement specific abstract data types and can be described and / or implemented using existing hardware at existing structural elements. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application specific integrated circuits, field programmable gate arrays (FPGAs), computers, etc.
[0099] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specifically noted or as is apparent from the discussion, terms such as processing or computing or calculating or determining display refer to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical electronic quantities within the computer system's registers and memories and transforms that data into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission or display devices.
[0100] It is also noted that the software-implemented aspects of the example embodiments are typically encoded on some form of non-transitory program storage medium or implemented on some type of transmission medium. The program storage medium may be magnetic (e.g., a floppy disk or hard disk) or optical (e.g., a compact disk read-only memory or CD ROM), and may be read-only or random access. Similarly, the transmission medium may be twisted pair, coaxial cable, optical fiber, or some other suitable transmission medium known in the art. The example embodiments are not limited by these aspects of any given implementation.
[0101] Finally, it should also be noted that although the appended claims set forth specific combinations of features described herein, the scope of the present disclosure is not limited to the specific combinations claimed below, but extends to cover any combination of features or embodiments disclosed herein, regardless of whether the specific combination is specifically listed in the appended claims.
Claims
1. A method comprising: determining historical user data associated with events occurring on the wearable device; determining a probability of interacting with an object on a display of the wearable device based on the historical user data; scaling a hit box associated with the object to form a scaled hit box; detecting user input within the scaled hit box based on eye tracking; as well as In response to detecting the user input, an action corresponding to the object is initiated.
2. The method according to claim 1, wherein The historical user data includes single-frame user interactions with objects on the display of the wearable device, and The determination of the probability of interacting with the object is based on a histogram of the historical user data.
3. The method according to claim 1 or 2, wherein The historical user data includes multiple frames of user interactions with an object on the display of the wearable device, The historical user data includes a time history, and The determination of the probability of interacting with the object is based on a prior distribution of object interactions at each frame and the temporal history.
4. A method according to any one of the preceding claims, wherein The historical user data includes multiple frames of user interactions with an object on the display of the wearable device, The historical user data includes a time history, and The determination of the probability of interacting with the object is based on a joint prior of object interaction at each frame and the temporal history.
5. A method according to any one of the preceding claims, wherein The historical user data includes focus trends, The scaling of the hit box associated with the object is based on the focus trend, and Based on the focus trend, the hit box is non-uniformly scaled.
6. The method according to any one of claims 1 to 5, wherein The determining of the probability of interacting with the object on the display of the wearable device includes determining a first probability of selecting a first object, the method further comprising: determining a second probability of interacting with a second object on a display of the wearable device based on the historical user data and the selection of the first object; scaling a second hit box associated with the second object; detecting a second user input within the scaled second hit box based on eye tracking; as well as In response to detecting the user input, a second action corresponding to the second object is initiated.
7. The method according to any one of claims 1 to 6, wherein The eye tracking comprises determining Cartesian coordinates on the display of the wearable device, and Filters the Cartesian coordinates by time.
8. The method according to any one of claims 1 to 7, further comprising: Storing historical user data associated with selected objects on the display of the wearable device.
9. A wearable device comprising: at least one processor; as well as at least one memory, the at least one memory comprising computer program code; The at least one memory and the computer program code are configured to, with the at least one processor, cause the wearable device to: determining historical user data associated with events occurring on the wearable device; determining a probability of interacting with an object on a display of the wearable device based on the historical user data; scaling a hit box associated with the object to form a scaled hit box; detecting user input within the scaled hit box based on eye tracking; as well as In response to detecting the user input, an action corresponding to the object is initiated.
10. The wearable device according to claim 9, wherein The historical user data includes single-frame user interactions with objects on the display of the wearable device, and The determination of the probability of interacting with the object is based on a histogram of the historical user data.
11. The wearable device according to claim 9 or 10, wherein The historical user data includes multiple frames of user interactions with an object on the display of the wearable device, The historical user data includes a time history, and The determination of the probability of interacting with the object is based on a prior distribution of object interactions at each frame and the temporal history.
12. The wearable device according to any one of claims 9 to 11, wherein The historical user data includes multiple frames of user interactions with an object on the display of the wearable device, The historical user data includes a time history, and The determination of the probability of interacting with the object is based on a joint prior of object interaction at each frame and the temporal history.
13. The wearable device according to any one of claims 9 to 12, wherein The historical user data includes focus trends, The scaling of the hit box associated with the object is based on the focus trend, and Based on the focus trend, the hit box is non-uniformly scaled.
14. The wearable device according to any one of claims 9 to 13, wherein The determining of the probability of interacting with the object on the display of the wearable device comprises determining a first probability of selecting a first object, the computer program code is further configured to: determining a second probability of interacting with a second object on a display of the wearable device based on the historical user data and the selection of the first object; scaling a second hit box associated with the second object; detecting a second user input within the scaled second hit box based on eye tracking; as well as In response to detecting the user input, a second action corresponding to the second object is initiated.
15. The wearable device according to any one of claims 9 to 14, wherein The eye tracking comprises determining Cartesian coordinates on the display of the wearable device, and Filters the Cartesian coordinates by time.
16. The wearable device according to any one of claims 9 to 15, wherein the computer program code is further configured to: Storing historical user data associated with selected objects on the display of the wearable device.
17. A non-transitory computer-readable storage medium comprising instructions stored thereon, which when executed by at least one processor are configured to cause a device to: determining historical user data associated with events occurring on the wearable device; determining a probability of interacting with an object on a display of the wearable device based on the historical user data; scaling a hit box associated with the object to form a scaled hit box; detecting user input within the scaled hit box based on eye tracking; as well as In response to detecting the user input, an action corresponding to the object is initiated.
18. The non-transitory computer-readable storage medium of claim 17, wherein The historical user data includes single-frame user interactions with objects on the display of the wearable device, and The determination of the probability of interacting with the object is based on a histogram of the historical user data.
19. The non-transitory computer-readable storage medium according to claim 17 or 18, wherein The historical user data includes multiple frames of user interactions with an object on the display of the wearable device, The historical user data includes a time history, and The determination of the probability of interacting with the object is based on a prior distribution of object interactions at each frame and the temporal history.
20. The non-transitory computer-readable storage medium according to any one of claims 17 to 19, wherein The determining of the probability of interacting with the object on the display of the wearable device includes determining a first probability of selecting a first object, the instructions being further configured to: determining a second probability of interacting with a second object on a display of the wearable device based on the historical user data and the selection of the first object; scaling a second hit box associated with the second object; detecting a second user input within the scaled second hit box based on eye tracking; as well as In response to detecting the user input, a second action corresponding to the second object is initiated.
21. A non-transitory computer-readable storage medium comprising instructions stored thereon, which when executed by at least one processor are configured to cause a computing system to perform the method according to any one of claims 1 to 8.
22. An apparatus comprising means for performing the method according to any one of claims 1 to 8.
23. A device comprising: at least one processor; as well as at least one memory, the at least one memory comprising computer program code; The at least one memory and the computer program code are configured to, together with the at least one processor, cause the apparatus to perform a method according to any one of claims 1 to 8.
