Calibrating gaze tracker

Abstract

The invention relates to a calibration gaze tracker. A method includes displaying a plurality of visual elements. The method includes determining an expected gaze target based on the respective feature values of the plurality of visual elements, the expected gaze target indicating a first display area where a user of the device intends to gaze while the plurality of visual elements are being displayed. The method includes obtaining an image via an image sensor, the image including a set of pixels corresponding to pupils of a user of the device. The method includes determining, by a gaze tracker, a measured gaze target based on the set of pixels corresponding to the pupil, the measured gaze target indicating a second display area in which the user is measuring gaze. The method includes adjusting a calibration parameter of the gaze tracker based on a difference between the first display area indicated by the expected gaze target and the second display area indicated by the measured gaze target.

Description

[0001] This application is a divisional application of Chinese invention patent application filed on March 27, 2023, with application number 202310304963.X and invention title "Calibration of Gaze Tracker". Cross-reference to related applications

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 324,351, filed March 28, 2022, and U.S. Provisional Patent Application No. 63 / 409,293, filed September 23, 2022, the full text of which is cited herein. Technical Field

[0003] This disclosure relates in its entirety to the calibration of gaze trackers. Background Technology

[0004] Some devices include displays that present visual content. Some devices manipulate visual content based on input. Incorrect input may trigger the device to manipulate visual content unintentionally. Some devices perform various operations based on input. Incorrect input may trigger the device to perform unpredictable operations. Attached Figure Description

[0005] Therefore, this disclosure will be understood by those skilled in the art, and a more detailed description can be made with reference to some exemplary embodiments, some of which are shown in the accompanying drawings.

[0006] Figures 1A to 1J It is a diagram based on an exemplary operating environment of some specific implementations.

[0007] Figure 2 This is a block diagram of a system for adjusting the calibration parameters of a gaze tracker based on some specific implementations.

[0008] Figure 3 This is a flowchart illustrating a method for adjusting the calibration parameters of a gaze tracker based on some specific implementations.

[0009] Figure 4 This is a block diagram of a device for adjusting the calibration parameters of a gaze tracker based on some specific implementations.

[0010] As is customary, the various features shown in the accompanying drawings may not be drawn to scale. Therefore, for clarity, the dimensions of various features may be arbitrarily expanded or reduced. Additionally, some drawings may not depict all components of a given system, method, or apparatus. Finally, similar reference numerals may be used throughout the specification and drawings to denote similar features. Summary of the Invention

[0011] The various embodiments disclosed herein include devices, systems, and methods for calibrating a gaze tracker. In various embodiments, the device includes a display, an image sensor, non-transitory memory, and one or more processors coupled to the display, image sensor, and non-transitory memory. In various embodiments, one method includes displaying a plurality of visual elements on the display. In some embodiments, the method includes determining a desired gaze target based on corresponding characteristic values ​​of the plurality of visual elements, the desired gaze target indicating a first display area that a user of the device intends to gaze at while the plurality of visual elements are being displayed. In some embodiments, the method includes acquiring an image via an image sensor, the image including a set of pixels corresponding to the pupil of a user of the device. In some embodiments, the method includes determining a measured gaze target by the gaze tracker based on the set of pixels corresponding to the pupil, the measured gaze target indicating a second display area that the user is measuringly gazing at. In some embodiments, the method includes adjusting calibration parameters of the gaze tracker based on the difference between the first display area indicated by the desired gaze target and the second display area indicated by the measured gaze target.

[0012] According to some embodiments, the device includes one or more processors, non-transitory memory, and one or more programs. In some embodiments, the one or more programs are stored in the non-transitory memory and executed by one or more processors. In some embodiments, the one or more programs include instructions for performing or causing to perform any of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to perform or cause to perform any of the methods described herein. According to some embodiments, a device includes one or more processors, non-transitory memory, and means for performing or causing to perform any of the methods described herein. Detailed Implementation

[0013] Numerous details have been described to provide a thorough understanding of the exemplary embodiments illustrated in the accompanying drawings. However, the drawings illustrate only some exemplary aspects of this disclosure and should not be considered limiting. Those skilled in the art will understand that other effective aspects and / or variations do not include all the specific details described herein. Furthermore, well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure further relevant aspects of the exemplary embodiments described herein.

[0014] A physical environment refers to the physical world that people can sense and / or interact with without the aid of electronic devices. A physical environment can include physical features such as physical surfaces or physical objects. For example, a physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and / or interact with a physical environment through senses such as sight, touch, hearing, taste, and smell. Conversely, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via electronic devices. For example, an XR environment can include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, etc. In the case of an XR system, a subset of a person's physical motion or a representation thereof is tracked, and in response, one or more features of one or more virtual objects simulated in the XR system are adjusted in a manner consistent with at least one physical law. For example, an XR system can detect head movement and, in response, adjust the graphical content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. For example, an XR system can detect movement of electronic devices (e.g., mobile phones, tablets, laptops, etc.) that present the XR environment, and in response, adjust the graphical content and sound field presented to the user in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), an XR system can adjust the characteristics of the graphical content in the XR environment in response to representations of physical motion (e.g., voice commands).

[0015] Many different types of electronic systems enable people to sense and / or interact with a variety of XR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays shaped like lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop / laptop computers. Head-mounted systems may have an integrated opaque display and one or more speakers. Alternatively, head-mounted systems may be configured to receive external opaque displays (e.g., smartphones). Head-mounted systems may incorporate one or more imaging sensors for capturing images or video of the physical environment, and / or one or more microphones for capturing audio of the physical environment. Head-mounted systems may have transparent or semi-transparent displays instead of opaque displays. Transparent or semi-transparent displays may have a medium through which light representing the image is directed to the person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, holographic medium, optical combiner, optical reflector, or any combination thereof. In some implementations, transparent or translucent displays can be configured to selectively become opaque. Projection-based systems can employ retinal projection technology, which projects graphic images onto the human retina. Projection systems can also be configured to project virtual objects onto a physical environment, such as as holograms or on a physical surface.

[0016] Some devices utilize gaze as input. Such devices include an image sensor and a gaze tracker. The image sensor captures one or more images of the user. The gaze tracker tracks the user's gaze by identifying pixels corresponding to the user's pupils. The gaze tracker determines the gaze direction based on the pixels corresponding to the pupils. The gaze tracker is calibrated so that it accurately tracks the user's gaze in a reliable manner. Calibration may need to be adjusted while the device is in use so that the gaze tracker continues to accurately track the gaze. For example, a head-mounted device with an eye-tracking camera may slide or move around on the user's head during use. In this example, the eye position relative to the eye-tracking camera may change when using the head-mounted device. Too much change in eye position relative to the eye-tracking camera can lead to inaccurate gaze tracking.

[0017] This disclosure provides a method, system, and / or apparatus for adjusting the calibration of a gaze tracker while using the gaze tracker, so that the gaze tracker continues to accurately track a user's gaze. The device adjusts the gaze tracker's calibration as a background operation while performing other tasks. The device adjusts the gaze tracker's calibration based on the difference between the expected gaze position and the measured gaze position. If the difference between the expected gaze position and the measured gaze position exceeds an acceptable threshold, the device adjusts the calibration parameters for the gaze position. The device adjusts the calibration parameters such that the difference between subsequent expected gaze positions and corresponding subsequent measured gaze positions is within the acceptable threshold.

[0018] The device can determine the expected gaze location based on user input. As an example, the device expects the user to gaze at a button when activated by pressing it (e.g., via a physical input device such as a mouse, keyboard, touchpad, or touchscreen), performing a gesture while gazing at the button, gazing at the button for a threshold duration, or via voice input (e.g., by saying "send" or "send message") (e.g., the "send" button in a messaging application). In this example, if the gaze tracker indicates that the user is gazing 10 pixels away from the button or 10 pixels away from the expected portion of the button when the button is activated by pressing, gesturing, gazing, or via voice input, the device determines that the gaze tracker may be generating an incorrect gaze target. Thus, the device adjusts the gaze tracker's calibration parameters based on this 10-pixel error. As another example, if the device is displaying a text string that the user expects to gaze at and the gaze tracker indicates that the user is gazing at a blank area 15 pixels away from the text string, the device determines that the gaze tracker may need to be recalibrated, and the device adjusts the gaze tracker's calibration parameters based on the 15-pixel error.

[0019] The calibration parameters can vary depending on the user's eye position relative to the device's image sensor. Since the gaze tracker uses the values ​​of the calibration parameters to generate a measured gaze target, the measured gaze target changes with the eye's position relative to the image sensor. If the device is a head-mounted device and it moves while worn on the user's head, the eye's position relative to the image changes. Adjusting the calibration parameters compensates for this movement of the head-mounted device on the user's head. The calibration parameters may include values ​​indicating the eye's position relative to the image sensor. Adjusting the calibration parameters may include changing those values ​​to new values ​​indicating the new position of the eye relative to the image sensor.

[0020] Adjusting calibration parameters while using the device reduces the need for dedicated recalibration operations. For example, adjusting calibration parameters as a background operation reduces the need for recalibration performed as a foreground operation, where the device guides the user to perform certain actions to recalibrate the gaze tracker. As an example, adjusting calibration parameters during regular device use reduces the need for guided recalibration operations, where the device prompts the user to gaze at a specific visual element and recalibrates the gaze tracker based on the difference between the measured gaze position and the position of the specific visual element.

[0021] Figure 1A This is an illustration of an example physical environment 10 according to some specific implementations. Although relevant features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure further relevant aspects of the exemplary specific implementations disclosed herein. For this purpose, as a non-limiting example, physical environment 10 includes electronic device 20 and user 22 of electronic device 20.

[0022] In some embodiments, electronic device 20 includes a handheld computing device that can be held by user 22. For example, in some embodiments, electronic device 20 includes a smartphone, tablet, media player, laptop, desktop computer, etc. In some embodiments, electronic device 20 includes a wearable computing device that can be worn by user 22. For example, in some embodiments, electronic device 20 includes a head-mounted device (HMD) or electronic watch. In various embodiments, electronic device 20 includes an image sensor 24 that captures images of at least one eye of user 22, a gaze tracker 26 that tracks the gaze of user 22 based on the images captured by image sensor 24, and a display that presents a graphical environment 40 (e.g., a GUI with various graphical user interface (GUI) elements). In some embodiments, electronic device 20 includes or is connected to a physical input device, such as a mouse, keyboard, touch-sensitive surface (e.g., touchpad), clicker device, etc. In some embodiments, the display includes a touchscreen display that can detect user input (e.g., tap input, long press input, drag input, etc.).

[0023] In some embodiments, electronic device 20 includes a smartphone or tablet, and image sensor 24 includes a front-facing camera that can capture images of user 22's eyes while user 22 is using electronic device 20. In some embodiments, electronic device 20 includes an HMD (Head-Down Device), and image sensor 24 includes a user-facing camera that captures images of user 22's eyes when the HMD is worn on user 22's head. In some embodiments, electronic device 20 is a laptop computer that includes a touch-sensitive surface (e.g., a touchpad) for receiving user input from user 22. In some embodiments, the laptop computer is connected to a separate physical input device for receiving user input from user 22, such as a mouse, touchpad, or keyboard. In some embodiments, electronic device 20 is a desktop computer that is connected to a separate physical input device (e.g., a mouse, touchpad, or keyboard) for receiving user input from user 22.

[0024] In some implementations, gaze tracker 26 acquires a set of one or more images captured by image sensor 24. Gaze tracker 26 identifies pixels corresponding to the eyes and / or pupils of user 22. Gaze tracker 26 tracks the gaze of user 22 based on the pixels corresponding to the eyes and / or pupils of user 22. In some implementations, gaze tracker 26 generates a gaze target including a gaze location value, a gaze intensity value, and a gaze duration value. The gaze location value indicates the coordinates of a display area within the graphics environment 40 that user 22 is gazing at. The gaze intensity value indicates the number of pixels that user 22 is gazing at. The gaze duration value indicates the duration for which user 22 has gazed at the display area indicated by the gaze location value.

[0025] In various embodiments, electronic device 20 calibrates gaze tracker 26 such that gaze tracker 26 reliably and accurately tracks the gaze of user 22. In some embodiments, gaze tracker 26 is associated with a set of one or more calibration parameters 28 (hereinafter referred to as "calibration parameters 28"). In such embodiments, calibrating gaze tracker 26 includes setting the values ​​of calibration parameters 28. Figure 1A In the example, calibration parameter 28 has a first value 30. In some implementations, the first value 30 includes a default value. In some implementations, the first value 30 varies with the intended position of the user 22's eye relative to the image sensor 24. For example, the first value 30 may correspond to the user 22's eye being aligned with the image sensor 24 (e.g., the eye intersecting the axis at the center of the image sensor 24's field of view). In other words, the first value 30 may indicate that the eye is at the center of the image sensor 24's field of view.

[0026] In some implementations, the graphics environment 40 includes a two-dimensional (2D) environment. In some implementations, the graphics environment 40 includes a three-dimensional (3D) environment, such as an XR environment. Figure 1A In the example, the graphical environment 40 includes various visual elements 50 (e.g., a first visual element 50a, a second visual element 50b, a third visual element 50c, and a fourth visual element 50d). In some embodiments, visual elements 50 include graphical objects (e.g., XR objects). In some embodiments, visual elements 50 include selectable functional representations (e.g., buttons) that the user 22 can select by providing user input (e.g., touch input via a touchpad or touchscreen, mouse click input via a mouse, key press input via a keyboard, gaze input, voice input via a microphone, etc.). In some embodiments, visual elements 50 include text 52 (e.g., a brief description of the function of the first visual element 50a). In some embodiments, visual elements 50 include graphics 54 (e.g., images, such as visual indications of the function of the second visual element 50b).

[0027] refer to Figure 1B In some specific implementations, visual element 50 is associated with a corresponding feature value 60. For example, first visual element 50a is associated with first feature value 60a, second visual element 50b is associated with second feature value 60b, third visual element 50c is associated with third feature value 60c, and fourth visual element 50d is associated with fourth feature value 60d.

[0028] like Figure 1B As shown, in some specific implementations, electronic device 20 determines the intended gaze target 70 based on feature value 60. Figure 1B In the example, the expected gaze target 70 indicates the expected gaze position 72 corresponding to the third visual element 50c. The expected gaze target 70 indicates the display area that the user 22 expects to gaze at. Figure 1B In the example, user 22 is expected to gaze at third visual element 50c because the expected gaze position 72 coincides with the position of third visual element 50c.

[0029] In some implementations, feature value 60 includes a corresponding saliency value for visual element 50, and electronic device 20 determines the expected gaze target 70 based on this saliency value. For example, electronic device 20 selects the position of third visual element 50c as the expected gaze position 72 because third visual element 50c has the highest saliency value among visual elements 50. In some implementations, electronic device 20 obtains (e.g., generates or receives) a saliency map of graphical environment 40, and electronic device 20 retrieves the saliency value of visual element 50 from this saliency map.

[0030] In some implementations, feature value 60 includes a corresponding position value for visual element 50, and electronic device 20 determines the intended gaze target 70 based on this position value. For example, electronic device 20 may select the position of a specific visual element 50 as the intended gaze position 72 because the specific visual element 50 has a position value within a threshold range of the intended gaze position value of user 22 (e.g., the specific visual element 50 is positioned near the center of the display area of ​​electronic device 20).

[0031] In some implementations, feature value 60 includes a corresponding color value for visual element 50, and electronic device 20 determines the intended gaze target 70 based on this color value. For example, electronic device 20 may select the location of a specific visual element 50 as the intended gaze location 72 because the specific visual element 50 has a color value that matches a threshold color value that is intended to attract the gaze of user 22 (e.g., the specific visual element 50 is red, while the remainder of the display area of ​​electronic device 20 is black; or the specific visual element 50 is displayed in color, while the remainder of the display area is displayed in black and white).

[0032] In some implementations, feature value 60 includes a corresponding movement value for visual element 50, and electronic device 20 determines the intended gaze target 70 based on this movement value. For example, electronic device 20 may select the position of a specific visual element 50 as the intended gaze position 72 because the specific visual element 50 has a movement value that matches a threshold movement value that is intended to attract the gaze of user 22 (e.g., the specific visual element 50 is moving while other visual elements 50 are stationary).

[0033] In some implementations, feature value 60 includes a corresponding user interaction value for visual element 50, and electronic device 20 determines the intended gaze target 70 based on this user interaction value. In some implementations, the user interaction value indicates the corresponding level of interaction with visual element 50 based on current and / or historical user input provided by user 22. As an example, electronic device 20 may select the location of a specific visual element 50 as the intended gaze location 72 because the specific visual element 50 has a user interaction value that matches a threshold interaction value, and user 22 is more likely to interact with (e.g., select) that specific visual element 50 (e.g., user 22 is more likely to select that specific visual element 50 because that specific visual element 50 is selected the most times among visual elements 50).

[0034] In some implementations, the expected gaze target 70 is associated with a confidence value that indicates the level of confidence (e.g., degree of certainty) associated with the expected gaze target 70. In some implementations, the confidence value varies with the eigenvalue 60. In some implementations, the confidence value varies with the variance of the eigenvalue 60 (e.g., is proportional to it). As an example, if the third eigenvalue 60c is the highest among the eigenvalues ​​60, and the difference between the third eigenvalue 60c and the second highest eigenvalue 60 is greater than a threshold difference, then the confidence value associated with the expected gaze target 70 can be set to a value greater than the threshold confidence value (e.g., the confidence value can be set to a value greater than 0.5, for example, the confidence value can be set to "1"). In this example, if the difference between the highest and second highest eigenvalues ​​in eigenvalue 60 is less than a threshold difference, the confidence value associated with the expected gaze target 70 can be set to a value less than the threshold confidence value (e.g., the confidence value can be set to a value less than 0.5, for example, the confidence value can be set to 0.2).

[0035] In some embodiments, the expected gaze position 72 corresponds to the location indicated by user input received via a physical input device. In some embodiments, the electronic device 20 detects user input at a specific location within the graphics environment 40 via a physical input device, and the electronic device 20 sets the specific location of the user input as the expected gaze position 72. In some embodiments, in response to detecting a mouse click via a mouse, the electronic device 20 sets the cursor position of the cursor as the expected gaze position 72. For example, in response to detecting a mouse click while the cursor is positioned over a third visual element 50c, the electronic device 20 may set the position (e.g., center) of the third visual element 50c as the expected gaze position 72. In some embodiments, in response to detecting a tap or press via a touch-sensitive surface such as a touchpad or touchscreen display, the electronic device 20 sets the cursor position of the cursor as the expected gaze position 72. For example, in response to detecting a tap or press via a touchpad or touchscreen display, the electronic device 20 may set the position (e.g., center) of the third visual element 50c as the expected gaze position 72.

[0036] In some implementations, in response to detecting a key press via the keyboard, electronic device 20 sets the position of the focus element to the expected gaze position 72. For example, when the focus element is on the third visual element 50c, in response to detecting a press of the Enter key, electronic device 20 may set the position (e.g., center) of the third visual element 50c to the expected gaze position 72. In some implementations, electronic device 20 sets the position of the focus element to the expected gaze position 72 in response to detecting voice input corresponding to a selection command. For example, when the focus element is on the third visual element 50c (e.g., when user 22 says "select"), electronic device 20 may set the position of the third visual element 50c to the expected gaze position 72 in response to detecting a selection voice command.

[0037] refer to Figure 1C Image sensor 24 captures image 78 of user 22, and gaze tracker 26 uses image 78 to generate measured gaze target 80. In some embodiments, measured gaze target 80 indicates measured gaze position 82. Measured gaze position 82 represents the display area that gaze tracker 26 has identified as corresponding to the gaze of user 22. In some embodiments, measured gaze target 80 includes measured gaze intensity, which indicates the number of pixels that gaze tracker 26 has identified as corresponding to the gaze of user 22. In some embodiments, measured gaze target 80 includes measured gaze duration, which indicates the duration that gaze tracker 26 has identified as corresponding to the gaze of user 22. Figure 1C As can be seen, the measured gaze position 82 may differ from the expected gaze position 72.

[0038] Return to reference Figure 1B In some embodiments, when a specific visual element 50 has a position value within a threshold range of the position value indicated by the measured gaze target 80, the electronic device 20 selects the position of the specific visual element 50 as the expected gaze position 72. In some embodiments, when the visual element 50 has a position value closest to the position indicated by the measured gaze target 80, the electronic device 20 selects the position of the specific visual element 50 as the expected gaze position 72. In some embodiments, when the measured gaze position 82 remains stationary (or below a threshold movement amount) for a threshold time length, and when the visual element 50 has a position value closest to the position indicated by the measured gaze target 80, the electronic device 20 selects the position of the specific visual element 50 as the expected gaze position 72. In some embodiments, when a selection input is received, when the visual element 50 has a position value closest to the position indicated by the measured gaze target 80, the electronic device 20 selects the position of the specific visual element 50 as the expected gaze position 72.

[0039] refer to Figure 1DIn various specific implementations, electronic device 20 determines the expected gaze target 70 ( Figure 1B (shown in) and Figure 1C and Figure 1D The differences between the measured gaze targets 80 are shown in the figure. Figure 1D In the example, electronic device 20 identifies a difference 90 between the expected gaze position 72 and the measured gaze position 82. This difference 90 may be caused by movement of electronic device 20 relative to the user 22's eye. For example, difference 90 may be caused by the eye no longer being in the center of the image sensor 24's field of view. After identifying difference 90, electronic device 20 determines a new value 32 for calibration parameter 28 (e.g., different from...). Figure 1C (The first value 30 shown is a second value). In some specific implementations, the new value 32 varies with the difference 90. The new value 32 can compensate for the eye not being in the center of the field of view of the image sensor 24. The electronic device 20 replaces the first value 30 with the new value 32.

[0040] Figure 1E This corresponds to the time period that occurs after the electronic device 20 sets the calibration parameter 28 to a new value 32. After setting the calibration parameter 28 to the new value 32, the image sensor 24 captures another image 98, and the gaze tracker 26 generates another measured gaze target 100 based on this other image 98. The measured gaze target 100 indicates another measured gaze position 102 that coincides with the expected gaze position 72. Setting the calibration parameter 28 to the new value 32 reduces (e.g., makes it smaller or eliminates) the gaze target 100. Figure 1D The difference shown is 90, and the accuracy of gaze tracker 26 is improved. (As shown) Figures 1C to 1E As shown, when electronic device 20 is performing non-calibration-related operations, electronic device 20 adjusts calibration parameter 28. In Figures 1C to 1E In the example, the adjustment of calibration parameter 28 is performed as a background operation rather than a foreground operation in order to reduce disruption to the operability of electronic device 20.

[0041] Advantageously, electronic device 20 adjusts calibration parameters 28 without displaying prompts to user 22 to adjust the position of electronic device 20 above his / her head. For example, electronic device 20 does not request user 22 to move electronic device 20 so that user 22's eyes are centered in the field of view of image sensor 24. Furthermore, electronic device 20 adjusts calibration parameters 28 without performing guided calibration, which may include prompting user 22 to look at specific visual elements 50 in order to adjust calibration parameters 28. The aforementioned presentation of guided calibration reduces disruption to the operability of electronic device 20, thereby increasing the usability of electronic device.

[0042] Figure 1F and Figure 1GThe electronic device 20 is shown to determine the intended gaze target 112 based on user input 110 provided by user 22. Figure 1G The sequence shown in the image. Figure 1F As shown, electronic device 20 detects user input 110 at a location corresponding to the fourth visual element 50d. In some embodiments, electronic device 20 includes a touchscreen display, and electronic device 20 detects user input 110 by detecting a tap on the touchscreen display. Alternatively, in some embodiments, electronic device 20 displays the graphical environment 40 as a virtual plane, and electronic device 20 detects user input 110 by detecting the intersection between a collider object representing user 22's finger (e.g., a finger) and the virtual plane of the graphical environment 40. For example, electronic device 20 detects user input 110 by detecting a 3D gesture performed by user 22. In some embodiments, electronic device 20 detects user input 110 by detecting voice input. For example, user 22 may utter a phrase corresponding to a request to select the fourth visual element 50d (e.g., user 22 may say "select the bottom right option"). In some embodiments, electronic device 20 detects user input 110 via a physical input device (e.g., a mouse, keyboard, touch-sensitive surface (e.g., touchpad), or clicker device). In some embodiments, the physical input device is connected to the electronic device 20 via a wire. Alternatively, in some embodiments, the physical input device provides instructions for user input 110 to the electronic device 20 via wireless communication.

[0043] In some implementations, feature value 60 indicates whether the corresponding visual element 50 has been selected. For example, feature value 60 may include a binary value, where a value of "0" indicates that the corresponding visual element 50 has not been selected, while a value of "1" indicates that the corresponding visual element 50 has been selected. Figure 1F In the example, the fourth feature value 60d may have a binary value "1" to indicate that the fourth visual element 50d has been selected, while the first feature value 60a, the second feature value 60b, and the third feature value 60c may have a binary value "0" to indicate that the first visual element 50a, the second visual element 50b, and the third visual element 50c have not been selected.

[0044] refer to Figure 1G In response to user input 110 detecting selection of fourth visual element 50d, electronic device 20 generates a anticipated gaze target 112, which indicates a anticipated gaze position 114 corresponding to the fourth visual element 50d. Figure 1G As can be seen, the expected gaze position 114 indicates that when user 22 is selecting the fourth visual element 50d, user 22 is expected to gaze at the fourth visual element 50d. Figure 1G In the example, if the difference between the measured gaze position and the expected gaze position 114 is greater than a threshold, the electronic device 20 adjusts... Figure 1A Calibration parameters 28 for the gaze tracker 26 shown.

[0045] Figures 1H to 1J The electronic device 20 is shown to determine the sequence of expected gaze targets based on the movement of specific visual elements 50. Figure 1H A fifth visual element 50e is shown moving in the direction indicated by arrow 120. For example, the fifth visual element 50e moves to the right of the graphics environment 40. In some embodiments, feature value 60 indicates the movement of the corresponding visual element 50. For example, feature value 60 may indicate the corresponding speed at which the corresponding visual element 50 is moving. Figure 1H In the example, visual elements 50a to 50d are stationary. Thus, feature values ​​60a to 60d can indicate a movement speed of 0. However, since the fifth visual element 50e is moving, the fifth feature value 60e can indicate the speed at which the fifth visual element 50e is moving. Alternatively, in some implementations, feature value 60 includes a binary value, where a value "0" indicates no movement and a value "1" indicates movement.

[0046] like Figure 1H As shown, the electronic device 20 determines a first expected gaze position 130a corresponding to the position of the fifth visual element 50e. In Figure 1H In the example, electronic device 20 selects the position of the fifth visual element 50e as the first expected gaze position 130a because feature value 60 indicates that the fifth visual element 50e is moving while the remaining visual elements 50a to 50d are stationary. Figure 1H As shown, electronic device 20 determines a first measured gaze position 140a, which is offset by a difference 150 from a first expected gaze position 130a. The difference 150 between the first expected gaze position 130a and the first measured gaze position 140a may be caused by changes in the position of the user 22's eyes relative to the image sensor 24. For example, in some embodiments, electronic device 20 includes a head-mounted device worn by the user 22 on his / her head, and the difference 150 may be due to the electronic device 20 sliding on the user 22's head as the user 22 moves.

[0047] refer to Figure 1I As the fifth visual element 50e moves across the graphics environment 40 in the direction indicated by arrow 120, the electronic device 20 can generate additional expected gaze targets and additional measured gaze targets to determine whether to adjust the calibration parameters 28 of the gaze tracker 26. For example, the electronic device 20 determines a second expected gaze position 130b corresponding to the new position of the fifth visual element 50e. Figure 1IIn the example, the previous position of the fifth visual element 50e is indicated by the dashed box 160. The electronic device 20 determines a second measured gaze position 140b, offset by a difference 150 from the second expected gaze position 130b. Due to the first measured gaze position 140a ( Figure 1H (as shown in the image) from the first expected gaze position 130a ( Figure 1H As shown in the figure, the difference 150 was offset and the second measured gaze position 140b was offset from the second expected gaze position 130b by a similar or identical difference 150, so the electronic device 20 determined to change the calibration parameter 28 of the gaze tracker 26 with a greater degree of certainty.

[0048] like Figure 1J As shown, the electronic device 20 adjusts the calibration parameter 28 by changing its value from a first value 30 to a third value 34. In some embodiments, the third value 34 varies with the difference 150 between the expected gaze positions 130a and 130b and the corresponding measured gaze positions 140a and 140b. For example, in some embodiments, the difference between the first value 30 and the third value 34 is related to... Figure 1H and Figure 1I The difference shown is proportional to 150.

[0049] Figure 1J The new position of the fifth visual element 50e, the third expected gaze position 130c, and the third measured gaze position 140c are shown. Figure 1J In the example, the previous position of the fifth visual element 50e is indicated by another dashed box 162. The electronic device 20 determines the third measured gaze position 140c after adjusting the calibration parameter 28. (As...) Figure 1J As can be seen, the third expected gaze position 130c and the third measured gaze position 140c are juxtaposed. In other words, the third measured gaze position 140c matches the third expected gaze position 130c. Since the gaze tracker 26 determines the third measured gaze position 140c after setting the calibration parameter 28 to the third value 34, the third measured gaze position 140c coincides with the third expected gaze position 130c. Therefore, the third measured gaze position 140c does not deviate from the third expected gaze position 130c.

[0050] Figure 2 It involves adjusting the calibration parameters of the gaze tracker according to certain specific implementations (e.g., Figure 1A The block diagram shows a system 200 for the calibration parameters 28 of the gaze tracker 26. In some embodiments, system 200 includes an expected gaze determiner 210, a measured gaze determiner 230, and a calibration parameter adjuster 250. In various embodiments, system 200 resides in... Figures 1A to 1J The electronic device shown is located at 20 locations (e.g., implemented therein).

[0051] In various specific implementations, the gaze determiner 210 is expected to acquire (e.g., receive or determine) feature values ​​220 associated with the corresponding visual element being displayed on the display (e.g., Figure 1B The characteristic value shown is 60). Figure 2 As shown, the anticipated gaze determiner 210 determines the anticipated gaze target 212 based on feature value 220. For example, the anticipated gaze determiner 210 determines... Figure 1B The intended gaze target 70 is shown. In some embodiments, the intended gaze target 212 includes the intended gaze location 212a (e.g., Figure 1B The expected gaze position 72, expected gaze intensity 212b, and / or expected gaze duration 212c are shown. In some embodiments, the expected gaze determiner 210 determines the expected gaze target 212 such that the expected gaze position 212a corresponds to the visual element with the maximum eigenvalue 220.

[0052] In some implementations, the anticipatory gaze determiner 210 determines a confidence score associated with the anticipatory gaze target 212. The confidence score indicates the degree of certainty in the anticipatory gaze target 212. In some implementations, the confidence score varies with the eigenvalue 220. For example, the confidence score may be based on the distribution of the eigenvalue 220. As an example, if the eigenvalue 220 has a large variance, the confidence score may be high. Conversely, if the eigenvalue 220 has a low variance, the confidence score may be low.

[0053] In some embodiments, feature value 220 includes a saliency value 220a indicating the corresponding saliency level of a visual element. In some embodiments, the saliency value 220a is based on the corresponding saliency of the visual element (e.g., a more salient visual element has a larger saliency value 220a compared to a less salient visual element). In some embodiments, the saliency value 220a is based on the corresponding obviousness of the visual element (e.g., a more obvious visual element has a larger saliency value 220a compared to a less obvious visual element). In some embodiments, the anticipated gaze determiner 210 obtains (e.g., receives or generates) a saliency map including the saliency value 220a. In some embodiments, the anticipated gaze determiner 210 determines the anticipated gaze target 212 such that the anticipated gaze location 212a corresponds to the visual element with the maximum saliency value 220a.

[0054] In some implementations, feature value 220 includes a position value 220b indicating the corresponding location of a visual element. In some implementations, a user is more likely to gaze at a specific location. For example, a user may be more likely to gaze at a visual element positioned towards the center of the display area. In this example, the anticipated gaze determiner 210 may generate an anticipated gaze target 212 such that the anticipated gaze position 212a points to a visual element near the center of the display area. More generally, in various implementations, the anticipated gaze determiner 210 generates an anticipated gaze target 212 such that the anticipated gaze position 212a corresponds to a visual element located within a portion of the display area where the user is more likely to gaze. In some implementations, the anticipated gaze determiner 210 identifies portions of the display area where the user is more likely to gaze based on historical gaze tracking data. For example, if historical gaze tracking data indicates that the user spends more time gazing at a specific portion of the display area (e.g., the center or upper right), the anticipated gaze determiner 210 determines that the specific portion of the display area is the portion the user is more likely to gaze at.

[0055] In some embodiments, feature value 220 includes a color value 220c indicating the corresponding color of the visual element. In some embodiments, a user is more likely to gaze at a colored visual element and less likely to gaze at a black-and-white visual element. More generally, in various embodiments, a user is more likely to gaze at certain colors (e.g., bright colors such as red and blue) and less likely to gaze at other colors (e.g., dark colors such as gray). In some embodiments, the anticipated gaze determiner 210 generates an anticipated gaze target 212 such that the anticipated gaze position 212a points to the location of a visual element having a color value 220c that matches a threshold color value (e.g., a preferred color, such as a bright color such as red or blue).

[0056] In some embodiments, feature value 220 includes a movement value 220d indicating the corresponding movement of a visual element. In some embodiments, movement value 220d includes a binary value indicating whether the corresponding visual element is moving (e.g., "0" indicates stationary, and "1" indicates movement). In some embodiments, the anticipated gaze determiner 210 determines that the user is more likely to gaze at a moving visual element and less likely to gaze at a stationary visual element. Therefore, in some embodiments, the anticipated gaze determiner 210 generates an anticipated gaze target 212 such that the anticipated gaze position 212a points to a visual element having a movement value 220d indicating movement (e.g., having a movement value 220d "1"). In some embodiments, movement value 220d includes a movement speed. In some embodiments, the anticipated gaze determiner 210 determines that the user is more likely to gaze at a visual element that is moving rapidly and less likely to gaze at a visual element that is moving slowly. Therefore, in some specific implementations, the expected gaze determiner 210 generates an expected gaze target 212 such that the expected gaze position 212a points to a visual element with a maximum movement value 220d.

[0057] In some implementations, feature value 220 includes an interaction value 220e indicating current and / or historical user interaction with a visual element. In some implementations, interaction value 220e includes a binary value indicating whether the user is currently interacting with the corresponding visual element. For example, interaction value 220e "1" may indicate that the device has detected user input selecting the corresponding visual element, and interaction value 220e "0" may indicate that the device has not detected user input selecting the corresponding visual element. In some implementations, the anticipated gaze determiner 210 determines that the user is more likely to gaze at a visual element that the user is currently interacting with, and less likely to gaze at a visual element that the user is not currently interacting with. Therefore, in some implementations, the anticipated gaze determiner 210 generates an anticipated gaze target 212 such that the anticipated gaze position 212a points to a visual element that the user has selected via user input (e.g., click input via mouse, key press input via keyboard, touch input via touchpad or touchscreen, gesture input via touchscreen or 3D gesture tracker, voice input via microphone, etc.). As an example, see [reference]. Figure 1G The expected gaze determiner 210 generates an expected gaze position 114 pointing to the fourth visual element 50d in response to user input 110 that selects the fourth visual element 50d.

[0058] In some implementations, the interaction value 220e indicates historical user interactions with the corresponding visual element. In some implementations, historical user interactions include historical user input for the visual element. In some implementations, the anticipated gaze determiner 210 determines that visual elements with more historical user interactions are more likely to be gazed upon than visual elements with fewer historical user interactions. Therefore, in some implementations, the anticipated gaze determiner 210 generates an anticipated gaze target 212 such that the anticipated gaze position 212a coincides with the position of the visual element with the maximum interaction value 220e.

[0059] In various specific implementations, the measured gaze determiner 230 acquires one or more images 232 of the user's eyes, and the measured gaze determiner 230 determines the measured gaze target 234 based on the images 232. In some specific implementations, the measured gaze determiner 230 receives the images 232 from an image sensor (e.g., a user-facing camera, such as an eye-tracking camera) that captures the images 232. In some specific implementations, the measured gaze determiner 230 implements... Figure 1A The gaze tracker 26 is shown. In some implementations, the measured gaze determiner 230 includes a set of one or more calibration parameters 240 (e.g., Figure 1A The calibration parameters shown are 28). Figure 2 In the example, one or more calibration parameters 240 in this group have an existing value 242. For example, as... Figure 1A As shown, calibration parameter 28 has a first value of 30.

[0060] In various implementations, the measured gaze determiner 230 determines the measured gaze target 234 based on an image 232 of the user's eyes. In some implementations, the measured gaze target 234 includes a measured gaze location 234a (e.g., Figure 1C The measured gaze position 82, measured gaze intensity 234b, and / or measured gaze duration 234c are shown. In some embodiments, the measured gaze target 234 varies with the existing value 242 of the calibration parameter 240. Thus, changing the value of the calibration parameter 240 triggers a change in the measured gaze target 234. In some embodiments, the measured gaze determiner 230 provides the measured gaze target 234 to the calibration parameter adjuster 250.

[0061] In some embodiments, the anticipated gaze determiner 210 generates an anticipated gaze target 212 such that the anticipated gaze position 212a points to a visual element that is close to or closest to the position indicated by the measured gaze target 234. In some embodiments, the anticipated gaze determiner 210 generates an anticipated gaze target 212 such that when the measured gaze target 234 remains stationary for a threshold time length (or less than a threshold movement amount), the anticipated gaze position 212a points to a visual element that is close to or closest to the position indicated by the measured gaze target 234. In some embodiments, the anticipated gaze determiner 210 generates an anticipated gaze target 212 such that when a selection input (e.g., touch input, gesture input, voice input, etc.) is received, the anticipated gaze position 212a points to a visual element that is close to or closest to the position indicated by the measured gaze target 234.

[0062] In some implementations, the calibration parameter adjuster 250 adjusts the calibration parameter 240 based on a comparison between the expected gaze target 212 and the measured gaze target 234. In some implementations, when the difference between the expected gaze target 212 and the measured gaze target 234 exceeds a threshold difference, the calibration parameter adjuster 250 generates a new value 252 for the calibration parameter 240. (See reference) Figure 1D The calibration parameter adjuster 250 changes the value of calibration parameter 28 from a first value 30 to a new value 32 based on the difference 90 between the expected gaze position 72 and the measured gaze position 82. In some embodiments, the calibration parameter adjuster 250 generates a new value 252 for calibration parameter 240 to reduce the difference between subsequent expected gaze targets and corresponding subsequent measured gaze targets. In some embodiments, the calibration parameter adjuster 250 generates the new value 252 as a background operation while the device continues to perform other operations (e.g., non-calibration-related operations, such as displaying visual content).

[0063] In some implementations, the new value 252 varies with the difference between the expected gaze target 212 and the measured gaze target 234. In some implementations, the difference between the existing value 242 and the new value 252 is proportional to the difference between the expected gaze target 212 and the measured gaze target 234. For example, the greater the difference between the expected gaze target 212 and the measured gaze target 234, the greater the difference between the existing value 242 and the new value 252. In some implementations, the calibration parameter adjuster 250 uses a lookup table (LUT) to determine the new value 252. For example, the LUT may list various values ​​representing the difference in change between the expected gaze target 212 and the measured gaze target 234, and the corresponding changes to be made to the existing value 242.

[0064] In some implementations, the calibration parameter adjuster 250 obtains a confidence score associated with the expected gaze target 212. For example, the expected gaze determiner 210 provides the calibration parameter adjuster 250 with the expected gaze target 212 and the confidence score associated with it. In some implementations, in response to a confidence score greater than a threshold confidence score, the calibration parameter adjuster 250 generates a new value 252 for the calibration parameter 240. In some implementations, in response to a confidence score less than a threshold confidence score, the calibration parameter adjuster 250 abandons the generation of the new value 252 for the calibration parameter 240.

[0065] Figure 3 This is a flowchart illustrating a method 300 for adjusting calibration parameters of a gaze tracker. In various specific embodiments, method 300 comprises a device including a display, an image sensor, non-transitory memory, and one or more processors coupled to the display, image sensor, and non-transitory memory (e.g., Figures 1A to 1J The electronic device 20 and / or shown Figure 2 The method 300 is executed by the system 200 shown. In some embodiments, the method 300 is executed by processing logic components, including hardware, firmware, software, or a combination thereof. In some embodiments, the method 300 is executed by a processor that executes code stored in a non-transitory computer-readable medium (e.g., memory).

[0066] As shown in box 310, in various specific embodiments, method 300 includes displaying multiple visual elements on a display. For example, as Figure 1A As shown, electronic device 20 displays visual element 50. In some specific embodiments, the visual element includes text (e.g., Figure 1A The text shown in 52) and / or the image shown (e.g., Figure 1A (See Figure 54). In some implementations, visual elements include GUI elements that are part of the GUI. For example, visual elements include selectable display representations (e.g., buttons) that can be selected by the user of the device.

[0067] As illustrated in box 320, in various specific embodiments, method 300 includes determining an expected gaze target for a first display area based on corresponding feature values ​​of the plurality of visual elements, whereby a user of the device is expected to gaze at or intend to gaze at the first display area while the plurality of visual elements are being displayed. For example, as Figure 1B As shown, the electronic device 20 determines the expected gaze target 70 based on the feature value 60 of the visual element 50.

[0068] As shown in box 320a, in some embodiments, the corresponding feature values ​​include corresponding saliency values. In such embodiments, method 300 includes determining the expected gaze target based on the corresponding saliency value. For example, as Figure 2 As shown, in some implementations, feature value 220 includes saliency value 220a, and the expected gaze determiner 210 determines the expected gaze target 212 based on saliency value 220a. In some implementations, it is expected that the user will gaze at the most salient visual element. As an example, if the device is displaying a virtual character, it is expected that the user will gaze at the virtual character's eyes rather than the virtual character's knees. Therefore, in this example, the virtual character's eyes may have a larger saliency value than the virtual character's knees.

[0069] In some implementations, the corresponding feature values ​​include positional values ​​indicating the placement of the respective visual elements. In such implementations, method 300 includes determining the intended gaze target based on the positional values ​​of the visual elements. For example, as Figure 2 As shown, in some implementations, feature value 220 includes position value 220b, and the expected gaze determiner 210 determines the expected gaze target 212 based on position value 220b. As an example, it is expected that the user will gaze at a visual element located near the center of the display area rather than near the edge of the display area, which may be in the user's peripheral vision. In this example, the expected gaze target points to a visual element near the center of the display area. In other implementations, it is expected that the user will gaze at a visual element to select that visual element, or it is expected that the user will gaze at a visual element while a separate selection input (e.g., touch input, gesture input, voice input, etc.) is provided.

[0070] In some implementations, the corresponding feature values ​​include color values ​​indicating the corresponding colors of the plurality of visual elements. In such implementations, method 300 includes determining the intended gaze target based on the color values ​​of the visual elements. For example, as Figure 2 As shown, in some specific implementations, feature value 220 includes color value 220c, and the expected gaze determiner 210 determines the expected gaze target 212 based on color value 220c. As an example, it can be expected that the user will gaze at the most vibrant visual element. As another example, it can be expected that the user will gaze at a visual element with a specific color (e.g., red, blue, etc.).

[0071] In some implementations, the corresponding feature values ​​include movement values ​​indicating the speed at which the plurality of visual elements are moving. In such implementations, method 300 includes determining the intended gaze target based on the movement values ​​of the visual elements. For example, as Figure 2 As shown, in some specific implementations, feature value 220 includes movement value 220d, and the expected gaze determiner 210 determines the expected gaze target 212 based on movement value 220d. As an example, it can be expected that the user will gaze at a moving visual element rather than a stationary visual element. For example, refer to... Figures 1H to 1JIt is expected that user 22 will gaze at the moving fifth visual element 50e instead of the stationary visual elements 50a to 50d.

[0072] In some implementations, the corresponding feature values ​​include interaction values ​​that indicate the corresponding level of interaction between the user and the plurality of visual elements. In such implementations, method 300 includes determining the intended gaze target based on the interaction values ​​associated with the visual elements. For example, as Figure 2 As shown, feature value 220 includes interaction value 220e, and the expected gaze determiner 210 determines the expected gaze target 212 based on interaction value 220e. As another example, see reference... Figure 1G When the user inputs 110 and is directed to the fourth visual element 50d, it is expected that the user 22 will gaze at the fourth visual element 50d.

[0073] In some implementations, determining the intended gaze target includes detecting input directed towards a first display area via a physical input device, and setting the first display area as the intended gaze target in response to detecting the input directed towards the first display area. For example, refer to Figure 1F and Figure 1G The electronic device 20 detects user input 110 via a physical input device and sets the position indicated by user input 110 as the expected gaze position 114.

[0074] In some implementations, determining the intended gaze target includes detecting a mouse click when the first display area corresponds to the cursor position, and setting the cursor position to the intended gaze target in response to the detected mouse click. For example, see Reference Figure 1F and Figure 1G In some specific implementations, electronic device 20 detects user input 110 via mouse. For example, user 22 uses mouse to move the cursor to fourth visual element 50d and clicks or double-clicks the fourth visual element 50d.

[0075] In some implementations, determining the intended gaze target includes detecting a tap input via a touch-sensitive surface when the first display area corresponds to the cursor position, and setting the cursor position to the intended gaze target in response to the detected tap input. For example, see reference... Figure 1F and Figure 1G In some specific implementations, the electronic device 20 detects user input 110 via a touchpad. For example, the user 22 uses the touchpad to move the cursor to the fourth visual element 50d and clicks or double-clicks the fourth visual element 50d.

[0076] In some implementations, determining the intended gaze target includes detecting a key press via the keyboard when the first display area corresponds to the focus indicator, and setting the first display area as the intended gaze target in response to detecting the key press. For example, see reference... Figure 1F and Figure 1G In some implementations, the electronic device 20 detects user input 110 via a keyboard. For example, the user 22 uses the arrow keys on the keyboard to move the focus indicator to the fourth visual element 50d and presses the input key to select the fourth visual element 50d.

[0077] In some implementations, determining the intended gaze target includes detecting voice input, including a selection command, via an audio sensor when the first display area corresponds to a focus indicator, and setting the first display area as the intended gaze target in response to detecting the voice input including the selection command. For example, refer to Figure 1F and Figure 1G In some implementations, electronic device 20 detects user input 110 by receiving voice input via a microphone. For example, if the fourth visual element 50d represents an icon for a messaging application, user 22 can say “open the messaging application”.

[0078] As shown in box 330, in various specific embodiments, method 300 includes acquiring an image via an image sensor, the image including a set of pixels corresponding to the pupil of a user of the device. For example, as Figure 1C As shown, image sensor 24 captures an image 78 of at least one eye of user 22. In some embodiments, the image sensor includes a user-facing camera that captures images after obtaining informed consent from the user. In some embodiments, the image sensor includes an eye-tracking camera that captures images after obtaining informed consent from the user.

[0079] As shown in box 340, in various specific embodiments, method 300 includes determining a measured gaze target by a gaze tracker based on the set of pixels corresponding to the pupil, the measured gaze target indicating a second display area that the user is measuringly gazing at. For example, refer to... Figure 1C The gaze tracker 26 determines the measured gaze target 80, which indicates the measured gaze position 82 that the user 22 of the electronic device 20 is currently gazing at. As another example, see [reference needed]. Figure 2 The measured gaze determiner 230 determines the measured gaze target 234 based on the image 232. In some specific implementations, the second display area indicated by the measured gaze target is a certain distance away from the first display area indicated by the expected gaze target. As an example, Figure 1D The difference 90 between the expected gaze position 72 and the measured gaze position 82 is shown. As another example, the measured gaze target can indicate that the user is gazing at a blank area that is several pixels away from the expected gaze position corresponding to the eyes of the virtual character being displayed.

[0080] As shown in box 340a, in some implementations, the plurality of visual elements includes a moving element that the user is expected to gaze at, the expected gaze target corresponding to the position of the moving element, and the measured gaze target indicating a position offset from the position of the moving element. For example, as Figure 1H As shown, visual element 50 includes a fifth visual element 50e that moves in the direction indicated by arrow 120, a first expected gaze position 130a that overlaps with the position of the fifth visual element 50e, and a first measured gaze position 140a that is offset by a difference 150 from the first expected gaze position 130a.

[0081] As shown in box 340b, in some embodiments, the plurality of visual elements includes a selectable display representation that the user is expected to gaze at when selecting the selectable display representation, the expected gaze target corresponding to the location of the selectable display representation, and a measured gaze target indicating a location offset from the location of the selectable display representation. For example, as... Figure 1G As shown, user 22 selects the fourth visual element 50d, and the expected gaze position 114 overlaps with the position of the fourth visual element 50d. The actual gaze position is ( Figure 1G (Not shown in the image) does not overlap with the fourth visual element 50d.

[0082] As shown in box 350, in various specific embodiments, method 300 includes adjusting calibration parameters of the gaze tracker based on the difference between a first display area indicated by a desired gaze target and a second display area indicated by a measured gaze target. For example, as Figures 1C to 1E As shown, the electronic device 20 adjusts the calibration parameter 28 by changing its value from a first value 30 to a new value 32 based on the difference 90 between the expected gaze position 72 and the measured gaze position 82. In various specific embodiments, the device adjusts the calibration parameter such that subsequently captured images produce a measured gaze position that overlaps with the expected gaze position. For example, as... Figure 1E As shown, changing the value of calibration parameter 28 from a first value 30 to a new value 32 causes the measured gaze position 102 to overlap with the expected gaze position 72. As described herein, in various specific implementations, the adjustment of the calibration parameter is performed as a background operation without prompting the user to view certain visual elements as part of the guided calibration process. Adjusting the calibration parameter as a background operation reduces disruption to device operability, thereby increasing device usability.

[0083] As shown in box 350a, in some specific implementations, adjusting the calibration parameters includes adjusting the calibration parameters when the distance between the first display area and the second display area is greater than a threshold distance. For example, refer to Figures 1D to 1EIn response to a difference 90 between the expected gaze position 72 and the measured gaze position 82 that is greater than a threshold distance, the electronic device 20 adjusts the calibration parameter 28. In some embodiments, method 300 includes abandoning the adjustment of the calibration parameter when the distance between the first display area and the second display area is less than a threshold distance.

[0084] In some implementations, the adjustment of calibration parameters is proportional to the distance between the first and second display areas. For example, the greater the distance between the first and second display areas, the greater the adjustment of the calibration parameters. As an example, refer to... Figure 1D The adjustment of calibration parameter 28 is proportional to the difference 90 between the expected gaze position 72 and the measured gaze position 82.

[0085] As shown in box 350b, in some embodiments, adjusting the calibration parameters includes recording changes in the pupil position from a first expected position to a second expected position. In some embodiments, the device includes a head-mounted device that is expected to be worn by the user in a particular manner. For example, the user's eyes are expected to be centered in the field of view of the eye-tracking camera. However, when the user moves, the head-mounted device may slide, and the eyes may no longer be centered in the field of view of the eye-tracking camera. In some embodiments, adjusting the calibration parameters compensates for the movement of the head-mounted device on the user's head. For example, the change in calibration parameters compensates for the eyes not being centered in the field of view of the eye-tracking camera.

[0086] As shown in box 350c, in some specific implementations, the gaze target is expected to be associated with a confidence score, and adjusting the calibration parameters includes adjusting the calibration parameters in response to a confidence score greater than a threshold confidence score and abandoning the adjustment in response to a confidence score less than a threshold confidence score. For example, as regarding Figure 2 In some specific implementations, when the expected gaze target 212 is associated with a confidence score greater than the threshold confidence score, the calibration parameter adjuster 250 generates a new value 252 for the calibration parameter 240, and when the expected gaze target 212 is associated with a confidence score less than the threshold confidence score, the calibration parameter adjuster 250 abandons generating the new value 252.

[0087] In some implementations, the confidence score varies with the density of the plurality of visual elements. For example, the confidence score may vary with the amount of spacing between the plurality of visual elements. As an example, when visual elements are positioned relatively close to each other, the confidence score of the expected gaze target may be low (e.g., below a threshold confidence score). In contrast, when visual elements are positioned relatively far from each other, the confidence score of the expected gaze target may be high (e.g., above a threshold confidence score).

[0088] In some specific implementations, the confidence score varies with the distance between the first display area and the second display area. For example, refer to... Figure 1D The confidence score of the expected gaze target 70 may vary with the difference 90 between the expected gaze position 72 and the measured gaze position 82. In some implementations, the confidence score is inversely proportional to the distance between the first display area and the second display area. For example, the greater the distance between the expected gaze position and the measured gaze position, the lower the confidence score of the expected gaze position.

[0089] As shown in box 350d, in some implementations, adjusting calibration parameters includes adjusting the calibration parameters in response to device movement data indicating that the device has moved beyond a threshold amount since a previous adjustment of the calibration parameters. As described herein, a head-mounted device may slide over a user's head due to movement while using the device. In some implementations, if sensor data from a visual-inertial ranging (VIO) system (e.g., accelerometer data from an accelerometer, gyroscope data from a gyroscope, and / or magnetometer data from a magnetometer) indicates that the head-mounted device and / or the user has moved beyond a threshold amount since a previous adjustment of the calibration parameters, the device may adjust the calibration parameters to continue providing accurate gaze tracking.

[0090] As shown in box 350e, in some specific implementations, adjusting the calibration parameters includes adjusting the calibration parameters in response to a second display area corresponding to a blank area. For example, refer to Figure 1C In response to the measured gaze position 82 pointing to a blank area where no visual element is being displayed, the electronic device 20 can adjust the calibration parameter 28. When the device displays visual elements, it is expected that the user will gaze at one or more of the displayed visual elements, rather than the blank area. Therefore, when gaze tracking indicates that the user is gazing at a blank area while the device is displaying visual elements, gaze tracking is likely to generate an incorrect measured gaze target, and this can be improved by adjusting the calibration parameter.

[0091] As shown in box 350f, in some embodiments, adjusting the calibration parameters includes adjusting the calibration parameters in response to a first saliency value in a first display area having a second saliency value greater than that in a second display area. In some embodiments, it is anticipated that the user will gaze at the visual element with the maximum saliency value. For example, refer to Figure 1B In response to the fact that the salience value of the second visual element 50b is greater than the salience values ​​of the remaining visual elements 50a, 50c and 50d, the electronic device 20 can select the position corresponding to the second visual element 50b as the expected gaze position 72.

[0092] In some implementations, method 300 includes determining a desired gaze target based on a measured gaze target. In some implementations, the measured gaze target indicates that the user's gaze is directed to a portion of the object within a threshold distance of the object's center. For example, the measured gaze target might indicate that the user is gazing towards the edge of a button rather than its center. In various implementations, it is expected that the user will gaze at or towards the center of the object. For example, it is expected that the user will gaze at or near the center of a button. In such implementations, method 300 includes determining that the desired gaze location corresponds to the center of the object the user is measuringly gazing at.

[0093] Figure 4 This is a block diagram based on some specific implementations of device 400. In some specific implementations, device 400 implements... Figures 1A to 1J The electronic device 20 and / or shown Figure 2 The system 200 is shown. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and to avoid obscuring more relevant aspects of the specific embodiments disclosed herein. Therefore, as a non-limiting example, in some embodiments, device 400 includes one or more processing units (CPUs) 401, a network interface 402, a programming interface 403, a memory 404, one or more input / output (I / O) devices 408, and one or more communication buses 405 for interconnecting these and various other components.

[0094] In some implementations, a network interface 402 is provided to establish and maintain a metadata tunnel between a cloud-hosted network management system and at least one private network including one or more compatible devices, among other uses. In some implementations, one or more communication buses 405 include circuitry for interconnecting and communicating between system components. Memory 404 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices, and may include non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 404 optionally includes one or more storage devices remotely located to one or more CPUs 401. Memory 304 includes a non-transitory computer-readable storage medium.

[0095] In some embodiments, memory 404 or a non-transitory computer-readable storage medium of memory 404 stores programs, modules, and data structures, or subsets thereof, including an optional operating system 406, an expected gaze determiner 210, a measured gaze determiner 230, and a calibration parameter adjuster 250. In various embodiments, device 400 executes... Figure 3 Method 300 is shown.

[0096] In some implementations, the expected gaze determiner 210 includes a tool for determining the expected gaze target (e.g., Figure 1B The expected gaze target 70 and / or Figure 2 The instructions 210a of the expected gaze target 212 shown, as well as heuristics and metadata 210b, are used. In some specific implementations, the expected gaze determiner 210 executes instructions 210a of the expected gaze target 212. Figure 3 The middle frame 320 represents at least some of the operations.

[0097] In some implementations, the measured gaze determiner 230 includes a method for determining the measured gaze target based on the value of calibration parameter 240 (e.g., Figure 1C The measured gaze target 80 and / or shown Figure 2 The instructions 230a of the measured gaze target 234 shown, as well as heuristics and metadata 230b, are used. In some specific implementations, the measured gaze determiner 230 executes instructions 230a of the measured gaze target 234. Figure 3 The middle frame 340 represents at least some of the operations.

[0098] In some specific implementations, the calibration parameter adjuster 250 includes a tool for adjusting the calibration parameter 240 (e.g., Figure 1A The calibration parameter 28 shown is indicated by instruction 250a, along with heuristics and metadata 250b. In some specific implementations, the calibration parameter adjuster 250 executes the instructions 250a, along with heuristics and metadata. Figure 3 Box 350 in the diagram represents at least some of the operations.

[0099] In some specific implementations, the one or more I / O devices 408 include features for obtaining input (e.g., Figure 1F The user input device 110 shown is an input device. In some embodiments, the input device includes a touchscreen (e.g., for detecting tap input), an image sensor (e.g., for detecting gesture input), and / or a microphone (e.g., for detecting voice input). In some embodiments, the one or more I / O devices 408 include an image for capturing the user's pupils (e.g., for capturing...). Figure 1C Image 78 shown Figure 1E The image shown is 98 and / or Figure 2 The image sensor (e.g., a user-facing camera, such as an eye-tracking camera) shown in image 232. In some specific implementations, the one or more I / O devices 408 include a display for displaying visual elements.

[0100] In various embodiments, the one or more I / O devices 408 include a video pass-through display that displays at least a portion of the physical environment surrounding the device 400 as an image captured by a camera. In various embodiments, the one or more I / O devices 408 include an optical pass-through display that is at least partially transparent and transmits light emitted or reflected by the physical environment.

[0101] It should be understood that Figure 4 This serves as a functional description of various features that may exist in a particular specific implementation, and differs from the structural diagrams of the specific implementations described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separated. For example, Figure 4 Some functional blocks shown individually can be implemented as a single block, and the various functions of a single functional block can be implemented in various specific implementations through one or more functional blocks. The actual number of blocks and the division of specific functions, as well as how features are allocated therein, will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software, and / or firmware chosen for that particular implementation.

[0102] The specific implementations described herein envision the use of gaze information to present prominent viewpoints and / or prominent information. Implementers should consider the extent to which gaze information is collected, analyzed, disclosed, transmitted, and / or stored in order to respect established privacy policies and / or privacy practices. These considerations should include practices that the application generally considers to comply with or exceed industry and / or government requirements for protecting user privacy. This disclosure also envisions that the use of user gaze information may be limited to the extent necessary to achieve the described implementations. For example, in an implementation where the user's device provides processing capabilities, gaze information may be processed locally on the user's device.

[0103] The various processes defined herein take into account options for obtaining and using users' personal information. For example, such personal information may be used to provide improved privacy screens on electronic devices. However, the extent to which such personal information is collected should be based on the user's informed consent. As described herein, users should understand and control the use of their personal information.

[0104] Personal information will be used by the appropriate parties only for lawful and reasonable purposes. Parties using such information will comply with privacy policies and practices that are at least in accordance with applicable laws and regulations. Furthermore, such policies should be comprehensive, user-accessible, and considered to meet or exceed government / industry standards. In addition, parties may not distribute, sell, or otherwise share such information except for any reasonable and lawful purpose.

[0105] However, users can limit the extent to which parties can access or otherwise obtain their personal information. For example, settings or other preferences can be adjusted so that users can decide whether their personal information can be accessed by various entities. Furthermore, while some of the characteristics defined herein are described in the context of the use of personal information, aspects of these characteristics can be implemented without the need for such information. For example, if user preferences, account names, and / or location history are collected, this information can be obfuscated or otherwise generalized so that it does not identify the corresponding user.

[0106] While various aspects of specific embodiments within the scope of the appended claims have been described above, it should be apparent that the various features of the above-described embodiments can be embodied in a wide variety of forms, and any particular structure and / or function described above are merely illustrative. Based on this disclosure, those skilled in the art will understand that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement an apparatus and / or practice a method. Furthermore, such an apparatus and / or such a method can be implemented using other structures and / or functions besides or different from one or more aspects set forth herein.

Claims

1. A method, the method comprising: At a device including a display, a gaze tracker, non-transitory memory, and one or more processors coupled to the display and the non-transitory memory: The cursor is displayed on the monitor; The first display area corresponding to the cursor is set as the expected gaze target, which indicates the place where the user of the device intends to gaze; The gaze tracker is used to determine a second display area, which corresponds to a measured gaze target on the display that the user is measuringly gazing at. as well as Without prompting for calibration, the calibration parameters of the gaze tracker are adjusted based on the difference between the first display area and the second display area.

2. The method of claim 1, wherein setting the first display area corresponding to the cursor as the expected gaze target comprises: Input directed at the first display area is detected via a physical input device; as well as In response to detecting the input directed at the first display area, the first display area is set as the intended gaze target.

3. The method of claim 1 or 2, wherein setting the first display area corresponding to the cursor as the expected gaze target comprises: When the cursor is located in the first display area, a mouse click is detected via the mouse. as well as In response to detecting the mouse click, the intended gaze target is set as the cursor.

4. The method of claim 1 or 2, wherein setting the first display area corresponding to the cursor as the expected gaze target comprises: When the first display area corresponds to the cursor, a tap input is detected via a touch-sensitive surface; as well as In response to detecting the tap input, the intended gaze target is set to the cursor.

5. The method of claim 1 or 2, wherein setting the first display area corresponding to the cursor as the expected gaze target comprises: When the first display area corresponds to the focus indicator, the key is pressed via the keyboard detection key; as well as In response to detecting the key press, the first display area is set as the intended gaze target.

6. The method of claim 1 or 2, wherein setting the first display area corresponding to the cursor as the expected gaze target comprises: When the first display area corresponds to the focus indicator, voice input including a selection command is detected via an audio sensor; as well as In response to detecting the voice input including the selection command, the first display area is set as the expected gaze target.

7. The method according to claim 1 or 2, wherein adjusting the calibration parameter includes adjusting the calibration parameter when the distance between the first display area and the second display area is greater than a threshold.

8. The method according to claim 1 or 2, wherein the adjustment of the calibration parameter is proportional to the distance between the first display area and the second display area.

9. The method of claim 1 or 2, wherein the expected gaze target is associated with a confidence score, and wherein adjusting the calibration parameter comprises: The calibration parameters are adjusted in response to the confidence score being greater than a threshold confidence score; as well as The adjustment of the calibration parameters is abandoned in response to the confidence score being less than the threshold confidence score.

10. The method of claim 10, wherein the confidence score varies with the distance between the first display area and the second display area.

11. The method of claim 1 or 2, wherein adjusting the calibration parameters includes adjusting the calibration parameters in response to the second display area corresponding to a blank area.

12. The method of claim 1 or 2, wherein adjusting the calibration parameter includes adjusting the calibration parameter in response to the first display area having a first significance value that is greater than a second significance value of the second display area.

13. The method of claim 1 or 2, wherein when a selection input is received, the expected gaze target is determined based on the measured gaze target.

14. An apparatus, the apparatus comprising: One or more processors; gaze tracker; monitor; Non-transitory memory; and One or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the device to: The cursor is displayed on the monitor; The first display area corresponding to the cursor is set as the expected gaze target, which indicates the place where the user of the device intends to gaze; The gaze tracker is used to determine a second display area, which corresponds to a measured gaze target on the display that the user is measuringly gazing at. as well as Without prompting for calibration, the calibration parameters of the gaze tracker are adjusted based on the difference between the first display area and the second display area.

15. The device of claim 14, wherein setting the first display area corresponding to the cursor as the intended gaze target comprises: Input directed at the first display area is detected via a physical input device; as well as In response to detecting the input directed at the first display area, the first display area is set as the intended gaze target.

16. The device of claim 14 or 15, wherein setting the first display area corresponding to the cursor as the intended gaze target comprises: When the cursor is located in the first display area, a mouse click is detected via the mouse. as well as In response to detecting the mouse click, the intended gaze target is set as the cursor.

17. The device of claim 14 or 15, wherein setting the first display area corresponding to the cursor as the intended gaze target comprises: When the first display area corresponds to the cursor, a tap input is detected via a touch-sensitive surface; as well as In response to detecting the tap input, the intended gaze target is set to the cursor.

18. The device of claim 14 or 15, wherein setting the first display area corresponding to the cursor as the intended gaze target comprises: When the first display area corresponds to the focus indicator, the key is pressed via the keyboard detection key; as well as In response to detecting the key press, the first display area is set as the intended gaze target.

19. The device of claim 14 or 15, wherein setting the first display area corresponding to the cursor as the intended gaze target comprises: When the first display area corresponds to the focus indicator, voice input including a selection command is detected via an audio sensor; as well as In response to detecting the voice input including the selection command, the first display area is set as the expected gaze target.

20. A non-transitory memory storing one or more programs, said one or more programs causing the device to: The cursor is displayed on the monitor; The first display area corresponding to the cursor is set as the expected gaze target, which indicates the place where the user of the device intends to gaze; The gaze tracker is used to determine a second display area, the second display area corresponding to a measured gaze target on the display that the user is measuringly gazing at; and Without prompting for calibration, the calibration parameters of the gaze tracker are adjusted based on the difference between the first display area and the second display area.

Images