Graphical user interface-based eye tracking calibration

Abstract

It is disclosed that an eye-tracking system can be calibrated at least in part on user interface elements presented on the display panel of a head-mounted display (HMD). The processor can present user interface elements on the display panel of the HMD and can receive user input data from a handheld controller indicating that a user wearing the HMD has provided user input associated with the user interface elements via the handheld controller. In response to receiving user input data from the handheld controller, the processor can receive eye data associated with one or more of the user's eyes from the eye-tracking sensors of the HMD and can calibrate the eye-tracking system at least in part on the eye data and location data indicating the location on the display panel where the user interface elements are presented.

Claims

1. It is a system, A head-mounted display (HMD), Display panel and A head-mounted display (HMD) equipped with eye-tracking sensors, Eye tracking system and, A handheld controller, Processor and A memory for storing computer executable instructions, wherein when the computer executable instructions are executed by the processor, the processor... The display panel will present user interface elements, The handheld controller receives user input data indicating that the user wearing the HMD has provided user input associated with the user interface elements via the handheld controller. In response to receiving user input data from the eye tracking sensor and the handheld controller, the system receives eye data associated with one or more of the user's eyes. A system comprising: a memory that causes the eye tracking system to be calibrated at least partially based on the eye data and location data indicating the location on the display panel where the user interface element is presented.

2. The system according to claim 1, wherein the user input is the selection of the user interface element.

3. The system according to claim 1, wherein the user input is to highlight the user interface element.

4. The system according to claim 1, wherein the user interface elements are presented during gameplay of a video game on the HMD.

5. The user input data is second user input data, and the user input is second user input, and when the computer executable instruction is executed by the processor, before receiving the second user input data from the handheld controller, the processor The system receives first user input data from the handheld controller, indicating that the user has provided first user input via the handheld controller, such as moving a pointer over the user interface element. In response to receiving the first user input data from the handheld controller, the user interface elements on the display panel are further highlighted. The system according to claim 1, wherein the second user input is the selection of the user interface element.

6. When the aforementioned computer executable instruction is executed by the processor, after calibrating the eye-tracking system, the processor is instructed to: Running an application that uses the aforementioned eye-tracking system, Presenting a second user interface element on the aforementioned display panel, The handheld controller receives second user input data indicating that the user has provided a second user input associated with the second user interface element via the handheld controller. Receiving second eye data associated with one or more of the user's eyes from the eye tracking sensor and in response to receiving the second user input from the handheld controller, The system according to claim 1, further comprising: recalibrating the eye tracking system based at least in part on the second eye data and the second location data indicating a second location on the display panel where the second user interface element is presented.

7. The eye tracking sensor is equipped with a camera, The system according to claim 1, wherein the eye data includes image data of one or more of the user's eyes.

8. It is a method, The processor presents user interface elements on the display panel of the head-mounted display (HMD), The processor receives user input data from the handheld controller indicating that the user wearing the HMD has provided user input associated with the user interface elements via the handheld controller. The processor receives eye data associated with one or more of the user's eyes from the eye tracking sensor of the HMD and from the handheld controller in response to receiving the user input data. A method comprising calibrating an eye-tracking system using the processor, at least in part, based on the eye data and location data indicating the location on the display panel where the user interface elements are presented.

9. The method according to claim 8, wherein the user input is the selection of the user interface element.

10. The method according to claim 8, wherein the user input is to highlight the user interface element.

11. The aforementioned user interface element is a weapon aiming reticle in a first-person perspective shooting video game, The method according to claim 8, wherein the user input is to fire the weapon.

12. The method according to claim 8, wherein the user interface elements are presented during gameplay of a video game on the HMD.

13. After the calibration, The aforementioned processor executes an application that uses the eye-tracking system, The processor presents a second user interface element on the display panel, The processor receives second user input data from the handheld controller indicating that the user has provided a second user input associated with the second user interface element via the handheld controller. The processor receives second eye data associated with one or more of the user's eyes in response to receiving the second user input data from the eye tracking sensor and the handheld controller. The method of claim 8, further comprising the processor recalibrating the eye tracking system at least in part on the second eye data and the second location data indicating a second location on the display panel where the second user interface element is presented.

14. The processor waits to execute the application that uses the eye tracking system until the calibration is complete. The method according to claim 8, further comprising: the processor starting execution of the application after calibration.

15. One or more non-temporary computer-readable media for storing computer executable instructions, wherein when the computer executable instructions are executed by a processor, Presenting user interface elements on the display panel of a head-mounted display (HMD), The handheld controller receives user input data indicating that the user wearing the HMD has provided user input associated with the user interface element via the handheld controller. Receiving eye data associated with one or more of the user's eyes from the eye tracking sensor of the HMD and in response to receiving the user input data from the handheld controller, One or more non-temporary computer-readable media that causes an operation to be performed, including calibrating an eye-tracking system based at least in part on the eye data and location data indicating the location on the display panel where the user interface element is presented.

16. The one or more non-temporary computer-readable media according to claim 15, wherein the user input is the selection of the user interface element.

17. The one or more non-temporary computer-readable media according to claim 15, wherein the user input is to highlight the user interface elements.

18. The user interface elements are presented during gameplay of a video game on the HMD in one or more non-temporary computer-readable media according to claim 15.

19. The aforementioned operation, Wait until the aforementioned calibration is complete before running the application that uses the eye tracking system, One or more non-temporary computer-readable media according to claim 15, further comprising: starting the execution of the application after the calibration.

20. The above operation, after the calibration, Running an application that uses the aforementioned eye-tracking system, Presenting a second user interface element on the aforementioned display panel, The handheld controller receives second user input data indicating that the user has provided a second user input associated with the second user interface element via the handheld controller. Receiving second eye data associated with one or more of the user's eyes from the eye tracking sensor and the handheld controller in response to receiving the second user input data, One or more non-temporary computer-readable media according to claim 15, further comprising recalibrating the eye-tracking system based at least in part on the second eye data and the second location data indicating a second location on the display panel where the second user interface element is presented.

Images