Optical systems and related methods to improve user experience and gaze interaction accuracy

Abstract

The optical system includes an eye tracker and a head-mounted display. The eye tracker's sensor module captures images of one or more of the user's eyes. The head-mounted display's processor provides a user interface containing one or more user interface objects based on one or more estimated gaze points of the user in the relevant eye images, calculates the distance between at least one estimated gaze point of the user and each user interface object, calculates a score for each user interface object based on the distance between the user's at least one estimated gaze point and each user interface object, and sets the user interface object with the highest score among the one or more user interface objects as a target user interface object for the relevant user's at least one estimated gaze point.

Description

[Technical Field]

[0001] This invention relates to an optical system and related methods for improving user experience and gaze interaction accuracy, and more particularly to an optical system and related methods for improving user experience and gaze interaction accuracy in an interactive virtual environment. [Background Technology]

[0002] Virtual reality (VR) uses computer technology to simulate a highly realistic three-dimensional space. When users wear special display devices to use VR applications, they experience the illusion of being in reality. Augmented reality (AR) is a technology that augments virtual information into real space. Compared to VR, which replaces real space, AR adds virtual objects to real space. Mixed reality (MR) merges the real and virtual worlds to create a new environment and virtual images that conform to general visual perception. In this environment, objects in the real world can coexist with objects in the digital world and interact with them in real time. Most current VR / AR / MR applications are operated using joysticks or touch panels, but these devices need to be carried around in different locations, which is inconvenient. If eye-tracking technology is applied to the VR / AR / MR field, users can easily turn their eyes into an interface, selecting or clicking objects by focusing on their gaze, focal points, and specific subtle movements.

[0003] Eye tracking is a technology that tracks a user's eye movements by measuring the gaze position or eye movement relative to the head. Existing computers display a user interface on the screen to provide the user with various visual data. A user interface typically contains multiple user interface elements (UI elements), each containing a graphic element and a hit box. The graphic element determines the appearance of the corresponding UI element, while the hit box is a virtual object invisible to the user and linked to a corresponding event handler. When a user's gaze command triggers a hit box on a UI element, a predetermined action is executed on the triggered UI element.

[0004] In existing optical systems, human-computer interaction based on eye-tracking gaze technology is provided according to the estimated absolute position of the gaze point. More specifically, the user needs to fix their gaze within the hit box of a specific user interface object to trigger that object. This technology places high demands on the accuracy of the eye tracker; even small errors can significantly impact the user experience. Therefore, there is a need for an optical system and method that can improve both the user experience and the accuracy of gaze-based interaction. [Summary of the Invention]

[0005] This invention provides an optical system for providing accurate eye tracking in an interactive virtual environment, comprising an eye tracker and a head-mounted display. The eye tracker includes a sensor module for capturing images of one or more eyes of a user. A processor in the head-mounted display provides a user interface based on one or more estimated gaze points of the user, wherein the user interface includes one or more user interface objects, and the one or more estimated gaze points of the user are calculated based on the one or more eye images of the user; calculates the distance between at least one estimated gaze point of the user and each user interface object; calculates a score for each user interface object based on the distance between the at least one estimated gaze point of the user and each user interface object; and sets the first user interface object with the highest score among the one or more user interface objects as a target user interface object associated with the at least one estimated gaze point of the user. A display screen in the head-mounted display is used to present the user interface.

[0006] The present invention also provides a method for providing accurate eye-tracking operation in an interactive virtual environment, comprising providing a user interface including one or more user interface objects, capturing one or more eye images of a user during eye-tracking operation, calculating one or more estimated fixation points of the user based on the one or more eye images of the user, determining the distance between at least one estimated fixation point of the user and each user interface object, determining a score for each user interface object based on the distance between the at least one estimated fixation point of the user and each user interface object, and setting a first user interface object with the highest score among the one or more user interface objects as a target user interface object associated with the at least one estimated fixation point of the user.

[0007] The present invention also provides an optical system for providing accurate eye tracking in an interactive virtual environment, comprising an eye tracker and a head-mounted display. The eye tracker includes a sensor module and a database. The sensor module is used to capture images of one or more eyes of a user. The database is used to store a set of preset trajectory patterns, each of which corresponds to a specific operational link of one or more user interface objects. A processor of the head-mounted display is used to provide a user interface based on one or more estimated gaze points of the user, wherein the user interface includes the one or more user interface objects, and the one or more estimated gaze points of the user are calculated based on the one or more eye images of the user; to determine a gaze trajectory of the user based on the one or more estimated gaze points of the user; and to execute the specific operational link of the at least one preset trajectory pattern when the user's gaze trajectory conforms to at least one preset trajectory pattern. A display screen of the head-mounted display is used to present the user interface.

[0008] The present invention also provides a method for providing accurate eye tracking in an interactive virtual environment, comprising providing a user interface including one or more user interface objects, storing a set of preset trajectory patterns, each preset trajectory pattern corresponding to a specific operational link associated with the one or more user interface objects, capturing one or more eye images of a user during an eye tracking operation, calculating one or more estimated gaze points of the user based on the one or more eye images of the user, determining a gaze trajectory of the user based on the one or more estimated gaze points of the user, and executing the specific operational link associated with the at least one preset trajectory pattern when the user's gaze trajectory conforms to at least one preset trajectory pattern. [Attached Image Description]

[0009] Figure 1 This is a functional block diagram of an optical system that can provide accurate eye tracking in an interactive virtual environment, according to an embodiment of the present invention. Figure 2 This is a functional block diagram of an optical system that can provide accurate eye tracking in an interactive virtual environment, according to another embodiment of the present invention. Figure 3 This is a flowchart illustrating a method for providing human-computer interaction based on gaze trajectory in an interactive virtual environment, as described in this embodiment of the invention. Figure 4A , 4B Figures 4C and 4C are schematic diagrams illustrating human-computer interaction based on gaze trajectory in an interactive virtual environment, as described in an embodiment of the present invention. Figure 5A , 5BFigures 5C and 5C are schematic diagrams illustrating human-computer interaction based on gaze trajectory in an interactive virtual environment, according to another embodiment of the present invention. Figure 6 This is a flowchart illustrating a method for providing human-computer interaction based on gaze score in an interactive virtual environment, as described in this embodiment of the invention. Figure 7 This is a schematic diagram illustrating human-computer interaction based on gaze score in an interactive virtual environment, as described in an embodiment of the present invention. Figure 8 This is a schematic diagram of the performance of the eye tracker obtained in the embodiment of the present invention. Figure 9 This is a flowchart illustrating a method for providing human-computer interaction based on eye-tracking gaze in an interactive virtual environment, as described in an embodiment of the present invention.

Detailed Implementation Methods

[0010] Figure 1 This is a functional block diagram of an optical system 100 that can provide accurate eye tracking in an interactive virtual environment, according to an embodiment of the present invention. Figure 2 This is a functional block diagram of an optical system 200 that can provide accurate eye tracking in an interactive virtual environment, according to another embodiment of the present invention.

[0011] exist Figure 1 In the illustrated embodiment, the optical system 100 includes a head-mounted display (HMD) 10 and an eye tracker 20. The HMD 10 includes a processor 12, a display screen 14, a memory 15, a sensor module 16, an input / output device 18, and a user interface 19. The memory includes a database 155, a first buffer 151, and a second buffer 152. The eye tracker 20 includes a processor 22, an illumination module 24, and a sensor module 26. The processor 12 controls the operation of the HMD 10, while the processor 22 controls the operation of the eye tracker 20.

[0012] exist Figure 2 In the illustrated embodiment, the optical system 200 is a head-mounted display 10, which includes a processor 12, a display screen 14, a memory 15, a sensor module 16, an input / output device 18, a user interface 19, and an eye tracker 20. The memory 15 includes a database 155, a first buffer 151, and a second buffer 152. The eye tracker 20 includes a processor 22, an illumination module 24, and a sensor module 26. The processor 12 controls the operation of the head-mounted display 10, while the processor 22 controls the operation of the eye tracker 20.

[0013] In other embodiments of the invention, the optical systems 100 and 200 may omit the processor 22. More specifically, the head-mounted display 10 and the eye tracker 20 may share the same processor 12, and the processor 12 controls the operation of the head-mounted display 10 and the eye tracker 20.

[0014] In optical systems 100 and 200, illumination module 24 may include one or more infrared light-emitting diodes (LEDs) to illuminate the user's eyes, ensuring sufficient contrast between the iris and pupil for users with different eye colors, especially against very bright or very dark backgrounds, thereby improving the accuracy of eye tracker 20 in capturing reflected light from the user's eyes. However, the implementation of illumination module 24 does not limit the scope of the invention.

[0015] In optical system 100 or 200, input / output device 18 can receive commands from the user. In one embodiment, input / output device 18 may include any type of handheld controller (e.g., game controller or game console) and / or any form of haptic feedback device (e.g., motion-sensing suit or motion-sensing gloves). Input / output device 18 can detect user motion signals and transmit them to controller 12 of optical system 100 or 200. In one embodiment, controller 12 can control the operation of optical system 100 or 200 according to user commands received by input / output device 18. In another embodiment, controller 12 can simultaneously control the operation of optical system 100 or 200 according to user commands received by input / output device 18 and user interface objects UIE1-UIE of user interface 19. N The received gaze command controls the operation of optical system 100 or 200.

[0016] Figure 3 This is a flowchart illustrating a method for providing human-computer interaction based on gaze trajectory in an interactive virtual environment, as described in this embodiment of the invention. Figure 3 The flowchart shown includes the following steps:

[0017] Step 310: Provide a user interface that contains one or more user interface objects.

[0018] Step 320: Store a set of preset trajectory patterns, each preset trajectory pattern corresponding to a specific operation link of one or more related user interface objects.

[0019] Step 330: Capture images of one or more of the user's eyes.

[0020] Step 340: Calculate and store one or more estimated fixation points of the user based on one or more eye images of the user.

[0021] Step 350: Calculate and store the user's gaze trajectory based on one or more estimated gaze points.

[0022] Step 360: Determine whether the user's gaze trajectory matches at least one preset trajectory pattern; if yes, proceed to step 370; if no, proceed to step 330.

[0023] Step 370: Execute the specific operation link of the relevant preset trajectory pattern.

[0024] In step 310, one or more user interface objects UIE1-UIE are included. N The user interface 19 can be displayed on the display screen 14, where N is a positive integer. In one embodiment, the user interface objects UIE1-UIE N This can be an abstract object that is not visible to the user. If the user interface object is UIE1-UIE... N As interactive objects, each user interface object is linked to an event handler, where each event handler is related to a specific operation of the optical system 100 or 200 and is controlled by the controller 12.

[0025] User interface objects UIE1-UIE N This can increase the interactivity of the user interface 19 and provide touch points for users to issue eye-tracking commands. The function of each user interface object may be related to input control, browsing control, and information display, but is not limited to the scope of the invention. Each user interface object includes an image object and a hit box. The image object can determine the appearance of the corresponding user interface object and may be related to the function of the corresponding user interface object. In one embodiment, the image object of the user interface object may be a checkbox, radio button, dropdown list, list box, toggle button, or date / time option related to input control. In another embodiment, the image object of the user interface object may be a breadcrumb, slider, pagination, icon, or image carousel related to browsing control. In yet another embodiment, the image object of the user interface object may be a tooltip, progress bar, notification, message box, or modal window displaying relevant information. However, the appearance of the image object of the user interface object does not limit the scope of this invention.

[0026] The hit box of a user interface object is a virtual object invisible to the user and linked to a corresponding event handler. When a gaze command from one or more estimated gaze points of the relevant user triggers the hit box of a user interface object, the processor 12 executes the action of the corresponding triggered user interface object. The present invention determines that a user interface object has been triggered by a gaze command when one or more triggering conditions are met. In the present invention, the triggering condition is determined to be met when the user's gaze trajectory conforms to a preset trajectory pattern containing one or more segments. The aforementioned segments may be the action of the user's gaze moving from a first point to a second point on the user interface 19. However, the type of triggering condition for each user interface object does not limit the scope of the present invention.

[0027] In step 320, the present invention stores a set of preset trajectory patterns in a database 155 of memory 15 in optical system 100 or 200, wherein each preset trajectory pattern corresponds to a specific operational link of one or more user interface objects. In one embodiment, the preset trajectory pattern may be an m-segment trajectory pattern (m is a positive integer), but is not limited thereto. For example, a two-segment trajectory pattern may sequentially include a first segment and a second segment, wherein the first segment corresponds to the user's action of moving their gaze from a first point on user interface 19 to a second point, and the second segment corresponds to the user's action of moving their gaze from a second point on user interface 19 to a third point. However, the value of m in the preset trajectory pattern does not limit the scope of the present invention.

[0028] In the optical system 100, the sensor module 26 of the eye tracker 20 includes at least one image sensor (eye sensor) for capturing one or more eye images of the user in step 330. The processor 22 of the eye tracker 20 can receive the one or more eye images captured by the sensor module 26 and calculate one or more estimated gaze points of the user based on the one or more eye images in step 340. Furthermore, the processor 22 of the eye tracker 20 can also calculate other eye-tracking related data based on the one or more eye images, such as the confidence and accuracy of the estimated gaze points, the eye position in three-dimensional space, and pupil-related information (e.g., pupil size). The algorithm for eye-tracking operation can be implemented as a program / software / solid that can be executed by the processor 22 of the eye tracker 20, but this does not limit the scope of the invention. Additionally, one or more estimated gaze points of the user can be stored in a first buffer 151 in memory 15 in step 340.

[0029] In the optical system 200, the sensor module 26 of the eye tracker 20 includes at least one image sensor (eye sensor) for capturing one or more eye images of the user in step 330. The processor 12 can receive the one or more eye images captured by the sensor module 26 and calculate one or more estimated gaze points of the user based on the one or more eye images in step 340. Furthermore, the processor 12 can also calculate other eye-tracking related data based on the one or more eye images, such as the confidence and accuracy of the estimated gaze points, the eye position in three-dimensional space, and pupil-related information (e.g., pupil size). The algorithm for eye-tracking operation can be implemented as a program / software / solid that can be executed by the processor 12, but this does not limit the scope of the invention. Additionally, the one or more estimated gaze points of the user can be stored in a first buffer 151 in memory 15 in step 340.

[0030] In optical systems 100 and 200, sensor module 16 includes at least one scene sensor, at least one sound sensor (e.g., a microphone), and / or at least one motion sensor (e.g., a gyroscope or accelerometer). The scene sensor can capture one or more scene images relative to the user's current field of view, the sound sensor can receive voice commands from the user, and the motion sensor can detect the user's movements (typically head movements).

[0031] In step 350, processor 12 can calculate the user's gaze trajectory based on one or more estimated fixation points. More specifically, processor 12 identifies gaze movement (segments) between two adjacent estimated fixation points within one or more estimated fixation points of the user, and provides the user's gaze trajectory by connecting all segments or connecting M consecutive segments, where M is a positive integer. Furthermore, the identified segments can be stored in a second buffer 152 in memory 15.

[0032] In step 360, the processor 12 determines whether the user's gaze trajectory conforms to at least one preset trajectory pattern. More specifically, the processor 12 compares the user's gaze trajectory identified in step 350 with the set of preset trajectory patterns stored in the database 155. If the user's gaze trajectory does not conform to any preset trajectory pattern, the invention will execute step 330 again to continue eye-tracking operations. If the user's gaze trajectory conforms to at least one preset trajectory pattern, the invention will execute step 370 to execute a specific operation link that conforms to the preset trajectory pattern. The specific operation link may include one or more predetermined actions. The predetermined actions include content selection, going to the previous page, going to the next page, settings, closing the page, returning, returning to the home page, displaying notification information, or screen locking, but are not limited to the scope of the invention.

[0033] Figures 4A-4CThe figure illustrates a human-computer interaction based on gaze trajectory in an interactive virtual environment according to an embodiment of the present invention. GZ1-GZ3 represent the user's estimated gaze points calculated in step 340. SG1 and SG2 represent segments within the user's gaze trajectory identified in step 350. Figure 4A In the initial state shown in the diagram, assume that user interface 19 contains four user interface components, UIE1-UIE4. Figure 4B and 4C During the subsequent human-computer interaction process shown in the figure, the user interface 19 will also present user interface components UIE5 and UIE6.

[0034] First, the user fixes their gaze on the center of user interface 19 in order to select user interface component UIE1, such as... Figure 4A As shown in the figure. Next, the user moves their gaze from the center of the user interface 19 to the top of the user interface 19 in order to select the user interface component UIE4, as shown. Figure 4B As shown in the figure. Finally, the user moves their gaze from the top of the user interface 19 to the upper left of the user interface 19 in order to select the user interface component UIE5, as shown. Figure 4C As shown in the figure.

[0035] Based on one or more eye images captured in step 330 by the sensor module 26 of the eye tracker 20, it is possible to record Figures 4A-4C The figure shows the user's eye / head movements. Based on the user's estimated gaze points GZ1-GZ3 calculated in step 340, the processor 12 can identify segments SG1 and SG2 in the user's gaze trajectory in step 350, where segment SG1 (upward gaze segment) is related to the user's action of moving their gaze from gaze point GZ1 to gaze point GZ2, and segment SG2 (leftward gaze segment) is related to the user's action of moving their gaze from gaze point GZ2 to gaze point GZ3.

[0036] If the user gaze trajectories of the included segments SG1 and SG2 conform to at least one preset trajectory pattern, the present invention will execute step 370 to perform a specific operational link of the relevant preset trajectory pattern. Figures 4A-4C In the embodiment shown in the figure, the specific operational link includes the "expand menu" action of the relevant user interface component UIE4, the "content selection" action of the relevant user interface components UIE5 and UIE6, and the "back" action of the relevant user interface component UIE5. More specifically, when the user's gaze trajectory containing segments SG1 and SG2 conforms to at least one preset trajectory pattern, the user interface components UIE4 and UIE5 will be triggered sequentially by the user's gaze trajectory.

[0037] Figures 5A-5CThis is a schematic diagram illustrating human-computer interaction based on gaze trajectory in an interactive virtual environment, according to another embodiment of the present invention. GZ1'-GZ3' represent the user's estimated gaze points calculated in step 340. SG1' and SG2' represent segments within the user's gaze trajectory identified in step 350. Figure 5A In the initial state shown, it is assumed that user interface 19 contains four user interface components, UIE1-UIE4. Figure 5B and 5C During the subsequent human-computer interaction process shown, the user interface 19 will also present user interface components UIE5 and UIE6.

[0038] Based on one or more eye images captured in step 330 by the sensor module 26 of the eye tracker 20, it is possible to record Figures 5A-5C The user's eye / head movements are shown. Based on the user's estimated fixation points GZ1'-GZ3' calculated in step 340, the processor 12 can identify segments SG1' and SG2' in the user's gaze trajectory in step 350, where segment SG1' (upward gaze segment) is related to the user's action of moving their gaze from fixation point GZ1' to fixation point GZ2', and segment SG2' (leftward gaze segment) is related to the user's action of moving their gaze from fixation point GZ2' to fixation point GZ3'.

[0039] If the accuracy of the eye tracker 20 is very low, Figures 5A-5C The predicted fixation points GZ1'-GZ3' shown will be significantly offset. Figures 4A-4C The figure shows the actual gaze points GZ1-GZ3 positions. However, when the user's gaze trajectory, including segments SG1' and SG2', conforms to at least one preset trajectory pattern, the present invention will still execute step 370 to perform a specific operational link of the relevant preset trajectory pattern. In other words, even if the precise coordinates of the user's gaze trajectory cannot be obtained due to the low accuracy of the eye tracker 20, the present invention can still perform a specific operational link of the relevant preset trajectory pattern.

[0040] Figure 6 This is a flowchart illustrating a method for providing human-computer interaction based on gaze score in an interactive virtual environment, as described in this embodiment of the invention. Figure 6 The flowchart shown includes the following steps:

[0041] Step 610: Provide a user interface that contains one or more user interface objects.

[0042] Step 620: Capture images of one or more of the user's eyes.

[0043] Step 630: Calculate a predicted fixation point for the user based on one or more eye images.

[0044] Step 640: Calculate the distance between the estimated gaze point and each user interface object.

[0045] Step 650: Calculate the performance map of the eye tracker 20 based on one or more eye images of the user.

[0046] Step 660: Determine the performance level of the eye tracker 20 at the location of each user interface object based on the performance graph of the eye tracker 20.

[0047] Step 670: Calculate the score for each user interface object based on the performance level of the eye tracker 20 at the location of each user interface object and the estimated distance between the gaze point and each user interface object.

[0048] Step 680: Set the user interface object with the highest score among one or more user interface objects as the target user interface object for the estimated gaze of the relevant user.

[0049] Step 690: Determine whether the target user interface object has been triggered; if yes, proceed to step 700; if no, proceed to step 610.

[0050] Step 700: Execute a predetermined action on the relevant target user interface object.

[0051] Figure 6 The implementation methods of steps 610, 620, and 630 are respectively and Figure 3 Steps 310, 330, and 340 are the same and will not be described again here. In step 640, the processor 12 calculates the distance between the estimated gaze point and each user interface object.

[0052] Figure 7 This is a schematic diagram illustrating human-computer interaction based on gaze score in an interactive virtual environment, as described in an embodiment of the present invention. For illustrative purposes, Figure 7 An embodiment with N=4 is shown, where the user interface 19 includes four user interface components UIE1-UIE4. GZ represents the user's estimated gaze point calculated in step 630, d1 represents the distance between the estimated gaze point GZ and the user interface object UIE1, d2 represents the distance between the estimated gaze point GZ and the user interface object UIE2, d3 represents the distance between the estimated gaze point GZ and the user interface object UIE3, and d4 represents the distance between the estimated gaze point GZ and the user interface object UIE4.

[0053] In step 650, the processor 12 calculates the performance graph of the eye tracker 20 based on images of one or more of the user's eyes and the user's initial visual range. The user's initial visual range refers to the visual range when the user is looking straight ahead, and can be calculated based on the physical specifications of the optical system 100 or 200 and the user's standard anthropometric data. The performance graph of the eye tracker 20 represents the operational performance of the eye tracker 20 in relation to the user's initial visual range. Since the sensor module 26 is fixed in position within the eye tracker 20, when the user wears the eye tracker 20, the position of the sensor module 26 only changes with the user's head movements, but is not affected by the user's eye movements. Therefore, the performance graph of the eye tracker 20 changes with the user's head movements, but is not related to the user's eye movements.

[0054] Figure 8 This is a schematic diagram of the performance of the eye tracker 20 obtained in step 650 of this embodiment of the invention. The vertical axis represents the user's vertical visual range (in degrees), the horizontal axis represents the user's horizontal visual range (in degrees), and the numbers represent the error rate of the eye tracker 20 at different positions within the user's visual range. Generally, the accuracy of the eye tracker 20 decreases as eccentricity increases. Figure 8 As shown, the average error rate of eye tracker 20 is approximately 1.88° within the central area of ​​the user's field of vision, approximately 1.96° within a visual angle of ±10° of the user's field of vision, and approximately 2° within a visual angle of ±20° of the user's field of vision. However, Figure 8 The figures shown are for illustrative purposes only and do not limit the scope of the invention.

[0055] In step 660, the processor 12 determines the performance level of the eye tracker 20 at the location of each user interface object based on the performance map of the eye tracker 20. See also... Figure 7 and Figure 8 For illustrative purposes, it is assumed that the eye tracker 20 has an error rate of approximately 2.92° at the location of the user interface object UIE1, approximately 2.13° at the location of the user interface object UIE2, approximately 1.61° at the location of the user interface object UIE3, and approximately 1.34° at the location of the user interface object UIE4.

[0056] In step 670, the processor 12 calculates a score for each user interface object based on the performance level of the eye tracker 20 at the location of each user interface object and the estimated distance between the gaze point and each user interface object. In one embodiment, the user interface object (UIE) nThe score SCn can be calculated using the following formula (1), where dn represents the estimated gaze point GZ and the user interface object UIE. n The distance between them, ERn represents the distance between the eye tracker 20 and the user interface object UIE. n The error rate at location, where W1 represents distance weight and W2 represents accuracy weight.

[0057] SCn=W1(1 / dn)+W2(ERn / dn)…(1)

[0058] In step 680, the processor 120 sets the user interface object with the highest score among one or more user interface objects as the target user interface object for the estimated gaze point GZ of the relevant user.

[0059] In one embodiment, distance can be the sole parameter determining human-computer interaction based on eye-tracking gaze; in this case, the distance weight W1 can be set to 1, and the accuracy weight W2 can be set to 0. Figure 7 and Figure 8 As shown, the user interface object UIE1 has a score of 0.8 (SC1), UIE2 has a score of approximately 0.67 (SC2), UIE3 has a score of 1 (SC3), and UIE4 has a score of approximately 0.909 (SC4). In this case, the user interface object UIE3 with the highest score will be set as the target user interface object for the relevant user's estimated gaze point GZ.

[0060] In another embodiment, both distance and accuracy can be parameters that determine human-computer interaction based on eye-tracking fixation. In this case, the distance weight W1 can be set to 1, and the accuracy weight W2 can be set to 1. Figure 7 and Figure 8 As shown, the user interface object UIE1 has a score SC1 of approximately 3.136, UIE2 has a score SC2 of approximately 2.087, UIE3 has a score SC3 of approximately 2.61, and UIE4 has a score SC4 of approximately 2.127. In this case, the user interface object UIE1 with the highest score will be set as the target user interface object for the relevant user's estimated gaze point GZ.

[0061] In step 690, the processor 12 determines whether the target user interface object has been triggered. When the gaze command of one or more estimated gaze points of the relevant user satisfies one or more triggering conditions, the present invention determines that the target user interface object has been triggered by the gaze command. The triggering conditions include the user's gaze point being within the target user interface object's hit box for more than the relevant target user interface object's gaze duration, the user pressing another button, the user issuing a voice command, the user blinking voluntarily, or the detection of a specific gaze point trajectory / pattern, but do not limit the scope of the present invention.

[0062] After determining that the target user interface object has been triggered, the processor 12 will execute a predetermined action corresponding to the target user interface object in step 700. The predetermined action includes content selection, going to the previous page, going to the next page, setting, closing the page, returning, returning to the home page, displaying notification information, or locking the screen, but does not limit the scope of the present invention.

[0063] Figure 9 This is a flowchart illustrating a method for providing human-computer interaction based on eye-tracking gaze in an interactive virtual environment, as described in an embodiment of the present invention. Figure 9 The flowchart shown includes the following steps:

[0064] Step 810: Provide a user interface that contains one or more user interface objects.

[0065] Step 820: Store a set of preset trajectory patterns, each preset trajectory pattern corresponding to a specific operation link of one or more related user interface objects.

[0066] Step 830: Capture images of one or more of the user's eyes.

[0067] Step 840: Calculate one or more predicted fixation points for the user based on one or more eye images of the user.

[0068] Step 850: Calculate the distance between at least one estimated gaze point and each user interface object.

[0069] Step 860: Calculate the performance map of the eye tracker 20 based on one or more eye images of the user.

[0070] Step 870: Determine the performance level of the eye tracker 20 at the location of each user interface object based on the performance graph of the eye tracker 20.

[0071] Step 880: Calculate a score for each user interface object based on the performance level of the eye tracker 20 at the location of each user interface object and the distance between at least one estimated gaze point and each user interface object.

[0072] Step 890: Set the user interface object with the highest score among one or more user interface objects as the target user interface object for at least one estimated gaze point of the relevant user.

[0073] Step 900: Calculate and store the user's gaze trajectory based on one or more estimated gaze points.

[0074] Step 910: Determine whether the user's gaze trajectory conforms to at least one preset trajectory pattern; if yes, proceed to step 920; if no, proceed to step 830.

[0075] Step 920: Determine whether the action of the relevant target user interface object conforms to a specific operation link of at least one preset trajectory pattern; if yes, proceed to step 930; if no, proceed to step 940.

[0076] Step 930: Perform the action on the relevant target user interface object.

[0077] Step 940: Perform an automatic correction procedure on the eye tracker 20; proceed to step 830.

[0078] exist Figure 9 In the embodiment shown, Figure 3 The human-computer interaction based on gaze trajectory shown Figure 6 The human-computer interactions based on gaze scores shown are all employed to provide accurate eye tracking in interactive virtual environments. In steps 850-890, the invention first determines the target user interface object most likely associated with an estimated user gaze point, such as... Figure 7 As shown. Next, in steps 900-910, the present invention calculates the user's gaze trajectory and compares it with a set of preset trajectory patterns. When it is determined that the user's gaze trajectory matches at least one preset trajectory pattern, step 920 then confirms the correlation between the action of the relevant target user interface object and the specific operational link of the relevant at least one preset trajectory pattern. When it is determined that the action of the relevant target user interface object matches the specific operational link of the relevant at least one preset trajectory pattern, the present invention executes the action of the relevant target user interface object in step 930.

[0079] When it is determined that the action of the relevant target user interface object does not conform to a specific operational link of at least one preset trajectory pattern, the present invention performs an automatic correction procedure on the eye tracker 20 in step 940. More specifically, the difference between the action of the relevant target user interface object and the specific operational link of the relevant at least one preset trajectory pattern may indicate a serious error in the operation of the eye tracker 20. In this case, the estimated gaze point calculated based on the measurement data of the eye tracker 20 may be significantly offset from the user's actual gaze point, but the movement relationship between adjacent gaze points remains correct. Therefore, the actual user interface object that the user intends to interact with can be regarded as a specific user interface object, and its related actions correspond to at least one preset trajectory pattern. In step 940, the present invention can perform an automatic correction procedure on the eye tracker 20 based on the geometric relationship between the specific user interface object and one or more estimated gaze points.

[0080] In this invention, the optical system can employ human-computer interaction based on gaze trajectory and human-computer interaction based on gaze score to reduce the accuracy requirements of eye-tracking operation. Therefore, this invention can provide accurate eye-tracking in an interactive virtual environment. The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should be included within the scope of the present invention. [Symbol Explanation]

[0081] 10: Head-mounted displays 12, 22: Processor 14: Display screen 15: Memory 16, 26: Sensor Module 18: Input / output devices 19: User Interface 20: Eye tracker 24: Lighting Module 30: User 100, 200: Optical system 151: First Buffer 152: Second Buffer 155: Database 310-370, 610-700, 810-940: Steps UIE1-UIE N User interface object GZ, GZ1, GZ2, GZ1', GZ2': Estimated gaze points SG1, SG2, SG1', SG2': Sections d1-d4: Distance

Claims

1. An optical system for providing accurate eye tracking in an interactive virtual environment, comprising: An eye-tracking device includes: A sensor module for capturing images of one or more of a user's eyes; and A head-mounted display comprising: A processor is used to: A user interface is provided based on one or more estimated gaze points of the user, wherein the user interface includes one or more user interface objects (UI elements), and the one or more estimated gaze points of the user are calculated based on one or more eye images of the user; Find at least one estimated gaze point for the user and the distance between each user interface object; A representation of the eye tracker is derived based on one or more eye images of the user. Based on the performance graph of the eye tracker, the performance level of the eye tracker at the location of each user interface object is determined; The score for each user interface object is calculated based on the performance level of the eye tracker at the location of each user interface object and the distance between the user's at least one predicted fixation point and each user interface object. Set the highest-scoring first user interface object among the one or more user interface objects as a target user interface object associated with the user's at least one estimated gaze point; and A display screen is used to present the user interface.

2. The optical system of claim 1, wherein the score of each user interface object is inversely proportional to the distance between the user's at least one estimated gaze point and each user interface object.

3. The optical system of claim 1, wherein the score of each user interface object is proportional to an error rate of the eye tracker at the location of each user interface object.

4. The optical system of claim 1, further comprising a database for storing a set of preset trajectory patterns, each preset trajectory pattern corresponding to a specific operational link associated with the one or more user interface objects, wherein the processor is further configured to: Based on one or more of the user's gaze points, determine the user's gaze trajectory; When the user's gaze trajectory matches at least one preset trajectory pattern, determine the correlation between an action of the target user interface object and a specific operational link of the at least one preset trajectory pattern; and When the action of the target user interface object conforms to the specific operation connection of the at least one preset trajectory pattern, the action of the target user interface object is executed.

5. The optical system of claim 4, wherein the processor is further configured to: When the action of the target user interface object does not conform to the specific operational connection of the at least one preset trajectory pattern, a geometric relationship is obtained between a second user interface object among the one or more user interface objects and the one or more gaze points, wherein an action of the second user interface object corresponds to the specific operational connection of the at least one preset trajectory pattern; and The eye tracker is calibrated based on this geometric relationship.

6. The optical system of claim 4, wherein the processor is further configured to: Identify one or more segments relating to the one or more predicted fixations of the user, wherein each segment is located between two adjacent predicted fixations of the user; and The user's gaze trajectory is provided by connecting M consecutive segments from one or more segments, where M is a positive integer.

7. The optical system of claim 6, further comprising: A first buffer for storing one or more estimated gaze points for the user; and A second buffer is used to store the identified one or more segments.

8. The optical system of claim 6, wherein the particular operational link includes one or more preset actions.

9. A method for providing accurate eye-tracking operation in an interactive virtual environment, comprising: Provide a user interface that contains one or more user interface objects; Capture images of one or more of a user's eyes during an eye tracking operation; Calculate one or more predicted fixation points for the user based on one or more eye images of the user; Find at least one estimated gaze point for the user and the distance between each user interface object; Based on one or more eye images of the user, a representation of the eye-tracking operation is derived; Based on the performance graph of the eye-tracking operation, determine the performance level of the eye-tracking operation at the location of each user interface object; A score for each user interface object is calculated based on the performance level of the eye tracking operation at the location of each user interface object and the distance between the user's at least one predicted fixation point and each user interface object; and Set the highest-scoring first user interface object among the one or more user interface objects as the target user interface object associated with the user's at least one estimated gaze point.

10. The method of claim 9, wherein the score of each user interface object is inversely proportional to the distance between the user's at least one estimated gaze point and each user interface object.

11. The method of claim 9, wherein the score of each user interface object is proportional to an error rate at which the eye tracking operates at the location of each user interface object.

12. The method of claim 9, further comprising: Store a set of preset trajectory patterns, each of which corresponds to a specific operational link of the one or more user interface objects. Based on one or more predicted gaze points of the user, calculate a gaze trajectory of the user and store the user's gaze trajectory; When the user's gaze trajectory matches at least one preset trajectory pattern, determine the correlation between an action of the target user interface object and a specific operation link of the at least one preset trajectory pattern; as well as When the action of the target user interface object conforms to the specific operation connection of the at least one preset trajectory pattern, the action of the target user interface object is executed.

13. The method of claim 12, further comprising: When the action of the target user interface object does not conform to the specific operational connection of the at least one preset trajectory pattern, a geometric relationship is obtained between a second user interface object among the one or more user interface objects and the one or more user gaze points, wherein an action of the second user interface object corresponds to the specific operational connection of the at least one preset trajectory pattern; and The eye-tracking operation is corrected based on this geometric relationship.

14. The method of claim 12, further comprising: Identify one or more segments relating to the user's one or more fixations, wherein each segment is located between two adjacent estimated fixations in one or more of the user's estimated fixations; and The user's gaze trajectory is provided by connecting M consecutive segments from one or more segments, where M is a positive integer.

15. The method of claim 14, wherein the particular operational link includes one or more preset actions.

Images