Optical systems and related methods to improve user experience and gaze interaction accuracy

Abstract

This invention discloses an optical system and related method for improving user experience and gaze interaction accuracy. In eye-tracking operation, one or more eye images of the user are first captured, and then one or more gaze points of the user are calculated based on these images. Next, a user interface including one or more user interface elements is provided based on the user's gaze points. A performance map of the eye-tracking operation is obtained based on the user's eye images, and at least one of the one or more user interface elements is adjusted based on the performance map of the eye tracker.

Description

Technical Field

[0001] This invention relates to an optical system and related methods for improving user experience and gaze interaction accuracy, and more particularly to an optical system and related methods for improving user experience and gaze interaction accuracy in interactive virtual environments. Background Technology

[0002] Virtual reality (VR) uses computer technology to simulate a highly realistic three-dimensional space. When users wear special display devices to use VR applications, they experience the illusion of being in reality. Augmented reality (AR) is a technology that augments virtual information into real space. Compared to VR, which replaces real space, AR adds virtual elements to real space. Mixed reality (MR) merges the real and virtual worlds to create a new environment and virtual images that conform to general visual perception. Elements in the real world can coexist with elements in the digital world and interact in real time. Most current VR / AR / MR applications are operated using joysticks or touch panels, but these devices need to be carried around in different locations, which is inconvenient. If eye-tracking technology is applied to the VR / AR / MR field, users can easily turn their eyes into the operating interface, selecting or clicking elements by focusing their gaze, focusing on specific subtle movements.

[0003] Eye tracking is a technology that tracks a user's eye movements by measuring the gaze position or eye movement relative to the head. Existing computers display a user interface on the screen to provide users with various visual data. A user interface typically contains multiple UI elements, each containing a graphic element and a hit box. The graphic element determines the appearance of the corresponding UI element, while the hit box is a virtual element invisible to the user and linked to a corresponding event handler. When a user's gaze command triggers a hit box on a UI element, a predetermined action is executed on the triggered UI element.

[0004] In interactive virtual environments based on eye-tracking gaze technology, the performance of eye tracking and the way the user interface is presented are crucial between the user and the optical system. Therefore, there is a need for optical systems and methods that can improve user experience and the accuracy of gaze interaction. Summary of the Invention

[0005] This invention provides an optical system for improving user experience and gaze interaction accuracy, comprising an eye tracker and a head-mounted display. The eye tracker includes a sensor module for capturing images of one or more eyes of a user. The head-mounted display includes a processor and a display screen. The processor provides a user interface based on one or more gaze points of the user, wherein the user interface includes one or more user interface elements, and the one or more gaze points of the user are calculated based on the one or more eye images of the user; calculates a rendering of the eye tracker based on the one or more eye images of the user; and adjusts at least one of the one or more user interface elements based on the rendering of the eye tracker. The display screen is used to display the user interface.

[0006] The present invention also provides a method for improving user experience and gaze interaction accuracy, comprising: capturing one or more eye images of a user in an eye-tracking operation; calculating one or more gaze points of the user based on the one or more eye images of the user; providing a user interface based on the one or more gaze points of the user, wherein the user interface includes one or more user interface elements; determining a performance map of the eye-tracking operation based on the one or more eye images of the user; and adjusting at least one of the one or more user interface elements based on the performance map of the eye tracker. Attached Figure Description

[0007] Figure 1 This is a functional block diagram of an optical system that can improve user experience and gaze interaction accuracy in an embodiment of the present invention.

[0008] Figure 2 This is a functional block diagram of an optical system that can improve user experience and gaze interaction accuracy in another embodiment of the present invention.

[0009] Figure 3 This is a flowchart illustrating the operation of an optical system that can improve user experience and gaze interaction accuracy in an embodiment of the present invention.

[0010] Figure 4 This is a schematic diagram of the user interface presented during the operation of the optical system in an embodiment of the present invention.

[0011] Figure 5 This is a schematic diagram of the performance of the eye tracker obtained in an embodiment of the present invention.

[0012] Figure 6A and 6B This is a schematic diagram illustrating the confidence value of the eye tracker's performance graph in an embodiment of the present invention.

[0013] Figures 7A-7BFigures 8A-8B and 9A-9B are schematic diagrams illustrating the adjustment of user interface element settings in embodiments of the present invention.

[0014] Figures 10A-10B Figures 11A-11B are schematic diagrams illustrating the adjustment of user interface element settings in embodiments of the present invention.

[0015] Figure 12A-12B Figures 13A-13B are schematic diagrams illustrating the adjustment of the layout of user interface elements in embodiments of the present invention.

[0016] Figures 14A-14D This is a schematic diagram illustrating how the user is informed of the poor performance of the eye tracker in an embodiment of the present invention.

[0017] Explanation of icon numbers:

[0018] 10: Head-mounted displays

[0019] 12, 22: Processor

[0020] 14: Display screen

[0021] 16, 26: Sensor Module

[0022] 18: Input / output devices

[0023] 19: User Interface

[0024] 20: Eye tracker

[0025] 24: Lighting Module

[0026] 30: User

[0027] 100, 200: Optical system

[0028] 310-420: Steps

[0029] UIE1-UIE N User interface elements

[0030] GZ1, GZ2: Focal points

[0031] d1, d2: Distance Detailed Implementation

[0032] Figure 1 This is a functional block diagram of an optical system 100 that can improve user experience and gaze interaction accuracy in an embodiment of the present invention. Figure 2 This is a functional block diagram of an optical system 200 that can improve user experience and gaze interaction accuracy in another embodiment of the present invention.

[0033] exist Figure 1In the illustrated embodiment, the optical system 100 includes a head-mounted display (HMD) 10 and an eye tracker 20. The HMD 10 includes a processor 12, a display screen 14, a sensor module 16, an input / output device 18, and a user interface 19. The eye tracker 20 includes a processor 22, an illumination module 24, and a sensor module 26. The processor 12 controls the operation of the HMD 10, while the processor 22 controls the operation of the eye tracker 20.

[0034] exist Figure 2 In the illustrated embodiment, the optical system 200 is a head-mounted display 10, which includes a processor 12, a display screen 14, a sensor module 16, an input / output device 18, a user interface 19, and an eye tracker 20. The eye tracker 20 includes a processor 22, an illumination module 24, and a sensor module 26. The processor 12 controls the operation of the head-mounted display 10, while the processor 22 controls the operation of the eye tracker 20.

[0035] In other embodiments of the invention, the optical systems 100 and 200 may omit the processor 22. More specifically, the head-mounted display 10 and the eye tracker 20 may share the same processor 12, and the processor 12 controls the operation of the head-mounted display 10 and the eye tracker 20.

[0036] Figure 3 This is a flowchart illustrating the operation of optical systems 100 and 200, which can improve user experience and gaze interaction accuracy in embodiments of the present invention. Figure 3 The flowchart shown includes the following steps:

[0037] Step 310: Capture images of one or more of the user's eyes; proceed to step 320.

[0038] Step 320: Calculate one or more fixation points of the user based on one or more eye images; proceed to step 330.

[0039] Step 330: Provide a user interface 19 containing one or more user interface elements based on one or more user gaze points; perform steps 340 and 370.

[0040] Step 340: Based on the information from user interface 19 and one or more user gaze points, identify a specific user interface element that interacts with the user from one or more user interface elements; proceed to step 350.

[0041] Step 350: Determine whether a specific user interface element is triggered based on one or more user gaze points; if yes, proceed to step 360; if no, proceed to step 310.

[0042] Step 360: Execute a predetermined action corresponding to a specific user interface element; proceed to step 310.

[0043] Step 370: Calculate the performance map of the eye tracker 20 based on one or more eye images of the user; proceed to step 380.

[0044] Step 380: Determine whether the confidence value of the performance graph of the eye tracker 20 is higher than a first threshold; if yes, proceed to step 400; if no, proceed to step 390.

[0045] Step 390: Determine whether the confidence value of the performance graph of the eye tracker 20 is higher than a second threshold; if yes, proceed to step 410; if no, proceed to step 420.

[0046] Step 400: Retain existing user interface elements; proceed to step 310.

[0047] Step 410: Adjust at least one user interface element among one or more user interface elements based on the performance graph of the eye tracker 20; perform step 310.

[0048] Step 420: Inform the user of the poor performance of the eye tracker 20 and notify the user to perform other actions.

[0049] In the optical system 100, the sensor module 26 of the eye tracker 20 includes at least one image sensor (eye sensor) for capturing one or more eye images of the user in step 310. The processor 22 of the eye tracker 20 can receive the one or more eye images captured by the sensor module 26 and calculate one or more gaze points of the user based on the one or more eye images in step 320. Furthermore, the processor 22 of the eye tracker 20 can also calculate other eye-tracking related data based on the one or more eye images, such as the confidence and accuracy of gaze point estimation, eye position in three-dimensional space, and pupil-related information (e.g., pupil size). The algorithm for eye tracking operation can be made into a program / software / firmware executable by the processor 22 of the eye tracker 20, but this does not limit the scope of the invention.

[0050] In the optical system 200, the sensor module 26 of the eye tracker 20 includes at least one image sensor (eye sensor) for capturing one or more eye images of the user in step 310. The processor 12 can receive the one or more eye images captured by the sensor module 26 and calculate one or more gaze points of the user based on the one or more eye images in step 320. Furthermore, the processor 12 can also calculate other eye-tracking related data based on the one or more eye images, such as the confidence and accuracy of gaze point estimation, eye position in three-dimensional space, and pupil-related information (e.g., pupil size). The eye-tracking algorithm can be made into a program / software / firmware executable by the processor 12, but this does not limit the scope of the invention.

[0051] In optical systems 100 and 200, sensor module 16 includes at least one scene sensor, at least one sound sensor (e.g., a microphone), and / or at least one motion sensor (e.g., a gyroscope or accelerometer). The scene sensor can capture one or more scene images of the current field of view in relation to the user, the sound sensor can receive voice commands issued by the user, and the motion sensor can detect the user's movements (typically head movements).

[0052] In optical systems 100 and 200, illumination module 24 may include one or more infrared light-emitting diodes (LEDs) to illuminate the user's eyes, ensuring sufficient contrast between the iris and pupil for users with different eye colors, especially against very bright or very dark backgrounds, thereby improving the accuracy of eye tracker 20 in capturing reflected light from the user's eyes. However, the implementation of illumination module 24 does not limit the scope of the invention.

[0053] In optical system 100 or 200, a virtual field of view and one or more user interface elements UIE1-UIE are included. N The user interface 19 can be displayed on the display screen 14, where N is a positive integer. In one embodiment, the user interface elements UIE1-UIE N These can be abstract elements that are not visible to the user. If the user interface elements are UIE1-UIE... N As interactive elements, each user interface element is linked to an event handler, where each event handler is related to a specific operation of the optical system 100 or 200 and is controlled by the processor 12. In this invention, the user interface elements UIE1-UIE N The appearance and functions can be dynamically adjusted based on the operating status of the eye tracker 20, thereby providing an optimized user experience.

[0054] User interface elements UIE1-UIE N This can increase the interactivity of the user interface 19 and provide touch points for users to issue eye-tracking commands. The function of each user interface element may be related to input control, browsing control, and message display, but is not limited to the scope of the invention. Each user interface element includes an image element and a hit box. The image element can determine the appearance of the corresponding user interface element and may be related to the function of the corresponding user interface element. In one embodiment, the image element of the user interface element may be a checkbox, radio button, dropdown list, list box, toggle button, or date / time option related to input control. In another embodiment, the image element of the user interface element may be a breadcrumb, slider, pagination, icon, or image carousel related to browsing control. In another embodiment, the image element of the user interface element may be a tooltip, progress bar, notification, message box, or modal window related to message display. However, the appearance of the graphic elements of the user interface elements does not limit the scope of this invention.

[0055] The hit box of a user interface element is a virtual element invisible to the user and is linked to a corresponding event handler. When a gaze command associated with one or more user gaze points triggers the hit box of a user interface element, the processor 12 executes the action corresponding to the triggered user interface element. The present invention determines that a user interface element has been triggered by a gaze command when one or more trigger conditions are met. These trigger conditions include at least one of the following: after one or more user gaze points are within the hit box of a user interface element, the user immediately or within a predetermined time interval confirms this via the input / output device 18 or sensor module 16 (e.g., a microphone); when one or more user gaze points are within the hit box of a user interface element for more than a predetermined gaze period; after one or more user gaze points are within the hit box of a user interface element, the user immediately or within a predetermined time interval confirms this via spontaneous eye movements (e.g., eye saccades or blinks); and when a gaze trajectory containing one or more user gaze points passes through the reaction boundary of the hit box of a user interface element. However, the type of trigger conditions associated with each user interface element does not limit the scope of the present invention.

[0056] In optical system 100 or 200, input / output device 18 can receive commands from the user. In one embodiment, input / output device 18 may include any type of handheld controller (e.g., gamepad or game console) and / or any form of haptic feedback device (e.g., motion-sensing suit or glove). Input / output device 18 can detect user motion signals and transmit them to processor 12 of optical system 100 or 200. In one embodiment, processor 12 can control the operation of optical system 100 or 200 according to user commands received by input / output device 18. In another embodiment, processor 12 can simultaneously control the operation of optical system 100 or 200 according to user commands received by input / output device 18 and user interface elements UIE1-UIE of user interface 19. N The received gaze command controls the operation of optical system 100 or 200.

[0057] In step 330, processor 12 may provide user interface 19 based on one or more user gaze points, wherein user interface 19 includes one or more user interface elements. In another embodiment, processor 12 may also provide user interface 19 based on one or more images related to the user's field of vision and the user's actions. Figure 4 This is a schematic diagram of the user interface 19 presented during the operation of optical systems 100 and 200 in an embodiment of the present invention. The existing information in the user interface 19 includes user interface elements UIE1-UIE. N The layout. For illustrative purposes, Figure 4 An embodiment with N=9 is shown, where user interface elements UIE1-UIE9 are set as an array. However, user interface elements UIE1-UIE... N The quantity and location do not limit the scope of this invention.

[0058] In step 340, the processor 12 identifies a specific user interface element that interacts with the user from one or more user interface elements based on the information from the user interface 19 and one or more user gaze points. The information from the user interface 19 includes user interface elements UIE1-UIE. N The current layout, for example Figure 4 The user interface elements UIE1-UIE9 are shown. Assuming that one or more gaze points calculated in step 320 are located on user interface element UIE6, the processor 12 will determine that the user is interacting with user interface element UIE6 through gaze commands.

[0059] In step 350, the processor 12 determines whether a specific user interface element has been triggered based on one or more of the user's gaze points. When a gaze command associated with one or more user-related gaze points satisfies one or more triggering conditions, the present invention determines that the specific user interface element has been triggered by the gaze command. These triggering conditions include the user's gaze point being within the hit box of the specific user interface element for more than the gaze period associated with the specific user interface element, the user pressing another button, the user issuing a voice command, the user blinking voluntarily, or the detection of a specific gaze point trajectory / pattern, but do not limit the scope of the present invention.

[0060] After determining that a specific user interface element has been triggered based on one or more user gaze points, the processor 12 will execute a predetermined action corresponding to the specific user interface element in step 360. The predetermined actions include content selection, going to the previous page, going to the next page, settings, closing the page, returning, returning to the home page, displaying a notification message, or locking the screen, but are not limited to the scope of this invention.

[0061] In step 370, the processor 12 calculates the performance graph of the eye tracker 20 based on images of one or more of the user's eyes and the user's initial visual range. The user's initial visual range refers to the visual range when the user is looking straight ahead, and can be calculated based on the physical specifications of the optical system 100 or 200 and the user's standard anthropometric data. The performance graph of the eye tracker 20 represents the operational performance of the user's initial visual range associated with the eye tracker 20. Since the sensor module 26 is fixed in position within the eye tracker 20, when the user wears the eye tracker 20, the position of the sensor module 26 only changes with the user's head movements, but is not affected by the user's eye movements. Therefore, the performance graph of the eye tracker 20 changes with the user's head movements, but is unrelated to the user's eye movements.

[0062] Figure 5 This is a schematic diagram of the performance of the eye tracker 20 obtained in step 370 of this embodiment of the invention. The vertical axis represents the user's vertical visual range (in degrees), the horizontal axis represents the user's horizontal visual range (in degrees), and the numbers represent the error rate of the eye tracker 20 at different positions within the user's visual range. Generally, the accuracy of the eye tracker 20 decreases as eccentricity increases. Figure 5 As shown, the average error rate of eye tracker 20 is approximately 1.88° in the central area of ​​the user's field of vision, approximately 1.96° within ±10° of the user's field of vision, and approximately 2° within ±20° of the user's field of vision.

[0063] Based on the performance chart of the eye tracker 20 obtained in step 370, the error rate of the eye tracker 20 during its initial use can be: Figure 5 The initial values ​​shown are then dynamically updated during the operation of optical systems 100 and 200. In one embodiment, the initial values ​​of the performance graph of eye tracker 20 can be adjusted by the user based on their own parameters, such as visual acuity and personal preferences. For example, assuming that a user has difficulty gazing at elements to the right due to poorer right-eye vision, the average error rate of eye tracker 20 within the user's right visual range can be manually increased.

[0064] In another embodiment, the initial values ​​of the performance graph of the eye tracker 20 can be automatically adjusted based on the relationship between the user and the optical system 100 / 200 structure. For example, the accuracy of the eye tracker 20 can be related to the distance between the user's eyes and the sensor module 26 in the eye tracker 20. For illustrative purposes, assume that the sensor module 26 in the eye tracker 20 is located below the center of the user's field of vision. Under the same degree of head / eye movement in the vertical direction, the accuracy of the eye tracker 20 will be higher when the user looks downward than when the user looks upward. Figure 5 As shown, the average error rate of eye tracker 20 is approximately 1.55° in the area below the user's field of vision, and approximately 2.44° in the area above the user's field of vision.

[0065] In one embodiment, the performance graph of the eye tracker 20 may be a single graph constructed based on a single performance parameter of the eye tracker 20, such as accuracy, confidence level, stability / accuracy, or sampling rate of eye tracking operation. In another embodiment, the performance graph of the eye tracker 20 may be constructed based on a plurality of performance parameters of the eye tracker 20, such as at least two of accuracy, confidence level, stability / accuracy, and sampling rate of eye tracking operation. In embodiments employing a plurality of performance parameters, each performance parameter may also have its own weighted value. These weighted values ​​may be automatically determined by the optical systems 100 and 200, or determined by the user based on their preferences or the type of application currently being used.

[0066] In another embodiment, the performance graph of the eye tracker 20 may comprise a plurality of graphs, each graph containing at least one subgraph. Each subgraph may be constructed based on a single performance parameter of the eye tracker 20, such as the accuracy, confidence level, stability / precision, or sampling rate of the eye tracking operation.

[0067] The confidence value of eye tracker 20 refers to the confidence value in predicting the user's gaze point. Eye tracker 20 can derive eye center point information based on images of one or more of the user's eyes. Based on the position of the user's eye center point, the possible pupil position and shape projected onto the image plane can be calculated and constrained to a very small parameter space. When the user's pupil image captured by sensor module 26 exceeds the parameter space, sensor module 20 can be considered to have low confidence and low accuracy.

[0068] Figure 6A and 6B This is a schematic diagram illustrating the confidence value of the eye-tracking device 20's performance in an embodiment of the present invention. Figure 6A As shown, when a large portion of the range of each gaze point GZ1 overlaps with a user element, the sensor module 20 can be considered to have high confidence and high accuracy. Figure 6B As shown, when most of the range of each gaze point GZ1 does not overlap with a user element, the sensor module 26 can be considered to have low confidence and low accuracy.

[0069] In another embodiment, the sensor module 26 of the eye tracker 20 may further include at least one inertial sensor for detecting a slippage that may alter the relationship between the user's eyes and the sensor module 26 in the eye tracker 20. For example, when the estimated user pupil position based on measurements from the sensor module 26 and the inertial sensor differs from the actual user pupil position, the sensor module 26 can be considered to have low confidence and low accuracy.

[0070] In steps 380 and 390, the processor 12 evaluates the confidence value of the performance map of the eye tracker 20 at the location of a specific user interface element. In embodiments where the performance map of the eye tracker 20 comprises a single image, the evaluation can be performed by comparing the confidence value of the eye tracker 20, a first threshold, and a second threshold. In embodiments where the performance map of the eye tracker 20 comprises multiple sub-images, the evaluation can be performed by comparing the confidence value of the eye tracker 20, multiple first thresholds, and multiple second thresholds, wherein each first threshold and second threshold is associated with a corresponding sub-image, and the confidence value derived from each sub-image is evaluated individually. To reduce the complexity of the comparison, the present invention can convert the confidence value obtained under the above comparison conditions into an abstract scale, wherein a larger value on the abstract scale corresponds to better performance of the eye tracker 20.

[0071] When it is determined in step 380 that the confidence value of the eye tracker 20 at the location of a specific user interface element is higher than a first threshold, the processor 12 executes step 400 to retain the existing settings of the user interface element.

[0072] When it is determined in steps 380 and 390 that the confidence value of the eye tracker 20 at the location of a specific user interface element is between a first threshold and a second threshold, the processor 12 executes step 410 to adjust the settings of at least one user interface element among one or more user interface elements based on the performance graph of the eye tracker 20.

[0073] In one embodiment, the method by which the processor 12 adjusts the settings of a user interface element includes adjusting the size of a corresponding image element of at least one user interface element, adjusting the size of a corresponding frame of at least one user interface element, and / or adjusting a gaze duration of a corresponding frame of at least one user interface element. As previously described, the frame of a user interface element is a virtual element that is not visible to the user and is linked to a corresponding event handler to await a specific trigger event from the user. The image element is the portion of the corresponding user interface element that is visible to the user. The specific user interface element is triggered when the user's gaze command remains within the frame of a particular user interface element for more than its corresponding gaze duration. When the position of a specific user interface element corresponds to a low confidence value region (e.g., low accuracy, low confidence value, or low sampling rate) on the performance map of the eye tracker 20, the present invention can increase the range of the hit box of the specific user interface element to make it easier for the user's gaze to hit the hit box of the specific user interface element, can decrease the range of the image element of the specific user interface element to make it easier for the user's gaze to hit the hit box of the specific user interface element, and / or can increase the gaze duration of the specific user interface element to reduce the probability of the hit box of the specific user interface element being triggered by misjudgment (especially when the position of the specific user interface element corresponds to a low sampling rate region on the performance map of the eye tracker 20).

[0074] Figures 7A-7B Figures 8A-8B and 9A-9B are schematic diagrams illustrating the adjustment of user interface element settings in step 410 of this embodiment of the invention. For illustrative purposes, it is assumed that three user interface elements UIE1-UIE3 are provided within the user's field of vision (FoV), and the position of each user interface element is not affected by the user's head movement. When the user 30 is looking straight ahead, user interface elements UIE1 and UIE3 are located within the peripheral field of vision of the user 30, while user interface element UIE2 is located within the central field of vision of the user 30, as shown below. Figure 7A As shown. When user 30 turns their gaze to the right and then turns their head, user interface elements UIE1 and UIE2 are located within the peripheral visible range of user 30, while user interface element UIE3 is located within the central visible range of user 30, as shown. Figure 7BAs shown. When one or more sensors in sensor modules 16 and 26 detect head / eye movement of user 30, the present invention updates the performance map of eye tracker 20 based on the new gaze point position of user 30, and updates the relationship between user 30's field of vision and user interface 19 based on user's head movement. As mentioned earlier, the accuracy of eye tracker 20 in the central field of vision of user 30 (lower eccentricity) is higher than that in the peripheral field of vision of user 30 (higher eccentricity). Therefore, the present invention can dynamically update user interface elements UIE1-UIE3 based on user's head / eye movement. For example, since eye tracker 20 has lower accuracy in the peripheral field of vision of user 30, each user interface element located in the peripheral field of vision of user 30 can contain a larger hit box (represented by a dashed rectangle), such as... Figure 7A and 7B As shown. Because the eye tracker 20 has low accuracy within the peripheral visual field of the user 30, each user interface element within the peripheral visual field of the user 30 can contain a small image element (represented by a dotted rectangle), such as Figure 8A and 8B As shown. When the sampling rate of the eye tracker 20 within the central visual field of the user 30 is higher than the sampling rate within the peripheral visual field of the user 30, each user interface element within the peripheral visual field of the user 30 can have a longer gaze duration, such as... Figure 9A and 9B As shown.

[0075] Figures 10A-10BFigures 11A-11B are schematic diagrams illustrating the adjustment of user interface element settings in step 410 of this embodiment of the invention. For illustrative purposes, it is assumed that two user interface elements, UIE1 and UIE2, are provided within the user's field of vision (FoV). When one or more sensors in sensor modules 16 and 26 detect head / eye movement of the user 30, the present invention updates the rendering of the eye tracker 20 based on the user 30's new gaze point position and updates the relationship between the user's field of vision and the user interface 19 based on the user's head movement. As previously mentioned, the accuracy of the eye tracker 20 can be a function of the distance between the user's eyes and sensor module 26. For illustrative purposes, it is assumed that the sensor module 26 of the eye tracker 20 is located below the central field of vision of the user 30, and the accuracy of the eye tracker 20 is linearly related to the distance between the user's eyes and sensor module 26, that is, the farther the sensor module 26 is from the user's eyes, the lower the accuracy of the eye tracker 20. When user 30 turns their gaze to the user interface element UIE1, the distance between the user's eyes and the sensor module 26 shortens, allowing the eye tracker 20 to have higher accuracy in positioning the user interface element UIE1. At this time, the user interface element UIE1 can contain a small hit box (represented by a dashed rectangle), such as... Figure 10A As shown. When user 30 turns their gaze to user interface element UIE2, the distance between the user's eyes and sensor module 26 increases, resulting in lower accuracy of eye tracker 20 in locating UIE2. In this case, user interface element UIE1 can contain a larger bounding box (represented by a dashed rectangle), such as... Figure 10B As shown. When user 30 turns their gaze to user interface element UIE1, the distance between the user's eyes and sensor module 26 shortens, allowing eye tracker 20 to have higher accuracy in positioning user interface element UIE1. At this time, user interface element UIE1 can contain larger image elements (represented by dotted rectangles), such as... Figure 11A As shown. When user 30 turns their gaze to user interface element UIE2, the distance between the user's eyes and sensor module 26 increases, resulting in lower accuracy of eye tracker 20 in locating UIE2. At this time, UIE2 may contain smaller image elements (represented by dotted rectangles), such as... Figure 11B As shown.

[0076] exist Figure 7A-10A In the embodiments shown in 7B-11B, when a first user interface element corresponds to a lower accuracy value on the rendering map and a second user interface element corresponds to a higher accuracy value on the rendering map, the difference between the first user interface element and the second user interface element can be achieved by adjusting only the first user interface element, adjusting only the second user interface element, or adjusting both the first user interface element and the second user interface element simultaneously.

[0077] After adjusting the settings of the user interface elements, the present invention can also adjust the layout of the user interface elements accordingly to avoid overlapping of the user interface elements. Figure 12A-12B Figures 13A-13B are schematic diagrams illustrating the adjustment of user interface element layout in embodiments of the present invention. For illustrative purposes, Figure 12A-12B Figures 13A-13B show an embodiment with N=6, where the user interface elements UIE1-UIE6 are set as an array.

[0078] exist Figure 12A In the diagram, the positions of user interface elements UIE1-UIE6 correspond to higher accuracy values ​​on the rendering map. Figure 12B In the diagram, the positions of user interface elements UIE1-UIE6 correspond to lower precision values ​​on the rendering map. Figure 12B The distance d2 between two adjacent user interface elements can be greater than Figure 12A The distance d1 between two adjacent user interface elements is set to prevent UIE1-UIE6 from overlapping after adjusting their settings (e.g., after shrinking image elements and / or enlarging the hitbox).

[0079] exist Figure 13A In the diagram, the positions of user interface elements UIE1-UIE6 correspond to higher accuracy values ​​on the rendering map. Figure 13B In the diagram, the positions of user interface elements UIE1-UIE6 correspond to lower precision values ​​on the rendering map. Figure 13B The number of rows / columns of the array formed by UIE1-UIE6 and the size of each UIE6 element can be changed to prevent UIE1-UIE6 from overlapping after adjusting the settings of UIE1-UIE6 (e.g., after enlarging an image element or shrinking the hitbox).

[0080] While modifying user interface elements can improve user experience in most cases, this is not applicable when the eye tracker 20 is performing poorly. In this invention, the value of the second threshold is less than the value of the first threshold, and it is used to define the minimum value required for the eye tracker 20 to provide adequate performance. When it is determined in steps 380 and 390 that the confidence value of the performance graph of the eye tracker 20 is not higher than the first and second thresholds, this invention informs the user of the poor performance of the eye tracker 20 in step 420 and notifies the user to perform other actions.

[0081] Figures 14A-14D This is a schematic diagram illustrating how a user is informed of a low performance of an eye tracker in an embodiment of the present invention. For illustrative purposes, Figures 14A-14DAn example with N=6 is shown, where the user interface elements UIE1-UIE6 are set as an array.

[0082] When the confidence value of the performance graph of the eye tracker 20 is determined to be higher than the first threshold and the second threshold, the user interface elements UIE1-UIE6 presented by the user interface 19 are as follows: Figure 14A As shown.

[0083] When the eye tracker 20 is deemed to be performing poorly, the appearance of each user interface element can be altered (e.g., changing color, fading color, brightening, or flashing) to indicate that the gaze interaction function of each user interface element has been disabled. Figure 14B As shown.

[0084] When the eye tracker 20 is deemed to be performing extremely poorly, the appearance of each user interface element can be blurred to indicate that the gaze interaction function of each user interface element has been turned off, such as... Figure 14C As shown.

[0085] When the eye tracker 20 is deemed to be performing extremely poorly, the position of each user interface element can be shuffled to indicate that the gaze interaction function of each user interface element has been disabled, such as... Figure 14D As shown.

[0086] In one embodiment, after informing the user of the poor performance of the eye tracker 20, the processor 12 may also disable gaze interaction and switch to another form of interaction (e.g., interaction based on the input / output device 18). In another embodiment, after informing the user of the poor performance of the eye tracker 20, the processor 12 may also request the user to recalibrate the eye tracker 20.

[0087] In this invention, the optical system presents one or more user interface elements on the user interface to provide user interaction functionality based on eye-tracking technology. During eye-tracking operation, the invention dynamically adjusts one or more user interface elements based on the performance graph of the eye tracker 20 / eye-tracking operation. Therefore, this invention provides an optical system and method that can improve user experience and the accuracy of gaze interaction.

[0088] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made in accordance with the claims of the present invention shall fall within the protection scope of the present invention.

Claims

1. An optical system that enhances user experience and gaze interaction accuracy, characterized by, Include: One-eye motion tracker includes: A first sensor module is used to capture images of one or more eyes of a user; and A head-mounted display, comprising: A first processor is used to: A user interface is provided based on one or more gaze points of the user, wherein the user interface includes one or more user interface elements, and the one or more gaze points of the user are calculated based on one or more eye images of the user; An eye tracker performance graph is constructed based on at least one of an accuracy, a confidence value, a stability, a precision, and a sampling rate, wherein the accuracy, the confidence value, the stability, the precision, and the sampling rate of the eye tracker are determined based on one or more eye images of the user. as well as Adjust at least one of the one or more user interface elements based on the performance graph of the eye tracker; and A display screen is used to present the user interface.

2. The optical system as described in claim 1, characterized in that: The eye tracker also includes a second processor for: Receive the one or more eye images captured by the first sensor module; and The user's one or more gaze points are calculated based on the one or more eye images; and The first processor is also used to receive the user's one or more gaze points from the second processor.

3. The optical system of claim 1, wherein, The first processor is also used for: Receive the one or more eye images captured by the first sensor module; and The user's one or more gaze points are calculated based on the images of one or more eyes.

4. The optical system as claimed in claim 1, characterized in that: Each user interface element contains an image element and a hitbox; and The first processor adjusts the at least one user interface element by adjusting the size of a corresponding image element of the at least one user interface element, adjusting the size of a corresponding hit box of the at least one user interface element, and / or adjusting a gaze duration of the corresponding hit box of the at least one user interface element based on the performance graph of the eye tracker.

5. The optical system of claim 1, wherein, The first processor is also used for: Based on the performance graph of the eye tracker, adjust the distance between two adjacent user interface elements, adjust the layout of the multiple user interface elements, and / or adjust the size of each user interface element.

6. The optical system of claim 1, wherein, The first processor is also used for: Based on information from the user interface and the user's one or more gaze points, identify a specific user interface element that interacts with the user from among the one or more user interface elements; Determine whether a particular user interface element is triggered by a gaze command from one or more gaze points associated with the user; and When a specific user interface element is triggered by the gaze command, a predetermined action is performed corresponding to that specific user interface element.

7. The optical system as claimed in claim 6, characterized in that: Each user interface element contains an image element and a hitbox; and The first processor is also used for: When a specific user interface element corresponds to a high confidence value on the performance graph of the eye tracker, the size of a corresponding image element of the specific user interface element is increased, the size of a corresponding hit box of the specific user interface element is reduced, and / or the gaze duration of the corresponding hit box of the specific user interface element is shortened. or When a specific user interface element corresponds to a low confidence value on the performance graph of the eye tracker, the size of a corresponding image element of the specific user interface element is reduced, the size of a corresponding hit box of the specific user interface element is increased, and / or the gaze duration of the corresponding hit box of the specific user interface element is increased.

8. The optical system of claim 6, wherein, The first processor is also used for: When the multiple user interface elements correspond to a low confidence value on the performance map of the eye tracker, the distance between the specific user interface element and an adjacent user interface element is increased, or the layout of the multiple user interface elements is adjusted to increase the distribution area of ​​the multiple user interface elements on the user interface.

9. The optical system of claim 1, wherein, The first processor is also used for: Determine whether a confidence value of the performance graph of the eye tracker is higher than a first threshold; Determine whether the confidence value of the performance graph of the eye tracker is higher than a second threshold value, wherein the second threshold value is lower than the first threshold value; as well as When the confidence value of the performance graph of the eye tracker is determined to be between the first threshold and the second threshold, the at least one user interface element is adjusted according to the performance graph of the eye tracker.

10. The optical system of claim 9, wherein, The first processor is also used for: When it is determined that the confidence value of the performance graph of the eye tracker is not higher than the second threshold, the user is notified of the low performance of the eye tracker.

11. The optical system of claim 10, wherein, The first processor is also used for: After notifying the user of the poor performance of the eye tracker, disable gaze interaction functionality for each user interface element and / or require the user to recalibrate the eye tracker.

12. A method of improving user experience and gaze interaction accuracy, characterized by, Include: Capture images of one or more of a user's eyes during eye tracking operations; Calculate one or more fixation points of the user based on the images of the user's one or more eyes; A user interface is provided based on the user's one or more gaze points, wherein the user interface contains one or more user interface elements; An eye-tracking performance graph is constructed based on at least one of an accuracy, a confidence score, a stability, a precision, and a sampling rate, wherein the accuracy, the confidence score, the stability, the precision, and the sampling rate are determined based on images of one or more of the user's eyes; and Adjust at least one of the one or more user interface elements based on the performance graph of the eye tracker.

13. The method of claim 12, wherein, Each user interface element contains an image element and a hit box, and the method further includes: Based on the performance graph of the eye-tracking operation, adjust the size of a corresponding image element of the at least one user interface element, adjust the size of a corresponding hit box of the at least one user interface element, and / or adjust the gaze duration of the corresponding hit box of the at least one user interface element.

14. The method of claim 12, wherein, Also includes: Based on the performance graph of the eye-tracking operation, adjust the distance between two adjacent user interface elements, adjust the layout of the multiple user interface elements, and / or adjust the size of each user interface element.

15. The method of claim 12, wherein, Also includes: Based on information from the user interface and the user's one or more gaze points, identify a specific user interface element that interacts with the user from among the one or more user interface elements; Determine whether a particular user interface element is triggered by a gaze command from one or more gaze points associated with the user; as well as When it is determined that a specific user interface element is triggered by the gaze command, a predetermined action is executed for that specific user interface element.

16. The method as described in claim 15, characterized in that, This particular user interface element contains an image element and a hit box, and the method also includes: When a specific user interface element corresponds to a high confidence value on the performance graph of the eye-tracking operation, increase the size of a corresponding image element of that specific user interface element, decrease the size of a corresponding hit box of that specific user interface element, and / or shorten the gaze duration of the corresponding hit box of that specific user interface element; or When a specific user interface element corresponds to a low confidence value on the performance graph of the eye-tracking operation, the size of a corresponding image element of the specific user interface element is reduced, the size of a corresponding hit box of the specific user interface element is increased, and / or the gaze duration of the corresponding hit box of the specific user interface element is increased.

17. The method as described in claim 15, characterized in that, Also includes: When the multiple user interface elements correspond to a low confidence value on the performance map of the eye tracker, the distance between the specific user interface element and an adjacent user interface element is increased or the layout of the multiple user interface elements is adjusted to increase the distribution area of ​​the multiple user interface elements on the user interface.

18. The method as described in claim 12, characterized in that, Also includes: Determine whether a confidence value of the performance graph of the eye-tracking operation is higher than a first threshold; Determine whether the confidence value of the performance graph of the eye-tracking operation is higher than a second threshold, wherein the second threshold is lower than the first threshold; as well as When the confidence value of the performance graph of the eye-tracking operation is determined to be between the first threshold and the second threshold, the at least one user interface element is adjusted according to the performance graph of the eye-tracking operation.

19. The method as described in claim 18, characterized in that, Also includes: When the confidence value of the performance graph of the eye tracker is determined to be no higher than the second threshold, the user is notified of the low performance of the eye tracker operation.

20. The method as described in claim 19, characterized in that, Also includes: When it is determined that the confidence value of the performance graph of the eye-tracking operation is not higher than the second threshold, the user is notified of the low performance of the eye-tracking operation by changing the appearance of each user interface element, blurring each user interface element, and / or randomly changing the position of each user interface element.

21. The method as described in claim 20, characterized in that, Also includes: After notifying the user of the poor performance of the eye-tracking operation, disable the gaze interaction functionality of each user interface element and / or require the user to recalibrate the eye-tracking operation.

Images