Information processing system using gaze dwell and high-speed eyelid closure

Abstract

To deal with the situation in which in a gaze-tracking interface, existing systems has relied solely on manual input operations using the fingers and this has been making it natural for users to experience malfunctions, physical fatigue, and increased time and effort.SOLUTION: The system detects an intentional fast blink (high-speed eyelid-closure motion) of a user by a sensor while the user is looking at a target of operation. Only when the eyelid-closure time is shorter than a preset threshold which is the line between the eyelid closure and an unconscious normal blink, the system recognizes the action as a decision (click) operation and executes processing. This enables high-speed and accurate hands-free operations.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a head-mounted display (HMD) with an eye-tracking function, and more particularly to an information processing system and method for performing input operations by combining gaze fixation and high-speed eyelid closing. [Background technology]

[0002] With the development of virtual reality (VR) and augmented reality (AR) technologies, there is a demand for hands-free computer operation techniques using HMDs. Many technologies have been proposed to date that allow users to position a cursor with their gaze and click by blinking (closing the eyelids).

[0003] Many input devices using eye-tracking technology and blink detection have been proposed. For example, Patent Document 1 (Japanese Patent Laid-Open Publication No. 2005-100366) describes a technique for determining input by utilizing the length of time the eyelids are closed. Additionally, US patents (US8456789, etc.) and Chinese patents (CN111897414, etc.) disclose systems that perform selection operations using eye tracking and blinking. However, these conventional technologies have not fully considered a configuration in which the speed of eyelid closing movement itself is precisely measured and used as an input trigger, or a mechanism for preventing malfunctions based on a combination of conditions including gaze retention. As a result, there were problems such as unconscious blinking being recognized as an incorrect operation, and it was difficult to achieve both reaction speed and operational reliability.

[0004] A comparison of the major prior art and this patent is shown in Table 1 below. The billing items were broken down into elements by function, and the items were investigated for matches, mismatches, and partial matches. Table 2 provides a detailed description of the functional elements. Legend: 〇 = Match △ = Partial match (similar elements present) X = Non-match (differentiable) Table 1 Comparison table TIFF2025172106000002.tif50159Table 2 Detailed description of functional elements TIFF2025172106000003.tif49126 Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention solves the above-mentioned conventional problems, and aims to provide an information processing system that enables fast response while preventing malfunctions by determining input operations based on the time of eyelid closing and (time and speed), (time and acceleration), or (rate of change of either time or speed acceleration). [Means for solving the problem]

[0006] The reason why no one has noticed this invention until now is thought to be because people have fallen into several "thinking traps (biases)." This is the trap of the common sense that "blinks = noise." For researchers specializing in EEG measurement and precise eye tracking, blinks have always been considered "annoying noise that needs to be removed." Because the researchers were so focused on "how to remove blinking from the data," it was difficult for them to come up with the reverse idea of ​​"utilizing" it. I also fell into the trap of the stereotype that "input = intentional action." Until now, computer interface designers have considered the act of "clicking" to be a substitute for clearly intended, conscious physical actions such as "pressing a button" or "picking with your fingers." As a result, people tended to seek solutions in clearly unusual, conscious actions such as "closing their eyes for a long time" or "blinking twice." As a result, it seems that they did not come up with the delicate and ergonomic idea of ​​incorporating a slight amount of intention (speed) into an everyday, unconscious action (normal blinking).

[0007] In order to solve the above problems, the present inventors have conducted extensive observations and analyses of the physiological phenomenon of human "blinking," and as a result have discovered something that no one has noticed before: the time (speed) required from closing to opening the eyelids is clearly different between "normal blinking that is performed unconsciously" and "fast blinking that is performed intentionally." Based on this finding, the present invention uses a completely new parameter, "speed," as an input trigger, instead of the conventional "number of times" or "length." Specifically, sensors built into the HMD constantly monitor the movement of the user's eyelids. Then, if the eyelid closure time is significantly shorter than the average time of a normal involuntary blink obtained for each individual (approximately 0.2 to 0.4 seconds) and is equal to or less than a predetermined threshold (e.g., less than 0.2 seconds), the input is confirmed as being below the threshold, and is defined as being valid as a "fast blink." Any eyelid closure longer than this threshold is considered an involuntary blink and no processing is performed. This extremely simple principle essentially eliminates the possibility of operational errors, while enabling high-speed input confirmation without slowing down the speed of thought. [Effects of the Invention]

[0008] According to the present invention, by combining the judgment based on the eyelid closing speed with the gaze retention condition, it is possible to prevent erroneous operations and achieve high-speed response at the same time. In addition, individual settings through calibration allow for flexible response to differences between users and environments. On first use, the user blinks normally several times and the system measures the average eyelid closure time. The system automatically sets the threshold for "rapid blinking" to a time significantly shorter than the average time (e.g., 50% of the average time). With zero learning cost, anyone can intuitively operate the device quickly and hands-free. There is no physical strain from moving your fingers or mental strain from closing your eyes for long periods of time, allowing users to immerse themselves in interacting with the computer without interrupting their train of thought. This means that the technology has immeasurable industrial potential, not only in the entertainment industry, such as e-sports, where high-speed operation is required, but also in medical and manufacturing settings where hands are often occupied, and as accessibility technology for people with physical disabilities. This is truly a disruptive invention that takes the fusion of humans and computers to a new level. [Brief explanation of the drawings]

[0009] [Figure 1] An image of locking your gaze onto the icon on the menu screen in the VR goggles and pressing the play button with a rapid blink. [Figure 2] An image of gazing at the camera icon on a smartphone's desktop and rapidly blinking to launch the camera app. [Figure 3] An image of the eyelid transition during two blinks. [Figure 4] It's like rapidly blinking to press the shutter when taking a group photo of four people. [Figure 5] Appearance of the Apple Vision Pro (taken from the official Apple website). [Figure 6] System processing concept diagram. [Figure 7] Conceptual diagram of basic processing flow. [Figure 8] Calibration process diagram. [Figure 9] Input judgment flow for closed eyelid + open eyelid measurement. [Figure 10] Input determination flow using double blinks. [Figure 11] Judgment flow using speed, acceleration, and rate of change. [Figure 12] Illustration of how it works with iris authentication. DETAILED DESCRIPTION OF THE INVENTION [Example]

[0010] Example 1 (basic use example) In this embodiment, the "Vision Pro" manufactured by Apple Inc. will be used as an example. The HMD is equipped with an eye-tracking camera and infrared sensors to detect the operator's gaze position and eyelid movement in real time. When the gaze stays on a specific icon for more than 0.5 seconds, the system locks the position and measures the time from the start of the eyelid closing action to the completion of the eyelid closing action. If the measurement time is less than 0.15 seconds, it is determined to be a rapid eyelid closure (rapid blink), and the corresponding processing (selection, activation, etc.) is executed. As a specific example, in FIG. 1, while the eye 2 is fixed with the line of sight 3 on the play button 4 in the main menu 1 displayed on the built-in display for a predetermined period of time or more, playback is initiated by a rapid blink. The VR goggles are equipped with all the necessary sensor functions, and can be realized through software development.

[0011] In FIG. 2, while an eye 2 is held with a line of sight 3 on a camera icon 4 in a main menu 1 displayed on a built-in display for a predetermined period of time or more, a rapid blink activates the camera app. Figure 6 is a conceptual diagram of the system processing, showing the flow of comparing the input with a threshold value after processing by the HMD and deciding whether to perform the processing. FIG. 7 is a conceptual diagram of a basic processing flow showing comparison with a threshold and branching of decision processing. The expression "multiply by a predetermined ratio" in the calibration process diagram of FIG. 8 means that the average blink rate is corrected (scaled) to determine the blink threshold for each user. Calibration calculation result example: The average eyelid closing time is set to 0.352 seconds, and this is multiplied by the ratio coefficient set on the system side = 0.8 (= 80% or less of the average is considered "fast"). Eyelid closure threshold = 0.352 seconds × 0.8 = 0.282 seconds For the user in question, the above 0.282 seconds is the eyelid closure threshold.

[0012] Example 2 (Improved accuracy by measuring eyelid opening behavior) In addition to the eyelid closing action, the time from the start of the eyelid opening action to the completion of the eyelid opening action is also measured. By confirming the input only when both are below the threshold, it is possible to further prevent malfunctions due to unintentional eyelid closure or physiological blinking. As a specific example, when taking a group photo using a camera equipped with a face recognition function such as a smartphone or digital camera, the conventional method of taking a selfie photo was by using a remote control or a timer, but now rapid blinking can be used as the shutter trigger. In this embodiment, if there is no gaze tracking function or if the upper limit of the gaze tracking distance is exceeded, the mode is automatically switched to face recognition mode. In Figure 4, by registering the photographer's face and eyelid shape 1 in the device in advance, the shutter can be triggered if the opening and closing of the eyelids can be captured by face recognition. FIG. 9 shows the input determination flow for the closed eyelid + open eyelid measurement in the above process.

[0013] Example 3 (double blink operation) The input is confirmed only when the set of fast eyelid closing + fast eyelid opening is performed twice in a row. In this case, the condition is that all four parameters (first eyelid closing, first eyelid opening, second eyelid closing, second eyelid opening) are less than the threshold value. FIG. 10 shows the input determination flow based on double blinks in the above process. Effective for preventing malfunctions during competitions and important operations. For example, if you want to perform the same action as double-clicking a computer mouse, you can achieve a similar effect by blinking twice quickly. The meanings of 1, 2, 3, 4, and 5 in Figure 3 are as follows: 1 Time from eyelid opening to complete closure 2 Time from closing to reopening the eyelids 3 Time from closing the eyelids to opening them completely 4 Time from reopening to closing the eyelids again 5. Time during re-closure By combining parameters 1 to 5, you can freely customize what behaviors are below the threshold. The simplest example of a combination is when the total time of 1, 2, 4, and 5 is below a threshold, which can be defined as a double-click operation.

[0014] Example 4 (Application to E-sports) By combining this invention with e-sports, skills can be activated and targets can be selected instantly during competition. This allows for intuitive and fast gameplay control without the need for a physical controller.

[0015] Multi-functionality can be achieved by combining the four parameters 1, 2, 3, and 4 in Figure 3. By combining the strengths and weaknesses of the four parameters, it is possible to design an infinite number of rich and intuitive "blink languages." Users can interact intelligently with a computer simply by moving their eyelids, just like an experienced pianist playing different keys on a keyboard. Examples of combinations are summarized in Table 3. Table 3 JPEG2025172106000004.jpg49115

[0016] Example 5 (Example of composite threshold determination) Advanced processing is possible, whereby the eyelid closure time and any of the motion characteristic values ​​(speed, acceleration, or rate of change) are combined to determine the threshold. FIG. 11 is a flowchart of this determination. The gaze dwell detection of 1 is a process in which the gaze detection means detects that the gaze of the operator has stayed on the target object for a predetermined time (for example, 0.5 seconds) or more. The eyelid closing action detection of 2 detects that the eyelid closing action has started by the eyelid closing detection means, This is a process for acquiring eyelid position data in time series. The motion profile measurement in 3 is a process of calculating the following motion characteristics from the acquired eyelid position data. (a) Speed: eyelid movement distance / elapsed time. (b) Acceleration: The change in velocity over time. (c) Rate of change: The gradient of velocity or acceleration over time (representing the initial rapid acceleration or deceleration characteristics). Step 4 determines whether the eyelid closing action time is less than a threshold value, and if it is less than the threshold value, the process proceeds to the subsequent processing, and if it is more than the threshold value, the process is terminated with the action being treated as invalid. 5 compares the measured speed with the speed threshold set in advance by calibration and treats it as valid input if it is above that threshold. 6 compares the measured acceleration with the acceleration threshold set in advance by calibration and if it is greater than that, treats it as a valid input. 7 compares the measured rate of change with a rate of change threshold set in advance by calibration, and if it is equal to or greater than that, treats it as a valid input. 8 is treated as invalid input and no processing is performed. 9 is treated as valid input and executes the process (e.g., selection, decision, operation start, etc.) corresponding to the target object. As for its effectiveness, even "blinks of a subtle speed" that are difficult to identify by measuring time alone can be identified with high accuracy by taking into account the motion profile of speed, acceleration, and rate of change. It can also adapt to variations in blinking speed due to fatigue and individual differences, helping to prevent malfunctions. It can also be applied to medical and other fields that require high-precision operation (e.g., surgical assistance VR).

[0017] Example 6 (UX improvement example by linking with iris authentication) The information processing system according to this embodiment is configured to include an iris authentication unit in addition to the gaze detection unit and eyelid closure detection unit. The iris authentication means acquires the iris pattern of the operator to authenticate the operator, and identifies the operator based on the authentication result. This makes it possible to automatically read and apply calibration data recorded in advance for each operator, that is, the high-speed eyelid-closing threshold calculated based on the normal blinking of that operator. As a result, even if multiple operators share the device, each operator does not need to perform the login operation or calibration procedure each time, and an individually adapted gaze input environment is provided immediately upon putting on the device. Furthermore, once the operator is identified through iris authentication, it is possible to automatically launch a specific application, such as an e-sports VR application, thereby achieving a seamless user experience (UX) without the need for login. The effects are as follows: 1. Personal optimization is automated by linking calibration values ​​to user accounts. 2. It can be shared by multiple users while maintaining high security, and is expected to have a wide range of applications in education and e-sports. 3. It can be used immediately after being attached, making it an ideal example of UX. 4. Iris recognition is a commonly used function, so it can be applied not only to Apple Vision Pro but also to other companies' HMDs. Figure 12 is a flow diagram when iris authentication and high-speed blink calibration are linked. Iris authentication is used to identify individuals, and if calibration has not been registered, the user is prompted to register it.If the eyelid closure threshold has been registered, apps such as e-sports will automatically launch, allowing the user to immediately begin playing. [Industrial Applicability]

[0018] It can be used as a data input method when your hands are full and mouse operation is difficult. It can also improve safe driving by allowing you to operate the car navigation screen with just your eyes and rapid blinking. When operating a remote surgical robot such as Da Vinci, auxiliary operations that are impossible because both hands are occupied can be compensated for by using gaze and rapid blinking. A surgeon in surgery pages through a patient's data display without putting down his scalpel.

[0019] A factory worker operates a machine with both hands and indicates the next process with his gaze and rapid blinking. People with disabilities in both hands can operate computers and connect with the world by moving only their hands from the neck up.

[0020] Below are the specifications for the Apple Vision Pro VR goggles available: (Table 4) Source URL https: / / www.apple.com / jp / apple-vision-pro / specs / Table 4. Specifications TIFF2025172106000005.tif9687

[0021] The role of each of the sensor functions in Table 4 is briefly explained below. Two high-resolution main cameras. Role: The main camera captures the real-world scenery and displays it on the display in front of your eyes (video pass-through). Because it has high resolution, it feels natural when reading text or looking at distant scenery. 6 world-facing tracking cameras. Role: The heart of the "spatial computer." It constantly monitors the surrounding space from various angles, up, down, left, right, and all directions. Lidar Scanner. Function: By shining an invisible infrared laser into the surrounding area and measuring the time it takes for the light to bounce back, it can instantly and precisely measure the "exact distance" to an object. Ambient Light Sensor. Function: Measures the "brightness" and "color temperature of light (white light or warm light)" of a room. Four eye-tracking cameras. Role: A dedicated camera that constantly monitors only the "eyeball" from the inside. The four sensors are capable of capturing blinks, eyelid movements, and pupil direction from any angle with millimeter-level accuracy. These are the main functions utilized in this patent. TrueDepth camera: What it does: A camera that precisely maps the three-dimensional shape of your face, just like Face ID on iPhone (registered trademark). Four inertial measurement units (IMUs). Role: A sensor similar to the semicircular canals that combines an accelerometer and a gyro sensor. Flicker Sensor. Function: Detects "flicker" caused by lighting (especially fluorescent lights and LEDs) flashing at speeds that are invisible to the human eye.

Claims

1. An information processing system including a head-mounted display having an eye gaze tracking function and a display unit, an eye gaze detection means for detecting an operator's eye gaze by the eye gaze tracking function of the head-mounted display, and an eyelid closure detection means for detecting an eyelid closure action of the operator, a) the line-of-sight detection means detects that the operator's line of sight is staying on a predetermined target object for a predetermined period of time or more; b) the eyelid closing detection means measures the time from the start of the eyelid closing action by the operator to the completion of the eyelid closing action during the dwell period; c) deterministically executing a process corresponding to the target object on the condition that the measured time is less than a predetermined eyelid closure threshold; d) the eyelid-closing threshold is set by multiplying an average value of a normal eyelid-closing action time measured in advance for each operator by a predetermined ratio coefficient; e) Eyelid closing actions above the threshold are invalidated as inputs; An information processing system comprising:

2. 2. The information processing system according to claim 1, wherein the eyelid closure detection means measures at least one of the eyelid movement speed, acceleration, or rate of change thereof during the eyelid closing action, and takes this into consideration in the threshold determination in step c).

3. 3. An information processing system according to claim 1, wherein the head-mounted display is equipped with a plurality of gaze detection cameras, and independently detects the closing of the eyelids of both eyes of the operator, and executes the input confirmation process only when the closing of the eyelids of both eyes is synchronized.

Images