Eye-tracking interaction device and eye-tracking interaction control system

By recognizing pupil size and gaze point using invisible light sources and image sensors, and combining this with physiological responses to determine user intent, the problem of misjudgment of gaze intent in existing eye-tracking interaction devices has been solved, achieving accurate user intent recognition and secure user interaction.

CN123114252APending Publication Date: 2026-11-17LENS TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611245228.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-17
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing eye-tracking interaction devices struggle to accurately determine a user's gaze intent when observing the far-field world, leading to misjudgments of near-field interface gaze, which impacts user experience and security.

Method used

An invisible light beam is emitted from an invisible light source to illuminate the eye. An image sensor collects the reflected light spot, and a processing unit identifies the pupil size and gaze point coordinates. Combining the pupil size threshold and constriction rate, the user's intention is determined, distinguishing between near-field interface gaze and far-field world gaze. The unmanned device is then controlled through an inertial measurement unit and a communication unit.

Benefits of technology

It enables accurate judgment of user intent during eye-tracking interaction, avoids accidental triggering of operation controls, and improves user experience and security, especially in high-security application scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114252A_ABST
    Figure CN123114252A_ABST
Patent Text Reader

Abstract

This invention relates to the field of human-computer interaction technology, providing an eye-tracking interaction device and control system. The device includes a processing unit, an invisible light source, an image sensor, a projection unit, and a display element. The invisible light source, under the control of the processing unit, emits an invisible light beam to illuminate the eye. The image sensor sends the acquired eye image to the processing unit. The processing unit controls the projection unit to project a virtual interactive interface onto the display element. The pupil is identified and its size is determined from the eye image. Based on the eye image, the user's gaze point coordinates are determined. If the gaze point coordinates are located within the near-field virtual interactive interface and the pupil size is less than a preset pupil size threshold, the user's intention is determined to be near-field interface fixation; otherwise, it is far-field world fixation. This invention can accurately determine whether the user's intention is near-field interface fixation or far-field world fixation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, and in particular to an eye-tracking interaction device and an eye-tracking interaction control system. Background Technology

[0002] Augmented Reality (AR) is a technology that overlays and blends virtual scenes or information with the real physical environment, presenting them to the user and creating a shared space between the virtual and real worlds. Currently, AR devices (such as glasses and helmets) can project virtual interactive interfaces onto transparent display elements (such as glasses lenses), allowing users to interact with the virtual interface on the display element through eye movements.

[0003] Existing eye-tracking interaction solutions mainly rely on eye gaze direction for intent recognition. When the user's gaze point is detected to fall on the virtual interactive interface of the display element, it is determined that the user's intent is near-field interface gaze; otherwise, it is far-field world gaze (also known as external world gaze).

[0004] However, when a user is gazing at the far-field world (also known as gazing at the external world), that is, when observing the real world in the far field, their gaze may simultaneously penetrate the virtual interactive interface in the near field, that is, the gaze point falls on the virtual interactive interface in the near field, leading to a misjudgment that the user's intention is to gaze at the near-field interface, that is, it is difficult to accurately determine whether the user's intention is to gaze at the near-field interface or to gaze at the far-field world. Summary of the Invention

[0005] This invention provides an eye-tracking interaction device and an eye-tracking interaction control system to solve the technical problem that existing eye-tracking interaction devices, which rely solely on gaze points for interaction control, cannot accurately determine the user's intent.

[0006] This invention provides an eye-tracking interaction device, comprising: a processing unit, an invisible light source, an image sensor, a projection unit, and a display element; An invisible light source emits an invisible beam of light to illuminate the eye under the control of the processing unit; The image sensor sends the acquired eye images to the processing unit; The processing unit controls the projection unit to project a virtual interactive interface onto the display element, identifies the pupil from the eye image and determines the pupil size, determines the user's gaze point coordinates based on the eye image, and determines the user's intention to gaze at the near-field interface if the gaze point coordinates are located within the near-field virtual interactive interface and the pupil size is less than a preset pupil size threshold. Otherwise, it is determined that the user's intention is to gaze at the far-field interface.

[0007] An eye-tracking interaction device according to the present invention further includes: a memory connected to a processing unit; The processing unit can also guide the user to gaze at the virtual interactive interface, and after receiving confirmation feedback from the user, control the image sensor to acquire the first eye calibration image, and determine the near-field pupil size from the image in the first eye calibration image; The processing unit can also guide the user to gaze at the far-field external world, and after receiving confirmation feedback from the user, control the image sensor to acquire a second eye calibration image and determine the far-field pupil size from the second eye calibration image; The processing unit calibrates a preset pupil size threshold based on near-field pupil size and far-field pupil size, and stores the preset pupil size threshold in memory.

[0008] An eye-tracking interaction device according to the present invention further includes: a memory connected to a processing unit, the memory being used to store a preset rate threshold and / or a preset time range; The processing unit can also determine the pupil size contraction rate and / or the contraction duration to less than a preset pupil size threshold, and retrieve the preset rate threshold and / or preset time range from the memory; and determine that the gaze point coordinates are located within the virtual interactive interface. If the pupil size contraction rate is greater than the preset rate threshold and / or the contraction duration is within the preset time range, the user's intention is determined to be near-field interface gaze; otherwise, it is far-field world gaze.

[0009] According to the eye-tracking interaction device provided by the present invention, after determining that the user's intention is to gaze at the near-field interface, the processing unit can also send a first opacity display instruction to the projection unit, and the projection unit projects the virtual interactive interface with the first opacity. After determining that the user's intention is to gaze at the far field world, the processing unit can also send a second opacity display instruction to the projection unit. The projection unit projects the virtual interactive interface with the second opacity, where the first opacity is greater than the second opacity.

[0010] According to the eye-tracking interaction device provided by the present invention, after determining that the user's intention is to gaze at the near-field interface, the processing unit can continuously monitor the pause position of the gaze point in the virtual interactive interface based on the coordinates of the gaze point; when the pause position is located within the area of ​​any control in the virtual interactive interface, a highlighting instruction is sent to the projection unit, and the projection unit highlights any control. The processing unit can also accumulate the duration of the gaze point on any control. If the duration of the gaze point exceeds the preset duration, the unit determines any control as the target control and generates the corresponding target instruction for the target control.

[0011] According to the eye-tracking interaction device provided by the present invention, the processing unit can also send a prompt window instruction to the projection unit when the dwell time exceeds the preset time, and the projection unit generates a user confirmation prompt window on the virtual interactive interface; The processing unit can also generate target instructions after receiving feedback from the user who has executed the instructions in the user confirmation prompt window.

[0012] According to an eye-tracking interaction device provided by the present invention, an invisible light source, an image sensor, and a projection unit are connected to a processing unit via a flexible circuit board.

[0013] According to the present invention, an eye-tracking interaction device is provided for interactive control of unmanned equipment, and further includes: an inertial measurement unit and a communication unit connected to the processing unit; The processing unit can also activate the inertial measurement unit when it determines that the user's intention is to gaze at the near-field interface. The inertial measurement unit sends the perceived head posture of the user to the processing unit. The processing unit generates gimbal control commands for the unmanned device based on the head posture and sends the gimbal control commands to the unmanned device through the communication unit.

[0014] The present invention also provides an eye-tracking interaction control system, comprising: an unmanned device and the aforementioned eye-tracking interaction device, wherein the eye-tracking interaction device is wirelessly connected to the unmanned device for controlling the unmanned device.

[0015] According to the provided eye-tracking interactive control system, it further includes: a control handle and / or a voice control unit, wherein the voice control unit is connected to the processing unit, and the control handle is connected to the processing unit through the communication unit; The processing unit can also activate the control handle and / or voice control unit and suppress the inertial measurement unit when it determines that the user's intention is to gaze at the far field world. The processing unit can also generate target instructions for controls corresponding to the gaze point when it is determined that the user's intention is to gaze at the near-field interface, activate the inertial measurement unit, and keep the control handle and / or voice control unit active. Based on the head posture, it generates gimbal control instructions for the unmanned device and sends the gimbal control instructions and target instructions to the unmanned device through the communication unit.

[0016] The eye-tracking interaction device and control system provided by this invention emit an invisible light beam from an invisible light source to illuminate the eye, generating a reflected light spot. An image sensor acquires an eye image including the reflected light spot. A processing unit controls a projection unit to project a virtual interactive interface onto a display element and identifies the pupil from the eye image and determines the pupil size. Based on the eye image, the coordinates of the user's gaze point are determined. If the gaze point coordinates are located within the near field of the virtual interactive interface and the pupil size is less than a preset pupil size threshold, the user's intention is determined to be near-field interface fixation; otherwise, it is far-field world fixation. This utilizes the physiological response of pupil constriction due to near reflection and relative pupil dilation when focusing on distant objects. When both conditions are met—the gaze point coordinates being located within the near field of the virtual interactive interface and the pupil size being less than a preset pupil size threshold—the user's intention is determined to be near-field interface fixation. Thus, during eye-tracking interaction, it can accurately determine whether the user's intention is near-field interface fixation or far-field world fixation. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the eye-tracking interaction device provided by the present invention.

[0019] Figure 2 This is a time series diagram of pupil diameter and fixation point when the eye-tracking interaction device provided by the present invention processes the user's switching between near-field interface fixation state and far-field world fixation state.

[0020] Figure 3 This is one of the schematic diagrams of the pupil diameter variation with gaze distance and threshold setting method in the eye-tracking interaction device provided by the present invention.

[0021] Figure 4 This is the second schematic diagram of the pupil diameter variation with gaze distance and threshold setting method in the eye-tracking interaction device provided by the present invention.

[0022] Figure 5 This is a schematic diagram of the structure of the AR glasses provided by the present invention.

[0023] Figure 6 This is a schematic diagram of the eye-tracking interactive control system provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] The eye-tracking interaction device of this invention embodiment is as follows: Figure 1 As shown, the device includes a processing unit 101, an invisible light source 102, an image sensor 103, a projection unit 104, and a display element 105, wherein the invisible light source 102, the image sensor 103, and the projection unit 104 are all connected to the processing unit 101. It is understood that the eye-tracking interaction device also includes a power supply for powering the processing unit 101, the invisible light source 102, the image sensor 103, and the projection unit 104.

[0026] The invisible light source 102 is used to emit an invisible light beam to illuminate the eye under the control of the processing unit 101. The processing unit 101 controls the illumination of the invisible light source 102 to ensure that its illumination angle covers the normal range of eye movement of the user, and to ensure that a stable reflected light spot (Purkinje image) can be formed on the cornea under different gaze directions. The reflected light spot is received by the image sensor 103 to assist in the positioning of the gaze point. Since it emits invisible light, it will not interfere with the user's normal field of vision.

[0027] For example, the invisible light source 102 can be an infrared light source or a near-infrared light source, such as an infrared LED light source emitting a wavelength of 850nm or 940nm. Provided that the solution of this application can be achieved, it can also be an infrared LED light source of other wavelengths.

[0028] Image sensor 103 is used to send acquired eye images to processing unit 101. Image sensor 103 can acquire eye images at a preset sampling frequency. The acquired eye images include normal eye images of the visible light portion, as well as the aforementioned reflected light spot images.

[0029] For example, the image sensor 103 can be a global shutter image sensor with a resolution of not less than 400×400 pixels and a sampling frequency of not less than 60Hz, preferably 100Hz. The global shutter image sensor can ensure that all pixels are exposed at the same time, which can avoid image distortion caused by rolling shutter during high-speed eye movement and is suitable for eye tracking.

[0030] The processing unit 101 is used to control the projection unit 104 to project a virtual interactive interface onto the display element 105, identify the pupil from the eye image and determine the pupil size, determine the user's gaze point coordinates based on the eye image, and determine the user's intention to gaze at the near-field interface if the gaze point coordinates are located within the near-field virtual interactive interface and the pupil size is less than a preset pupil size threshold. Otherwise, it is determined that the user's intention is to gaze at the far-field interface.

[0031] Specifically, based on the current application scenario, the processing unit 101 controls the projection unit 104 to project the virtual interactive interface corresponding to the current application scenario onto the display element 105. For example, if the current scenario is an eye-tracking control scenario for a drone, the virtual interactive interface can display controls for controlling the drone's flight attitude and the captured images transmitted back by the drone.

[0032] After receiving the eye image, the processing unit 101 identifies the pupil and determines its size. For example, morphological processing can be used to remove interfering points (such as eyelashes and eyelid edges) from the eye image. Then, adaptive threshold binarization is used to separate the pupil region (the pupil is dark black, while the iris and sclera are light-colored). Finally, ellipse fitting (least squares method) is performed on the edge points of the pupil region to extract the pupil edge from the eye image, thereby calculating the pupil diameter (units can be pixels or mm). In this embodiment, the pupil diameter is used as the pupil size; however, the pupil radius or area can also be used. It is understood that the eye image is acquired at a certain sampling rate; therefore, the pupil size is also a quantity that changes over time. For example, the pupil diameter can be represented as a time sequence D(t) over time t.

[0033] The processing unit 101 can detect in parallel the bright reflective spots formed on the cornea by invisible light sources in the eye image. For example, it uses grayscale peak detection to locate the center of the spot, calculates the relative position vector (Δx, Δy) between the center of the spot and the center of the pupil, and uses a pre-calibrated polynomial mapping model (or neural network model) to convert the relative position vector into the user's gaze point coordinates G(x,y) on the display element 105.

[0034] After calculating the gaze point coordinates and aperture size, the processing unit 101 determines whether the gaze point coordinates are located within the near-field virtual interactive interface and whether the pupil size is less than a preset pupil size threshold. If so, it determines that the user's intention is near-field interface gaze; otherwise, it is far-field world gaze. The coordinate range of the display element area where the virtual interactive interface is located, as well as the coordinates of controls within it, can be obtained through the projection unit 104.

[0035] Specifically, the processing unit 101 first determines whether the coordinates of the gaze point fall within the virtual interactive interface on the display element 105. If the coordinates of the gaze point are outside the virtual interactive interface, the user's intention is far-field gazing, meaning the user is in a far-field gazing state. If the coordinates of the gaze point are within the near-field virtual interactive interface, it indicates that the user may have a near-field interface gazing intention, or the user may be observing the far-field external world and their line of sight happens to pass through the area of ​​the virtual interactive interface. Therefore, in order to accurately determine whether the user's intention is near-field interface gazing or far-field gazing, the processing unit 101 further determines whether the pupil size is less than a preset pupil size threshold. This is because when the human eye gazes at targets at different distances, not only does eyeball convergence occur, but pupil size also changes. That is, when gazing at near objects, the pupil constricts due to the near reflex; when gazing at distant objects, the pupil dilates relatively. Moreover, this physiological response is controlled by the autonomic nervous system and is difficult to suppress subjectively. Therefore, this embodiment can utilize this physiological response of the pupil. When the pupil size is less than a preset pupil size threshold, it indicates that the user intends to gaze at the near-field virtual interactive interface, rather than having their line of sight pass through the virtual interactive interface when observing the far-field external world. Thus, during eye-tracking interaction, it is possible to accurately determine whether the user intends to gaze at the near-field interface or the far-field world.

[0036] like Figure 2 As shown, when the user's state switches from far-field world gaze to near-field interface gaze, the pupil constricts and becomes smaller, and the change in pupil diameter is shown by the blue curve. At the same time, when the coordinates of the gaze point are also located within the near-field virtual interactive interface, as shown by the green curve, it is determined that the user's intention is near-field interface gaze.

[0037] In this embodiment, the eye-tracking interaction device uses an invisible light source 102 to emit an invisible light beam to illuminate the eye, generating a reflected light spot. An image sensor 103 acquires an eye image including the reflected light spot. A processing unit 101 controls a projection unit 104 to project a virtual interactive interface onto a display element 105, and identifies the pupil from the eye image and determines the pupil size. Based on the eye image, the user's gaze point coordinates are determined. If the gaze point coordinates are located within the near-field virtual interactive interface and the pupil size is less than a preset pupil size threshold, the user's intention is determined to be near-field interface gazing. Otherwise, it is far-field world gazing. This utilizes the physiological response of pupil constriction due to near reflection and relative pupil dilation when gazing at distant objects. When both conditions are met—the gaze point coordinates being located within the near-field virtual interactive interface and the pupil size being less than a preset pupil size threshold—the user's intention is determined to be near-field interface gazing. Thus, during eye-tracking interaction, it can accurately determine whether the user's intention is near-field interface gazing or far-field world gazing.

[0038] In some embodiments, the eye-tracking interaction device further includes a memory 106 connected to the processing unit 101.

[0039] The processing unit 101 can also guide the user to gaze at the virtual interactive interface. After receiving confirmation feedback from the user, it controls the image sensor 103 to acquire a first eye calibration image and determines the near-field pupil size from the first eye calibration image. For example, the processing unit 101 can control the projection unit 104 to project a virtual light spot onto the virtual interactive interface. The equivalent optical distance of the virtual light spot is approximately 0.3m. The processing unit 101 can also prompt the user to first gaze at the virtual light spot appearing on the near-field virtual interactive interface and maintain this gaze for a certain duration (e.g., 3 seconds, or other durations that can be adjusted according to the actual usage scenario) so that the pupil remains stable after contraction. After receiving confirmation feedback from the user that the prompt action has been completed, the processing unit 101 controls the image sensor 103 to acquire the first eye calibration image, which is a pupil image that remains stable after pupil contraction. The processing unit 101 can determine the near-field pupil size from this first eye calibration image, such as the near-field pupil diameter D. near For example: D near =2.1mm.

[0040] The processing unit 101 can also guide the user to focus on the far-field external world. After receiving confirmation feedback from the user, it controls the image sensor to acquire a second eye calibration image and determines the far-field pupil size from the second eye calibration image. For example, the processing unit 101 can prompt the user to focus on an actual target in the far-field external world, such as an actual target 10m away, and maintain focus on the actual target for a certain period of time (e.g., 3s) so that the pupil remains stable after dilation. After receiving confirmation feedback from the user that the prompting action has been completed, it controls the image sensor 103 to acquire a second eye calibration image, which is a pupil image after dilation and stability. The processing unit 101 can determine the far-field pupil size from this second eye calibration image, such as the far-field pupil diameter D. far For example: D far =4.8mm.

[0041] Processing unit 101 calibrates a preset pupil size threshold based on near-field and far-field pupil sizes, and stores the preset pupil size threshold in memory 106. For example, the preset pupil size threshold can be the average or median of the near-field and far-field pupil sizes, such as... Figure 3 As shown, for D near and D far The preset pupil size threshold D is obtained by taking the average value. th =3.4mm.

[0042] It is understandable that after the memory 106 stores the preset pupil size threshold, the processing unit 101 can obtain the preset pupil size threshold from the memory 106 during the subsequent actual eye-tracking interaction process, so as to compare it with the actually calculated pupil size.

[0043] In this embodiment, the processing unit 101 first calibrates the near-field pupil size and the far-field pupil size, and then calibrates the preset pupil size threshold by combining the near-field pupil size and the far-field pupil size. Using the mean or median value as the preset pupil size threshold is simple and easy to implement, and the near-field interface gaze and the far-field world gaze are regarded as two equivalent states, thereby obtaining a reasonable pupil size threshold.

[0044] It should be noted that after the user wears the eye-tracking interaction device, the user is prompted to calibrate the preset pupil size threshold, so as to accurately calibrate the preset pupil size threshold according to different users and different lighting environments.

[0045] Furthermore, the processing unit 101 calibrates a preset pupil size threshold based on near-field pupil size and far-field pupil size, specifically including: The processing unit 101 calculates the size difference between the far-field pupil size and the near-field pupil size, and obtains a size difference weighted value based on a preset size difference weight. The preset pupil size threshold is obtained by subtracting the size difference weighted value from the far-field pupil size.

[0046] Taking pupil diameter as an example, specifically, the preset pupil size threshold D is calibrated according to the following formula. th : D th =D far -α(D far -D near ).

[0047] Where α is the size difference weight, and its value ranges from (0, 0.5) ∪ (0.5, 1).

[0048] For example, the value of α ranges from (0.2, 0.5), such as: Figure 4 As shown, when α=0.3, which is the 70th percentile of far-field pupil size, D th =4.0mm; the pupil size easily crosses this threshold when changing from large to small. In the state of far-field world fixation, the user can enter the near-field control area with slight pupil contraction, and the interaction response is sensitive. It is more difficult to cross this threshold when changing from small to large pupil size, which can avoid frequent state jumps caused by slight pupil fluctuations, and the near-field interface fixation state is highly stable.

[0049] For example, α takes values ​​in the range (0.6, 0.8), D thThe pupil size will be 3.4mm lower than the average. It is less likely to cross this threshold when the pupil size changes from large to small. When looking at the far field world, the user must deeply constrict the pupil to enter the near field control area. This ensures that only pupils that are significantly constricted and continuously focused can be judged as near field interface gaze, greatly reducing the risk of accidental control touch. When the pupil size changes from small to large, it is relatively easy to cross this threshold. The user can relax their gaze to return to the external world gaze state, which perfectly meets the needs of high-safety scenarios such as drones.

[0050] In this embodiment, the preset pupil size threshold is jointly calibrated based on the far-field pupil size, the size difference between the far-field pupil size and the near-field pupil size, and the size difference weight. The preset pupil size threshold can be adjusted by setting the size difference weight based on the specific application scenario (light intensity or the degree of safety requirements of the application scenario), thereby realizing personalized adaptive pupil size threshold setting.

[0051] In some embodiments, the memory 106 is used to store a preset rate threshold and / or a preset time range, which are preset in the memory 106.

[0052] The processing unit 101 can also determine the pupil size contraction rate and / or the contraction duration to less than a preset pupil size threshold, and obtain the preset rate threshold and / or preset time range from the memory 106; and determine that the gaze point coordinates are located within the virtual interactive interface. If the pupil size contraction rate is greater than the preset rate threshold and / or the contraction duration is within the preset time range, the user's intention is determined to be near-field interface gaze; otherwise, it is far-field world gaze.

[0053] Specifically, after the processing unit 101 determines the pupil size contraction rate and / or the contraction duration to less than a preset pupil size threshold, it retrieves the pre-stored preset rate threshold and / or preset time range from the memory 106 and compares them with the contraction rate and contraction duration, respectively.

[0054] Under constant lighting conditions, when the human eye shifts from viewing a distant target to viewing a nearby target, the duration of pupil constriction is typically between 300ms and 800ms. Within this time window, the change in pupil diameter (D...) far -D nearDividing the pupil constriction rate (approximately 2mm~3mm) by the time, the calculated physiological rate of pupil constriction naturally falls within the range of 2.5mm / s~10mm / s. Preferably, the preset rate threshold can be 3mm / s~8mm / s, for example, 3mm / s. The pupil constriction rate can be calculated using dD(t) / dt. By judging the pupil constriction rate, the processing unit 101 can eliminate pupil drift caused by slowly varying factors such as ambient light (whose rate is usually less than 0.1mm / s). That is, during the process of viewing the far-field external world through the area of ​​the virtual interactive interface, it avoids the situation where the pupil slowly constricts to a size less than the preset pupil size threshold due to increased ambient light, which might be mistakenly interpreted as the user needing to focus on the near-field interface.

[0055] The preset time range can be set to 0.3s~0.8s to exclude abrupt pupil changes caused by transient interference such as blinking and strong flashes of light (whose contraction time is usually less than 0.2s). Similarly, when viewing the distant external world through the area of ​​the virtual interactive interface, it avoids situations where blinking or strong flashes cause the pupil to slowly and violently contract to a size less than the preset pupil size threshold, which might be mistakenly interpreted as the user needing to focus on the near-field interface.

[0056] Preferably, the processing unit 101 determines that the user's intention is near-field interface fixation if the pupil size contraction rate is greater than a preset rate threshold and contracts to less than a preset pupil size threshold within a preset time range; otherwise, it determines that the user is far-field world fixation. For example: Figure 2 As shown, the pupil size contraction rate is 6.75 mm / s, and the time taken to contract to below the preset pupil size threshold is 400 ms (0.4 s), which determines that the user's intention is to gaze at the near-field interface.

[0057] In this embodiment, by pre-storing the pupil contraction rate and / or a preset time range in the memory 106 to limit the pupil contraction process and the size of the pupil after contraction, pupil drift caused by slow-changing factors such as ambient light or fast-changing factors such as flashes and blinks can be avoided, thereby more accurately determining the user's intention.

[0058] In some embodiments, after determining that the user's intention is to gaze at the near-field interface, the processing unit 101 may send a first opacity display instruction to the projection unit 104, and the projection unit 104 may project the virtual interactive interface with the first opacity.

[0059] After determining that the user's intention is to gaze at the far field world, the processing unit 101 can also send a second opacity display instruction to the projection unit 104. The projection unit 104 projects the virtual interactive interface with the second opacity, where the first opacity is greater than the second opacity.

[0060] Specifically, after determining that the user's intention is to gaze at the near-field interface, the processing unit 101 sends a first opacity display command to the projection unit 104. Upon receiving the first opacity display command, the projection unit 104 increases the projection brightness and sharpness, so that the opacity of the virtual interactive interface projected on the display element 105 reaches the first opacity, thereby projecting a clearer virtual interactive interface on the display element 105. After determining that the user's intention is to gaze at the far-field world, the processing unit 101 sends a second opacity display command to the projection unit 104. Upon receiving the second opacity display command, the projection unit 104 decreases the projection brightness and sharpness, so that the opacity of the virtual interactive interface projected on the display element 105 decreases to the second opacity, thereby projecting a near-transparent virtual interactive interface on the display element 105, reducing the obstruction of the virtual interactive interface to the user's line of sight when gazing at the far-field world.

[0061] In this embodiment, under different user intentions, the processing unit 101 sends different transparency display instructions to the projection unit 104, so that the projection unit 104 projects a virtual interactive interface on the display element 105 with different opacities to adapt to the user intention, thereby improving the user interaction experience under any user intention.

[0062] In some embodiments, after determining that the user's intention is near-field interface gazing, the processing unit 101 may continuously monitor the pause position of the gaze point within the virtual interactive interface based on the gaze point coordinates; when the pause position is within the area of ​​any control in the virtual interactive interface, a highlighting instruction is sent to the projection unit 104, and the projection unit 104 highlights any control. For example, the processing unit 101 may include the gaze point coordinates in the highlighting instruction, so that the projection unit 104, upon receiving the highlighting instruction, determines which control to highlight.

[0063] Specifically, when a user is in a near-field viewing state, the coordinates of the gaze point change continuously with the user's eye movements. The processing unit 101 can continuously monitor the gaze point coordinates to determine the pause position of the gaze point within the virtual interactive interface. The area of ​​any control within the virtual interactive interface is predetermined. The processing unit 101 can determine whether the pause position is within the area of ​​any control by comparing the gaze point coordinates with this area. If the pause position is within the area of ​​any control, it means that the user is currently viewing that control. Therefore, the projection unit 104 is controlled to highlight that control, thus providing feedback confirmation to the user and improving the user's interactive experience.

[0064] The processing unit 101 can also accumulate the duration of the gaze point's dwell time on any control. If the dwell time exceeds a preset duration, the processing unit determines that any control is a target control and generates a corresponding target instruction for the target control. To avoid the user accidentally triggering any control when the gaze point only briefly stays in the area of ​​any control without any intention to trigger it, the processing unit 101 judges the duration of the gaze point's dwell time on any control, that is, it judges the duration of the user's gaze on any control. If the dwell time exceeds a preset duration, it means that the user intends to trigger any control. Therefore, the processing unit determines that any control is a target control and generates a corresponding target instruction for the target control.

[0065] For example: When a user glances at control 1, control 1 is highlighted. If the user determines that control 1 is not the control to be triggered, their gaze will move away from control 1 (the preset duration is not reached, and control 1 is only highlighted and then returns to normal display). The user then looks at control 2, which is highlighted. If control 2 is the control that the user wants to trigger, the gaze will stay on control 2 for the preset duration. The processing unit 101 confirms that control 2 is the target control. The user only needs to gaze at control 2 for the preset duration, and the processing unit 101 will automatically trigger control 2 and generate the corresponding target instruction for control 2.

[0066] In this embodiment, the processing unit 101 controls the projection unit 104 to highlight the control and provide the user with gaze point feedback, thereby improving the user's interactive experience. At the same time, the target control is determined only when the gaze point stays on any control for a longer than a preset time, avoiding the generation of target commands by mistake and improving operational safety. This is especially true in application scenarios with high safety requirements, such as drones, to avoid safety problems caused by the generation of control commands by mistake.

[0067] In some embodiments, if the dwell time exceeds a preset time, the processing unit 101 may also send a prompt window instruction to the projection unit 104, and the projection unit 104 may generate a user confirmation prompt window on the virtual interactive interface. That is, after receiving the prompt window instruction, the projection unit 104 generates a user confirmation prompt window on the virtual interactive interface. For example, the user confirmation prompt window is used to confirm or cancel controls, or to display a preset confirmation action for user confirmation, such as blinking, micro-gesture, or voice confirmation.

[0068] The processing unit 101 can also generate a target instruction after receiving feedback information that the user has executed the prompt in the user confirmation prompt window. For example, the processing unit 101 receives feedback information related to the user staring at the confirmation control for a long time, or receives feedback information related to the user's preset confirmation action, such as detecting the user blinking, and then confirms that the user's intention is to trigger any of the controls.

[0069] In this embodiment, the processing unit 101 determines whether the duration of the gaze point on any control is greater than a preset duration. If it is greater than the preset duration, the projection unit 104 controls the projection unit 104 to generate a user confirmation prompt window on the virtual interactive interface for secondary confirmation. This avoids the accidental triggering of the target command when the user gazes at any control for a long time without deciding whether to trigger it, thus further improving the safety of operation.

[0070] In some embodiments, the invisible light source 102, image sensor 103, and projection unit 104 are connected to the processing unit 101 via a flexible circuit board. The eye-tracking interaction device can be AR glasses or a helmet. Such AR devices are generally small in size, with limited and dispersed usable space. For example, in AR glasses, the usable space is typically limited to certain parts of the temples and frame. In this embodiment, connecting the invisible light source 102, image sensor 103, and projection unit 104 to the processing unit 101 via a flexible circuit board allows for a more flexible layout of the components, forming a compact and wearable hardware architecture.

[0071] In some embodiments, the eye-tracking interaction device is used for interactive control of unmanned devices, such as unmanned aerial vehicles (UAVs), unmanned vehicles (UGVs), unmanned ships (USVs), and robots. Accordingly, the eye-tracking interaction device further includes an inertial measurement unit 107 and a communication unit 108 connected to the processing unit 101.

[0072] The processing unit 101 can also activate the inertial measurement unit 107 when it determines that the user's intention is to gaze at the near-field interface. That is, the inertial measurement unit 107 is activated when the user's intention is to gaze at the near-field interface and goes into sleep mode when the user is gazing at the far-field world. The inertial measurement unit 107 sends the perceived head posture of the user to the processing unit 101. The processing unit 101 generates gimbal control commands for the unmanned device based on the head posture and sends the gimbal control commands to the unmanned device through the communication unit 108.

[0073] For unmanned equipment control scenarios, such as controlling a drone, the drone's flight status can be controlled via a joystick, voice, and eye-tracking. Footage captured by the drone can also be transmitted back to the processing unit 101 via the communication unit 108. The processing unit 101 controls the projection unit 104 to project the footage onto a virtual interactive interface on the display element 105. When the user's intention is to gaze at the near-field interface, indicating an intention to view the captured footage, the processing unit 101 will activate the inertial measurement unit 107 to control the drone gimbal's head posture and transmit footage from different perspectives. However, when the user's intention is to gaze at the far-field world, i.e., when the user looks at the drone through the display element 105, they are not viewing the virtual interactive interface or controls. Even if head movements cause changes in the image, the user will not perceive them. Therefore, when the user's intention is to gaze at the far-field world, the inertial measurement unit 107 is kept in a dormant state to save power and avoid invalid gimbal control command transmissions.

[0074] like Figure 5 As shown, the following description uses an eye-tracking interaction device as an example of AR glasses: The processing unit 101, memory 106, inertial measurement unit 107, and communication unit 108 are integrated on the main circuit board 100, which is located in the temple of the AR glasses. The invisible light source 102 and image sensor 103 are located in the frame above the nose pads, and can be arranged side-by-side. The projection unit 104 is located in the upper frame of the AR glasses, and the display element 105 is the lens of the AR glasses.

[0075] Processing unit 101 can be an ARM Cortex-M7 or other microcontroller / SoC, possessing digital signal processing capabilities and rich peripheral interfaces, including: SPI (Serial Peripheral Interface), I... 2 C (Inter-Integrated Circuit), UART (Universal Asynchronous Receiver / Transmitter), and GPIO (General Purpose Input / Output), etc. The processing unit 101 is responsible for running the firmware program in the memory 106, including: executing core algorithms such as image processing, pupil detection, gaze point calculation, state discrimination, and instruction mapping. The memory 106 is connected to the processing unit 101 via the SPI bus; the inertial measurement unit 107 is connected via I... 2The C-bus connects to the processing unit 101; the communication unit 108 connects to the processing unit 101 via the UART interface; the invisible light source 102 and the image sensor 103 are connected to the processing unit 101 via the first flexible circuit board 109, the invisible light source 102 is connected to the GPIO interface of the processing unit 101, and the image sensor 103 is connected to the DCMI (Digital Camera Interface) or a dedicated parallel interface of the processing unit 101.

[0076] It is understood that the layout structure of the components of AR glasses is not limited to that in this embodiment.

[0077] The present invention also provides an eye-tracking interaction control system, such as Figure 6 As shown, the system includes: an unmanned device 201 and an eye-tracking interaction device as described in the above embodiment. The eye-tracking interaction device is wirelessly connected to the unmanned device and is used to control the unmanned device. Specifically, the processing unit 101 is wirelessly connected to the unmanned device through the communication unit 108 to perform eye-tracking manipulation on the unmanned device or receive information transmitted back by the unmanned device, such as transmitted captured images.

[0078] For example, in the field of drone applications, the virtual interactive interface also displays operation controls for controlling the movement state of the drone, such as: take-off, landing, left turn, right turn, ascent, descent, return to home, etc. When the user looks at any operation control, the processing unit 101 detects that the user's gaze point stays on any operation control for a longer than a preset time, generates a target instruction for that operation control, and sends the target instruction to the drone through the communication unit 108 to control the movement state of the drone.

[0079] In the eye-tracking interaction control system of this embodiment, the eye-tracking interaction device can accurately determine whether the user's intention is to gaze at the near-field interface or the far-field world during the eye-tracking interaction process, thus avoiding the safety problem caused by the accidental triggering of operation controls when gazing at the far-field world.

[0080] In some embodiments, the eye-tracking interactive control system further includes a handle 202 and / or a voice control unit 203. The voice control unit 203 is connected to the processing unit 101, and the handle 202 is connected to the processing unit 101 via a communication unit 108.

[0081] The processing unit 101 is also configured to activate the control handle 202 and / or the voice control unit 203 and suppress the inertial measurement unit 107 when it is determined that the user's intention is to gaze at the far field world.

[0082] The processing unit 101 is also used to generate target instructions for the controls corresponding to the gaze point when it is determined that the user's intention is to gaze at the near-field interface, and to activate the inertial measurement unit 107, while keeping the control handle 201 and / or voice control unit 203 active, to generate gimbal control instructions for the unmanned device based on the head posture, and to send the gimbal control instructions and target instructions to the unmanned device through the communication unit.

[0083] In this embodiment, multimodal control of the unmanned device is achieved through the control handle 202, the voice control unit 203, and the eye-tracking and head posture control of the eye-tracking interaction device. When the user's intention is to gaze at the near-field interface, the control handle 202 and / or the voice control unit 203 remain active, thereby decoupling the view control and flight control. In the near-field interface gaze state, the head controls the gimbal view, allowing the user to shoot wherever they look, while the control handle 202 and / or the voice control unit 203 controls the drone's flight. The two do not interfere with each other, greatly reducing the operator's difficulty of operation.

[0084] Furthermore, in this embodiment, different control modes are automatically activated under different user intentions. That is, by determining different user intentions, the automatic switching of control modes of unmanned equipment is realized. Compared with the prior art, which requires users to manually complete the switching of each control mode (such as: button switching, menu selection, etc.), the cognitive burden and operation steps of switching control modes are reduced.

[0085] In some embodiments, when the processing unit 101 determines that the user's intention is near-field interface fixation, and multiple control modal target instructions are received simultaneously, the gimbal control instruction and the target instruction of the fixation point corresponding to the control are sent to the unmanned device first. Since the user prioritizes the unmanned device's camera feed and eye-controlled operation during near-field interface fixation, when multiple control modal control instructions are received simultaneously, the gimbal control instruction and the target instruction of the fixation point corresponding to the control need to be processed first.

[0086] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0087] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0088] All actions involving the acquisition of signal information or data in this invention are carried out in compliance with the relevant data protection laws and policies of the country where the device is located, and with the authorization granted by the owner of the device.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An eye movement interaction device, characterized by, include: Processing unit, invisible light source, image sensor, projection unit, and display element; The invisible light source emits an invisible beam of light to illuminate the eye under the control of the processing unit; The image sensor sends the acquired eye image to the processing unit; The processing unit controls the projection unit to project a virtual interactive interface onto the display element, identifies the pupil from the eye image and determines the pupil size, determines the user's gaze point coordinates based on the eye image, and determines the user's intention to gaze at the near-field interface if the gaze point coordinates are located within the near-field virtual interactive interface and the pupil size is less than a preset pupil size threshold; otherwise, it is determined that the user's intention is to gaze at the far-field world.

2. The eye-tracking interaction device according to claim 1, characterized in that, Also includes: A memory connected to the processing unit; The processing unit can also guide the user to look at the virtual interactive interface, and after receiving confirmation feedback from the user, control the image sensor to acquire a first eye calibration image and determine the near-field pupil size from the first eye calibration image; The processing unit can also guide the user to gaze at the far-field external world, and after receiving confirmation feedback from the user, control the image sensor to acquire a second eye calibration image, and determine the far-field pupil size from the second eye calibration image; The processing unit calibrates the preset pupil size threshold based on the near-field pupil size and the far-field pupil size, and stores the preset pupil size threshold in the memory.

3. The eye-tracking interaction device according to claim 1, characterized in that, Also includes: A memory connected to the processing unit, the memory being used to store a preset rate threshold and / or a preset time range; The processing unit can also determine the pupil size contraction rate and / or the contraction duration to less than the preset pupil size threshold, and obtain the preset rate threshold and / or preset time range from the memory; and determine that the gaze point coordinates are located within the virtual interactive interface. If the pupil size contraction rate is greater than the preset rate threshold and / or the contraction duration is within the preset time range, the user's intention is determined to be near-field interface gaze; otherwise, it is far-field world gaze.

4. The eye-tracking interaction device according to claim 1, characterized in that, The processing unit can also send a first opacity display instruction to the projection unit after determining that the user's intention is to gaze at the near-field interface, and the projection unit projects the virtual interactive interface with the first opacity. The processing unit can also send a second opacity display instruction to the projection unit after determining that the user's intention is far-field world gazing. The projection unit projects the virtual interactive interface with the second opacity, where the first opacity is greater than the second opacity.

5. The eye-tracking interaction device according to claim 1, characterized in that, After determining that the user's intention is to gaze at the near-field interface, the processing unit can also continuously monitor the pause position of the gaze point in the virtual interactive interface based on the coordinates of the gaze point; when the pause position is located within the area of ​​any control in the virtual interactive interface, a highlighting instruction is sent to the projection unit, and the projection unit highlights the control. The processing unit can also accumulate the duration of the gaze point on any control. If the duration of the gaze point is greater than a preset duration, the unit determines that any control is a target control and generates a target instruction corresponding to the target control.

6. The eye-tracking interaction device according to claim 5, characterized in that, The processing unit can also send a prompt window instruction to the projection unit when the dwell time exceeds the preset time, and the projection unit generates a user confirmation prompt window on the virtual interactive interface; The processing unit can also generate the target instruction after receiving feedback information from the user who has executed the prompt in the user confirmation prompt window.

7. The eye-tracking interaction device according to claim 1, characterized in that, The invisible light source, image sensor, and projection unit are connected to the processing unit via a flexible circuit board.

8. The eye-tracking interaction device according to any one of claims 1 to 7, characterized in that, The interactive control for unmanned equipment also includes: an inertial measurement unit and a communication unit connected to the processing unit; The processing unit can also activate the inertial measurement unit when it determines that the user's intention is to gaze at the near-field interface. The inertial measurement unit sends the perceived head posture of the user to the processing unit. The processing unit generates gimbal control commands for the unmanned device based on the head posture and sends the gimbal control commands to the unmanned device through the communication unit.

9. An eye-tracking interactive control system, characterized in that, include: An unmanned device and an eye-tracking interaction device according to any one of claims 1 to 8, wherein the eye-tracking interaction device is wirelessly connected to the unmanned device and is used to control the unmanned device.

10. The eye-tracking interactive control system according to claim 9, characterized in that, Also includes: A control handle and / or a voice control unit, wherein the voice control unit is connected to the processing unit, and the control handle is connected to the processing unit via a communication unit; The processing unit can also activate the control handle and / or voice control unit and suppress the inertial measurement unit when it determines that the user's intention is far-field world gaze. The processing unit can also generate target instructions for the controls corresponding to the gaze point when it is determined that the user's intention is to gaze at the near-field interface, activate the inertial measurement unit, and keep the control handle and / or voice control unit active. It can also generate gimbal control instructions for the unmanned device based on the head posture, and send the gimbal control instructions and the target instructions to the unmanned device through the communication unit.