Eye movement interaction control method and apparatus

Abstract

The application relates to the technical field of eye movement control, and provides an eye movement interaction control method and device, the method comprising the following steps: acquiring an eye image of a user; identifying a pupil from the eye image and determining a pupil size; determining a gaze point coordinate of the user based on the eye image; in the case that the gaze point coordinate is located in a virtual interaction interface of a near field and the pupil size is smaller than a preset pupil size threshold, determining that the user's intention is near field interface gaze; otherwise, the user's intention is far field world gaze. The application utilizes the physiological reaction that the pupil shrinks due to near reflex and the pupil is relatively enlarged when gazing at a distant object, determines that the user's intention is near field interface gaze when the two conditions that the gaze point coordinate is located in the virtual interaction interface of the near field and the pupil size is smaller than the preset pupil size threshold are simultaneously met, and thus the user's intention of near field interface gaze or far field world gaze can be accurately determined in the eye movement interaction process.

Description

Technical Field

[0001] This invention relates to the field of eye-tracking control technology, and more particularly to an eye-tracking interactive control method and device. Background Technology

[0002] Augmented Reality (AR) is a technology that overlays and merges virtual scenes or information with the real physical environment, presenting them to the user and creating a shared space between the virtual and real worlds. Currently, AR devices (such as glasses and helmets) can project virtual interactive interfaces onto transparent displays (such as eyeglass lenses), allowing users to interact with these interfaces through eye movements. Existing eye-tracking interaction solutions primarily rely on the direction of eye gaze. When the camera captures the user's gaze point falling on the near-field virtual interactive interface, it determines whether the user intends to interact with the AR device or view content from the near-field interface (such as footage transmitted from an unmanned device). For example, in applications where AR devices control unmanned equipment (such as drones, unmanned vehicles, unmanned boats, or robots), the AR device displays a virtual interactive interface on a transparent display. This virtual interface contains multiple virtual controls for controlling the unmanned equipment. When the user's gaze point falls on the area of ​​a particular virtual control, it is determined that the user intends to trigger that button's function to control the unmanned equipment.

[0003] However, when a user is gazing at the far-field world (also known as external world gazing), that is, observing the real world in the far field, their line of sight may simultaneously pass through the near-field virtual interactive interface, meaning the gaze point falls on the near-field virtual interactive interface. This can lead to a misjudgment that the user's intention is near-field interface gazing, making it difficult to accurately determine whether the user's intention is near-field interface gazing or far-field world gazing. For example, when a user is gazing at a drone in the distant sky, and their visual direction happens to pass through the area where a virtual control is located, meaning the gaze point falls on that virtual control area, the user's interaction intention may be misjudged as operating the virtual control, leading to accidental triggering. This is unacceptable in application scenarios with high safety requirements, such as operating drones. Summary of the Invention

[0004] This invention provides an eye-tracking interactive control method and device to solve the technical problem that existing interactive control methods based on gaze points are unable to accurately determine the user's intent.

[0005] This invention provides an eye-tracking interaction control method, comprising the following steps: Obtain the user's eye image; Identify the pupil and determine its size from the eye image; The coordinates of the user's gaze point are determined based on the eye image; If the gaze point coordinates are located within the near-field virtual interactive interface and the pupil size is less than a preset pupil size threshold, then the user's intention is determined to be near-field interface gaze; otherwise, it is far-field world gaze.

[0006] According to an eye-tracking interaction control method provided by the present invention, the preset pupil size threshold is determined based on the following method: Acquire the near-field pupil size when the user is looking at a virtual interactive interface in the near field, and the far-field pupil size when the user is looking at the external real world in the far field; The preset pupil size threshold is calibrated based on the near-field pupil size and the far-field pupil size.

[0007] According to an eye-tracking interaction control method provided by the present invention, a preset pupil size threshold is calibrated based on the near-field pupil size and the far-field pupil size, including: Calculate the size difference between the far-field pupil size and the near-field pupil size, and obtain a weighted value for the size difference based on a preset size difference weight; The preset pupil size threshold is obtained by subtracting the size difference weighted value from the far-field pupil size.

[0008] According to an eye-tracking interaction control method provided by the present invention, if the gaze point coordinates are located within a near-field virtual interaction interface and the pupil size is less than a preset pupil size threshold, then the user's intention is determined to be near-field interface gaze; otherwise, it is far-field world gaze. The method includes: If the gaze point coordinates are determined to be within the near-field virtual interactive interface, and the pupil size contraction rate is greater than a preset rate threshold and / or contracts to less than the preset pupil size threshold within a preset time range, the user's intention is determined to be near-field interface gaze; otherwise, it is far-field world gaze.

[0009] According to an eye-tracking interaction control method provided by the present invention, the method further includes: after determining that the user's intention is to gaze at a near-field interface, displaying the virtual interaction interface with a first opacity; and after determining that the user's intention is to gaze at a far-field world, displaying the virtual interaction interface with a second opacity, wherein the first opacity is greater than the second opacity.

[0010] According to an eye-tracking interaction control method provided by the present invention, after determining that the user's intention is to gaze at a near-field interface, the method further includes: Based on the coordinates of the gaze point, the pause position of the gaze point within the virtual interactive interface is continuously monitored. When the pause position is located within the area of ​​any control in the virtual interactive interface, that control is highlighted. If the gaze point lingers on any control for a duration longer than a preset duration, the control is identified as the target control, and a corresponding target instruction is generated for the target control.

[0011] According to an eye-tracking interactive control method provided by the present invention, when the gaze point lingers on any control for a duration longer than a preset duration, the method determines that any control is a target control and generates a corresponding target instruction for the target control, including: If the duration of the gaze on any of the controls exceeds a preset duration, a user confirmation prompt window is generated on the virtual interactive interface; After receiving feedback from the user that they have followed the prompts in the user confirmation window, the target instruction is generated.

[0012] An eye-tracking interactive control method according to the present invention, used for interactive control of unmanned equipment, further includes: When it is determined that the user's intention is to gaze at the far field world, the first set of control modes is activated and the second set of control modes is suppressed. The first set of control modes includes at least: manual control mode and voice control mode for controlling the motion state of the unmanned device. The second set of control modes includes at least: eye-tracking control mode for controlling the motion state of the unmanned device and head posture control mode for controlling the motion state of the gimbal of the unmanned device. If the user's intent is determined to be near-field interface gaze, the second control modality set is activated, while the first control modality set remains active.

[0013] According to an eye-tracking interaction control method provided by the present invention, the method further includes: when it is determined that the user's intention is to gaze at a near-field interface, and when multiple target instructions of control modes are received simultaneously, prioritizing the sending of the target instructions corresponding to the second set of control modes to the unmanned device.

[0014] The present invention also provides an eye-tracking interaction control device, comprising: The eye image acquisition module is used to acquire images of the user's eyes. A pupil size determination module is used to identify the pupil and determine the pupil size from the eye image; A gaze point determination module is used to determine the coordinates of the user's gaze point based on the eye image; The user intent determination module is used to determine that the user intent is near-field interface gaze if the gaze point coordinates are located within the near-field virtual interactive interface and the pupil size is less than a preset pupil size threshold; otherwise, it is far-field world gaze.

[0015] The eye-tracking interaction control method and device provided by this invention acquires an image of the user's eyes; identifies the pupil from the eye image and determines the pupil size; determines the coordinates of the user's gaze point based on the eye image; and determines the user's intention to gaze at the near-field interface if the gaze point coordinates are located within the near-field virtual interaction interface and the pupil size is less than a preset pupil size threshold. Otherwise, it is determined to be far-field world gaze. This method utilizes the physiological response of pupil constriction due to near reflex and relative pupil dilation when gazing at distant objects to determine whether the user's intention is near-field interface gaze or far-field world gaze during eye-tracking interaction. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the eye-tracking interactive control method provided by the present invention.

[0018] Figure 2 This is a time series diagram of pupil diameter and fixation point when the user switches between near-field interface fixation state and far-field world fixation state in the eye-tracking interaction control method provided by the present invention.

[0019] Figure 3 This is one of the schematic diagrams of the pupil diameter variation with gaze distance and the threshold setting method in the eye-tracking interaction control method provided by the present invention.

[0020] Figure 4 This is the second schematic diagram of the pupil diameter variation with gaze distance and the threshold setting method in the eye-tracking interaction control method provided by the present invention.

[0021] Figure 5 This is a schematic diagram of the eye-tracking interactive control device provided by the present invention.

[0022] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] The eye-tracking interactive control method of this invention can be integrated into an AR device, such as AR glasses. An AR device typically includes a projection unit, a display element (e.g., a lens), an infrared light source, an image sensor (e.g., a global shutter infrared CMOS sensor), and a processing unit (e.g., an ARM Cortex-M7 MCU). The projection unit, infrared light source, and image sensor are all connected to the processing unit. The projection unit projects the virtual interactive interface to be displayed onto the display element. Specifically, this method is integrated into the processing unit, which is the executing entity. The specific process is as follows: Figure 1 As shown, the procedure includes steps S110 to S140.

[0025] Step S110: Acquire the user's eye image. Specifically, the eye image can be acquired in real time through the image sensor of the AR device, with a sampling rate of 100Hz and a resolution of 400×400. The processing unit acquires the user's eye image acquired by the image sensor.

[0026] Step S120: Identify the pupil from the eye image and determine the pupil size. For example, the pupil edge can be extracted from the eye image using thresholding and ellipse fitting algorithms to calculate the pupil diameter (units can be pixels or mm). The pupil diameter is used to measure pupil size. Alternatively, the pupil radius or area can also be used. It is understood that the eye image is acquired at a certain sampling rate; therefore, pupil size is also a quantity that changes over time. For example, the pupil diameter can be represented as a time sequence D(t).

[0027] Step S130: Determine the user's gaze coordinates based on the eye image. The gaze coordinates represent the user's viewing position on the display element. For example, this can be achieved by using corneal reflection to detect the bright reflective spot formed on the cornea by an infrared light source, calculating the relative position vector between the center of the bright reflective spot and the center of the pupil, and then converting the relative position vector into the user's gaze coordinates in the display area on the display element using a pre-calibrated polynomial mapping model.

[0028] Step S140: If the gaze point coordinates are located within the near-field virtual interactive interface and the pupil size is less than a preset pupil size threshold, then the user's intention is determined to be near-field interface gaze; otherwise, it is far-field world gaze. The virtual interactive interface can be an interface displaying controls in a specific application, or it can be an information display interface. For example, in an application scenario using AR devices to control a drone, the virtual interactive interface can display controls for controlling the drone and the drone's captured footage. The coordinate range of the display element area where the virtual interactive interface is located, as well as the coordinates of the controls within it, can be obtained through the projection unit.

[0029] In this step, the first step is to determine whether the gaze point coordinates fall within the near-field virtual interactive interface on the display element. If the gaze point coordinates are outside the near-field virtual interactive interface, the user's intention is far-field gazing, meaning the user is in a far-field gazing state. If the gaze point coordinates are within the near-field virtual interactive interface, it indicates that the user may have a near-field interface gazing intention, or they may be observing the far-field external world and their line of sight happens to pass through the area of ​​the virtual interactive interface. Therefore, to accurately determine whether the user's intention is near-field interface gazing or far-field world gazing, this step further determines whether the pupil size is less than a preset pupil size threshold. This is because when the human eye gazes at targets at different distances, not only does eyeball convergence occur, but pupil size also changes. That is, when gazing at near objects, the pupil constricts due to the near reflex; when gazing at distant objects, the pupil relatively dilates. Moreover, this physiological response is controlled by the autonomic nervous system and is difficult to suppress subjectively. Therefore, this step can utilize the physiological response of the pupil. If the pupil size is less than the preset pupil size threshold, it indicates that the user intends to gaze at the near-field virtual interactive interface, rather than having their line of sight pass through the virtual interactive interface when observing the far-field external world. Thus, during eye-tracking interaction, it is possible to accurately determine whether the user intends to gaze at the near-field interface or the far-field world.

[0030] like Figure 2 As shown, when the user's state switches from far-field world gaze to near-field interface gaze, the pupil constricts and becomes smaller, and the change in pupil diameter is shown by the blue curve. At the same time, when the coordinates of the gaze point are also located within the near-field virtual interactive interface, as shown by the green curve, it is determined that the user's intention is near-field interface gaze.

[0031] The eye-tracking interaction control method of this embodiment acquires the user's eye image; identifies the pupil from the eye image and determines the pupil size; determines the user's gaze point coordinates based on the eye image; if the gaze point coordinates are located within the near-field virtual interaction interface and the pupil size is less than a preset pupil size threshold, then the user's intention is determined to be near-field interface gaze; otherwise, it is far-field world gaze. That is, it utilizes the physiological response of pupil constriction due to near reflex and relative pupil dilation when gazing at distant objects to determine whether the user's intention is near-field interface gaze when both conditions are met: the gaze point coordinates are located within the near-field virtual interaction interface and the pupil size is less than a preset pupil size threshold. Thus, it can accurately determine whether the user's intention is near-field interface gaze or far-field world gaze during eye-tracking interaction.

[0032] In some embodiments, the preset pupil size threshold is determined based on the following method: The system acquires the near-field pupil size when the user is focused on a virtual interactive interface in the near field, and the far-field pupil size when the user is focused on the external real world in the far field. For example, after wearing glasses, the user, following voice prompts, first focuses on a virtual light point appearing on the near-field virtual interactive interface. The equivalent optical distance of the virtual light point is approximately 0.3m. The user maintains focus on the virtual light point for a certain duration (e.g., 3 seconds) to allow the pupil to constrict and stabilize. Then, the near-field pupil size is recorded, such as the near-field pupil diameter D. near For example: D near =2.1mm. Then, the user is prompted to focus on a real target in the far field of the external world, such as a target 10m away. The user is instructed to maintain focus on the target for a certain duration (e.g., 3 seconds) to allow the pupil to dilate and stabilize. The far field pupil size is then recorded, such as the far field pupil diameter D. far For example: D far =4.8mm.

[0033] A preset pupil size threshold is calibrated based on near-field and far-field pupil sizes. For example, the average or median of the near-field and far-field pupil sizes can be used as the preset pupil size threshold. Figure 3 As shown, for D near and D far The preset pupil size threshold D is obtained by taking the average value. th =3.4mm.

[0034] In this embodiment, the near-field pupil size and far-field pupil size are first calibrated, and then the preset pupil size threshold is calibrated by the near-field pupil size and the far-field pupil size. The method of using the mean or median value as the preset pupil size threshold is simple and easy to implement, and the near-field interface gaze and the far-field world gaze are regarded as two equivalent states, thereby obtaining a reasonable pupil size threshold.

[0035] It should be noted that after users wear AR devices, they are prompted to calibrate the preset pupil size threshold, so as to accurately calibrate the preset pupil size threshold according to different users and different lighting environments.

[0036] In some embodiments, a preset pupil size threshold is calibrated based on near-field pupil size and far-field pupil size, specifically including: Calculate the size difference between the far-field pupil size and the near-field pupil size, and obtain the weighted value of the size difference according to the preset size difference weight.

[0037] The preset pupil size threshold is obtained by subtracting the size difference weighting value from the far-field pupil size.

[0038] Taking pupil diameter as an example, specifically, the preset pupil size threshold D is calibrated according to the following formula. th : D th =D far -α(D far -D near ).

[0039] Where α is the size difference weight, and its value ranges from (0, 0.5) ∪ (0.5, 1).

[0040] For example, the value of α ranges from (0.2, 0.5), such as: Figure 4 As shown, when α=0.3, which is the 70th percentile of far-field pupil size, D th =4.0mm. When the pupil size increases from small to large, it is less likely to exceed this threshold, thus preventing the pupil from dilating and causing the user to exit the near-field viewing state due to the dimming of the external ambient light. When the pupil size decreases from large to small, it is easier to exceed this threshold, making it easier for the user to switch to near-field viewing when in the far-field viewing state.

[0041] For example, α takes values ​​in the range (0.6, 0.8), D th It will be 3.4mm lower than the average value. The pupil size is less likely to cross this threshold when it changes from large to small. In the state of far-field world gaze, the user must deeply constrict the pupil to trigger near-field interface gaze. That is, it ensures that only the user's intention with obviously constricted pupil can be judged as near-field interface gaze, which greatly reduces the risk of accidental touch of the control. This just meets the needs of high-safety scenarios such as drones.

[0042] In this embodiment, the preset pupil size threshold is jointly calibrated based on the far-field pupil size, the size difference between the far-field pupil size and the near-field pupil size, and the size difference weight. The preset pupil size threshold can be adjusted by setting the size difference weight based on the specific application scenario (light intensity or the degree of safety requirements of the application scenario), thereby realizing personalized adaptive pupil size threshold setting.

[0043] In some embodiments, step S140, if the gaze point coordinates are located within the near-field virtual interactive interface and the pupil size is less than a preset pupil size threshold, then it is determined that the user's intention is near-field interface gaze; otherwise, it is far-field world gaze. Specifically, this includes: If the gaze point coordinates are located within the near-field virtual interactive interface, and the user's intention is determined to be near-field interface gaze if the pupil size contraction rate is greater than a preset rate threshold and / or contracts to less than a preset pupil size threshold within a preset time range; otherwise, it is far-field world gaze.

[0044] Under constant lighting conditions, when the human eye shifts from viewing a distant target to viewing a nearby target, the duration of pupil constriction is typically between 300ms and 800ms. Within this time window, the change in pupil diameter (D...) far -D near Dividing the pupil constriction rate (approximately 2mm~3mm) by the time, the calculated physiological rate of pupil constriction naturally falls within the range of 2.5mm / s~10mm / s. Preferably, the preset rate threshold can be 3mm / s~8mm / s, for example, 3mm / s. The pupil constriction rate can be calculated using dD(t) / dt. By judging the pupil constriction rate, pupil drift caused by slowly varying factors such as ambient light (whose rate is usually less than 0.1mm / s) can be eliminated. That is, when viewing the distant external world through the area of ​​the virtual interactive interface, the situation where the pupil slowly constricts to a size less than the preset pupil size threshold due to increased ambient light is avoided, which might mistakenly indicate that the user needs to focus on the near-field interface.

[0045] The preset time range can be set to 0.3s~0.8s to exclude abrupt pupil changes caused by transient interference such as blinking and strong flashes of light (whose contraction time is usually less than 0.2s). Similarly, when viewing the distant external world through the area of ​​the virtual interactive interface, it avoids situations where blinking or strong flashes cause the pupil to slowly and violently contract to a size less than the preset pupil size threshold, which might be mistakenly interpreted as the user needing to focus on the near-field interface.

[0046] Preferably, if the pupil size contraction rate is greater than a preset rate threshold and contracts to less than a preset pupil size threshold within a preset time range, the user's intention is determined to be near-field interface fixation; otherwise, it is far-field world fixation. For example: Figure 2 As shown, the pupil size contraction rate is 6.75 mm / s, and the time taken to contract to below the preset pupil size threshold is 400 ms (0.4 s), which determines that the user's intention is to gaze at the near-field interface.

[0047] In this embodiment, the pupil contraction process and the size of the pupil after contraction are limited by the pupil contraction rate and / or a preset time range. This can avoid pupil drift caused by slow-changing factors such as ambient light or fast-changing factors such as flashes of light and blinking, thereby more accurately determining the user's intention.

[0048] In some embodiments, the eye-tracking interaction control method further includes: after determining that the user's intention is to gaze at a near-field interface, displaying a virtual interactive interface with a first opacity, and after determining that the user's intention is to gaze at a far-field world, displaying a virtual interactive interface with a second opacity, wherein the first opacity is greater than the second opacity.

[0049] Specifically, when the user's intention is determined to be near-field viewing, the virtual interactive interface is displayed with a first opacity, meaning higher opacity, allowing the user to clearly see the virtual interactive interface. When the user's intention is determined to be far-field viewing, the virtual interactive interface is displayed with a second opacity, meaning lower opacity, making the virtual interactive interface closer to transparent, reducing obstruction of the user's line of sight when viewing from the far field. Of course, when the user's intention switches from far-field viewing to near-field viewing, the opacity of the virtual interactive interface increases from the second opacity to the first opacity; when the user's intention switches from near-field viewing to far-field viewing, the opacity of the virtual interactive interface decreases from the first opacity to the second opacity.

[0050] In this embodiment, the display opacity of the virtual interactive interface varies under different user intentions, thereby improving the user interaction experience regardless of the user intention.

[0051] In some embodiments, after determining that the user intent is a near-field interface gaze, the method further includes the following steps: Step S150: Continuously monitor the pause position of the gaze point within the virtual interactive interface based on the gaze point coordinates. When the user is in a near-field gaze state, the gaze point coordinates change continuously with the user's eye movements. By continuously monitoring the gaze point coordinates, the pause position of the gaze point within the virtual interactive interface can be determined.

[0052] Step S160: When the pause position is within the area of ​​any control in the virtual interactive interface, highlight that control. The area of ​​any control in the virtual interactive interface is predetermined. By comparing the gaze point coordinates with this area, it can be determined whether the pause position is within the area of ​​any control. If the pause position is within the area of ​​any control, it means that the user is currently viewing that control. Therefore, highlighting that control provides feedback and confirmation to the user, thereby improving the user's interactive experience.

[0053] Step S170: If the duration of the gaze point's stay on any control exceeds a preset duration, determine that control as the target control and generate the corresponding target instruction for the target control. In this step, to avoid the user accidentally triggering any control when the gaze point only briefly stays within the area of ​​any control without any intention to trigger it, the duration of the gaze point's stay on any control is judged. That is, the user's gaze duration on any control is judged. If the duration of the stay exceeds the preset duration, it indicates that the user intends to trigger that control. Therefore, the control is determined as the target control, and the corresponding target instruction for the target control is generated.

[0054] For example: When a user glances at control 1, control 1 is highlighted. If the user determines that control 1 is not the control to be triggered, and moves their gaze away from control 1 (without reaching the preset time, control 1 is only highlighted and then returns to normal display), and then looks at control 2, control 2 is highlighted. If control 2 is the control that the user wants to trigger, then control 2 is confirmed as the target control. The user only needs to look at control 2 for the preset time, and control 2 will be automatically triggered, generating the corresponding target command for control 2.

[0055] In this embodiment, the user is given gaze feedback by highlighting the control, which improves the user interaction experience. At the same time, the target control is determined only when the gaze stays on any control for a longer than a preset time, so as to avoid generating the target command by mistake and improve the operation safety.

[0056] In some embodiments, step S170, where the duration of the gaze point on any control exceeds a preset duration, determines any control as the target control and generates a corresponding target instruction for the target control, specifically includes: If the gaze remains on any control for a duration exceeding a preset duration, a user confirmation prompt window is generated in the virtual interactive interface. For example, the user confirmation prompt window is used to confirm or cancel the control, or to display a preset confirmation action for the user, such as blinking, micro-gesture, or voice confirmation.

[0057] After receiving feedback from the user confirming the action in the confirmation prompt window, a target instruction is generated. For example, if the user stares at the confirmation control for an extended period of time, or if feedback related to a pre-defined confirmation action by the user is received, such as detecting a blink, then it is confirmed that the user's intention is to trigger either control.

[0058] In this embodiment, when the gaze point stays on any control for a longer than a preset time, a user confirmation prompt window is generated in the virtual interactive interface for secondary confirmation. This avoids the accidental triggering of the target command when the user gazes at any control for a long time without deciding whether to trigger it, thus further improving operational safety.

[0059] In some embodiments, the method of the present invention is used for the interactive control of unmanned devices. Taking drones as an example, in existing related technologies, the control of AR-based drones includes multiple control modes: voice, eye tracking, manual (remote control or joystick), and head posture, among others. The head posture control mode is used to control the gimbal movement of the drone, while voice, eye tracking, and manual control modes can control the drone's overall motion. Currently, switching between control modes requires manual operation by the user (e.g., button switching, menu selection), increasing the cognitive burden and operational steps involved in switching control modes.

[0060] In this embodiment, when it is determined that the user's intention is far-field gaze, a first set of control modes is activated and a second set of control modes is suppressed. The first set of control modes includes at least: a manual control mode and a voice control mode for controlling the motion state of the unmanned device, and the second set of control modes includes at least: an eye-tracking control mode for controlling the motion state of the unmanned device and a head posture control mode for controlling the gimbal motion state of the unmanned device.

[0061] If the user's intent is determined to be near-field interface gaze, the second control modality set is activated while the first control modality set remains active.

[0062] Specifically, when the user's intention is to gaze at the far field world, the user observes the drone through the display element of the AR device. This means that the user is only focused on the drone's flight attitude and position at this time. Moreover, when watching the drone, the user cannot observe the images transmitted back by the drone in the virtual interactive interface on the display element. Therefore, when used for gazing at the far field world, only the first set of control modes, which includes the manual control mode and the voice control mode for controlling the motion state of the unmanned device, is activated.

[0063] When the user's intention is to gaze at the near-field interface, the user focuses on controls in the virtual interactive interface or the footage transmitted back by the drone. These controls include operational controls for the drone's movement, such as takeoff, landing, left turn, right turn, ascent, descent, and return-to-home. In this case, the user prioritizes the footage captured by the drone. Therefore, the eye-tracking control mode for controlling the drone's movement and the head-gesture control mode for controlling the drone's gimbal movement are activated. Through eye-tracking—that is, gazing at any of the aforementioned operational controls—the user controls the drone's flight attitude, while head movements control the movement of the drone's gimbal, thereby controlling the shooting angle and obtaining different captured footage.

[0064] Of course, when the user's intention is to gaze at the near-field interface, the manual control mode and voice control mode can remain active, thereby decoupling the view control and flight control. In the near-field interface gaze state, the head controls the gimbal view, so that you can shoot wherever you look, and the drone can be controlled manually / by voice without interference, which greatly reduces the cognitive load of the operator.

[0065] Furthermore, by switching between near-field interface gaze state and far-field world gaze state, the control mode is automatically switched. When looking at the aircraft, the system automatically enters the far-field world gaze state, supporting manual and voice control flight. When looking at the screen, it automatically enters the near-field interface gaze state and automatically switches the control mode. The gimbal can be controlled by head rotation, and the drone can be controlled by eye control. The two states can be switched freely while the control mode is automatically switched, thus solving the fundamental contradiction that one cannot simultaneously look at the aircraft and look at the screen in the traditional drone control process.

[0066] For example, when controlling a drone to navigate through a narrow space using AR glasses, the drone control process is as follows: Far-field phase: The operator looks up and observes the relative position of the drone and the trees through the lens, then manually controls the drone to fly towards the gap entrance using the remote control. At this time, the operator is in a direct far-field observation state, with manual and voice control activated, while head posture control is suppressed.

[0067] Near-field phase: As the operator approaches the gap entrance, they switch their gaze to the transmitted image on the lens, their pupils constrict, and their fixation point is within the transmitted image, thus entering the near-field interface fixation state and triggering the near-field gimbal control mode. The operator controls the gimbal by rotating their head to observe the positions of the trees on both sides of the gap, while simultaneously fine-tuning the flight attitude using the remote controller for precise passage.

[0068] Far-field confirmation: After passing through, the operator looks up to observe that the drone has passed safely, the pupils dilate, and the system automatically switches back to the far-field world gaze state to continue the next stage of operation.

[0069] In this embodiment, different sets of control modes are automatically activated under different user intentions. That is, by determining different user intentions, the automatic switching of control modes of unmanned equipment is realized, reducing the cognitive load of the operator.

[0070] In some embodiments, the eye-tracking interaction control method further includes: when it is determined that the user's intention is near-field interface fixation, and multiple control modal target commands are received simultaneously, prioritizing the sending of the target command corresponding to the second control modal set to the unmanned device. Since the user prioritizes the unmanned device's captured image and eye-tracking control method during near-field interface fixation, when multiple control modal control commands are received simultaneously, the control command corresponding to the second control modal set needs to be processed first.

[0071] The eye-tracking interaction control device provided by the present invention is described below. The eye-tracking interaction control device described below can be referred to in correspondence with the eye-tracking interaction control method described above.

[0072] The eye-tracking interaction control device of this invention, such as Figure 5 As shown, it includes the following modules: The eye image acquisition module 510 is used to acquire the user's eye image.

[0073] The pupil size determination module 520 is used to identify the pupil from an eye image and determine the pupil size.

[0074] The gaze point determination module 530 is used to determine the coordinates of the user's gaze point based on the eye image.

[0075] The user intent determination module 540 is used to determine that the user intent is near-field interface gaze if the gaze point coordinates are located within the near-field virtual interactive interface and the pupil size is less than a preset pupil size threshold; otherwise, it is far-field world gaze.

[0076] In some embodiments, the preset pupil size threshold is determined based on the following method: Obtain the near-field pupil size when the user is looking at a virtual interactive interface in the near field, and the far-field pupil size when the user is looking at the external real world in the far field.

[0077] A preset pupil size threshold is calibrated based on near-field and far-field pupil sizes.

[0078] In some embodiments, a preset pupil size threshold is calibrated based on near-field pupil size and far-field pupil size, including: Calculate the size difference between the far-field pupil size and the near-field pupil size, and obtain the weighted value of the size difference according to the preset size difference weight.

[0079] The preset pupil size threshold is obtained by subtracting the size difference weighting value from the far-field pupil size.

[0080] In some embodiments, the user intent determination module 540 is specifically used to: determine that the gaze point coordinates are located within the near field of the virtual interactive interface, and determine that the user intent is near field interface gaze if the pupil size contraction rate is greater than a preset rate threshold and / or contracts to less than a preset pupil size threshold within a preset time range; otherwise, it is far field world gaze.

[0081] In some embodiments, the eye-tracking interaction control device further includes: a display adjustment module, configured to display a virtual interactive interface with a first opacity after determining that the user's intention is to gaze at a near-field interface, and to display the virtual interactive interface with a second opacity after determining that the user's intention is to gaze at a far-field world, wherein the first opacity is greater than the second opacity.

[0082] In some embodiments, the eye-tracking interaction control device further includes: The gaze point monitoring module is used to continuously monitor the pause position of the gaze point within the virtual interactive interface based on the gaze point coordinates after determining that the user's intention is to gaze at the near-field interface. The control highlighting module is used to highlight any control when the pause position is within the area of ​​any control in the virtual interactive interface; The target instruction generation module is used to identify any control as the target control and generate the corresponding target instruction for the target control when the gaze point stays on any control for a duration longer than a preset duration.

[0083] In some embodiments, the target instruction generation module is specifically used to generate a user confirmation prompt window in the virtual interactive interface when the gaze point stays on any control for a duration longer than a preset duration; and to generate a target instruction after receiving feedback information from the user that the user has executed the prompt in the user confirmation prompt window.

[0084] In some embodiments, the eye-tracking interaction control device is used for the interaction control of unmanned equipment, and further includes: a control mode switching module, configured to activate a first control mode set and suppress a second control mode set when it is determined that the user's intention is far-field world gaze, the first control mode set including at least: a manual control mode and a voice control mode for controlling the motion state of the unmanned equipment, and the second control mode set including at least: an eye-tracking control mode for controlling the motion state of the unmanned equipment and a head posture control mode for controlling the gimbal motion state of the unmanned equipment; and to activate the second control mode set and keep the first control mode set active when it is determined that the user's intention is near-field interface gaze.

[0085] In some embodiments, the eye-tracking interaction control device further includes a target instruction sending module, which, when determining that the user's intention is to gaze at the near-field interface, prioritizes sending the target instruction corresponding to the second set of control modes to the unmanned device when multiple control modes are received simultaneously.

[0086] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute an eye-tracking interaction control method, which includes: Obtain the user's eye image; Identify the pupil and determine its size from an eye image; Determine the coordinates of the user's gaze point based on eye images; If the gaze point coordinates are located within the near-field virtual interactive interface and the pupil size is less than a preset pupil size threshold, then the user's intention is determined to be near-field interface gaze; otherwise, it is far-field world gaze.

[0087] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the eye-tracking interaction control method provided by the above methods, the method including: Obtain the user's eye image; Identify the pupil and determine its size from an eye image; Determine the coordinates of the user's gaze point based on eye images; If the gaze point coordinates are located within the near-field virtual interactive interface and the pupil size is less than a preset pupil size threshold, then the user's intention is determined to be near-field interface gaze; otherwise, it is far-field world gaze.

[0089] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the eye-tracking interaction control method provided by the methods described above, the method comprising: Obtain the user's eye image; Identify the pupil and determine its size from an eye image; Determine the coordinates of the user's gaze point based on eye images; If the gaze point coordinates are located within the near-field virtual interactive interface and the pupil size is less than a preset pupil size threshold, then the user's intention is determined to be near-field interface gaze; otherwise, it is far-field world gaze.

[0090] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0091] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0092] All actions involving the acquisition of signal information or data in this invention are carried out in compliance with the relevant data protection laws and policies of the country where the device is located, and with the authorization granted by the owner of the device.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An eye-tracking interaction control method, characterized in that, include: Obtain the user's eye image; Identify the pupil and determine its size from the eye image; The coordinates of the user's gaze point are determined based on the eye image; If the gaze point coordinates are located within the near-field virtual interactive interface and the pupil size is less than a preset pupil size threshold, then the user's intention is determined to be near-field interface gaze; otherwise, it is far-field world gaze.

2. The eye-tracking interactive control method according to claim 1, characterized in that, The preset pupil size threshold is determined based on the following method: Acquire the near-field pupil size when the user is looking at a virtual interactive interface in the near field, and the far-field pupil size when the user is looking at the external real world in the far field; The preset pupil size threshold is calibrated based on the near-field pupil size and the far-field pupil size.

3. The eye-tracking interactive control method according to claim 2, characterized in that, The preset pupil size threshold is calibrated based on the near-field pupil size and the far-field pupil size, including: Calculate the size difference between the far-field pupil size and the near-field pupil size, and obtain a weighted value for the size difference based on a preset size difference weight; The preset pupil size threshold is obtained by subtracting the size difference weighted value from the far-field pupil size.

4. The eye-tracking interaction control method according to claim 1, characterized in that, If the gaze point coordinates are located within the near-field virtual interactive interface and the pupil size is less than a preset pupil size threshold, then the user's intention is determined to be near-field interface gaze; otherwise, it is far-field world gaze, including: If the gaze point coordinates are determined to be within the near-field virtual interactive interface, and the pupil size contraction rate is greater than a preset rate threshold and / or contracts to less than the preset pupil size threshold within a preset time range, the user's intention is determined to be near-field interface gaze; otherwise, it is far-field world gaze.

5. The eye-tracking interaction control method according to claim 1, characterized in that, Also includes: After determining that the user's intention is to gaze at the near-field interface, the virtual interactive interface is displayed with a first opacity. After determining that the user's intention is to gaze at the far-field world, the virtual interactive interface is displayed with a second opacity, where the first opacity is greater than the second opacity.

6. The eye-tracking interactive control method according to claim 1, characterized in that, After determining that the user's intent is near-field interface gaze, the following is also included: Based on the coordinates of the gaze point, the pause position of the gaze point within the virtual interactive interface is continuously monitored. When the pause position is located within the area of ​​any control in the virtual interactive interface, that control is highlighted. If the gaze point lingers on any control for a duration longer than a preset duration, the control is identified as the target control, and a corresponding target instruction is generated for the target control.

7. The eye-tracking interaction control method according to claim 6, characterized in that, If the gaze point lingers on any control for a duration exceeding a preset duration, the control is identified as the target control, and a corresponding target instruction is generated for the target control, including: If the duration of the gaze on any of the controls exceeds a preset duration, a user confirmation prompt window is generated on the virtual interactive interface; After receiving feedback from the user that they have followed the prompts in the user confirmation window, the target instruction is generated.

8. The eye-tracking interaction control method according to any one of claims 1 to 7, characterized in that, Interactive control for unmanned equipment also includes: When it is determined that the user's intention is to gaze at the far field world, the first set of control modes is activated and the second set of control modes is suppressed. The first set of control modes includes at least: manual control mode and voice control mode for controlling the motion state of the unmanned device. The second set of control modes includes at least: eye-tracking control mode for controlling the motion state of the unmanned device and head posture control mode for controlling the motion state of the gimbal of the unmanned device. If the user's intent is determined to be near-field interface gaze, the second control modality set is activated, while the first control modality set remains active.

9. The eye-tracking interactive control method according to claim 8, characterized in that, Also includes: When it is determined that the user's intention is to gaze at the near-field interface, and multiple target commands of control modes are received simultaneously, the target command corresponding to the second set of control modes is sent to the unmanned device first.

10. An eye-tracking interactive control device, characterized in that, include: The eye image acquisition module is used to acquire images of the user's eyes. A pupil size determination module is used to identify the pupil and determine the pupil size from the eye image; A gaze point determination module is used to determine the coordinates of the user's gaze point based on the eye image; The user intent determination module is used to determine that the user intent is near-field interface gaze if the gaze point coordinates are located within the near-field virtual interactive interface and the pupil size is less than a preset pupil size threshold; otherwise, it is far-field world gaze.

Images