Eye movement pupil adaptive interaction method, device, equipment and storage medium

By extracting multidimensional micro-features and dynamic pupil changes from AR eye-tracking data, and adaptively adjusting the intention confidence and cognitive load state, the accuracy and naturalness of AR eye-tracking interaction are improved, solving the problems of false triggering and insufficient interaction accuracy in existing technologies.

CN123114265APending Publication Date: 2026-11-17GUDONG INTELLIGENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611483616.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-22
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing AR eye-tracking interaction technologies suffer from false triggering issues, making it difficult to accurately distinguish between intentional interaction and unintentional gaze. Furthermore, existing technologies struggle to effectively utilize micro-features of eye movement for interaction judgment, resulting in insufficient interaction accuracy and adaptability.

Method used

By acquiring user eye-tracking data and extracting multidimensional eye-tracking micro-features, combined with pupil dynamic changes, the interaction judgment threshold is adaptively adjusted using intent confidence and cognitive load status. Pupil response confirmation is achieved by using unified sub-perceptual micro-modulation below the conscious perception threshold, thus realizing implicit confirmation of interaction intent.

Benefits of technology

It improves the accuracy and adaptability of AR eye-tracking interaction, reduces the risk of accidental triggering, reduces the user's operational burden, and enhances the naturalness and adaptability of the interaction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114265A_ABST
    Figure CN123114265A_ABST
Patent Text Reader

Abstract

The application provides an eye movement pupil self-adaptive interaction method, device, equipment and storage medium. The application determines an interaction intention by extracting multi-dimensional eye movement micro-features representing user eye movement behavior and pupil dynamic change, and applies a unified sub-perception modulation below a user consciousness perception threshold to a target UI element after determining the selection intention. An implicit confirmation is realized by detecting a pupil response feature caused by the micro-modulation, without the need for the user to perform an additional explicit operation, which is conducive to reducing accidental triggering caused by unintentional fixation. Further, a cognitive load state is estimated according to user eye movement and / or pupil dynamic change, and a classification threshold of intention recognition and a confirmation threshold of target confirmation are adaptively determined according to the cognitive load state, so that the interaction judgment condition can adapt to the change of the user cognitive state, thereby improving the accuracy, naturalness and adaptive ability of AR eye movement interaction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of augmented reality human-computer interaction technology, and in particular to an eye-tracking pupil adaptive interaction method, apparatus, device, and storage medium. Background Technology

[0002] With the development of augmented reality (AR) technology, AR glasses are increasingly being applied to scenarios such as information prompts, navigation, industrial assistance, and remote collaboration. Since AR glasses typically use a head-mounted display, traditional touch, button, or handheld control methods are limited by device form factor and user scenarios. Therefore, eye-tracking-based natural interaction methods have gained attention. Existing eye-tracking interaction technologies usually obtain the user's gaze position through eye tracking and map the gaze position to interactive objects in the AR interface, thereby enabling gaze-based target selection, interface browsing, and other interactive operations.

[0003] However, existing gaze-based interaction methods are prone to false triggering. When browsing an AR interface, a user's gaze may naturally fall on an interactive object, but this gaze does not necessarily indicate an intention to perform an action. If the determination of whether a user needs to perform an action is based solely on the gaze location or gaze duration, it is difficult to accurately distinguish between intentional interaction and unintentional gaze. While using a fixed dwell time as a trigger condition can reduce false triggering to some extent, shorter dwell times are prone to misoperation, while longer dwell times increase interaction waiting time and visual fatigue. At the same time, eye movement behavior varies among different users and under different usage states, making it difficult to adapt to the dynamic changes in actual interaction processes using fixed judgment conditions.

[0004] To further improve the accuracy of eye-tracking interaction, some existing technologies utilize physiological information such as pupil changes to help determine the user's attentional state or interaction events. However, confirmation usually still requires visual changes perceptible to the user, additional explicit actions, or other auxiliary interaction methods. While these methods can increase the basis for confirmation, they may alter the visual presentation of the original AR interface or require the user to perform additional actions, thus increasing the interaction burden. Furthermore, when relying solely on pupil changes for interaction determination, the dynamic characteristics of the user's pupils may change under different cognitive states. Fixed response determination conditions are easily affected by the user's current cognitive state, thereby reducing the stability of interaction judgment.

[0005] In addition, some existing eye-tracking interaction systems mainly use relatively coarse-grained eye movement information such as gaze position and eye movement speed for interaction judgment. They have limited comprehensive utilization of eye movement micro-features such as micro-salivation, gaze stability, pupil dynamic changes and blinking, and therefore find it difficult to further utilize these micro-behavioral features to distinguish different user interaction intentions.

[0006] In summary, the problems existing in the current technology urgently need to be solved. Summary of the Invention

[0007] This invention provides an eye-tracking pupil adaptive interaction method, device, equipment, and storage medium to overcome the deficiencies in the prior art and improve the accuracy, naturalness, and adaptability of AR eye-tracking interaction.

[0008] This invention provides an eye-tracking pupil adaptive interaction method, comprising: Acquire eye movement data of the user's eyes, and extract multidimensional eye movement micro-features based on the eye movement data to characterize the user's eye movement behavior and pupil dynamic changes; Based on the multidimensional eye-tracking micro-features, the user's interaction intent and corresponding intent confidence are determined, and based on the intent confidence and a pre-determined classification threshold, it is determined whether the user has generated a selection intent. When it is determined that the selection intent has been generated, the target UI element is determined according to the user's current gaze position. A uniform sub-sensory micro-modulation below the user's conscious perception threshold is applied to the target area corresponding to the target UI element. Pupil dynamic data after the uniform sub-sensory micro-modulation is applied is obtained, and pupil response characteristics are determined based on the pupil dynamic data. The pupil response feature is matched with a predetermined confirmation threshold and a preset confirmation response feature. When the pupil response feature meets the confirmation threshold and the confirmation response feature, the target UI element is confirmed and the corresponding interactive operation of the target UI element is executed. The classification threshold and the confirmation threshold are determined based on the user's cognitive load status, which is estimated based on the user's eye movement and / or pupil dynamic changes.

[0009] According to the eye-tracking pupil adaptive interaction method provided by the present invention, the step of acquiring eye-tracking data of the user's eyes and extracting multidimensional eye-tracking micro-features based on the eye-tracking data to characterize the user's eye-tracking behavior and pupil dynamic changes specifically includes: The user's eye images are captured by an eye-tracking component installed on AR glasses, and the user's eye movement data is determined based on the eye images; Based on the eye-tracking data, microsaccade events are detected during the user's gaze, and the microsaccade rate and microsaccade direction distribution characteristics are determined based on the microsaccade events. The gaze stability characteristics are determined based on changes in the user's gaze position, and the pupil dynamic characteristics are determined based on changes in pupil diameter over time. Blinking features are determined based on changes in the user's eyelid state; The microscan video rate, the microscan gaze direction distribution features, the gaze stability features, the pupil dynamic features, and the blink features constitute the multidimensional eye movement micro-features.

[0010] According to an eye-tracking pupil-adaptive interaction method provided by the present invention, the step of acquiring user eye images through an eye-tracking component mounted on AR glasses and determining user eye movement data based on the eye images specifically includes: The AR glasses collect images of the user's eyes by using a first infrared light source, a second infrared light source, and an infrared camera. The first infrared light source and the second infrared light source illuminate the user's eyes in an alternating pulse pattern, and the infrared camera collects images of the user's eyes in the corresponding illumination sequence. Pupil features and corneal reflection features are extracted from the eye image, and eye movement data are determined based on the pupil features and corneal reflection features.

[0011] According to the eye-tracking pupil adaptive interaction method provided by the present invention, the step of determining the user's interaction intent and corresponding intent confidence based on the multi-dimensional eye-tracking micro-features, and determining whether the user has generated a selection intent based on the intent confidence and a pre-determined classification threshold, and determining the target UI element based on the user's current gaze position when the selection intent is determined, specifically includes: The multidimensional eye-tracking micro-features are input into a pre-trained intent classification model to obtain the classification results and intent confidence scores corresponding to multiple candidate interaction intents. The user's interaction intent is then determined based on the intent confidence scores corresponding to each candidate interaction intent. The intent confidence level corresponding to the interaction intent is compared with the classification threshold. When the intent confidence level reaches the classification threshold, it is determined that the user has generated a selection intent. After determining the intention to make the selection, the user's current gaze position is obtained, and the UI element corresponding to the current gaze position is determined as the target UI element based on the spatial correspondence between the current gaze position and each UI element in the AR interface.

[0012] According to the eye-tracking pupil adaptive interaction method provided by the present invention, the step of applying a uniform sub-perceptual micro-modulation below the user's conscious perception threshold to the target area corresponding to the target UI element, acquiring pupil dynamic data after the application of the uniform sub-perceptual micro-modulation, and determining pupil response features based on the pupil dynamic data specifically includes: A uniform sub-sensory micro-modulation is applied to the target area corresponding to the target UI element, so that the displayed content in the target area changes according to the same modulation rule, and the modulation amplitude of the uniform sub-sensory micro-modulation is lower than the user's conscious perception threshold. Using the application time of the unified sub-sensory micro-modulation as a time reference, the user's pupil diameter data is acquired before the application time to determine the pupil diameter reference value, and pupil dynamic data within a preset response time window is acquired after the application time. Based on the pupil diameter baseline value and the pupil dynamic data within the preset response time window, the pupil response features are extracted. The pupil response features include at least the maximum pupil diameter dilation, the initial rate of change of pupil dilation, and the time of occurrence of the peak pupil dilation.

[0013] According to an eye-tracking pupil adaptive interaction method provided by the present invention, the step of matching the pupil response features with a pre-determined confirmation threshold and preset confirmation response features specifically includes: The maximum pupil diameter dilation, the initial rate of change of pupil dilation, and the time of peak pupil dilation are determined based on the pupil response characteristics. The maximum dilation of the pupil diameter is compared with the confirmation threshold, and the initial rate of change of pupil dilation and the time of occurrence of the peak pupil dilation are matched with the confirmation response features, respectively. When the maximum dilation of the pupil diameter reaches the confirmation threshold, and the initial rate of change of pupil dilation and the time of occurrence of the peak pupil dilation both satisfy the confirmation response characteristics, the pupil response characteristics are determined to satisfy the confirmation conditions.

[0014] According to the eye-tracking pupil adaptive interaction method provided by the present invention, the classification threshold and the confirmation threshold are determined through the following steps: Multiple cognitive load indicators are extracted based on the eye movement data and / or the multidimensional eye movement micro-features, and the multiple cognitive load indicators are fused to obtain a cognitive load characterization value; The user's cognitive load status is determined based on the cognitive load characterization values; The corresponding classification threshold and confirmation threshold are determined based on the cognitive load status; The classification threshold is used to determine whether the interaction intent meets the selection intent criteria, and the confirmation threshold is used to determine whether the pupil response feature meets the confirmation criteria.

[0015] The present invention also provides an eye-tracking pupil adaptive interaction method apparatus, comprising: The data acquisition module is used to acquire eye movement data of the user's eyes, and extract multidimensional eye movement micro-features based on the eye movement data to characterize the user's eye movement behavior and pupil dynamic changes; The intent determination module is used to determine the user's interaction intent and corresponding intent confidence based on the multidimensional eye-tracking micro-features, and to determine whether the user has generated a selection intent based on the intent confidence and a pre-determined classification threshold. When it is determined that the user has generated the selection intent, the target UI element is determined according to the user's current gaze position. The pupil response module is used to apply a uniform sub-sensory micro-modulation below the user's conscious perception threshold to the target area corresponding to the target UI element, acquire pupil dynamic data after the uniform sub-sensory micro-modulation is applied, and determine pupil response characteristics based on the pupil dynamic data. An interactive operation module is used to match the pupil response features with a predetermined confirmation threshold and a preset confirmation response feature. When the pupil response features meet the confirmation threshold and the confirmation response feature, the module determines that the target UI element has been confirmed and executes the interactive operation corresponding to the target UI element. The classification threshold and the confirmation threshold are determined based on the user's cognitive load status, which is estimated based on the user's eye movement and / or pupil dynamic changes.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the eye-tracking pupil adaptive interaction method as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the eye-tracking pupil adaptive interaction method as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the eye-tracking pupil adaptive interaction method as described above.

[0019] The eye-tracking pupil adaptive interaction method, device, equipment, and storage medium provided by this invention first extract multidimensional eye-tracking micro-features from the user's eye-tracking data to characterize eye-tracking behavior and pupil dynamic changes. Based on the multidimensional eye-tracking micro-features, the user's interaction intention and corresponding intention confidence are determined. This eliminates the need to judge whether the user needs to interact based solely on a single gaze position. Instead, it uses multidimensional eye-tracking micro-features to comprehensively identify the user's interaction intention, which helps improve the ability to distinguish between intentional interaction and unintentional gaze.

[0020] After determining that the user intends to select and identifies the target UI element, this invention further applies a uniform sub-sensory micro-modulation below the user's conscious perception threshold to the target area corresponding to the target UI element, and acquires pupil dynamic data after the micro-modulation. The pupil dynamic data is used to determine pupil response characteristics, which are then matched with a confirmation threshold and confirmation response characteristics to determine whether the target UI element has been confirmed. Thus, without requiring additional explicit user action, interactive confirmation can be completed using the pupil dynamic response generated by the uniform sub-sensory micro-modulation of the target area. This reduces the user's operational burden, adds physiological response-level confirmation criteria to gaze-based target selection, and reduces false triggers caused by interactions based solely on gaze behavior.

[0021] Furthermore, this invention sets the classification threshold and confirmation threshold to be determined based on the user's cognitive load state, and estimates the cognitive load state using dynamic changes in the user's eye movements and / or pupils. This allows the classification threshold used for interaction intent recognition and the confirmation threshold used for target UI element confirmation to be adjusted according to changes in the user's cognitive state. Therefore, compared to interaction methods that use fixed judgment thresholds, this invention can adopt corresponding interaction judgment conditions for different user cognitive states, thereby improving the adaptability of interaction judgment to changes in user state.

[0022] In summary, this invention combines multidimensional eye-tracking micro-feature interaction intent recognition, pupil response implicit confirmation based on unified sub-sensory micro-modulation, and threshold adaptive adjustment based on cognitive load state. This enables the user's "whether they have an interaction intent" and "whether they confirm the execution of the interaction operation" to be judged through different levels of information, and allows the relevant judgment conditions to be adjusted according to the user's state, thereby improving the accuracy, naturalness, and adaptability of AR eye-tracking interaction. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the eye-tracking pupil adaptive interaction method provided by the present invention; Figure 2 This is a schematic diagram of the alternating pulse irradiation timing provided by the present invention; Figure 3 This is a schematic diagram of the pupil orientation response characteristic curve provided by the present invention; Figure 4This is a schematic diagram of the adaptive UI information density of AR glasses provided by the present invention. Figure 5 This is a schematic diagram of the structure of the eye-tracking pupil adaptive interaction method device provided by the present invention; Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] To address the problems in existing technologies, this invention proposes an eye-tracking pupil-adaptive interaction method to improve the accuracy, naturalness, and adaptability of AR eye-tracking interaction. The eye-tracking pupil-adaptive interaction method is described below, as follows: Figure 1 As shown, including but not limited to the following steps: Step 110: Obtain eye movement data of the user's eyes, and extract multidimensional eye movement micro-features based on the eye movement data to characterize the user's eye movement behavior and pupil dynamic changes.

[0027] In step 110, eye-tracking data of the user's eyes is continuously acquired first through an eye-tracking component installed on the AR glasses. This eye-tracking data is used to characterize the user's eye movements and pupil dynamics during AR interface interaction, and serves as input data for subsequent interaction intent recognition and cognitive load estimation.

[0028] After acquiring the eye-tracking data, it is preprocessed to reduce interference caused by device movement, image noise, and transient abnormal data during the eye-tracking signal acquisition process. The eye-tracking data is then continuously processed according to a preset data processing cycle. Based on the processed eye-tracking data, multidimensional eye-tracking micro-features reflecting user eye-tracking behavior and pupil dynamic changes are extracted, enabling these micro-features to describe the user's current visual attention and interaction state from different dimensions.

[0029] The multidimensional eye-movement micro-features include at least features characterizing the micro-movement properties of eye movements, features characterizing the stability of gaze behavior, and features characterizing pupil dynamics and eyelid activity. Unlike methods that only use gaze position as the basis for interaction, this embodiment characterizes the micro-changes during eye movements, enabling subsequent interaction intent recognition to comprehensively utilize the dynamic behavioral information exhibited by the user during gaze.

[0030] The multidimensional eye-tracking micro-features are organized according to a unified data structure and form a feature representation corresponding to the current moment or the current time period, providing input for the determination of subsequent interaction intent and intent confidence.

[0031] Step 120: Determine the user's interaction intent and corresponding intent confidence based on the multidimensional eye-tracking micro-features, and determine whether the user has generated a selection intent based on the intent confidence and a pre-determined classification threshold. When it is determined that the user has generated the selection intent, determine the target UI element based on the user's current gaze position.

[0032] In step 120, the multidimensional eye-tracking micro-features obtained in step 110 are input into a pre-established interaction intent recognition model to analyze the user's current eye-tracking behavior in order to determine the user's interaction intent and the confidence level of the intent corresponding to the interaction intent.

[0033] The interaction intent is used to characterize the interaction purpose corresponding to the user's current eye movement behavior. The interaction intent recognition model outputs the probability that the user belongs to different interaction intents based on the input multidimensional eye movement micro-features, and determines the current interaction intent and its corresponding intent confidence based on the output results. The intent confidence is used to characterize the degree of matching between the current multidimensional eye movement micro-features and the determined interaction intent.

[0034] After determining the interaction intent and intent confidence, the intent confidence is compared with a pre-determined classification threshold. When the intent confidence reaches the classification threshold, it is determined that the user has a selection intent; when the intent confidence does not reach the classification threshold, the target element selection confirmation process is not initiated, in order to avoid the user accidentally triggering the interaction operation due to a glance while browsing the AR interface normally.

[0035] After determining that the user intends to make a selection, the interface area the user is currently focusing on is determined based on the user's current gaze position. This gaze position is then mapped to the positions of predefined UI elements in the AR interface, thereby identifying the target UI element corresponding to the gaze position. This target UI element serves as the object of the subsequent implicit confirmation process.

[0036] Through the above processing, this embodiment first uses eye-tracking micro-features to make a preliminary judgment on whether the user has an interactive intention, and then determines the specific target UI element after the selection intention conditions are met. In this way, it distinguishes between "the user is looking at a certain interface element" and "the user wants to perform an operation on the interface element", providing a target object for the subsequent confirmation process.

[0037] Step 130: Apply a uniform sub-sensory micro-modulation below the user's conscious perception threshold to the target area corresponding to the target UI element, obtain pupil dynamic data after the uniform sub-sensory micro-modulation is applied, and determine pupil response characteristics based on the pupil dynamic data.

[0038] In step 130, after determining the target UI element, the processor controls the display component of the AR glasses to apply uniform sub-sensory micro-modulation to the target area corresponding to the target UI element.

[0039] The unified sub-perceptual micro-modulation refers to a subtle modulation applied to the displayed content in the target area. The modulation amplitude is below the user's conscious perception threshold, making it difficult for the user to directly perceive the modulation under normal circumstances. Simultaneously, the unified sub-perceptual micro-modulation applies to the target area according to a uniform modulation rule, rather than creating a user-recognizable selection indication by setting different display features for different display objects.

[0040] Simultaneously or subsequently, the user's pupil dynamics data is acquired via an eye-tracking component. This pupil dynamics data characterizes the dynamic changes in the user's pupils before and after the application of the unified subsensory micromodulation.

[0041] The processor uses the application time of the unified subsensory micromodulation as a time reference to compare and analyze the pupil dynamic data before and after the application, and extracts pupil response features to describe changes in pupil response. These pupil response features characterize the dynamic response of the pupil after being subjected to the unified subsensory micromodulation, including features that describe the degree of pupil dilation, the process of dilation change, and the timing of dilation.

[0042] Through the above processing, the target UI element itself does not directly require the user to perform explicit confirmation actions such as looking again, pausing, or performing gestures. Instead, it applies uniform sub-perceptual micro-modulation below the user's conscious perception threshold to the target area and detects the corresponding dynamic changes in the pupils, providing a basis for subsequent interaction confirmation using pupil responses.

[0043] Step 140: Match the pupil response feature with a predetermined confirmation threshold and a preset confirmation response feature. When the pupil response feature meets the confirmation threshold and the confirmation response feature, determine that the target UI element is confirmed and execute the interactive operation corresponding to the target UI element.

[0044] The classification threshold and the confirmation threshold are determined based on the user's cognitive load status, which is estimated based on the user's eye movement and / or pupil dynamic changes.

[0045] In step 140, the pupil response features obtained in step 130 are verified. The processor matches the pupil response features with a predetermined verification threshold and a preset verification response feature to determine whether the acquired pupil response meets the response conditions required for verification of the target UI element.

[0046] The confirmation threshold characterizes the response amplitude condition required for the pupil response to reach the confirmation level, while the confirmation response feature characterizes the dynamic pupil change characteristics corresponding to the confirmation of the target UI element. By simultaneously considering both the amplitude of the pupil response and the feature changes during the response process, misjudgments caused by relying solely on a single pupil change can be avoided.

[0047] When the pupil response characteristics meet the confirmation threshold and the confirmation response characteristics, it is determined that the user has confirmed the target UI element, and the corresponding AR interaction is executed according to the pre-associated interactive operation of the target UI element. For example, when the target UI element corresponds to the operation of opening, switching, selecting, or entering a certain function interface, the corresponding operation is executed after confirming the target UI element.

[0048] If the pupil response feature does not meet the confirmation threshold and / or the confirmation response feature, the interactive operation corresponding to the target UI element will not be executed, thereby avoiding the operation being triggered simply because the user is browsing or briefly looking at the target UI element.

[0049] During the above interaction, the classification threshold and the confirmation threshold are not fixed, but are determined based on the user's cognitive load state. The user's cognitive load state is estimated based on the user's eye movement and / or pupil dynamic changes. The processor determines the current corresponding cognitive load state based on the obtained user eye movement and / or pupil dynamic changes, and determines the classification threshold for interaction intent recognition and the confirmation threshold for target UI element confirmation based on different cognitive load states.

[0050] Therefore, as the user's cognitive state changes, the judgment conditions used for interaction intent recognition and target UI element confirmation can change accordingly, allowing the same user to use different interaction judgment sensitivities under different cognitive states. By using multidimensional eye-tracking micro-features for interaction intent recognition and pupil response features for target confirmation, and adaptively determining the judgment thresholds used for both based on cognitive load status, we can improve the adaptability of AR eye-tracking interaction to real-time changes in the user's state while reducing misoperations caused by unintentional gaze.

[0051] It should be noted that each processing step in this embodiment can be executed by the local processor of the AR glasses, or by an external computing device that is communicatively connected to the AR glasses, or by the AR glasses and the external computing device working together. Without changing the core technical concept of this invention, the above processing location and specific implementation method can be adjusted according to the device's computing power and application scenario.

[0052] As a further optional embodiment, the step of acquiring eye movement data of the user's eyes and extracting multidimensional eye movement micro-features based on the eye movement data to characterize the user's eye movement behavior and pupil dynamic changes specifically includes: The user's eye images are captured by an eye-tracking component installed on AR glasses, and the user's eye movement data is determined based on the eye images; Based on the eye-tracking data, microsaccade events are detected during the user's gaze, and the microsaccade rate and microsaccade direction distribution characteristics are determined based on the microsaccade events. The gaze stability characteristics are determined based on changes in the user's gaze position, and the pupil dynamic characteristics are determined based on changes in pupil diameter over time. Blinking features are determined based on changes in the user's eyelid state; The microscan video rate, the microscan gaze direction distribution features, the gaze stability features, the pupil dynamic features, and the blink features constitute the multidimensional eye movement micro-features.

[0053] In this embodiment, an eye-tracking component mounted on AR glasses continuously acquires images of the user's eyes and processes these images to obtain eye-tracking data reflecting eye movements and pupil changes. Based on this data, the user's eye-tracking behavior during AR interface browsing is analyzed. The continuous eye-tracking process is divided into corresponding gaze processes, and subtle changes in eye position are detected during gaze to identify micro-salivation events generated by the user in a gaze state. Based on the number of detected micro-salivation events and their direction distribution within a preset time period, the micro-salivation rate and direction distribution characteristics are determined, thereby characterizing the user's micro-eye-tracking behavior during gaze.

[0054] Furthermore, the fixation stability characteristics are determined based on the changes in the user's gaze position over time to characterize the stability of the user's visual attention area during the fixation process; at the same time, the dynamic changes of the pupil over time are analyzed based on continuously collected pupil diameter data to determine the corresponding pupil dynamic characteristics to characterize the pupil change state of the user during the current interaction process.

[0055] Furthermore, blinking events are identified based on changes in eyelid state in eye-tracking data, and blinking features are determined based on blinking events within a preset time period to characterize the user's eye activity state during the current interaction. The obtained micro-scanning video rate, micro-scanning gaze direction distribution features, gaze stability features, pupil dynamic features, and blinking features are combined to form multi-dimensional eye-tracking micro-features corresponding to the current user state.

[0056] In this embodiment, the user's eye movement behavior is not determined solely based on a single gaze position. Instead, multidimensional eye movement micro-features are constructed from multiple dimensions, such as eye movement micro-movements, gaze stability, pupil dynamics, and eyelid movements. This provides richer behavioral feature information for subsequent determination of interaction intent based on eye movement micro-features.

[0057] As a further optional embodiment, the step of acquiring the user's eye images through an eye-tracking component mounted on the AR glasses and determining the user's eye movement data based on the eye images specifically includes: The AR glasses collect images of the user's eyes by using a first infrared light source, a second infrared light source, and an infrared camera. The first infrared light source and the second infrared light source illuminate the user's eyes in an alternating pulse pattern, and the infrared camera collects images of the user's eyes in the corresponding illumination sequence. Pupil features and corneal reflection features are extracted from the eye image, and eye movement data are determined based on the pupil features and corneal reflection features.

[0058] refer to Figure 2 , Figure 2 This diagram illustrates the alternating pulse illumination sequence. In this embodiment, a first infrared light source and a second infrared light source, mounted on the AR glasses, alternately illuminate the user's eyes. An infrared camera captures images of the user's eyes under the corresponding illumination sequence, ensuring that the acquired eye images correspond to different infrared illumination conditions. Based on the eye images captured under different illumination conditions, image analysis is performed on the user's eye region to obtain pupil features and corneal reflection features that describe the eye's condition.

[0059] Specifically, the location and morphological changes of the pupil region are determined based on the eye image, and corresponding pupil features are determined based on the pupil region; simultaneously, corneal reflection features are determined based on the reflection area formed by infrared light on the user's corneal surface. By jointly processing the pupil features and the corneal reflection features, eye movement data corresponding to changes in the user's gaze are determined.

[0060] During the acquisition of eye-tracking data, alternating illumination from a first infrared light source and a second infrared light source allows the pupil region and corneal reflection region to acquire corresponding imaging features under different lighting conditions. This enables the acquisition of image information suitable for pupil detection and gaze localization. The eye-tracking data determined based on the pupil and corneal reflection features can be used to characterize changes in the user's gaze and dynamic changes in the pupil, and serves as input for subsequent multidimensional eye-tracking micro-feature extraction.

[0061] In this embodiment, the user's eye images are acquired by using different infrared illumination conditions that work alternately, and eye movement data is determined by combining pupil features and corneal reflection features. This improves the stability of eye movement data acquisition and provides a data foundation for subsequent interactive intent recognition and pupil response analysis.

[0062] As a further optional embodiment, the step of determining the user's interaction intent and corresponding intent confidence based on the multidimensional eye-tracking micro-features, and determining whether the user has generated a selection intent based on the intent confidence and a pre-determined classification threshold, and determining the target UI element based on the user's current gaze position when the selection intent is determined, specifically includes: The multidimensional eye-tracking micro-features are input into a pre-trained intent classification model to obtain the classification results and intent confidence scores corresponding to multiple candidate interaction intents. The user's interaction intent is then determined based on the intent confidence scores corresponding to each candidate interaction intent. The intent confidence level corresponding to the interaction intent is compared with the classification threshold. When the intent confidence level reaches the classification threshold, it is determined that the user has generated a selection intent. After determining the intention to make the selection, the user's current gaze position is obtained, and the UI element corresponding to the current gaze position is determined as the target UI element based on the spatial correspondence between the current gaze position and each UI element in the AR interface.

[0063] In this embodiment, the multidimensional eye-tracking micro-features obtained in step 110 are first input into a pre-trained intent classification model. The intent classification model analyzes the user's current eye-tracking behavior based on the combination relationships between different eye-tracking micro-features and outputs multiple candidate interaction intents and the intent confidence scores corresponding to each candidate interaction intent. The candidate interaction intents are used to characterize different interaction purposes that may correspond to the user's current eye-tracking behavior. The processor determines the user's current interaction intent based on the intent confidence scores corresponding to each candidate interaction intent and uses the corresponding confidence scores as the basis for subsequent intent selection.

[0064] After determining the user's interaction intent, the intent confidence level corresponding to the interaction intent is compared with a pre-determined classification threshold to determine whether the current eye movement behavior meets the criteria for a selection intent. When the intent confidence level reaches the classification threshold, it is determined that the user has generated a selection intent, thus entering the target UI element determination process; when the intent confidence level does not reach the classification threshold, the current eye movement behavior is not determined as a selection intent, to avoid the user triggering a selection operation due to brief fixation or natural eye movement during normal browsing of the AR interface.

[0065] After determining that the user intends to make a selection, the user's current gaze position is obtained, and the current gaze position is spatially mapped to UI elements in the AR interface to determine the UI element pointed to by the current gaze position. Specifically, the UI element corresponding to the gaze position can be determined based on the spatial relationship between the interface area where the current gaze position is located and the display area corresponding to each UI element, and the determined UI element is used as the target UI element for subsequent implicit confirmation processes.

[0066] In this embodiment, multidimensional eye-tracking micro-features are used to identify user interaction intentions, and the intention confidence is used to further filter the selection intentions. After determining that the user has a selection intention, the specific target UI element is determined based on the current gaze position. This combines the user's intention judgment with the target location, avoiding the direct triggering of interactive operations based solely on the user's gaze position.

[0067] As a further optional embodiment, the step of applying a uniform sub-sensory micro-modulation below the user's conscious perception threshold to the target area corresponding to the target UI element, acquiring pupil dynamic data after the application of the uniform sub-sensory micro-modulation, and determining pupil response characteristics based on the pupil dynamic data specifically includes: A uniform sub-sensory micro-modulation is applied to the target area corresponding to the target UI element, so that the displayed content in the target area changes according to the same modulation rule, and the modulation amplitude of the uniform sub-sensory micro-modulation is lower than the user's conscious perception threshold. Using the application time of the unified sub-sensory micro-modulation as a time reference, the user's pupil diameter data is acquired before the application time to determine the pupil diameter reference value, and pupil dynamic data within a preset response time window is acquired after the application time. Based on the pupil diameter baseline value and the pupil dynamic data within the preset response time window, the pupil response features are extracted. The pupil response features include at least the maximum pupil diameter dilation, the initial rate of change of pupil dilation, and the time of occurrence of the peak pupil dilation.

[0068] In this embodiment, after identifying the target UI element, to avoid directly executing interactive operations based on the user's natural gaze behavior, a uniform sub-sensory micro-modulation is first applied to the target area corresponding to the target UI element. This uniform sub-sensory micro-modulation acts on the displayed content within the target area and modulates the entire target area according to a uniform modulation rule, ensuring that the displayed content at different locations within the target area does not form differentiated visual features used to distinguish different interactive objects. Simultaneously, the modulation amplitude of the uniform sub-sensory micro-modulation is set below the user's conscious perception threshold, thereby providing stimulation conditions for subsequently obtaining the user's physiological response without requiring the user to actively perceive the micro-modulation.

[0069] The application time of the unified subsensory micromodulation is recorded and used as the time reference for subsequent pupil dynamic analysis. Before the application time, the user's pupil diameter data is acquired, and a pupil diameter reference value is determined based on the pre-application pupil diameter data to characterize the user's pupil state before the application of the unified subsensory micromodulation. Subsequently, after the application time, pupil dynamic data is continuously acquired within a preset response time window to obtain the changes in the user's pupil after the application of the unified subsensory micromodulation.

[0070] After obtaining pupil dynamic data before and after application, the pupil diameter change within a preset response time window is compared with the baseline pupil diameter value, and a time-series analysis is performed on the pupil change process to extract the pupil response features. These pupil response features comprehensively describe the amplitude, process, and timing of the pupil response change, including at least the maximum pupil diameter dilation, the initial rate of pupil dilation, and the peak pupil dilation time. The maximum dilation characterizes the maximum degree of change in the pupil relative to the baseline state before the application of the unified subsensory micromodulation; the initial rate of change characterizes the speed of change after the pupil begins to dilate; and the peak time characterizes the time when the pupil dilation response reaches its maximum.

[0071] In this embodiment, the method described above is not merely to detect whether the pupil has changed, but to establish dynamic pupil response data with a clear time reference based on unified subsensory micro-modulation. Furthermore, pupil response features are constructed from multiple dimensions, including the amplitude, trend, and temporal characteristics of the pupil response. This provides a basis for subsequent confirmation and determination of target UI elements based on the pupil response features, thereby improving the reliability of the determination when the pupil response is used as an interactive confirmation signal.

[0072] As a further optional embodiment, the step of matching the pupil response features with a predetermined confirmation threshold and a preset confirmation response feature specifically includes: The maximum pupil diameter dilation, the initial rate of change of pupil dilation, and the time of peak pupil dilation are determined based on the pupil response characteristics. The maximum dilation of the pupil diameter is compared with the confirmation threshold, and the initial rate of change of pupil dilation and the time of occurrence of the peak pupil dilation are matched with the confirmation response features, respectively. When the maximum dilation of the pupil diameter reaches the confirmation threshold, and the initial rate of change of pupil dilation and the time of occurrence of the peak pupil dilation both satisfy the confirmation response characteristics, the pupil response characteristics are determined to satisfy the confirmation conditions.

[0073] In this embodiment, reference Figure 3 , Figure 3 This is a schematic diagram of the pupil orientation response characteristic curve. After obtaining the pupil response characteristics corresponding to the target UI element, multi-dimensional matching is performed on the pupil response characteristics to determine whether the user has generated a confirmation response to the target UI element. First, based on the pupil response characteristics, the maximum dilation of the pupil diameter during the response process, the initial rate of change of pupil dilation, and the time of peak pupil dilation are determined. The maximum dilation is used to characterize the amplitude of the pupil response, the initial rate of change is used to characterize the trend of change after the pupil begins to dilate, and the time of peak pupil dilation is used to characterize the temporal characteristics of the pupil response.

[0074] Subsequently, the maximum dilation of the pupil diameter is compared with a predetermined confirmation threshold to determine whether the change in the pupil response meets the confirmation requirements of the target UI element. At the same time, the initial rate of change of the pupil dilation and the time of the peak occurrence of the pupil dilation are matched with preset confirmation response features to determine whether the change process and occurrence sequence of the pupil response are consistent with the preset confirmation response.

[0075] When the maximum dilation of the pupil diameter reaches the confirmation threshold, and both the initial rate of change of pupil dilation and the time of peak pupil dilation occurrence satisfy the confirmation response characteristics, the pupil response characteristics are determined to meet the confirmation conditions. Based on this, it is determined that the user has confirmed the target UI element, and the corresponding interactive operation is then executed. By simultaneously matching the amplitude, trend, and time characteristics of the pupil response change, confirmation judgment can be avoided based solely on a single pupil diameter change, thereby improving the reliability of the confirmation result.

[0076] If the maximum dilation of the pupil diameter does not reach the confirmation threshold, or if at least one of the initial rate of change of pupil dilation and the time of occurrence of the peak pupil dilation does not meet the confirmation response characteristics, then it is determined that the pupil response characteristics do not meet the confirmation conditions, and the interactive operation corresponding to the target UI element is not executed, thereby avoiding non-target pupil changes generated by the user during natural browsing from being misjudged as confirmation responses.

[0077] As a further optional embodiment, the classification threshold and the confirmation threshold are determined through the following steps: Multiple cognitive load indicators are extracted based on the eye movement data and / or the multidimensional eye movement micro-features, and the multiple cognitive load indicators are fused to obtain a cognitive load characterization value; The user's cognitive load status is determined based on the cognitive load characterization values; The corresponding classification threshold and confirmation threshold are determined based on the cognitive load status; The classification threshold is used to determine whether the interaction intent meets the selection intent criteria, and the confirmation threshold is used to determine whether the pupil response feature meets the confirmation criteria.

[0078] In this embodiment, during the process of interaction intent recognition and target UI element confirmation, multiple cognitive load indicators that can reflect the user's current cognitive state are determined based on the user's eye movement data and / or multidimensional eye movement micro-features, and the multiple cognitive load indicators are fused to obtain a cognitive load representation value that characterizes the user's current cognitive load level.

[0079] Specifically, each cognitive load indicator reflects the user's current cognitive state from different perspectives. The processor comprehensively processes the obtained multiple cognitive load indicators to reduce the bias caused by individual differences or the current interaction scenario, and determines the user's current cognitive load state based on the fusion result. The cognitive load state is used to characterize the user's current cognitive load level.

[0080] After determining the cognitive load state, a classification threshold and a confirmation threshold corresponding to the current cognitive load state are determined based on a pre-established correspondence between cognitive load states and interaction judgment parameters. The classification threshold is used to determine the confidence level of the interaction intent to ascertain whether the user has achieved their selection intent; the confirmation threshold is used to determine the pupil response characteristics corresponding to the target UI element to ascertain whether the pupil response meets the confirmation condition.

[0081] As the user's cognitive load changes, the corresponding classification and confirmation thresholds can be adjusted accordingly, allowing the recognition of interaction intent and confirmation of target UI elements to adopt judgment conditions that match the user's current cognitive state. For example, when the user's cognitive load changes, the strictness of the judgment of selection intent can be changed by adjusting the classification threshold, and the confirmation sensitivity of pupil response can be changed by adjusting the confirmation threshold.

[0082] By incorporating the user's current cognitive load into the interaction intent recognition and pupil implicit confirmation process, the classification threshold and confirmation threshold are no longer determined by fixed criteria, but can be adaptively determined based on the user's real-time cognitive state. This improves the adaptability of the interaction process to changes in the user's state and achieves a dynamic balance between interaction accuracy and operational sensitivity.

[0083] refer to Figure 4 , Figure 4 This is a schematic diagram illustrating the adaptive information density of AR glasses' UI. Besides adjusting the classification and confirmation thresholds, the information presentation of the AR interface can also be adaptively adjusted based on the determined cognitive load state. Specifically, the number of interactive elements simultaneously presented in the AR interface, the display status of non-core information, and the spatial spacing between different UI elements are adjusted according to different cognitive load states, thereby changing the information density of the AR interface. When the user is in a high cognitive load state, the number of simultaneously presented non-core interactive elements can be reduced, and the spacing between UI elements can be appropriately increased to reduce visual information interference; when the user is in a low cognitive load state, more interactive elements can be displayed to provide more complete interactive information. Figure × illustrates the adaptive change of AR glasses' UI information density under different cognitive load states.

[0084] Furthermore, to ensure that the interaction judgment conditions under different cognitive load states are adapted to the user's current state, a mapping relationship between cognitive load states and interaction parameters can be pre-established. The interaction parameters include at least a classification threshold for determining the selection intent and a confirmation threshold for determining whether the target UI element has been confirmed, and the corresponding interaction parameters are determined from the mapping relationship based on the user's current cognitive load state.

[0085] In one implementation, cognitive load is divided into multiple load levels, and corresponding classification and confirmation thresholds are configured for each load level. When a user's cognitive load changes, the corresponding classification and confirmation thresholds are redefined based on the changed cognitive load, allowing the user to apply interaction judgment conditions with varying degrees of stringency under different cognitive load states. Compared to interaction methods with fixed thresholds, this method can dynamically adjust the judgment conditions for selection intent and goal confirmation based on the user's current cognitive state.

[0086] Table 1 shows the mapping relationship between cognitive load state and interaction parameters, where different cognitive load states correspond to different classification thresholds and confirmation thresholds. This mapping relationship allows for adjustment of the interaction decision sensitivity based on the user's current cognitive load state without altering the interaction flow.

[0087]

[0088] Table 1 The eye-tracking pupil adaptive interaction method and apparatus provided by the present invention are described below, such as... Figure 5 As shown, the eye-tracking pupil adaptive interaction method device described below can be referred to in correspondence with the eye-tracking pupil adaptive interaction method described above.

[0089] An eye-tracking pupil adaptive interaction method device, comprising: The data acquisition module 510 is used to acquire eye movement data of the user's eyes and extract multidimensional eye movement micro-features based on the eye movement data to characterize the user's eye movement behavior and pupil dynamic changes. The intent determination module 520 is used to determine the user's interaction intent and corresponding intent confidence based on the multidimensional eye-tracking micro-features, and to determine whether the user has generated a selection intent based on the intent confidence and a pre-determined classification threshold. When it is determined that the user has generated the selection intent, the target UI element is determined according to the user's current gaze position. The pupil response module 530 is used to apply a uniform sub-perceptual micro-modulation below the user's conscious perception threshold to the target area corresponding to the target UI element, acquire pupil dynamic data after the uniform sub-perceptual micro-modulation is applied, and determine pupil response features based on the pupil dynamic data. The interactive operation module 540 is used to match the pupil response feature with a predetermined confirmation threshold and a preset confirmation response feature. When the pupil response feature meets the confirmation threshold and the confirmation response feature, the target UI element is confirmed and the interactive operation corresponding to the target UI element is executed. The classification threshold and the confirmation threshold are determined based on the user's cognitive load status, which is estimated based on the user's eye movement and / or pupil dynamic changes.

[0090] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute an eye-tracking pupil adaptive interaction method, which includes: Acquire eye movement data of the user's eyes, and extract multidimensional eye movement micro-features based on the eye movement data to characterize the user's eye movement behavior and pupil dynamic changes; Based on the multidimensional eye-tracking micro-features, the user's interaction intent and corresponding intent confidence are determined, and based on the intent confidence and a pre-determined classification threshold, it is determined whether the user has generated a selection intent. When it is determined that the selection intent has been generated, the target UI element is determined according to the user's current gaze position. A uniform sub-sensory micro-modulation below the user's conscious perception threshold is applied to the target area corresponding to the target UI element. Pupil dynamic data after the uniform sub-sensory micro-modulation is applied is obtained, and pupil response characteristics are determined based on the pupil dynamic data. The pupil response feature is matched with a predetermined confirmation threshold and a preset confirmation response feature. When the pupil response feature meets the confirmation threshold and the confirmation response feature, the target UI element is confirmed and the corresponding interactive operation of the target UI element is executed. The classification threshold and the confirmation threshold are determined based on the user's cognitive load status, which is estimated based on the user's eye movement and / or pupil dynamic changes.

[0091] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0092] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the eye-tracking pupil adaptive interaction method provided by the above methods, the method comprising: Acquire eye movement data of the user's eyes, and extract multidimensional eye movement micro-features based on the eye movement data to characterize the user's eye movement behavior and pupil dynamic changes; Based on the multidimensional eye-tracking micro-features, the user's interaction intent and corresponding intent confidence are determined, and based on the intent confidence and a pre-determined classification threshold, it is determined whether the user has generated a selection intent. When it is determined that the selection intent has been generated, the target UI element is determined according to the user's current gaze position. A uniform sub-sensory micro-modulation below the user's conscious perception threshold is applied to the target area corresponding to the target UI element. Pupil dynamic data after the uniform sub-sensory micro-modulation is applied is obtained, and pupil response characteristics are determined based on the pupil dynamic data. The pupil response feature is matched with a predetermined confirmation threshold and a preset confirmation response feature. When the pupil response feature meets the confirmation threshold and the confirmation response feature, the target UI element is confirmed and the corresponding interactive operation of the target UI element is executed. The classification threshold and the confirmation threshold are determined based on the user's cognitive load status, which is estimated based on the user's eye movement and / or pupil dynamic changes.

[0093] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the eye-tracking pupil adaptive interaction method provided by the methods described above, the method comprising: Acquire eye movement data of the user's eyes, and extract multidimensional eye movement micro-features based on the eye movement data to characterize the user's eye movement behavior and pupil dynamic changes; Based on the multidimensional eye-tracking micro-features, the user's interaction intent and corresponding intent confidence are determined, and based on the intent confidence and a pre-determined classification threshold, it is determined whether the user has generated a selection intent. When it is determined that the selection intent has been generated, the target UI element is determined according to the user's current gaze position. A uniform sub-sensory micro-modulation below the user's conscious perception threshold is applied to the target area corresponding to the target UI element. Pupil dynamic data after the uniform sub-sensory micro-modulation is applied is obtained, and pupil response characteristics are determined based on the pupil dynamic data. The pupil response feature is matched with a predetermined confirmation threshold and a preset confirmation response feature. When the pupil response feature meets the confirmation threshold and the confirmation response feature, the target UI element is confirmed and the corresponding interactive operation of the target UI element is executed. The classification threshold and the confirmation threshold are determined based on the user's cognitive load status, which is estimated based on the user's eye movement and / or pupil dynamic changes.

[0094] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0095] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An eye movement pupillary adaptive interactive method, characterized in that, include: Acquire eye movement data of the user's eyes, and extract multidimensional eye movement micro-features based on the eye movement data to characterize the user's eye movement behavior and pupil dynamic changes; Based on the multidimensional eye-tracking micro-features, the user's interaction intent and corresponding intent confidence are determined, and based on the intent confidence and a pre-determined classification threshold, it is determined whether the user has generated a selection intent. When it is determined that the selection intent has been generated, the target UI element is determined according to the user's current gaze position. A uniform sub-sensory micro-modulation below the user's conscious perception threshold is applied to the target area corresponding to the target UI element. Pupil dynamic data after the uniform sub-sensory micro-modulation is applied is obtained, and pupil response characteristics are determined based on the pupil dynamic data. The pupil response feature is matched with a predetermined confirmation threshold and a preset confirmation response feature. When the pupil response feature meets the confirmation threshold and the confirmation response feature, the target UI element is confirmed and the corresponding interactive operation of the target UI element is executed. The classification threshold and the confirmation threshold are determined based on the user's cognitive load status, which is estimated based on the user's eye movement and / or pupil dynamic changes.

2. The eye movement pupillary adaptive interactive method of claim 1, wherein, The step of acquiring eye movement data of the user's eyes and extracting multidimensional eye movement micro-features based on the eye movement data to characterize the user's eye movement behavior and pupil dynamic changes specifically includes: The user's eye images are captured by an eye-tracking component installed on AR glasses, and the user's eye movement data is determined based on the eye images; Based on the eye-tracking data, microsaccade events are detected during the user's gaze, and the microsaccade rate and microsaccade direction distribution characteristics are determined based on the microsaccade events. The gaze stability characteristics are determined based on changes in the user's gaze position, and the pupil dynamic characteristics are determined based on changes in pupil diameter over time. Blinking features are determined based on changes in the user's eyelid state; The microscan video rate, the microscan gaze direction distribution features, the gaze stability features, the pupil dynamic features, and the blink features constitute the multidimensional eye movement micro-features.

3. The eye-gaze pupillary adaptive interactive method of claim 2, wherein, The step of acquiring user eye images through an eye-tracking component mounted on AR glasses and determining user eye movement data based on the eye images specifically includes: The AR glasses collect images of the user's eyes by using a first infrared light source, a second infrared light source, and an infrared camera. The first infrared light source and the second infrared light source illuminate the user's eyes in an alternating pulse pattern, and the infrared camera collects images of the user's eyes in the corresponding illumination sequence. Pupil features and corneal reflection features are extracted from the eye image, and eye movement data are determined based on the pupil features and corneal reflection features.

4. The eye movement pupillary adaptive interactive method of claim 1, wherein, The step of determining the user's interaction intent and corresponding intent confidence based on the multidimensional eye-tracking micro-features, and determining whether the user has generated a selection intent based on the intent confidence and a pre-determined classification threshold, and determining the target UI element based on the user's current gaze position when the selection intent is determined, specifically includes: The multidimensional eye-tracking micro-features are input into a pre-trained intent classification model to obtain the classification results and intent confidence scores corresponding to multiple candidate interaction intents. The user's interaction intent is then determined based on the intent confidence scores corresponding to each candidate interaction intent. The intent confidence level corresponding to the interaction intent is compared with the classification threshold. When the intent confidence level reaches the classification threshold, it is determined that the user has generated a selection intent. After determining the intention to make the selection, the user's current gaze position is obtained, and the UI element corresponding to the current gaze position is determined as the target UI element based on the spatial correspondence between the current gaze position and each UI element in the AR interface.

5. The eye movement pupillary adaptive interactive method of claim 1, wherein, The step of applying a uniform sub-sensory micro-modulation below the user's conscious perception threshold to the target area corresponding to the target UI element, acquiring pupil dynamic data after the application of the uniform sub-sensory micro-modulation, and determining pupil response characteristics based on the pupil dynamic data specifically includes: A uniform sub-sensory micro-modulation is applied to the target area corresponding to the target UI element, so that the displayed content in the target area changes according to the same modulation rule, and the modulation amplitude of the uniform sub-sensory micro-modulation is lower than the user's conscious perception threshold. Using the application time of the unified sub-sensory micro-modulation as a time reference, the user's pupil diameter data is acquired before the application time to determine the pupil diameter reference value, and pupil dynamic data within a preset response time window is acquired after the application time. Based on the pupil diameter baseline value and the pupil dynamic data within the preset response time window, the pupil response features are extracted. The pupil response features include at least the maximum pupil diameter dilation, the initial rate of change of pupil dilation, and the time of occurrence of the peak pupil dilation.

6. The eye movement pupillary adaptive interactive method of claim 1, wherein, The step of matching the pupil response features with a predetermined confirmation threshold and a preset confirmation response feature specifically includes: The maximum pupil diameter dilation, the initial rate of change of pupil dilation, and the time of peak pupil dilation are determined based on the pupil response characteristics. The maximum dilation of the pupil diameter is compared with the confirmation threshold, and the initial rate of change of pupil dilation and the time of occurrence of the peak pupil dilation are matched with the confirmation response features, respectively. When the maximum dilation of the pupil diameter reaches the confirmation threshold, and the initial rate of change of pupil dilation and the time of occurrence of the peak pupil dilation both satisfy the confirmation response characteristics, the pupil response characteristics are determined to satisfy the confirmation conditions.

7. The eye movement pupillary adaptive interactive method of claim 1, wherein, The classification threshold and the confirmation threshold are determined through the following steps: Multiple cognitive load indicators are extracted based on the eye movement data and / or the multidimensional eye movement micro-features, and the multiple cognitive load indicators are fused to obtain a cognitive load characterization value; The user's cognitive load status is determined based on the cognitive load characterization values; The corresponding classification threshold and confirmation threshold are determined based on the cognitive load status; The classification threshold is used to determine whether the interaction intent meets the selection intent criteria, and the confirmation threshold is used to determine whether the pupil response feature meets the confirmation criteria.

8. An eye-gaze pupillary adaptive interaction method, characterized in that, include: The data acquisition module is used to acquire eye movement data of the user's eyes, and extract multidimensional eye movement micro-features based on the eye movement data to characterize the user's eye movement behavior and pupil dynamic changes; The intent determination module is used to determine the user's interaction intent and corresponding intent confidence based on the multidimensional eye-tracking micro-features, and to determine whether the user has generated a selection intent based on the intent confidence and a pre-determined classification threshold. When it is determined that the user has generated the selection intent, the target UI element is determined according to the user's current gaze position. The pupil response module is used to apply a uniform sub-sensory micro-modulation below the user's conscious perception threshold to the target area corresponding to the target UI element, acquire pupil dynamic data after the uniform sub-sensory micro-modulation is applied, and determine pupil response characteristics based on the pupil dynamic data. An interactive operation module is used to match the pupil response features with a predetermined confirmation threshold and a preset confirmation response feature. When the pupil response features meet the confirmation threshold and the confirmation response feature, the module determines that the target UI element has been confirmed and executes the interactive operation corresponding to the target UI element. The classification threshold and the confirmation threshold are determined based on the user's cognitive load status, which is estimated based on the user's eye movement and / or pupil dynamic changes.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the eye-tracking pupil adaptive interaction method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the eye-tracking pupil adaptive interaction method as described in any one of claims 1 to 7.