Attention monitoring method and device based on multi-modal sensing, equipment and medium

Abstract

The invention discloses an attention monitoring method, device and equipment based on multi-modal sensing and a medium, and relates to the technical field of attention monitoring. The method comprises the following steps: collecting eye movement image data of eyeballs of a user; extracting fixation point stability features, eye movement track features, blink mode features and pupil dynamic features based on the eye movement image data; determining a concentration score of the user based on the fixation point stability feature, the blink mode feature, the pupil dynamic feature and the eye movement track feature; and if the concentration score is lower than a preset threshold value, sending reminding information to a user through the wearable device. The attention distraction behavior of the user in a continuous high attention demand scene can be accurately identified and prompted in time.

Description

Technical Field

[0001] This invention relates to the field of attention monitoring technology, and in particular to an attention monitoring method, device, equipment and medium based on multimodal sensing. Background Technology

[0002] In group teaching environments, children's lack of concentration remains a widespread and unresolved educational challenge. In traditional teaching settings, teachers face multiple students simultaneously, making it difficult to monitor each student's attention level in real time, let alone intervene promptly when attention wanders. Furthermore, parents lack objective and continuous data on their children's classroom performance after class, hindering their ability to accurately understand their learning status and thus making it difficult to help their children develop effective attention improvement mechanisms.

[0003] In recent years, with the continuous development of sensing technology and artificial intelligence algorithms, eye-tracking technology has been gradually applied to the fields of behavior analysis and state recognition.

[0004] However, most existing eye-tracking solutions still have significant limitations. For example, the invention patent with publication number CN118838498A proposes an interactive control method, device, smart wearable device, and storage medium for smart wearable devices. It mainly focuses on basic eye-tracking trajectory capture functions, such as detecting the wearer's eye interaction indicators (e.g., blink frequency, fixation time). Its recognition capabilities are still relatively simple, lacking effective judgment of complex behaviors at the cognitive level, such as attention drift and subtle distractions. Some systems can only adjust the interface display content based on eye position information, without involving in-depth attention analysis and active intervention mechanisms.

[0005] Overall, existing eye-tracking technologies primarily focus on determining the specific location of visual attention and are only used to assess user visual fatigue. They have not yet established a reliable mapping model from eye movement habits to concentration levels, and therefore cannot accurately identify and provide real-time reminders for distracted attention in high cognitive load scenarios such as classrooms. Summary of the Invention

[0006] This invention provides an attention monitoring method, device, equipment, and medium based on multimodal sensing, aiming to solve the problem of how to accurately identify and promptly remind users of their distracted behavior in scenarios with continuous high attention demands.

[0007] In a first aspect, embodiments of the present invention provide an attention monitoring method based on multimodal sensing, comprising: Collect eye movement image data of the user's eyes; Based on the eye-tracking image data, fixation stability features, eye movement trajectory features, blinking pattern features, and pupil dynamic features are extracted. The user's attention score is determined based on the fixation point stability feature, the blink pattern feature, the pupil dynamic feature, and the eye movement trajectory feature; If the focus score is lower than a preset threshold, a reminder message will be sent to the user via the wearable device.

[0008] A further technical solution is that the fixation point stability features include the duration of fixation and the spatial distribution dispersion of the fixation point, and the extraction process of the fixation point stability features includes the following steps: Identify and locate multiple fixation points from the continuous eye-tracking image data; The duration of fixation is recorded; The spatial distribution dispersion of the gaze points is calculated based on the identified multiple gaze points.

[0009] A further technical solution is that the eye movement trajectory features include the amplitude and frequency of rapid eye saccades and the fractal dimension of the eye movement path. The extraction process of the eye movement trajectory features includes the following steps: Identify rapid eye saccades between adjacent fixations; The amplitude of the rapid eye movement is calculated, and the number of rapid eye movements per unit time is counted to obtain the frequency of the rapid eye movement; The eye movement path of rapid eye saccades within a preset time period is analyzed, and the fractal dimension is calculated.

[0010] A further technical solution is that the blinking pattern features include blinking frequency and the duration of a single blink, and the extraction process of the blinking pattern features includes the following steps: Identify blinking states from the continuous eye movement image data; The number of blinks within a preset unit of time is counted to calculate the blink frequency; The duration of a single blink is calculated by measuring the time it takes for the eyes to go from closed to fully open during a single blink.

[0011] A further technical solution is that the pupil dynamic features include the fluctuation frequency and amplitude of the pupil diameter, and the extraction process of the pupil dynamic features includes the following steps: Based on the eye-tracking image data, the sequence of pupil diameter changes over time is tracked and recorded to form pupil diameter time-series data; The fluctuation frequency of the pupil diameter is extracted based on the pupil diameter time series data; Calculate the difference between the maximum and minimum pupil diameter values ​​within a predetermined time window to obtain the range of pupil diameter variation.

[0012] A further technical solution is that determining the user's attention score based on the fixation point stability feature, the blink pattern feature, the pupil dynamic feature, and the eye movement trajectory feature includes: Sub-scores are calculated for the fixation stability feature, the blink pattern feature, the pupil dynamics feature, and the eye movement trajectory feature, respectively. The attention score is obtained by weighted summing of the sub-scores of the fixation stability feature, the blink pattern feature, the pupil dynamics feature, and the eye movement trajectory feature.

[0013] A further technical solution is that determining the user's attention score based on the fixation point stability feature, the blink pattern feature, the pupil dynamic feature, and the eye movement trajectory feature includes: The fixation stability features, blink pattern features, pupil dynamic features, and eye movement trajectory features are input as input data into a pre-trained large model, and the large model outputs the user's attention score; wherein, the large model is trained using a refined dataset corresponding to the user.

[0014] Secondly, embodiments of the present invention also provide an attention monitoring device based on multimodal sensing, comprising: The acquisition unit is used to acquire eye movement image data of the user's eyes; The extraction unit is used to extract fixation stability features, eye movement trajectory features, blink pattern features, and pupil dynamic features based on the eye movement image data. The determining unit is used to determine the user's attention score based on the fixation point stability feature, the blink pattern feature, the pupil dynamic feature, and the eye movement trajectory feature; The reminder unit is used to send a reminder message to the user via the wearable device if the focus score is lower than a preset threshold.

[0015] Thirdly, embodiments of the present invention also provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method.

[0016] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the above-described method.

[0017] This invention provides an attention monitoring method, device, equipment, and medium based on multimodal sensing, comprising: acquiring eye movement image data of a user's eyeballs; extracting fixation stability features, eye movement trajectory features, blinking pattern features, and pupil dynamic features based on the eye movement image data; determining the user's attention score based on the fixation stability features, blinking pattern features, pupil dynamic features, and eye movement trajectory features; and issuing a reminder message to the user through the wearable device if the attention score is lower than a preset threshold.

[0018] This invention provides an attention monitoring method based on multimodal sensing. It collects user eye movement data non-invasively through wearable devices using methods such as infrared light sources and image sensors, and simultaneously extracts four types of features: fixation stability, blink patterns, pupil dynamics, and eye movement trajectory. These features are then fused to generate an attention score, thereby achieving objective and continuous quantification of attention status. This method significantly improves the accuracy and robustness in identifying subtle attentional distractions and provides immediate alerts via the device when attention decline is detected, forming a closed loop of perception-assessment-intervention. It is effectively applicable to scenarios requiring sustained focus, such as classrooms and remote work, providing users with reliable attention maintenance support. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating an attention monitoring method based on multimodal sensing provided in an embodiment of the present invention; Figure 2 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0023] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0024] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0025] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0026] Please see Figure 1 This invention provides an attention monitoring method based on multimodal sensing for wearable devices, comprising the following steps: S1 collects eye movement image data of the user's eyes.

[0027] In a specific implementation, the wearable device may be, for example, smart glasses, which include: Infrared light sources, such as infrared LEDs or laser diodes, are used to project light signals of a specific wavelength onto the surface of the eyeball. Image sensors, including cameras (CMOS / CCD), photodiode arrays, and MEMS micromirrors, are responsible for capturing reflected light signals.

[0028] By projecting light signals onto the user's eyes through an infrared light source integrated into the wearable device and collecting eye movement image data using an image sensor, the monitoring process can be carried out continuously in the user's natural state without the need for the user to actively cooperate or change their behavior. This greatly reduces the interference of the monitoring behavior itself on attention and provides a foundation for obtaining real and continuous eye movement data, which is a prerequisite for achieving reliable attention assessment.

[0029] S2, Based on the eye movement image data, extract fixation stability features, eye movement trajectory features, blink pattern features, and pupil dynamic features.

[0030] In practice, based on the collected eye-tracking image data, the system simultaneously extracts fixation stability features, blink pattern features, pupil dynamic features, and eye movement trajectory features. This multimodal feature extraction strategy constitutes the core advantage of this method. Each feature reflects the cognitive state from a different physiological perspective: fixation stability features characterize the ability to focus and maintain visual attention; blink pattern features reflect the inhibitory activity and fatigue state of the central nervous system; pupil dynamic features are directly related to cognitive load and the excitation level of the autonomic nervous system; and eye movement trajectory features reflect the efficiency and pattern of visual information acquisition.

[0031] By integrating the above four types of features, the system can construct a comprehensive and complementary attention evaluation model, which significantly improves the robustness and accuracy of state recognition and effectively avoids misjudgments caused by the susceptibility of a single feature to environmental interference, individual differences, or specific accidental actions.

[0032] In some preferred embodiments, the fixation stability features include the duration of fixation and the spatial distribution dispersion of fixation. The extraction process of the fixation stability features includes the following steps: identifying and locating multiple fixations from continuous eye movement image data; counting the duration of fixation; and calculating the spatial distribution dispersion of fixation based on the identified multiple fixations.

[0033] In practice, by identifying and locating multiple fixation points from continuous eye-tracking image data, the system can discretize the continuous visual perception process into analyzable event units. The duration of fixation points is crucial because individuals need time to process specific information when focused, resulting in relatively long and stable fixation durations; conversely, when attention is scattered, the visual system cannot effectively lock onto information points, leading to short and unpredictable fixation durations.

[0034] Furthermore, calculating the spatial dispersion of these gaze points (e.g., the magnitude of change over a preset time period) provides crucial information from another dimension: when highly focused, gaze points are tightly clustered in areas relevant to the target task; while when distracted, gaze points wander aimlessly, resulting in a significant increase in spatial dispersion. By combining the complementary parameters of duration (stability in the time dimension) and spatial dispersion (concentration in the spatial dimension), the system can effectively distinguish between "effective visual search" and "aimless gaze wandering," thereby more accurately quantifying the degree of focus of the user's visual attention and providing a solid and nuanced data foundation for subsequent comprehensive judgment.

[0035] Furthermore, in some preferred embodiments, the blinking pattern features include blinking frequency and duration of a single blink. The extraction process of the blinking pattern features includes the following steps: identifying blinking states from continuous eye movement image data; counting the number of blinking states within a preset unit time to calculate the blinking frequency; and calculating the duration of a single blink from when the eyes are closed to when they are fully open.

[0036] In practice, identifying blinking states from continuous eye movement image data is the first step in objectifying blinking behavior. When the process from eye closing to eye opening is completed, a blinking state is identified, and one blinking state corresponds to one blink.

[0037] Counting the number of blinks per unit time to calculate blink frequency has direct psychophysiological significance: when an individual is focused on a high cognitive load task, the brain subconsciously inhibits blinking activity to maintain the continuity of visual information input, so the blink frequency will decrease significantly; conversely, when attention is interrupted or in a relaxed or absent-minded state, the blink frequency will return to the baseline or even increase.

[0038] More precisely, the duration of a single blink from closing to opening can be calculated; this parameter reflects the "quality" of the blink. Rapid, unconscious blinks and deliberate, slow eye-closing movements may indicate different physiological states; for example, fatigue may lead to an abnormally prolonged blink duration. By simultaneously monitoring both blink frequency and single blink duration, the blink pattern indicator can be made more specific and accurate in assessing cognitive attention.

[0039] Furthermore, in some preferred embodiments, the pupil dynamic features include the fluctuation frequency and amplitude of the pupil diameter. The extraction process of the pupil dynamic features includes the following steps: based on the eye movement image data, tracking and recording the sequence of pupil diameter changes over time to form pupil diameter time-series data; extracting the fluctuation frequency of the pupil diameter based on the pupil diameter time-series data; calculating the difference between the maximum and minimum values ​​of the pupil diameter within a predetermined time window to obtain the amplitude of the pupil diameter change.

[0040] In practice, pupil diameter time-series data is generated by tracking pupil diameter, transforming transient pupil size changes into a data stream suitable for time-series analysis. Furthermore, fluctuation frequencies are extracted from this pupil diameter time-series data. Existing research indicates that these subtle, high-frequency fluctuations are closely related to neural activity in the brain responsible for attention control. During periods of focused attention, the nervous system's control over the pupil is enhanced, resulting in more stable fluctuation patterns; conversely, when attention is diverted, the fluctuation frequency and patterns exhibit characteristic changes.

[0041] Simultaneously, the difference between the maximum and minimum pupil diameter within a predetermined time window, i.e., the amplitude of change, directly reflects the degree of cognitive effort: when facing highly difficult tasks or with high concentration, the pupils will significantly dilate due to increased cognitive load (increased amplitude); while when interest is lost or attention is wandering, the pupils may be in a relatively static or abnormally constricted state. Therefore, by simultaneously analyzing the fluctuation frequency (reflecting the stability of control) and amplitude of change (reflecting the intensity of engagement), this scheme can provide an objective, continuous, and difficult-to-subjectively conceal quantitative indicator of the user's intrinsic cognitive effort at the level of the autonomic nervous system, greatly enhancing the depth and reliability of attention state assessment.

[0042] Furthermore, in some preferred embodiments, the eye movement trajectory features include the amplitude and frequency of rapid eye saccades and the fractal dimension of the eye movement path. The extraction process of the eye movement trajectory features includes the following steps: identifying rapid eye saccades between adjacent fixation points; calculating the amplitude of the rapid eye saccades and counting the number of rapid eye saccades per unit time to obtain the frequency of the rapid eye saccades; analyzing the eye movement path of the rapid eye saccades within the preset time and calculating the fractal dimension.

[0043] In practice, identifying rapid saccades between adjacent fixation points is fundamental to analyzing eye movement patterns. Calculating the amplitude and frequency of saccades provides basic kinematic indicators: when focused, saccades are typically small in amplitude and moderate in frequency, aiming to efficiently acquire specific information; while when distracted, large, irregular saccades or high-frequency, small, ineffective saccades may occur.

[0044] The next step is to calculate the fractal dimension of the eye-tracking path. Fractal dimension is a mathematical tool used to describe pattern complexity; applying it to eye-tracking path analysis quantifies the "chaos" of eye-tracking patterns. A smooth, efficient, and purposefully directed task-oriented eye-tracking path has a low fractal dimension; conversely, a chaotic, random, and aimless eye-tracking path (similar to Brownian motion) has a high fractal dimension.

[0045] In a preferred embodiment of the present invention, the fractal dimension of the eye-tracking path is obtained through box counting, specifically including the following steps: First, the coordinate sequence of the eye-tracking path is normalized on a two-dimensional plane and mapped to a unit square region to eliminate the scale effect caused by individual or acquisition device differences, ensuring the standardization and comparability of the calculation results. Next, a grid system covering the unit square is constructed, and the eye-tracking path is covered multiple times using a series of progressively decreasing square grid sizes. During each coverage, the number of grids containing at least one eye-tracking path point is counted. Then, different grid sizes and their corresponding grid numbers are recorded, forming multiple sets of data pairs. The data pairs are each double-logarithmic, and linear fitting is performed using the least squares method to obtain a fitted straight line in logarithmic coordinates. Finally, the absolute value of the slope of the fitted straight line is determined as the estimated fractal dimension of the eye-tracking path. The fractal dimension quantitatively describes the complexity and space-filling ability of eye-tracking paths: the higher the fractal dimension, the more tortuous, chaotic, and irregular the eye-tracking path, which is highly correlated with a state of distracted attention; the lower the fractal dimension, the smoother, more direct, and more efficient the eye-tracking path, which corresponds to highly focused visual search behavior.

[0046] Therefore, by combining traditional amplitude and frequency parameters with advanced path fractal dimension analysis, this scheme can go beyond simple statistics on the "quantity" of eye movements and achieve in-depth evaluation of the "quality" and "pattern" of eye movements. This is particularly effective in identifying distracted states where the gaze is moving but is actually an ineffective search, providing a unique judgment dimension that is difficult to replace with other features.

[0047] S3. Determine the user's attention score based on the fixation point stability feature, the blink pattern feature, the pupil dynamic feature, and the eye movement trajectory feature.

[0048] In practice, the user's attention score is determined based on the extracted fixation stability features, blink pattern features, pupil dynamic features, and eye movement trajectory features. The physiological features are transformed into quantitative cognitive state indicators through a specific algorithm model, realizing an objective and continuous measurement of attention level, making the originally subjective and vague "attention" state measurable and comparable.

[0049] In some preferred embodiments, the above step of "determining the user's attention score based on the fixation stability feature, the blink pattern feature, the pupil dynamic feature, and the eye movement trajectory feature" specifically includes the following steps: calculating sub-scores for the fixation stability feature, the blink pattern feature, the pupil dynamic feature, and the eye movement trajectory feature respectively; and performing a weighted summation of the sub-scores for the fixation stability feature, the blink pattern feature, the pupil dynamic feature, and the eye movement trajectory feature to obtain the attention score.

[0050] In specific implementation, sub-scores are calculated for the fixation stability feature, blink pattern feature, pupil dynamics feature, and eye movement trajectory feature, respectively. This means that the system first independently evaluates and quantifies each type of feature. This allows for separate verification and processing of the signal quality for each modality, avoiding the fatal impact of a single feature anomaly on the overall system and improving the system's robustness. Specifically, scoring rules are pre-defined for the fixation stability feature, blink pattern feature, pupil dynamics feature, and eye movement trajectory feature, and sub-scores for each feature are calculated based on the corresponding scoring rules.

[0051] Alternatively, a scoring model (e.g., a neural network model, which is not specifically limited in this invention) can be pre-trained for the fixation stability feature, blink pattern feature, pupil dynamics feature, and eye movement trajectory feature, respectively. The scoring model is trained on a large amount of training data and can accurately predict sub-scores. Further, sub-scores for the fixation stability feature, blink pattern feature, pupil dynamics feature, and eye movement trajectory feature are calculated based on the corresponding scoring model.

[0052] Furthermore, the attention score is obtained by weighted summation of the various sub-scores. The weights used in the weighted summation can be set by those skilled in the art, and are not specifically limited in this invention. Specifically, initially, this invention comprehensively allocates the weights of the four factors based on the verification experimental data. As the user gradually uses the product, some weights containing more typical features will be gradually and adaptively adjusted to make the final evaluation model more suitable for the user's actual eye habits.

[0053] The system can dynamically adjust the contribution weight of different features in the final decision based on specific application scenarios, task types, or even the physiological characteristics of individual users. For example, in reading scenarios, the weights of fixation stability and eye movement trajectory can be set higher; while in monitoring scenarios requiring alertness, the weight of pupil dynamic features may be more important. This configurable weighting mechanism gives the attention assessment model a high degree of adaptability and personalization potential, rather than a rigid, fixed algorithm.

[0054] Ultimately, the comprehensive attention score obtained through weighted summation maximizes the integration of complementarity and redundancy of multi-source information, amplifying consensus judgments while suppressing accidental noise. This results in a more accurate and stable overall attention state indicator than any single sub-score, providing a scientific and reliable basis for subsequent reminder decisions.

[0055] In some preferred embodiments, the above step "determining the user's attention score based on the fixation stability feature, the blink pattern feature, the pupil dynamic feature, and the eye movement trajectory feature" specifically includes the following steps: inputting the fixation stability feature, the blink pattern feature, the pupil dynamic feature, and the eye movement trajectory feature as input data into a pre-trained large model, and having the large model output the user's attention score; wherein, the large model is trained using a refined dataset corresponding to the user.

[0056] In practice, pre-trained large models, through learning on massive datasets, have developed a powerful ability to extract deep-seated correlations from high-dimensional, non-linear data. Compared to traditional methods that use fixed formulas or weights for weighted summation, large models can automatically discover subtle and complex interactions between different eye-movement features and their non-linear mapping relationships with attentional states. This allows them to identify highly personalized attentional distraction patterns that are difficult for traditional algorithms to capture. For example, some users may exhibit habitual rapid blinking when deep thinking, while others may show specific rhythmic fluctuations in their pupils. These unique "biometric signatures" can be effectively learned and modeled by large models.

[0057] Furthermore, the innovation of this embodiment lies in providing a technical foundation for achieving personalized and precise adaptation. The system can fine-tune and optimize the large model based on continuous usage data from different users. Its operating mechanism is as follows: the initial pre-trained large model serves as a general starting point with good prior knowledge; during usage by a specific user, the system continuously collects that user's multimodal eye-tracking data and their subjective feedback on their attention state. This data constitutes a refined dataset for that user. Through further training based on this refined dataset (i.e., transfer learning or online learning), the large model can dynamically adjust its internal parameters, thereby generating a dedicated attention assessment model specifically adapted to the user's unique eye habits and cognitive physiological characteristics. This process effectively solves the problem of assessment bias caused by individual physiological differences. For example, for a user who habitually blinks when highly focused, the general model may misjudge it as an interruption of attention, but the model after personalized learning will recognize this as a sign of their focused state, thereby automatically reducing the weight of blinking features in negative judgments or establishing a new positive association pattern for it.

[0058] Ultimately, this mechanism of continuous learning and personalized adaptation based on a large model forms a closed loop of "use-feedback-optimization," enabling the attention monitoring system to continuously evolve over time, becoming more and more accurate with each use. The overall technical effect is that the system's assessment of user attention states possesses unprecedented individual relevance, environmental adaptability, and long-term accuracy, significantly improving user experience and the effectiveness of interventions, and overcoming the inherent limitations of fixed-parameter models when facing a broad user base.

[0059] S4. If the focus score is lower than a preset threshold, a reminder message is sent to the user through the wearable device.

[0060] In practice, when the focus score falls below a preset threshold, a reminder message (such as tactile, visual, or auditory feedback) is sent to the user through wearable devices, completing a closed loop from "perception" to "intervention". This provides immediate and subtle feedback in the early stages of attention decline caused by external interference or internal factors, enabling users to adjust their attention state autonomously in the context of continuous task flow, thereby effectively preventing the continuation and deterioration of attention loss and maintaining high cognitive performance.

[0061] In some preferred embodiments, the method further includes the following steps: recording and storing data on the change of the focus score over time to generate a focus history record; sending the focus history record to a terminal device and visually displaying it in the form of a time curve through a graphical user interface.

[0062] In practice, the data on how focus scores change over time is recorded and stored. This ensures that each real-time assessment is not an isolated data point, but rather forms a long-term historical archive reflecting the patterns of user attention changes. Based on these historical focus records, in-depth longitudinal analysis can be performed, such as identifying which times of day a user's attention is most focused, or their focus response patterns for specific types of tasks.

[0063] Furthermore, by sending historical attention records to the terminal device and visualizing them as time-lapse graphs through a graphical user interface, the data becomes more intuitive and actionable. This visualization allows users (such as students), parents, or educators to clearly see the trend of attention changes over time, rather than simply receiving a binary judgment of "whether attention is currently focused." This not only provides objective evidence for communication and feedback, but more importantly, it helps identify patterns of attention fluctuations and reveals potential influencing factors. This provides data-driven decision support for developing personalized attention improvement strategies (such as adjusting learning plans and optimizing task assignments), ultimately forming a complete closed loop from real-time intervention to long-term behavioral improvement.

[0064] This invention proposes an attention monitoring method based on multimodal sensing, comprising: acquiring eye movement image data of a user's eyes; extracting fixation stability features, eye movement trajectory features, blinking pattern features, and pupil dynamic features based on the eye movement image data; determining the user's attention score based on the fixation stability features, blinking pattern features, pupil dynamic features, and eye movement trajectory features; and issuing a reminder message to the user through the wearable device if the attention score is lower than a preset threshold. This invention provides an attention monitoring method based on multimodal sensing, which non-invasively acquires user eye movement data through an infrared light source and image sensor on a wearable device, and simultaneously extracts four types of features: fixation stability, blinking pattern, pupil dynamics, and eye movement trajectory, fusing them to generate an attention score, thereby achieving objective and continuous quantification of attention status. This method can significantly improve the accuracy and robustness of identifying subtle attentional distractions, and issues an immediate reminder through the device when attention decline is detected, forming a closed loop of perception-assessment-intervention. It is effectively applicable to scenarios requiring sustained focus, such as classrooms and remote work, providing users with reliable attention maintenance support.

[0065] Corresponding to the above-described attention monitoring method based on multimodal sensing, the present invention also provides an attention monitoring device based on multimodal sensing. This attention monitoring device includes a unit for executing the aforementioned attention monitoring method based on multimodal sensing, and can be configured in a desktop computer, tablet computer, laptop computer, or other terminal. Specifically, the attention monitoring device based on multimodal sensing includes: The acquisition unit is used to acquire eye movement image data of the user's eyes; The extraction unit is used to extract fixation stability features, eye movement trajectory features, blink pattern features, and pupil dynamic features based on the eye movement image data. The determining unit is used to determine the user's attention score based on the fixation point stability feature, the blink pattern feature, the pupil dynamic feature, and the eye movement trajectory feature; The reminder unit is used to send a reminder message to the user via the wearable device if the focus score is lower than a preset threshold.

[0066] In some preferred embodiments, the fixation stability features include the duration of fixation and the spatial dispersion of the fixation points. The extraction process of the fixation stability features includes the following steps: Identify and locate multiple fixation points from the continuous eye-tracking image data; The duration of fixation is recorded; The spatial distribution dispersion of the gaze points is calculated based on the identified multiple gaze points.

[0067] In some preferred embodiments, the blinking pattern features include blinking frequency and duration of a single blink, and the extraction process of the blinking pattern features includes the following steps: Identify blinking states from the continuous eye movement image data; The number of blinks within a preset unit of time is counted to calculate the blink frequency; The duration of a single blink is calculated by measuring the time it takes for the eyes to go from closed to fully open during a single blink.

[0068] In some preferred embodiments, the pupil dynamic features include the fluctuation frequency and amplitude of the pupil diameter, and the extraction process of the pupil dynamic features includes the following steps: Based on the eye-tracking image data, the sequence of pupil diameter changes over time is tracked and recorded to form pupil diameter time-series data; The fluctuation frequency of the pupil diameter is extracted based on the pupil diameter time series data; Calculate the difference between the maximum and minimum pupil diameter values ​​within a predetermined time window to obtain the range of pupil diameter variation.

[0069] In some preferred embodiments, the eye movement trajectory features include the amplitude and frequency of rapid eye saccades and the fractal dimension of the eye movement path. The extraction process of the eye movement trajectory features includes the following steps: Identify rapid eye saccades between adjacent fixations; The amplitude of the rapid eye movement is calculated, and the number of rapid eye movements per unit time is counted to obtain the frequency of the rapid eye movement; The eye movement path of rapid eye saccades within a preset time period is analyzed, and the fractal dimension is calculated.

[0070] In some preferred embodiments, determining the user's attention score based on the fixation stability feature, the blink pattern feature, the pupil dynamics feature, and the eye movement trajectory feature includes: Sub-scores are calculated for the fixation stability feature, the blink pattern feature, the pupil dynamics feature, and the eye movement trajectory feature, respectively. The attention score is obtained by weighted summing of the sub-scores of the fixation stability feature, the blink pattern feature, the pupil dynamics feature, and the eye movement trajectory feature.

[0071] In some preferred embodiments, determining the user's attention score based on the fixation stability feature, the blink pattern feature, the pupil dynamics feature, and the eye movement trajectory feature includes: The fixation stability features, blink pattern features, pupil dynamic features, and eye movement trajectory features are input as input data into a pre-trained large model, and the large model outputs the user's attention score; wherein, the large model is trained using a refined dataset corresponding to the user.

[0072] It should be noted that those skilled in the art will clearly understand that the specific implementation process of the above-mentioned attention monitoring device based on multimodal sensing and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0073] The aforementioned attention monitoring device based on multimodal sensing can be implemented as a computer program, which can, for example... Figure 2 It runs on the computer device shown.

[0074] Please see Figure 2 , Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a terminal or a server. The terminal can be an electronic device with communication functions, such as a smartphone, tablet, laptop, desktop computer, personal digital assistant, or wearable device. The server can be a standalone server or a server cluster composed of multiple servers.

[0075] The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0076] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, it causes the processor 502 to execute an attention monitoring method based on multimodal sensing.

[0077] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0078] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute an attention monitoring method based on multimodal sensing.

[0079] The network interface 505 is used for network communication with other devices. Those skilled in the art will understand that the above structure is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. A specific computer device 500 may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.

[0080] The processor 502 is used to run a computer program 5032 stored in the memory to perform the following steps: Collect eye movement image data of the user's eyes; Based on the eye-tracking image data, fixation stability features, eye movement trajectory features, blinking pattern features, and pupil dynamic features are extracted. The user's attention score is determined based on the fixation point stability feature, the blink pattern feature, the pupil dynamic feature, and the eye movement trajectory feature; If the focus score is lower than a preset threshold, a reminder message will be sent to the user via the wearable device.

[0081] In some preferred embodiments, the fixation stability features include the duration of fixation and the spatial dispersion of the fixation points. The extraction process of the fixation stability features includes the following steps: Identify and locate multiple fixation points from the continuous eye-tracking image data; The duration of fixation is recorded; The spatial distribution dispersion of the gaze points is calculated based on the identified multiple gaze points.

[0082] In some preferred embodiments, the blinking pattern features include blinking frequency and duration of a single blink, and the extraction process of the blinking pattern features includes the following steps: Identify blinking states from the continuous eye movement image data; The number of blinks within a preset unit of time is counted to calculate the blink frequency; The duration of a single blink is calculated by measuring the time it takes for the eyes to go from closed to fully open during a single blink.

[0083] In some preferred embodiments, the pupil dynamic features include the fluctuation frequency and amplitude of the pupil diameter, and the extraction process of the pupil dynamic features includes the following steps: Based on the eye-tracking image data, the sequence of pupil diameter changes over time is tracked and recorded to form pupil diameter time-series data; The fluctuation frequency of the pupil diameter is extracted based on the pupil diameter time series data; Calculate the difference between the maximum and minimum pupil diameter values ​​within a predetermined time window to obtain the range of pupil diameter variation.

[0084] In some preferred embodiments, the eye movement trajectory features include the amplitude and frequency of rapid eye saccades and the fractal dimension of the eye movement path. The extraction process of the eye movement trajectory features includes the following steps: Identify rapid eye saccades between adjacent fixations; The amplitude of the rapid eye movement is calculated, and the number of rapid eye movements per unit time is counted to obtain the frequency of the rapid eye movement; The eye movement path of rapid eye saccades within a preset time period is analyzed, and the fractal dimension is calculated.

[0085] In some preferred embodiments, determining the user's attention score based on the fixation stability feature, the blink pattern feature, the pupil dynamics feature, and the eye movement trajectory feature includes: Sub-scores are calculated for the fixation stability feature, the blink pattern feature, the pupil dynamics feature, and the eye movement trajectory feature, respectively. The attention score is obtained by weighted summing of the sub-scores of the fixation stability feature, the blink pattern feature, the pupil dynamics feature, and the eye movement trajectory feature.

[0086] In some preferred embodiments, determining the user's attention score based on the fixation stability feature, the blink pattern feature, the pupil dynamics feature, and the eye movement trajectory feature includes: The fixation stability features, blink pattern features, pupil dynamic features, and eye movement trajectory features are input as input data into a pre-trained large model, and the large model outputs the user's attention score; wherein, the large model is trained using a refined dataset corresponding to the user.

[0087] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0088] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program may be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0089] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program. When executed by a processor, the computer program causes the processor to perform the following steps: Collect eye movement image data of the user's eyes; Based on the eye-tracking image data, fixation stability features, eye movement trajectory features, blinking pattern features, and pupil dynamic features are extracted. The user's attention score is determined based on the fixation point stability feature, the blink pattern feature, the pupil dynamic feature, and the eye movement trajectory feature; If the focus score is lower than a preset threshold, a reminder message will be sent to the user via the wearable device.

[0090] In some preferred embodiments, the fixation stability features include the duration of fixation and the spatial dispersion of the fixation points. The extraction process of the fixation stability features includes the following steps: Identify and locate multiple fixation points from the continuous eye-tracking image data; The duration of fixation is recorded; The spatial distribution dispersion of the gaze points is calculated based on the identified multiple gaze points.

[0091] In some preferred embodiments, the blinking pattern features include blinking frequency and duration of a single blink, and the extraction process of the blinking pattern features includes the following steps: Identify blinking states from the continuous eye movement image data; The number of blinks within a preset unit of time is counted to calculate the blink frequency; The duration of a single blink is calculated by measuring the time it takes for the eyes to go from closed to fully open during a single blink.

[0092] In some preferred embodiments, the pupil dynamic features include the fluctuation frequency and amplitude of the pupil diameter, and the extraction process of the pupil dynamic features includes the following steps: Based on the eye-tracking image data, the sequence of pupil diameter changes over time is tracked and recorded to form pupil diameter time-series data; The fluctuation frequency of the pupil diameter is extracted based on the pupil diameter time series data; Calculate the difference between the maximum and minimum pupil diameter values ​​within a predetermined time window to obtain the range of pupil diameter variation.

[0093] In some preferred embodiments, the eye movement trajectory features include the amplitude and frequency of rapid eye saccades and the fractal dimension of the eye movement path. The extraction process of the eye movement trajectory features includes the following steps: Identify rapid eye saccades between adjacent fixations; The amplitude of the rapid eye movement is calculated, and the number of rapid eye movements per unit time is counted to obtain the frequency of the rapid eye movement; The eye movement path of rapid eye saccades within a preset time period is analyzed, and the fractal dimension is calculated.

[0094] In some preferred embodiments, determining the user's attention score based on the fixation stability feature, the blink pattern feature, the pupil dynamics feature, and the eye movement trajectory feature includes: Sub-scores are calculated for the fixation stability feature, the blink pattern feature, the pupil dynamics feature, and the eye movement trajectory feature, respectively. The attention score is obtained by weighted summing of the sub-scores of the fixation stability feature, the blink pattern feature, the pupil dynamics feature, and the eye movement trajectory feature.

[0095] In some preferred embodiments, determining the user's attention score based on the fixation stability feature, the blink pattern feature, the pupil dynamics feature, and the eye movement trajectory feature includes: The fixation stability features, blink pattern features, pupil dynamic features, and eye movement trajectory features are input as input data into a pre-trained large model, and the large model outputs the user's attention score; wherein, the large model is trained using a refined dataset corresponding to the user.

[0096] The storage medium is a physical, non-transient storage medium, such as a USB flash drive, external hard drive, read-only memory (ROM), magnetic disk, or optical disk, or any other physical storage medium capable of storing program code. The computer-readable storage medium can be non-volatile or volatile.

[0097] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0098] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0099] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0100] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0101] As a preferred embodiment of the present invention, the smart glasses have a flexible configuration in computer device applications to adapt to different application scenarios and power consumption requirements. As an example, this solution is compatible with the following two operating states: The first scenario is local computing mode: the smart glasses themselves integrate a computing unit (such as a microprocessor, application-specific integrated circuit, etc.). In this mode, human physiological data (such as eye movement data, blink frequency, fixation point, etc.) collected by the eye-tracking sensors and other related sensors on the glasses are directly calculated and analyzed locally, and an attention state judgment result is generated accordingly. Subsequently, the smart glasses will directly activate the built-in reminder device (such as a miniature indicator light, speaker, or vibration motor) to remind the user based on the result.

[0102] The second scenario is a distributed computing mode: the smart glasses primarily function as a data acquisition terminal, while the computing unit resides in an external device connected to it, such as a smartphone, dedicated controller, computer, or other wearable device. In this mode, the smart glasses transmit the collected human physiological data to the external device via wireless or wired communication modules (such as Bluetooth or Wi-Fi). The computing unit in the external device performs complex calculations and analysis of the data, derives an attention state judgment result, and then sends this result or corresponding control commands back to the smart glasses, which ultimately execute the reminder action.

[0103] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0104] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Since these modifications and variations fall within the scope of the claims and their equivalents, this invention also intends to include these modifications and variations.

[0105] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An attention monitoring method based on multimodal sensing for wearable devices, characterized in that, include: Collect eye movement image data of the user's eyes; Based on the eye-tracking image data, fixation stability features, eye movement trajectory features, blinking pattern features, and pupil dynamic features are extracted. The user's attention score is determined based on the fixation point stability feature, the blink pattern feature, the pupil dynamic feature, and the eye movement trajectory feature; If the focus score is lower than a preset threshold, a reminder message will be sent to the user via the wearable device.

2. The attention monitoring method based on multimodal sensing according to claim 1, characterized in that, The fixation stability features include the duration of fixation and the spatial dispersion of the fixation points. The extraction process of the fixation stability features includes the following steps: Identify and locate multiple fixation points from the continuous eye-tracking image data; The duration of fixation is recorded; The spatial distribution dispersion of the gaze points is calculated based on the multiple gaze points identified.

3. The attention monitoring method based on multimodal sensing according to claim 2, characterized in that, The eye movement trajectory features include the amplitude and frequency of rapid eye saccades and the fractal dimension of the eye movement path. The extraction process of the eye movement trajectory features includes the following steps: Identify rapid eye saccades between adjacent fixations; The amplitude of the rapid eye movement is calculated, and the number of rapid eye movements per unit time is counted to obtain the frequency of the rapid eye movement; The eye movement path of rapid eye saccades within a preset time period is analyzed, and the fractal dimension is calculated.

4. The attention monitoring method based on multimodal sensing according to claim 1, characterized in that, The blinking pattern features include blinking frequency and duration of a single blink. The extraction process of the blinking pattern features includes the following steps: Identify blinking states from the continuous eye movement image data; The number of blinks within a preset unit of time is counted to calculate the blink frequency; The duration of a single blink is calculated by measuring the time it takes for the eyes to go from closed to fully open during a single blink.

5. The attention monitoring method based on multimodal sensing according to claim 1, characterized in that, The pupil dynamic features include the fluctuation frequency and amplitude of the pupil diameter. The extraction process of the pupil dynamic features includes the following steps: Based on the eye-tracking image data, the sequence of pupil diameter changes over time is tracked and recorded to form pupil diameter time-series data; The fluctuation frequency of the pupil diameter is extracted based on the pupil diameter time series data; Calculate the difference between the maximum and minimum pupil diameter within a predetermined time window to obtain the range of change in pupil diameter.

6. The attention monitoring method based on multimodal sensing according to claim 1, characterized in that, The process of determining a user's attention score based on the fixation point stability feature, the blink pattern feature, the pupil dynamic feature, and the eye movement trajectory feature includes: Sub-scores are calculated for the fixation stability feature, the blink pattern feature, the pupil dynamics feature, and the eye movement trajectory feature, respectively. The attention score is obtained by weighted summing of the sub-scores of the fixation stability feature, the blink pattern feature, the pupil dynamics feature, and the eye movement trajectory feature.

7. The attention monitoring method based on multimodal sensing according to claim 1, characterized in that, The process of determining a user's attention score based on the fixation point stability feature, the blink pattern feature, the pupil dynamic feature, and the eye movement trajectory feature includes: The fixation stability features, blink pattern features, pupil dynamic features, and eye movement trajectory features are input as input data into a pre-trained large model, and the large model outputs the user's attention score; wherein, the large model is trained using a refined dataset corresponding to the user.

8. An attention monitoring device based on multimodal sensing, characterized in that, include: The acquisition unit is used to acquire eye movement image data of the user's eyes; The extraction unit is used to extract fixation stability features, eye movement trajectory features, blink pattern features, and pupil dynamic features based on the eye movement image data. The determining unit is used to determine the user's attention score based on the fixation point stability feature, the blink pattern feature, the pupil dynamic feature, and the eye movement trajectory feature; The reminder unit is used to send a reminder message to the user via the wearable device if the focus score is lower than a preset threshold.

9. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the method as described in any one of claims 1-7.

Images