ADHD screening method and system based on mobile terminal eye movement tracking and multi-modal agent

Abstract

The invention discloses an ADHD screening method and system based on mobile terminal eye movement tracking and a multi-modal agent. The method comprises the following steps: (1) collecting and preprocessing mobile terminal eye movement data; (2) performing fixation point estimation based on a lightweight MGazeNet model; (3) eye movement feature extraction based on a multi-normal form interaction task; and (4) carrying out comprehensive diagnosis analysis based on machine learning and a large model agent. The invention further provides an ADHD screening system based on mobile terminal eye movement tracking and the multi-modal intelligent agent. The ADHD screening system sequentially comprises the following modules: (1) a front-end data acquisition and preprocessing module; (2) an end side eye movement tracking reasoning module; (3) an interactive task control and feature calculation module; and (4) a multi-mode intelligent diagnosis and analysis module. The ADHD screening method effectively solves the problems that traditional ADHD screening depends on expensive professional equipment and subjective scales, so that the popularity rate is low, and objectivity is poor, screening portability and diagnosis objective accuracy are improved, and the intelligent level of ADHD auxiliary diagnosis and user interaction experience are improved.

Description

Technical Field

[0001] This invention relates to the fields of human-computer interaction, eye tracking, and artificial intelligence-assisted diagnosis, specifically to an ADHD screening method and system based on mobile eye tracking and multimodal intelligent agents. Background Technology

[0002] Attention deficit hyperactivity disorder (ADHD) is a common neurodevelopmental disorder characterized primarily by inattention, hyperactivity, and impulsivity. Despite its high prevalence in both children and adults, current diagnosis and screening face numerous challenges. First, traditional ADHD diagnosis relies heavily on clinical interviews with psychiatrists and subjective scales (such as the SWAN scale), lacking objective, quantifiable, and specific physiological indicators, which easily leads to misdiagnosis or missed diagnosis. Second, the uneven distribution of medical resources, with specialized hospitals capable of diagnosing adult ADHD being extremely scarce, makes it difficult for many patients to receive timely assessment. Furthermore, existing eye-tracking studies typically rely on expensive infrared eye trackers or wearable devices, making them difficult to implement in large-scale populations.

[0003] With the development of deep learning and mobile computing technologies, gaze estimation techniques based on ordinary cameras (webcams / front-facing mobile phone cameras) have become possible. Meanwhile, large language models have shown great potential in medical question answering and report generation. However, existing solutions often lack dedicated interaction paradigms for core ADHD symptoms (such as inhibitory control deficits) and lack intelligent systems capable of integrating objective eye-tracking data with subjective scales and providing personalized feedback. Therefore, there is an urgent need for a low-cost, high-precision, portable, and intelligent ADHD screening solution. Summary of the Invention

[0004] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide an ADHD screening method and system based on mobile eye tracking and multimodal intelligent agents.

[0005] This paper proposes a technical solution that utilizes the front-facing camera of a mobile device to collect eye-tracking data, combines it with the lightweight deep learning model MGazeNet for high-precision gaze estimation, stimulates ADHD features through specific visual cognitive tasks, and finally uses machine learning and a large language model agent for comprehensive evaluation. This solution requires no additional hardware; the entire process from data collection and feature analysis to report generation can be completed using only a smartphone.

[0006] This invention provides an ADHD screening method based on mobile eye tracking and multimodal intelligent agents, comprising the following steps: (1) Mobile eye-tracking data acquisition and preprocessing; capture the user's facial video stream through the front camera of the mobile device, perform face detection and key point localization, and crop out the facial and eye area images; establish the mapping relationship between the user's gaze and the screen coordinates through the calibration process; (2) Deep learning-based gaze point estimation; The lightweight convolutional neural network model MGazeNet is used to extract features and perform regression analysis on the preprocessed image data, and output the coordinates of the user's gaze point on the screen; The model combines the linear adaptive batch normalization (LABN) module and the squeeze excitation (SE) module to adapt to the computing resources of mobile devices and improve accuracy; (3) Interactive cognitive tasks and eye movement feature extraction; guide users to complete specific visual cognitive interactive tasks, including forward saccade task (PST), reverse saccade task (AST) and memory-guided saccade task (MGST); extract eye movement feature indicators reflecting attention control, inhibitory control and working memory ability based on collected fixation point data; (4) Comprehensive diagnostic analysis based on machine learning and intelligent agents; use machine learning algorithms to classify and evaluate the extracted eye movement features and identify ADHD risk features; input the eye movement feature data into the intelligent agent driven by the large language model, combine it with the medical knowledge base to perform reasoning, and generate personalized screening reports and intervention suggestions.

[0007] Specifically, step (1) includes: (11) During system initialization, check camera permissions, ensure sufficient lighting, and have the user adjust their posture and remain stable so that their face is in the center of the viewfinder of the front-facing camera of the device; keep the distance between the eyes and the screen within the preset range (about 50 cm) to ensure the quality of eye movement data acquisition. (12) Use the front camera of the mobile device to obtain a real-time RGB video stream and perform face detection on each frame of the image; use the facial key point detection algorithm to extract the feature landmarks of the face, which cover the key positions of the chin, mouth, nose, eyes and eyebrows. (13) Before entering the task, the user needs to complete the five-point calibration mode according to the screen instructions. The specific operation is as follows: the user needs to look at the colored dots or crosshairs of five specific positions (the four corners and the center of the screen) that appear randomly or sequentially on the screen until the system prompts "calibration successful", so as to cooperate with the system to establish the mapping relationship between the gaze point and the screen coordinates. (14) During the calibration and task preparation phase, the user can check whether the gaze position captured by the system is consistent with the actual gaze position by observing the real-time gaze point feedback (GazePoint) displayed on the screen. If there is a significant deviation, the above calibration steps are repeated.

[0008] Preferably, a dynamic personalized calibration method combining IMU sensor data is designed, which can adapt to accuracy errors caused by large attitude changes.

[0009] Step (2) specifically includes: (21) The preprocessed data is constructed into four input tensors: Face Patch, Left Eye Patch, Right Eye Patch, and Face Grid Vector; the Face Grid Vector is used to represent the relative position and bounding box information of the face and eyes in the original full-frame image; (22) Construct a weight-shared Siamese CNN as an eye feature extractor to process both left and right eye image patches simultaneously; before inputting the network, perform a horizontal flip operation on the right eye image patch to eliminate the mirror difference between the left and right eyes, so that the model can learn a unified eye feature representation. (23) Use a fully connected layer to extract facial spatial feature vectors from facial image patches and facial mesh vectors; use this facial spatial feature vector as a guiding condition and input it into the Linear Adaptive Batch Normalization (LABN) layer to recalibrate the intermediate layer feature map of the eye feature extraction network, so as to introduce the constraints of head pose and spatial position on facial features. (24) In the feature fusion stage, the squeeze excitation (SE) module is introduced to adaptively weight the binocular feature maps; the SE module automatically learns and enhances the feature channels that contribute highly to the prediction of gaze direction by explicitly modeling the correlation between channels, and suppresses irrelevant features. (25) The multimodal feature vectors after LABN recalibration and SE weighted fusion are concatenated and regression prediction is performed through a fully connected layer; the model output is defined in the Unified Prediction Space, that is, the horizontal and vertical coordinate offsets (x, y) of the gaze point relative to the physical position of the device camera, in centimeters (cm), in order to solve the generalization problem caused by the differences in screen size and camera position of different devices.

[0010] Step (3) specifically includes: (31) Perform the forward saccade task (PST): The system presents a central fixation point (FP) for 1000ms, then the FP disappears and a 200ms time gap is created. After that, a target stimulus is randomly presented on the left or right side of the screen. Users are required to shift their gaze to the target location as quickly and accurately as possible to measure attentional orientation speed and visual response time. (32) Perform the AntiSaccade Task (AST): The system presents the central fixation point FP, followed by the target stimulus at the edge; the user is required to suppress spontaneous saccades toward the target when the stimulus is perceived, and instead perform saccades in the opposite direction symmetrical about the vertical meridian of the target position; this is used to assess the user's ability to suppress the response and the level of impulse control, and to record the direction error rate. (33) Perform the Memory Guided Saccade Task (MGST): The system presents the central FP for 1000ms, followed by a target stimulus flashing at the edge for 500ms and then disappearing. During this period, the user must keep his gaze on the central FP (delay period of 1000ms). After the central FP disappears, the user is asked to move his gaze to the spatial location where the target was previously displayed based on his memory. This is used to assess the user's visuospatial working memory ability. (34) Definition of eye movement events and calculation of basic indicators: Eye movement events are defined based on the collected time-series coordinate data: a sequence of fixation points that are close in space and last for more than 70ms is defined as fixation, and a rapid movement between two fixation points is defined as saccade; 20 basic eye movement features are calculated, including: saccade latency, saccade amplitude / degree, saccade duration, fixation duration, number of fixation points, number of saccades, maximum fixation / saccade time, average X / Y coordinates, and the mean and standard deviation of the above indicators; (35) The Random Forest algorithm was used to sort and filter the extracted original eye movement features by importance. Redundant features were eliminated by cross-validation to construct the optimal feature subset for ADHD classification. The core indicators that reflect attention stability (such as fixation duration SD) and inhibition control defects (such as reverse saccade error rate) were retained.

[0011] Step (4) specifically includes: (41) Based on the optimal eye movement feature subset after screening, a classification model is constructed by integrating two decision tree algorithms: random forest and extreme random tree. The weighted average of the predicted probabilities of each base classifier is calculated using a soft voting mechanism, and the probability score and risk level (low, medium, high) of the user having ADHD are output. (42) Intelligent agent Prompt construction and multimodal input: The calculated objective eye movement indicators (such as average reaction time, mean saccade angle, and fixation duration SD) and the results of the SWAN subjective scale filled in by the user are formatted to construct a prompt word containing role setting (ADHD diagnosis expert), task instructions and structured data JSON; the prompt is input into an intelligent agent driven by a large language model; (43) The agent calls the medical knowledge base plugin and workflow, and performs reasoning based on the input feature data. Specifically, it analyzes the cognitive functions in three dimensions: attention control ability: evaluated based on the reaction time and success rate of the forward saccade task; response inhibition ability: evaluated based on the direction error rate and saccade angle deviation of the reverse saccade task; spatial working memory ability: evaluated based on the fixation accuracy and delayed response of the memory-guided saccade task. (44) Store users’ eye movement data, diagnostic conclusions, recommendations and next steps in a relational database; establish users’ personal health records to support longitudinal efficacy evaluation and disease tracking by comparing historical data.

[0012] This invention also provides an attention deficit hyperactivity disorder (ADHD) screening system based on mobile eye tracking and multimodal intelligent agents, comprising the following modules: (1) Front-end data acquisition and preprocessing module, running on mobile terminal, configured to: call device camera to capture real-time video stream, use face key point detection algorithm to locate facial features; perform region of interest (ROI) extraction, crop out face image block, left eye image block and right eye image block, and calculate face grid vector to preserve spatial position information; perform image normalization processing to eliminate differences in head posture and device holding angle; (2) End-side eye-tracking inference module, integrating the lightweight deep learning model MGazeNet, configured to: receive preprocessed multi-stream data input, extract binocular features through Siamese CNN; recalibrate and fuse features using the Linear Adaptive Batch Normalization (LABN) module and the Squeeze Excitation (SE) module; perform regression calculation in a unified prediction space, and output the user's high-precision gaze coordinates (x, y) in the screen coordinate system in real time. (3) Interactive task control and feature calculation module, configured to: render and present three visual cognitive task stimuli: forward saccade (PST), reverse saccade (AST), and memory-guided saccade (MGST); execute a five-point calibration procedure to establish the mapping relationship between gaze and screen; record temporal eye movement data during the task in real time, identify fixation and saccade events through algorithms, and calculate 20 eye movement feature indicators including reaction time, saccade latency, and orientation error rate; (4) Multimodal intelligent diagnostic analysis module, deployed in the cloud or on a server, configured for: Risk assessment unit: using a soft voting ensemble learning algorithm, combined with random forest and extreme random tree classifiers, to calculate the probability and level of ADHD risk based on eye movement features; Intelligent agent reasoning unit: constructing an intelligent agent based on a large language model, integrating a medical knowledge base plugin; receiving structured eye movement feature data and user subjective scale data, and generating a natural language report containing cognitive function analysis, risk assessment and intervention suggestions through deep reasoning.

[0013] The present invention also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to implement the method of the present invention.

[0014] This invention presents an ADHD screening method based on mobile eye tracking and a multimodal intelligent agent. It utilizes the lightweight deep learning model MGazeNet to accurately capture gaze coordinates on mobile devices, and combines LABN and SE modules to adapt to edge computing resources and improve regression accuracy. Simultaneously, it extracts key eye-tracking features using interactive cognitive tasks such as forward, backward, and memory-guided learning. Finally, it integrates machine learning algorithms with a large language model-driven intelligent agent to perform comprehensive risk assessment and multimodal reasoning.

[0015] Furthermore, the developed mobile-based ADHD intelligent screening system boasts low cost, high accessibility, and a superior interactive experience, enabling objective quantification and intuitive presentation of users' cognitive functions such as attention and inhibitory control. This system provides users with a novel way to monitor their mental health and receive personalized intervention anytime, anywhere, representing an innovative exploration and practice of artificial intelligence and eye-tracking interaction technology in the auxiliary diagnosis of mental illnesses.

[0016] The working principle of this invention is: This system combines clinical neuropsychology paradigms with lightweight deep learning models and achieves automated diagnosis through a large language model agent. The system's workflow is mainly divided into three stages: 1. High-precision end-to-end eye tracking and gaze point estimation: The system first uses the front-facing camera of a mobile device (such as a smartphone) to capture the user's facial video stream in real time. In the preprocessing stage, facial landmark detection technology is used to locate and crop images of the face, left eye, and right eye, and multi-stream input data is constructed by combining facial mesh vectors. Subsequently, the data is fed into the lightweight deep learning model MGazeNet deployed on the mobile device. This model uses a Siamese convolutional neural network to extract eye features and uses a linear adaptive batch normalization module and a squeeze excitation module to recalibrate and fuse the features, effectively overcoming the limitations of mobile device computing power and the impact of user head posture changes on accuracy. Finally, the model regresses and outputs the user's high-precision gaze point coordinates (x, y) in the screen coordinate system in a unified prediction space, and further eliminates individual differences through a five-point calibration process to ensure data accuracy. 2. Cognitive Interaction Task Execution and Biomarker Extraction: Based on real-time fixation points, the system guides users to complete three specific visual cognitive interaction tasks to induce ADHD-related eye movement behavior characteristics: Forward Sagging Task (PST): measures the user's speed of attention orientation (reaction time) to sudden visual stimuli; Reverse Sagging Task (AST): requires users to suppress instinctive saccades towards a target and look in the opposite direction to assess reaction inhibition ability (one of the core deficits of ADHD); Memory-Guided Sagging Task (MGST): assesses the user's visuospatial working memory ability. The system records temporal eye movement data in real time during the tasks, automatically identifies fixation and saccade events, and extracts 20 eye movement feature indicators, including saccade latency, orientation error rate, and fixation duration stability, to construct an objective feature pool reflecting the user's cognitive function status. 3. Multimodal Intelligent Analysis and Personalized Diagnostic Report Generation: In the analysis phase, the system uses a soft-voting ensemble learning strategy, combined with classifiers such as random forests, to calculate the risk probability of eye movement features and output the ADHD risk level. Simultaneously, the system combines these objective eye-tracking indicators with subjective scale data filled in by the user, transforming them into structured multimodal cue words, which are then input into an intelligent agent driven by a large language model. The agent invokes a built-in medical knowledge base for deep reasoning, comprehensively analyzing the user's strengths and weaknesses in three dimensions: attention control, inhibitory function, and working memory. Ultimately, the agent generates a natural language report containing detailed data interpretation, risk assessment conclusions, and personalized intervention suggestions (such as specific attention training methods or medical guidance), and answers the user's questions through dialogue interaction, achieving a complete closed loop from "data perception" to "intelligent diagnosis and treatment."

[0017] The innovation of this invention is: (1) Low cost and high accessibility: This invention eliminates the hardware dependence of traditional eye-tracking research on infrared eye trackers (such as Tobii) or head-mounted devices. It innovatively utilizes the front-facing camera of smartphones, which has a very high penetration rate, as the sole data acquisition device, and solves the hardware difference problem through software algorithms. This design brings medical-grade precision ADHD screening technology from professional laboratories to homes and personal terminals, greatly reducing the screening threshold and solving the problems of uneven distribution of medical resources and difficulty in accessing medical care.

[0018] (2) Novel MGazeNet model: In response to the challenges of limited computing power and variable environment on mobile devices, a linear adaptive batch normalization module is innovatively introduced to recalibrate the eye feature map using facial pose features, which solves the error caused by head movement; combined with the squeeze excitation module, the weight of key feature channels is adaptively enhanced, which significantly improves the regression accuracy in complex backgrounds, solves the problem of slow inference and low accuracy of traditional models on mobile devices, and realizes real-time edge inference.

[0019] (3) Neuropsychology-driven multi-task fusion screening paradigm: It integrates three eye saccade tasks: forward, backward and memory-guided, to comprehensively assess the three core cognitive functions of attention, inhibitory control and working memory, which is more effective in diagnosis than single tasks.

[0020] (4) Dual-engine architecture of "eye-tracking computing + intelligent agent reasoning": This invention innovatively constructs a two-layer diagnostic architecture of "machine learning + large language model". Machine learning is responsible for accurate risk classification, while the large model intelligent agent is responsible for deep semantic understanding and humanized interaction, realizing the leap from "data computing" to "intelligent diagnosis and treatment".

[0021] The advantages of this invention are: (1) Strong objectivity and accurate data: Compared with the problem that the purely subjective scales commonly used in clinical practice (such as the SWAN scale) are easily affected by the patient's subjective feelings or parental bias, the eye movement data (such as saccade latency, orientation error rate, and fixation duration SD) provided by this invention are physiological evidence that cannot be forged. Through five-point calibration and model optimization, the system achieves a fixation point estimation accuracy with an average error as low as 1.16cm on mobile devices, and selects the most discriminative feature subset through the random forest algorithm, which significantly improves the objectivity and accuracy of the screening results.

[0022] (2) Superior User Experience and Interaction Design: Transforming tedious medical tests into simple interactive tasks such as "looking at animal head images" and "remembering locations" greatly reduces the psychological burden and testing resistance of users, making it particularly suitable for potential ADHD individuals with difficulty concentrating. The system supports background operation, capturing eye-tracking heatmaps of users while browsing other applications, achieving a natural state assessment without conscious awareness. The intelligent agent dialogue mode breaks the stereotype of traditional medical reports, enabling natural language question answering and follow-up questions, providing empathetic feedback and explanations, and enhancing user trust.

[0023] (3) Excellent real-time performance and edge performance: Thanks to the lightweight design of the MGazeNet model and the optimized deployment of the MNN framework, this system exhibits extremely high inference efficiency on mobile devices. Actual test data shows that the inference speed can reach 92-110 FPS on iOS devices and 64-82 FPS on Android devices. This high frame rate performance ensures accurate capture of rapid eye saccade events, avoids data loss due to lag, and ensures the timeliness and smoothness of screening.

[0024] (4) Full-process closed-loop health management service This invention is not limited to a single risk assessment, but provides a complete closed-loop service system. The system establishes personal health records and persistently stores historical data through a database, supporting users to track their condition and compare treatment effects over a long period of time. The intelligent agent not only provides diagnostic conclusions, but also provides targeted training suggestions (such as puzzles, Sudoku, meditation) or medical guidance (recommending specialist doctors) based on the assessment results, realizing a complete closed-loop health management from "discovering problems" to "solving problems". Attached Figure Description

[0025] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a flowchart of the process of the method of the present invention.

[0027] Figure 2 This is a schematic diagram of the network structure of the MGazeNet eye-tracking model of the present invention.

[0028] Figure 3 The flowchart illustrates the three interactive cognitive tasks (PST, AST, MGST) in this invention.

[0029] Figure 4 This is a schematic diagram of the technical architecture and module composition of the system of the present invention.

[0030] Figure 5 This is a schematic diagram of the interaction logic and workflow of the intelligent agent in this invention. Detailed Implementation

[0031] The ADHD screening method and system based on mobile eye tracking and multimodal intelligent agents of the present invention will be described in detail below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments, and should not be construed as limiting the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Example 1

[0033] See Figure 1 The present invention provides a flowchart illustrating an ADHD screening method based on mobile eye tracking and multimodal intelligent agents, and details the steps involved: (1) Mobile eye-tracking data acquisition and preprocessing; During system initialization, camera permissions are checked to ensure sufficient lighting. The user adjusts their posture and remains stable, positioning their face in the center of the front-facing camera's viewfinder. The distance between the eyes and the screen is maintained within a preset range (approximately 50 cm) to ensure the quality of eye-tracking data acquisition. The mobile device's front-facing camera is used to acquire a real-time RGB video stream, and face detection is performed on each frame. Facial landmarks are extracted using a facial keypoint detection algorithm, covering key positions of the chin, mouth, nose, eyes, and eyebrows. Before entering the task, the user must complete a five-point calibration mode as instructed on the screen. Specifically, the user must sequentially gaze at five randomly or sequentially appearing colored dots or crosshairs on the screen (the four corners and the center of the screen) until the system prompts "calibration successful," thus establishing a mapping relationship between the gaze point and screen coordinates. During the calibration and task preparation phase, the user observes the real-time gaze point feedback displayed on the screen to confirm whether the gaze position captured by the system matches the actual gaze position. If there is a significant deviation, the calibration steps are repeated.

[0034] (2) Fog point estimation based on deep learning; See Figure 2The preprocessed four-channel data (left eye, right eye, face, and Face Grid) are input into the MGazeNet model. Features are extracted from the face and eye images using a CNN encoder. To eliminate differences between the left and right eye images, the right eye image is horizontally flipped before input. During feature extraction, a linear adaptive batch normalization layer recalibrates the eye feature maps based on facial pose features, while a squeeze excitation module adaptively enhances the weights of key feature channels. Finally, the fused feature vector is passed through a fully connected layer to regress and output the user's gaze coordinates (x, y) on the screen.

[0035] (3) Interactive cognitive tasks and eye-tracking feature extraction; See Figure 3 The system guides users through three tasks: Forward Sagging Task (PST): A fixation point is displayed in the center of the screen, disappears, and then reappears at the edge; the user must quickly look at it. The saccade latency is recorded, reflecting the speed of attentional orientation. Reverse Sagging Task (AST): When the target appears, the user must look in the opposite direction. The orientation error rate is recorded, reflecting inhibitory control ability. ADHD patients typically have a higher error rate. Memory-Guided Sagging Task (MGST): The target flashes and disappears; the user must look at it from memory after a delay. Fixation accuracy is recorded, reflecting working memory. Based on temporal coordinate data, the system calculates 20 feature indicators, including average reaction time, saccade amplitude, and standard deviation of fixation duration, and uses a random forest algorithm to select the most discriminative core features.

[0036] (4) Integrated diagnostic analysis based on machine learning and intelligent agents; See Figure 5 First, an ensemble learning classifier (combining a soft voting mechanism with random forests and Extra Trees) is used to analyze the features, outputting the ADHD risk probability (low / medium / high). Then, the risk level, detailed eye-tracking indicators (such as "reverse saccade error rate 50%)," and the user's subjective questionnaire data are encapsulated into a structured prompt and input into a cloud-based AI agent. The agent accesses a medical knowledge base, analyzes the data, and concludes that "this user has weak inhibitory control and a moderate risk of ADHD," and generates a conversational report containing "recommendation for reverse saccade training" and "schedule an appointment with a specialist" to provide feedback to the user.

[0037] Example 2

[0038] See Figure 4 This invention provides an ADHD screening system based on mobile eye tracking and multimodal intelligent agents, used to implement the method of Embodiment 1. For ease of explanation, only the parts related to the embodiments of the present invention are shown. The functional modules / units in this system can be hardware modules / units or software modules / units, mainly including:

[0039] (1) Data acquisition module, responsible for controlling the front camera of the mobile device, image cropping and five-point calibration interaction; (2) Eye-tracking model inference module, which is deployed on the mobile phone using the MNN or TFLite framework, is responsible for converting the image into screen gaze coordinates in real time to ensure low latency; (3) Task control module, responsible for rendering PST, AST and MGST task animations, and recording timestamps and eye-tracking data; (4) Intelligent analysis module, deployed in the cloud. It includes a feature calculation unit, a machine learning classifier, and an LLM agent interface. It receives task data uploaded from the client side, completes the entire process from feature extraction to natural language report generation, and stores the results in the database to establish user health records.

[0040] Example 3

[0041] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the ADHD screening method based on mobile eye tracking and multimodal intelligent agents as described in Example 1.

[0042] Example 4

[0043] A computing device includes a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in Embodiment 1.

[0044] In the embodiments of this invention, the modules can be integrated into one unit, deployed separately, or further divided into multiple sub-modules. The modules can be distributed within the system of the embodiments as described, or they can be modified to reside in one or more systems different from those of the embodiments of this invention.

[0045] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, embodiments of the present invention are not limited to any particular combination of hardware and software.

[0046] The above-disclosed embodiments are merely specific examples of the present invention, but the present invention is not limited thereto. Those skilled in the art can make various modifications and variations to the present invention without departing from its spirit and scope. Obviously, all such modifications and variations should fall within the protection scope claimed by the present invention.

Claims

1. An ADHD screening method based on mobile eye-tracking and multimodal intelligent agents, characterized in that, Includes the following steps: (1) Mobile eye-tracking data acquisition and preprocessing; capture the user's facial video stream through the front camera of the mobile device, perform face detection and key point localization, and crop out the facial and eye area images; establish the mapping relationship between the user's gaze and the screen coordinates through the calibration process; (2) Deep learning-based gaze point estimation; The lightweight convolutional neural network model MGazeNet is used to extract features and perform regression analysis on the preprocessed image data, and output the coordinates of the user's gaze point on the screen; The model combines the linear adaptive batch normalization LABN module and the squeeze excitation SE module. (3) Interactive cognitive tasks and eye-tracking feature extraction; Guide users to complete specific visual cognitive interaction tasks, including forward saccade task (PST), reverse saccade task (AST), and memory-guided saccade task (MGST); extract eye movement feature indicators reflecting attention control, inhibitory control, and working memory ability based on collected fixation point data. (4) Comprehensive diagnostic analysis based on machine learning and intelligent agents; use machine learning algorithms to classify and evaluate the extracted eye movement features and identify ADHD risk features; input the eye movement feature data into an intelligent agent driven by a large language model, combine it with a medical knowledge base for reasoning, and generate personalized screening reports and intervention suggestions.

2. The ADHD screening method based on mobile eye tracking and multimodal intelligent agents according to claim 1, characterized in that, Step (1) specifically includes: (11) During system initialization, check camera permissions, ensure sufficient lighting, and allow the user to adjust their posture and remain stable so that their face is in the center of the viewfinder of the device's front camera. Keep the distance between the eyes and the screen within the preset range to ensure the quality of eye movement data acquisition. (12) Use the front camera of the mobile device to obtain a real-time RGB video stream and perform face detection on each frame of the image; use the facial key point detection algorithm to extract the feature landmarks of the face, which cover the key positions of the chin, mouth, nose, eyes and eyebrows. (13) Before entering the task, the user needs to complete the five-point calibration mode according to the screen instructions. The specific operation is as follows: the user needs to look at the five specific positions that appear randomly or sequentially on the screen, including the four corners and the center point of the screen, with colored dots or crosshairs, until the system prompts "calibration successful", so as to cooperate with the system to establish the mapping relationship between the gaze point and the screen coordinates. (14) During the calibration and task preparation phase, the user can check whether the gaze position captured by the system is consistent with the actual gaze position by observing the real-time gaze point feedback displayed on the screen. If there is a significant deviation, the above calibration steps are repeated.

3. The ADHD screening method based on mobile eye tracking and multimodal intelligent agents as described in claim 1, characterized in that, Step (2) specifically includes: (21) The preprocessed data is constructed into a four-way input tensor, namely: face image block, left eye image block, right eye image block and face mesh vector; the face mesh vector is used to represent the relative position and bounding box information of the face and eyes in the original full frame image; (22) Construct a weight-shared twin convolutional neural network as an eye feature extractor to process left and right eye image patches simultaneously; before inputting the network, perform a horizontal flip operation on the right eye image patch to eliminate the mirror difference between the left and right eyes, so that the model can learn a unified eye feature representation. (23) Use a fully connected layer to extract facial spatial feature vectors from facial image patches and facial mesh vectors; use the facial spatial feature vector as a guiding condition and input it into the linear adaptive batch normalized LABN layer to recalibrate the intermediate layer feature map of the eye feature extraction network, so as to introduce the constraints of head pose and spatial position on facial features. (24) In the feature fusion stage, the squeeze excitation SE module is introduced to adaptively weight the binocular feature maps; the SE module automatically learns and enhances the feature channels that contribute highly to the prediction of gaze direction by explicitly modeling the correlation between channels, and suppresses irrelevant features. (25) The multimodal feature vectors after LABN recalibration and SE weighted fusion are concatenated and regression prediction is performed through a fully connected layer; the model output is defined in a unified prediction space, that is, the horizontal and vertical coordinate offsets (x, y) of the gaze point relative to the physical position of the device camera are output, which solves the generalization problem caused by the differences in screen size and camera position of different devices.

4. The ADHD screening method based on mobile eye tracking and multimodal intelligent agents according to claim 1, characterized in that, Step (3) specifically includes: (31) Perform forward eye saccade task (PST): The system presents a central fixation point (FP) for 1000ms, then the FP disappears and a 200ms time gap is created. After that, a target stimulus is randomly presented on the left or right side of the screen. The user is required to shift their gaze to the target location as quickly and accurately as possible to measure attentional orientation speed and visual response time. (32) Perform the reverse saccade task AST: The system presents the central fixation point FP, followed by the target stimulus at the edge; the user is required to suppress spontaneous saccades toward the target when the stimulus is perceived, and instead perform saccades in the opposite direction symmetrical about the vertical meridian to the target position; the user's reaction inhibition ability and impulse control level are assessed in this way, and the direction error rate is recorded. (33) Perform the memory-guided saccade task MGST: The system presents the central fixation point FP for 1000ms, followed by a target stimulus flashing at the edge for 500ms and then disappearing. During this period, the user must keep their gaze on the central FP, with a delay of 1000ms. After the central FP disappears, the user is required to move their gaze to the spatial location where the target previously appeared based on memory. This is used to assess the user's visuospatial working memory ability. (34) Definition of eye movement events and calculation of basic indicators: eye movement events are defined based on the collected time-series coordinate data: a sequence of fixation points that are close in space and last for more than 70ms is defined as fixation, and rapid movement between two fixation points is defined as saccade; various basic eye movement features are calculated, including: saccade latency, saccade amplitude, saccade duration, fixation duration, number of fixation points, number of saccades, maximum fixation / saccade time, average X / Y coordinates, and the mean and standard deviation of the above indicators; (35) The Random Forest algorithm was used to sort and filter the extracted original eye movement features by importance. Redundant features were eliminated by cross-validation to construct the optimal feature subset for ADHD classification, and the core indicators reflecting attention stability and inhibition control defects were retained.

5. The ADHD screening method based on mobile eye tracking and multimodal intelligent agents according to claim 1, characterized in that, Step (4) specifically includes: (41) Based on the optimal eye movement feature subset after screening, a classification model is constructed by integrating two decision tree algorithms: random forest and extreme random tree; a soft voting mechanism is used to calculate the weighted average of the predicted probabilities of each base classifier, and output the probability score and risk level of the user having ADHD. (42) Intelligent agent Prompt construction and multimodal input: The calculated objective eye-tracking indicators and the results of the SWAN subjective scale filled in by the user are formatted and processed to construct prompt words containing role settings, task instructions and structured data JSON; the prompt is input into an intelligent agent driven by a large language model; (43) The agent calls the medical knowledge base plugin and workflow, and performs reasoning based on the input feature data. Specifically, it analyzes the cognitive functions in three dimensions: attention control ability: evaluated based on the reaction time and success rate of the forward saccade task; response inhibition ability: evaluated based on the direction error rate and saccade angle deviation of the reverse saccade task; spatial working memory ability: evaluated based on the fixation accuracy and delayed response of the memory-guided saccade task. (44) Store users’ eye movement data, diagnostic conclusions, recommendations and next steps in a relational database; establish users’ personal health records to support longitudinal efficacy evaluation and disease tracking by comparing historical data.

6. An ADHD screening system based on mobile eye-tracking and multimodal intelligent agents, characterized in that... include: The front-end data acquisition and preprocessing module runs on a mobile terminal and is configured to: call the device's camera to capture real-time video streams and use facial landmark detection algorithms to locate facial features; The system extracts regions of interest (ROIs), crops facial, left eye, and right eye image patches, and calculates facial mesh vectors to preserve spatial location information. It also performs image normalization to eliminate differences in head pose and device grip angle. The edge-side eye-tracking inference module integrates the lightweight deep learning model MGazeNet and is configured to: receive preprocessed multi-stream data input and extract binocular features through a Siamese convolutional neural network; The features are recalibrated and fused using the linear adaptive batch normalization (LABN) module and the squeeze excitation (SE) module; regression calculation is performed in a unified prediction space, and the high-precision gaze coordinates (x, y) of the user in the screen coordinate system are output in real time. The interactive task control and feature calculation module is configured to: render and present three visual cognitive task stimuli: forward saccade (PST), reverse saccade (AST), and memory-guided saccade (MGST); execute a five-point calibration procedure to establish a mapping relationship between gaze and screen; record temporal eye movement data in real time during the task, identify fixation and saccade events through algorithms, and calculate various eye movement feature indicators, including reaction time, saccade latency, and orientation error rate. The multimodal intelligent diagnostic analysis module, deployed in the cloud or on a server, is configured for: a risk assessment unit that uses a soft voting ensemble learning algorithm combined with random forest and extreme random tree classifiers to calculate the probability and level of ADHD risk based on eye movement features; and an intelligent agent reasoning unit that constructs an intelligent agent based on a large language model and integrates a medical knowledge base plugin; receiving structured eye movement feature data and user subjective scale data, and generating a natural language report containing cognitive function analysis, risk assessment, and intervention suggestions through deep reasoning.

7. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the ADHD screening method based on mobile eye tracking and multimodal intelligent agents as described in any one of claims 1-5.

8. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and when the processor executes the executable code, it implements the method described in any one of claims 1-5.

Images