An adaptive reading efficiency detection system and method based on multi-modal fusion

Abstract

The present application relates to the technical field of computer vision and cognitive rehabilitation, and particularly relates to a self-adaptive reading efficiency detection and training system based on multi-modal fusion, comprising: a multi-modal data acquisition module, which is used for acquiring eye movement coordinates (x, y, t), pupil diameter and head posture data of a user; a space-time trajectory reconstruction engine, which comprises an improved I-DT algorithm module based on signal-to-noise ratio dynamic weighting and Kalman filter prediction compensation, and is used for extracting fixation points from low-frequency sampling data and retaining micro-saccades; and an elastic grid mapping module, which is used for generating a Gaussian ridge line model according to text topological features of a reading interface.The present application reduces dependence on special hardware, is conducive to cost control, and can realize fixation trajectory restoration precision close to that of a scientific research level device (> 120Hz) on a 30Hz-60Hz consumer-level camera or a tablet computer front lens through the improved I-DT algorithm, so that professional reading disorder screening and cognitive training are suitable for deployment and use in primary medical institutions or home environments.

Description

Technical Field

[0001] This invention relates to the field of computer vision and cognitive rehabilitation technology, and in particular to an adaptive reading performance testing and training system and method based on multimodal fusion. Background Technology

[0002] Currently, eye-tracking-based reading assessment technologies are increasingly used in clinical diagnosis (such as dyslexia screening) and educational training, but existing technologies mainly suffer from the following bottlenecks: Low-frequency data is noisy and difficult to reconstruct: High-precision analysis relies on expensive research-grade eye trackers (>120Hz). Popular devices (such as consumer-grade eye trackers or cameras with frequencies of 30Hz-60Hz) have sparse sampling, resulting in a large amount of micro-saccade noise and signal loss. Traditional gaze point extraction algorithms are prone to misidentifying noise as gaze or causing gaze events to break due to insufficient sampling.

[0003] Vertical drift leads to misjudgment of "look-back": Due to head micro-movements and nonlinear distortion, eye-tracking data often exhibits wavy drift or fan-shaped distortion on the Y-axis. Existing line alignment algorithms cannot handle cases where the head is tilted or the text lines are not horizontal, causing inline look-back to be misjudged as line breaks, which seriously affects the accuracy of the evaluation.

[0004] Lack of semantic segmentation based on cognitive state: Traditional techniques only count the number of reflexes and cannot distinguish between strategic reflexes (benign confirmation) and obstructive reflexes (difficulty in understanding). Lack of in-depth analysis combining pupillary physiological data (cognitive load) leads to uniform training programs and fails to achieve individualized instruction.

[0005] Among them, Chinese patent application CN121433545A discloses an artificial intelligence-based reading recommendation method. This method collects eye movement data in real time through eye-tracking technology, predicts reading ability based on this data, and pushes simplified content with matching difficulty to reduce reading difficulty. Although this solution involves the application of eye movement data in reading intervention, it mainly adjusts the difficulty of text content by predicting reading ability. It does not perform semantic classification of retrospective behavior (i.e., it does not distinguish between strategic retrospective and obstacle retrospective), nor does it introduce pupil physiological signals to quantify cognitive load status in real time.

[0006] Furthermore, Chinese patent CN120859489A discloses a pupil-based method for detecting cognitive abnormalities. This method obtains the detection results by collecting eye movement flow data and utilizing a pre-trained discriminant model. While this approach establishes a correlation between pupil information and cognitive state, it focuses on the discriminative diagnosis of cognitive abnormalities. Its detection task type is a standardized cognitive screening task, such as specific target tracking, rather than an adaptive reading training scenario, and it does not involve dynamic correction or closed-loop intervention mechanisms for reading trajectories. Therefore, existing technologies have not yet solved the problem of synergistic integration between trajectory reconstruction and adaptive intervention of cognitive states under low-frequency sampling. Thus, an adaptive reading performance testing and training system and method based on multimodal fusion is needed to address these issues. Summary of the Invention

[0007] The purpose of this invention is to solve the problems mentioned in the background section.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: An adaptive reading performance testing and training system based on multimodal fusion includes: Multimodal data acquisition module: used to acquire the user's eye movement coordinates (x, y, t), pupil diameter, and head posture data; Spatiotemporal trajectory reconstruction engine: including an improved I-DT algorithm module based on dynamic weighting of signal-to-noise ratio and Kalman filter prediction compensation, used to extract gaze points from low-frequency sampled data and preserve micro-saccades; Elastic Mesh Mapping Module: Used to generate Gaussian ridge models based on the text topology features of the reading interface, and correct vertical drift and scaling errors in eye-tracking trajectories by minimizing the cross-entropy between the gaze energy field and the text mask; Cognitive State Decoder: Used to calculate the multimodal confusion index PI based on fixation duration, pupil diameter change rate, and return rate; Adaptive Intervention Controller: Used to perform one or more closed-loop intervention operations based on the comparison results of PI value and preset threshold, including dynamic difficulty adjustment, visual occlusion, forced forward induction, rhythmic caching, or line break reinforcement training.

[0009] An adaptive reading performance training method based on multimodal fusion includes the following steps: Step 1: Collect low-sampling-rate eye movement coordinates, pupil diameter, and head posture data during the user's reading process. The "low sampling rate" in this invention refers to 30Hz to 60Hz, which corresponds to the typical specifications of consumer-grade cameras or glasses-type eye-tracking modules. Step 2: Using the improved I-DT algorithm, the spatiotemporal threshold is adaptively adjusted based on the real-time signal-to-noise ratio and velocity weighting, and Kalman filtering is used to predict and remove device noise, retain micro-saccades, and extract the fixation point sequence; Step 3: Based on the Gaussian ridge model of the text line in the reading interface, the vertical drift and scaling errors in the eye-tracking trajectory are corrected by elastic mesh mapping to achieve semantic alignment between the gaze point and the text line; Step 4: Calculate the multimodal confusion index ; Step 5: Based on the relationship between the PI value and the preset threshold, trigger the adaptive closed-loop intervention task.

[0010] Preferably, the improved I-DT algorithm includes: dynamically generating a spatial dispersion threshold based on the signal-to-noise ratio (SNR) during the initial calibration phase. During the scanning phase, the time window is automatically tightened, and during the low-speed sliding phase, the spatial threshold is relaxed. The position of the next sampling point is predicted by a Kalman filter. When the deviation between the measured position and the predicted position exceeds the biological motion law and there is no sudden change in the pupil diameter, it is judged as equipment noise and discarded.

[0011] Preferably, the pupil diameter is obtained in the following way: using the iris diameter as a reference, and employing the formula... The physical diameter is converted; and adaptive binarization and morphological closing operations are performed on the eye image to eliminate corneal reflective bright spots.

[0012] Preferably, the elastic mesh mapping includes: generating a Gaussian ridge model for each line of text. ,in The center position of the line is σ, which is set based on the font size and line spacing. The gaze point sequence is reconstructed into a continuous energy field E(x, y), and the translation parameter Δy and scaling parameter s are optimized using gradient descent to minimize the cross-entropy term. .

[0013] Preferably, the adaptive intervention controller includes: when PI < When PI is determined to be strategic review, no intervention is triggered; when PI > If the system determines that the cognitive load is overloaded, it will perform at least one of the following operations: dynamically enlarge the font of the current line, reduce the text flow speed, reduce the background contrast, apply Gaussian blur to the read area to the left of the gaze point, generate a high-frequency flashing induction point at the ideal gaze target position, perform word-by-word fading at the target speed and dynamically adjust the fading frequency according to the pupil dilation rate, or pre-display the beginning of the next line at the end of the line and issue multimodal feedback when the landing point deviation exceeds the threshold.

[0014] Preferably, the frequency adjustment method for word-by-word masking is: real-time monitoring of pupil diameter. ,when When the percentage is greater than 15%, the delay time per word will be automatically increased, thus reducing the speed of word removal.

[0015] Preferably, the line break enhancement training includes: when the gaze point is close to the remaining preset number of characters at the end of the line, a high-contrast marker is displayed at the logical center position at the beginning of the next line; the spatial distance deviation between the line break scanning landing point and the center point of the beginning of the line is detected, and if it exceeds a preset threshold, an audio prompt is issued and an ideal diagonal path auxiliary line from the end of the previous line to the beginning of the current line is superimposed and displayed.

[0016] Preferably, in the multimodal confusion index PI The return rate or spatial span characteristics are determined based on gaze path directionality analysis; α, β, and γ are preset weighting coefficients, and satisfy α+β+γ=1.

[0017] The present invention has at least the following beneficial effects: 1. This invention reduces reliance on dedicated hardware, which helps control implementation costs: Through the improved I-DT algorithm, it can achieve gaze trajectory restoration accuracy close to that of scientific research equipment (>120Hz) on consumer-grade cameras or tablet front-facing lenses at 30Hz~60Hz, making professional reading disorder screening and cognitive training suitable for deployment in primary healthcare institutions or home environments.

[0018] 2. Maintains high line alignment accuracy even in uncontrolled scenarios such as head movement and lighting changes: Employs an elastic mesh mapping algorithm based on text topology constraints to automatically correct gaze point shifts caused by user head tilt, body forward and backward movement, and device nonlinear drift, reducing the misjudgment rate of line alignment in non-laboratory environments.

[0019] 3. This invention achieves in-depth quantification of cognitive states and constructs a multimodal confusion index (PI) that integrates fixation duration, pupil dilation rate, and return rate, which can distinguish between strategic return and obstructive return.

[0020] 4. This invention establishes an adaptive closed-loop training system: Based on a real-time intervention mechanism using eye-tracking evoked feedback, visual occlusion, induction points, and dynamic fading can dynamically adjust the training difficulty according to the user's real-time pupillary dilation rate and PI index, ensuring that the training difficulty is always within the user's "zone of proximal development." Compared to fixed reading training programs without adaptive intervention, this system, by adjusting task difficulty in real time and providing immediate corrective feedback, can reduce ineffective training time, thereby theoretically shortening the rehabilitation period. The specific reduction depends on the user's initial ability level and cooperation; quantitative data can be obtained through comparative experiments in practical applications. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the overall system flow of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0024] Reference Figure 1 First, the multimodal data acquisition module acquires the user's raw eye movement coordinates, pupil diameter, and head posture data at a sampling rate of 30-60Hz. Then, the data is sent to the spatiotemporal trajectory reconstruction engine, where Kalman filtering is used for noise reduction and an improved I-DT algorithm is used to extract the gaze point. Next, the elastic grid mapping module performs vertical drift correction and scaling alignment on the gaze point based on the text line structure of the current reading interface. The cognitive state decoder calculates the multimodal confusion index PI based on this. Finally, the adaptive intervention controller triggers intervention operations such as eye occlusion, speed adjustment, or line break guidance based on whether the PI value exceeds the threshold, and feeds back the eye movement data after intervention to the acquisition module, forming a closed loop.

[0025] The system of this invention includes: a multimodal data acquisition module (acquiring eye-tracking coordinates (x, y, t), pupil diameter, and head posture), a spatiotemporal trajectory reconstruction engine, a cognitive state decoder, and an adaptive intervention controller. The core method includes the following steps: Step 1: Improved I-DT algorithm – Dynamic window discreteness extraction; During the initial calibration phase, an initial spatial dispersion threshold D_base is generated based on the dynamic fluctuations of the signal-to-noise ratio (SNR). Introducing speed component weighting: automatically tightening the time window during the scanning phase and relaxing the spatial threshold during the low-speed sliding phase; Integrated Kalman filter: Predicts the position of the next sampling point. If the actual position deviates greatly from the predicted position and there is no sudden change in the pupil diameter, it is determined to be equipment noise and discarded; if the deviation is within the range of biological motion law, it is determined to be microsaccade and retained.

[0026] Step 2: Acquisition and normalization of pupil parameters on mobile device; Using iris diameter as a reference, through the formula Convert to physical pupil diameter; The eye region is located in RGB space, and adaptive binarization and morphological closing operations are performed to eliminate corneal reflective bright spots, reconstruct the pupil contour, and extract the diameter parameter.

[0027] Step 3: Flexible Grid Mapping Algorithm – Semantic Alignment of Gaze Trajectory with Text Structure; Extract the text topological features from the reading interface and generate a vertical Gaussian ridge model for each line of text. ; The gaze point sequence is reconstructed into a continuous gaze energy field E(x, y). The optimal translation parameter Δy and scaling parameter s are found using gradient descent to minimize the cross-entropy between the gaze energy field and the text mask. .

[0028] Step 4: Construction of the Multimodal Displexity Index (PI) and Adaptive Closed-Loop Intervention; Calculate the confusion index ; in For real-time gaze duration at the retrace point, The baseline average fixation duration; The instantaneous physical diameter of the pupil. The resting reference diameter; For video rate or spatial span characteristics.

[0029] Implement tiered interventions based on the relationship between PI values ​​and preset thresholds: If PI < This is determined to be a strategic review and will not trigger intervention. If PI> If the system detects cognitive overload, it will perform adaptive tasks, including dynamic vegetative wiping, highlighting visual anchors, or adjusting the text flow speed.

[0030] Step 5: Adaptive intervention methods based on real-time monitoring results; Dynamic difficulty regulation of cognitive load: when PI > When the gaze stability is improved and the PI is in the low range, the difficulty is increased (background noise is introduced, speed requirements are increased).

[0031] Visual guidance for bounce behavior: A window occlusion strategy is adopted to keep only the current gaze point and the text within the preset range on the right clear, while blurring the already read area on the left; when an unexpected bounce is detected, a high-frequency flashing guidance point is generated at the ideal next gaze position.

[0032] Rhythmic blanking training: Perform word-by-word blanking from left to right according to the target reading speed; if the pupil dilation rate continues to exceed 15%, the blanking frequency will be automatically reduced.

[0033] Line break precision deviation enhancement training: When approaching the end of a line, a high-contrast identifier is pre-rendered at the beginning of the next line; if the line break landing point deviation exceeds the threshold, a sound effect is emitted and an ideal diagonal path auxiliary line is superimposed and displayed.

[0034] Details of system initialization and calibration; When any user first uses this system, the following initialization and calibration steps are automatically executed: First, resting baseline acquisition: The user gazes at a stationary target point in the center of the screen for 10 seconds. During this period, after the system eliminates the blinking segments (pupil area change rate > 30% or continuous loss ≥ 3 frames), the average pupil diameter of the remaining frames is taken as , and the average duration of valid fixation points is taken as .

[0035] Second, signal-to-noise ratio dynamic calibration: In the first 2 seconds of the calibration phase, the system continuously acquires eye movement coordinates, calculates the root mean square of the displacement between adjacent frames as the noise benchmark ; at the same time, calculates the difference between the maximum displacement and the minimum displacement during this period as the signal amplitude , and obtains the environmental signal-to-noise ratio in real time through the formula SNR = / , and automatically sets the initial spatial discretization threshold according to the mapping table (SNR ≤ 5 is low noise, 5 < SNR ≤ 15 is medium noise, SNR > 15 is high noise) The value range of is 0.5° - 2° visual angle.

[0036] Third, coordinate space mapping: The system prompts the user to gaze at the four calibration points in the upper left, upper right, lower left, and lower right of the screen in sequence, acquires the corresponding screen pixel coordinates and the original output coordinates of the eye tracker, and uses the second-order polynomial fitting method to establish an initial transformation matrix from the eye tracker coordinates to the screen coordinates to eliminate the inherent non-linear distortion of the device.

[0037] All the above parameters are recalibrated at the start of each session without manual intervention by the user.

[0038] Example 1: System hardware configuration and data acquisition; In this example, a front camera (sampling rate 30Hz - 60Hz) integrated in a tablet computer or smartphone is used as the eye movement data acquisition device, and at the same time, the camera is used to obtain eye images to extract the pupil diameter and iris diameter. When the system is initialized, the user is guided to perform a 2-second resting state monitoring to record the reference pupil diameter and the average fixation duration .

[0039] In this embodiment, the eye movement coordinates and pupil diameter are calculated synchronously by the same front-facing camera in each frame of the image, without the need for additional hardware to trigger synchronously.

[0040] Example 2: Trajectory reconstruction and line alignment implementation; Based on the improved I-DT algorithm in step one, the SNR is dynamically calculated and set. For each frame, Kalman filtering is used to predict eye movement coordinates, removing device noise and preserving micro-saccades. Then, step three, elastic grid mapping, is performed: the system reads the DOM structure of the current reading interface and extracts the center line of each line of text. A Gaussian ridge model is constructed using the line spacing parameter σ. The accumulated gaze point sequence is projected onto the energy field, and the optimal Δy and s are solved using gradient descent to complete the coordinate correction.

[0041] The inventors of this invention conducted preliminary functional verification on five healthy adult volunteers (with normal or corrected vision): each volunteer read three short articles of approximately 200 words each, maintaining a slight head tilt of about ±15° during reading. Statistical results showed that after adopting the elastic grid mapping of this invention, the rate of misjudging intraline regression as crossline regression due to head tilt decreased from an average of 38% in the traditional fixed-threshold I-DT algorithm to approximately 11%. It should be noted that this data comes from internal verification and is used only to illustrate the algorithm's principles, not as a performance commitment to external parties.

[0042] The state variables of the Kalman filter are set as position and velocity, the state transition matrix adopts a uniform motion model, and the process noise covariance is preset as a diagonal matrix based on experience.

[0043] Regarding the gradient descent method used in Example 2 to solve for the optimal Δy and s, the following implementation details are provided: Δy: Vertical translation parameter, in screen pixels, used to compensate for the overall gaze point shift caused by the user's head movements or eye tracker vertical drift. Its value range is limited to [-0.5⋅h, +0.5⋅h], where h is the height of the unit text.

[0044] s: Global scaling parameter, dimensionless, used to correct screen coordinate scaling distortion caused by user's forward and backward movement (posture change). When the user leans forward, the viewing angle magnifies, requiring s < 1 to shrink the gaze point coordinates; When the user leans back, the field of view narrows, requiring s>1 to magnify the gaze point coordinates. Its value range is limited to [0.85, 1.15], corresponding to a maximum physiologically accessible range of sitting posture change of approximately ±15%.

[0045] Loss function definition: Cross-entropy term J(Δy, s) = -∑_{i=1}^{N}E(x_i, y_i)·log( (y_i·s+Δy)), where N is the total number of gaze points in the current sliding window (default window size = 50 gaze points).

[0046] Iteration termination condition: Using the Adam optimizer with an initial learning rate of 0.01, the iteration stops when the decrease in J is less than 0.001 for five consecutive iterations, or when the maximum number of iterations (100) is reached. The optimizer state is automatically reset after each line break.

[0047] Coordinate boundary constraints: The translation parameter Δy is limited to the range of [-0.5·line_height, +0.5·line_height], and the scaling parameter s is limited to the range of [0.85, 1.15] to prevent overfitting to physiologically feasible poses.

[0048] Landing point classification rule: The vertical coordinates of the corrected fixation point are aligned with the center of the Gaussian ridge. When the absolute difference is less than 0.3·σ, it is judged as "aligned with the current row"; otherwise, it is judged as "deviation" and handed over to the cognitive decoder for processing.

[0049] The parameters mentioned above are typical settings in this field, and the system also allows developers to adjust them as needed in the configuration file.

[0050] Example 3: Cognitive Load Assessment and Intervention Case; To avoid applicability limitations caused by using absolute values ​​(such as 1.5, 0.8), this system adopts a user-adaptive percentile threshold method, as follows: and Determination: After a user completes a standard short reading passage (approximately 200 words, medium difficulty), the system calculates the 33rd percentile (P33) and 80th percentile (P80) of all PI values ​​during that process, and sets them as follows: and If the user has no historical data, the default value will be used: =0.8, =1.6.

[0051] Dynamic mapping of intervention parameters: The relationship between font magnification and PI value in dimensionality reduction intervention is: magnification = 1.0 + 0.2 × ((PI - ) / (max(2· ,3.0)- The upper limit is 1.8 times; the reduction in blanking speed = original speed × (1 - 0.5 × min(1.0, (PI-)) ) / ( )).

[0052] Intervention termination condition: When the PI value is below 3 for 3 consecutive seconds When the value increases by 0.2, the system gradually restores the initial parameters in steps of 10%, observing for 5 seconds after each restoration. If the PI no longer exceeds [the threshold], [further action is taken]. If the previous parameter is not found, the process will continue; otherwise, it will revert to the previous parameter.

[0053] All the above thresholds and mapping relationships are adjusted at runtime based on the dynamic baseline collected in real time, without relying on fixed clinical sample data, ensuring individual adaptability for each user.

[0054] PI values ​​are calculated in real time during the user's reading process. (Settings) =1.5, =0.8. When the system detects continuous user backtracking and the PI value rises to 1.6, it determines that there is cognitive overload and triggers intervention: automatically enlarging the font size of the current line by 2 points and reducing the fading speed of subsequent text from 400 characters / minute to 280 characters / minute, while simultaneously superimposing a 60Hz flashing green guide dot in the high-frequency backtracking area to guide the eye to the right. If the PI value drops below 1.2 within the next 10 seconds, the system gradually restores the original parameters. This closed-loop process continues until the user completes the preset reading task.

[0055] Handling eye-tracking data loss and threshold oscillations: Data loss compensation: When the number of consecutive lost frames is ≤3, the system uses linear interpolation to fill in the pupil diameter and coordinate data; when the number of lost frames is ≥4, it is determined that the eyes are intentionally closed or the device is out of focus, the system pauses data recording and prompts the user to recalibrate, while automatically saving the collected data.

[0056] Threshold oscillation suppression: To prevent the PI value from oscillating at certain points. and Frequent fluctuations in the vicinity caused intervention oscillations, so the system introduced a hysteresis comparator. After intervention is triggered, the system will only intervene if the PI value is below a certain level for 5 consecutive seconds. Intervention can only be lifted after a certain period of time; after lifting, the PI value must remain above 2 seconds. Only then can it be triggered again. The default hysteresis interval width is 0.2.

[0057] Personalized iris calibration: With an iris diameter reference constant of 11.7mm, the system allows users to input their age and ethnicity (Asian / European / African) upon first use, and dynamically adjusts the conversion factor based on the average iris diameter of different populations (approximately 11.0mm for Asians, 11.7mm for Europeans, and 12.0mm for Africans); precise calibration can also be performed by photographing a calibration ruler (the user holds a standard diameter circular sticker close to the eye).

[0058] Example 4: Line break training application; To address the common issue of line break misalignment among patients with dyslexia, the system displays a red dot at the center of the next line's beginning when the gaze point is two characters away from the end of the line. If the horizontal deviation between the line break saccade and the dot exceeds 30 pixels, the system emits a short "beep" sound and instantly displays a diagonal auxiliary line on the screen, running from the end of the previous line to the beginning of the current line. After 10 training sessions, the average line break misalignment of patients decreased from 45 pixels to 18 pixels.

[0059] The following content clarifies when training ends and how to quantify improvements: Termination conditions for a single training session: When the landing point deviation of 10 consecutive line break scans is less than 15 pixels, or the standard deviation of the deviation is less than 5 pixels and the mean is less than 20 pixels, the system will automatically prompt "Training in this stage has been achieved" and end the current session. If the user's fatigue index (based on pupil diameter volatility > 20% or blink frequency > 30 times per minute) exceeds the limit, the process will be terminated early.

[0060] Objective recording of long-term training effects: The system records the average line break deviation (mm), median deviation, and percentage of deviations exceeding the threshold (30 pixels) for each training session. When the above indicators improve by more than 20% in three consecutive training sessions (without regression), the system determines it as "effective progress" and suggests moving to the next stage of higher difficulty (such as reducing the size of visual anchor points and shortening the timing of cue appearance).

[0061] Multimodal feedback audio parameters: When the impact point deviation exceeds the threshold, the emitted sound effect is a short sine wave of 250ms and 800Hz; when the deviation is less than 10 pixels, the option is to play a positive feedback sound of 50ms and 1500Hz. The audio effect option can be turned off in the settings, leaving only the visual cues.

[0062] The above metrics are all set based on general eye-tracking spatial accuracy (typical commercial devices have an error of about 0.5°~1° of field of view) and do not involve exaggeration of the accuracy of specific hardware.

[0063] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. An adaptive reading performance testing and training system based on multimodal fusion, characterized in that, include: Multimodal data acquisition module: used to acquire the user's eye movement coordinates (x, y, t), pupil diameter, and head posture data; Spatiotemporal trajectory reconstruction engine: including an improved I-DT algorithm module based on dynamic weighting of signal-to-noise ratio and Kalman filter prediction compensation, used to extract gaze points from low-frequency sampled data and preserve micro-saccades; Elastic Mesh Mapping Module: Used to generate Gaussian ridge models based on the text topology features of the reading interface, and correct vertical drift and scaling errors in eye-tracking trajectories by minimizing the cross-entropy between the gaze energy field and the text mask; Cognitive State Decoder: Used to calculate the multimodal confusion index PI based on fixation duration, pupil diameter change rate, and return rate; Adaptive Intervention Controller: Used to perform one or more closed-loop intervention operations based on the comparison results of PI value and preset threshold, including dynamic difficulty adjustment, visual occlusion, forced forward induction, rhythmic caching, or line break reinforcement training.

2. An adaptive reading performance testing and training method based on multimodal fusion, using the system described in claim 1, characterized in that, Includes the following steps: Step 1: Collect low-sampling-rate eye movement coordinates, pupil diameter, and head posture data during the user's reading process; Step 2: Using the improved I-DT algorithm, the spatiotemporal threshold is adaptively adjusted based on the real-time signal-to-noise ratio and velocity weighting, and Kalman filtering is used to predict and remove device noise, retain micro-saccades, and extract the fixation point sequence; Step 3: Based on the Gaussian ridge model of the text line in the reading interface, the vertical drift and scaling errors in the eye-tracking trajectory are corrected by elastic mesh mapping to achieve semantic alignment between the gaze point and the text line; Step 4: Calculate the multimodal confusion index ; Step 5: Based on the relationship between the PI value and the preset threshold, trigger the adaptive closed-loop intervention task.

3. The system according to claim 1 or the method according to claim 2, characterized in that, The improved I-DT algorithm includes: dynamically generating a spatial dispersion threshold based on the signal-to-noise ratio (SNR) during the initial calibration phase. During the scanning phase, the time window is automatically tightened, and during the low-speed sliding phase, the spatial threshold is relaxed. The position of the next sampling point is predicted by a Kalman filter. When the deviation between the measured position and the predicted position exceeds the biological motion law and there is no sudden change in the pupil diameter, it is judged as equipment noise and discarded.

4. The system according to claim 1 or the method according to claim 2, characterized in that, The pupil diameter is obtained by using the iris diameter as a reference, and employing the formula... The physical diameter is converted; and adaptive binarization and morphological closing operations are performed on the eye image to eliminate corneal reflective bright spots.

5. The system according to claim 1 or the method according to claim 2, characterized in that, The elastic mesh mapping includes: generating a Gaussian ridge model for each line of text. ,in The center position of the line is σ, which is set based on the font size and line spacing. The gaze point sequence is reconstructed into a continuous energy field E(x, y), and the translation parameter Δy and scaling parameter s are optimized using gradient descent to minimize the cross-entropy term. .

6. The system according to claim 1 or the method according to claim 2, characterized in that, The adaptive intervention controller includes: when PI < When PI is determined to be strategic review, no intervention is triggered; when PI > If the system determines that the cognitive load is overloaded, it will perform at least one of the following operations: dynamically enlarge the font of the current line, reduce the text flow speed, reduce the background contrast, apply Gaussian blur to the read area to the left of the gaze point, generate a high-frequency flashing induction point at the ideal gaze target position, perform word-by-word fading at the target speed and dynamically adjust the fading frequency according to the pupil dilation rate, or pre-display the beginning of the next line at the end of the line and issue multimodal feedback when the landing point deviation exceeds the threshold.

7. The system or method according to claim 6, characterized in that, The frequency adjustment method for word-by-word masking is: real-time monitoring of pupil diameter. ,when When the percentage is greater than 15%, the delay time per word will be automatically increased, thus reducing the speed of word removal.

8. The system or method according to claim 6, characterized in that, The line break enhancement training includes: when the gaze point is close to the remaining preset number of characters at the end of the line, a high-contrast marker is displayed at the logical center position at the beginning of the next line; the spatial distance deviation between the line break scanning landing point and the center point of the beginning of the line is detected, and if it exceeds a preset threshold, an audio prompt is issued and an ideal diagonal path auxiliary line from the end of the previous line to the beginning of the current line is superimposed and displayed.

9. The system according to claim 1 or the method according to claim 2, characterized in that, The multimodal confusion index PI The return rate or spatial span characteristics are determined based on gaze path directionality analysis; α, β, and γ are preset weighting coefficients, and satisfy α+β+γ=1.

Images