A child visual cognitive ability evaluation system and method based on desktop touch and multi-modal behavior fusion, a terminal and a storage medium

Abstract

The application relates to the technical field of data processing, and discloses a children's visual cognitive ability evaluation system and method based on desktop touch and multi-modal behavior fusion, a terminal and a storage medium, the system comprises a touch interaction collection module, which presents a multi-stage evaluation task and generates an interaction log containing a time stamp; a video collection module, which synchronously records a face video; an interaction feature extraction module, which extracts behavior indexes such as accuracy and reaction time from the log; a view point estimation module, which maps the video into a two-dimensional gaze track through deep learning; a time sequence alignment module, which aligns and intercepts an effective decision time window with the log time stamp as a boundary; a line of sight analysis module, which counts the first view point arrival time and the gaze duration ratio and the like line of sight features in the time window; and an evaluation result output module, which fuses the behavior indexes and the line of sight features to generate a cognitive evaluation result. The application deeply aligns and fuses the system interaction event stream and the behavior data, and improves the cognitive ability evaluation precision.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a children's visual cognitive ability assessment system, visual cognitive ability assessment method, terminal, and computer-readable storage medium based on desktop touch and multimodal behavior fusion. Background Technology

[0002] The assessment of children's visual cognitive abilities is usually combined with systematic clinical observation by professionals of children's early social interactions and behavioral patterns. However, this traditional assessment model has significant limitations: on the one hand, traditional scales are easily influenced by the subjective bias of interventionists, resulting in highly subjective and inconsistent assessment results; on the other hand, systematic clinical observation is highly dependent on professional medical resources, consuming a large amount of medical human resources and making it difficult to meet the universal assessment needs of a large group of children.

[0003] In recent years, computer technology and multimodal behavioral analysis have provided new approaches to assessing developmental behavioral disorders in children. Machine-assisted methods can alleviate the shortage of professional physicians and improve the objectivity of assessments. However, traditional single-modal systems that rely solely on screen clicks can only reflect the final success or failure of a task, ignoring core behavioral markers such as frequent eye wandering and attention deficit difficulties that accompany children's cognitive processes. They fail to connect underlying interactions with external performance, resulting in highly limited and one-sided assessment results. Deep multimodal fusion of internal system data and behavioral data is also impossible.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide a visual cognitive ability assessment system and method for children based on desktop touch control and multimodal behavior fusion. This aims to solve the problems in the existing technology where early cognitive behavior assessment mainly relies on standardized assessment scales and manual systematic clinical observation, resulting in poor specialization in assessing underlying cognitive mechanisms, low automation, and difficulty in deep temporal alignment of internal and external modal data, leading to inaccurate assessment results.

[0006] To achieve the above objectives, the present invention provides a children's visual cognitive ability assessment system based on desktop touch control and multimodal behavior fusion, the children's visual cognitive ability assessment system based on desktop touch control and multimodal behavior fusion includes: The touch interaction acquisition module is used to present children with multi-stage visual cognitive assessment tasks and collect the children's tapping operations on the touch screen to generate an interaction log. A video capture module is used to synchronously record the child's facial video during the visual cognitive assessment task; An interaction feature extraction module, connected to the touch interaction acquisition module, is used to extract internal interaction features from the interaction log. The internal interaction features include: accuracy, cross-stage accuracy difference, average reaction time, cross-stage reaction time difference, and reaction time variation coefficient. The viewpoint estimation module, connected to the video acquisition module, is used to perform deep learning processing on the facial video, extract facial feature points and three-dimensional viewpoint vectors, and map the three-dimensional viewpoint vectors to a two-dimensional gaze trajectory on the screen based on the facial feature points. The temporal alignment module is connected to the touch interaction acquisition module and the viewpoint estimation module respectively. It is used to align the two-dimensional gaze trajectory on the time axis with the stimulus presentation timestamp and response timestamp in the interaction log as the boundary, and extract the effective decision time window corresponding to each matching task from the aligned two-dimensional gaze trajectory. The gaze analysis module, connected to the time alignment module, is used to statistically analyze the gaze behavior characteristics of the gaze falling on the preset interest area of ​​the screen within the effective decision time window. The gaze behavior characteristics include: the arrival time of the target first viewpoint, the gaze duration ratio, the dwell frequency, the number of gaze wanderings, and the total scanning path length. The assessment result output module is connected to the interaction feature extraction module and the gaze analysis module, respectively, and is used to generate and output the child's cognitive assessment results based on the internal interaction features and the gaze behavior features.

[0007] Optionally, in the aforementioned child visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion, the touch interaction acquisition module includes: a color attribute recognition unit, a shape attribute recognition unit, a round control unit, a voice guidance unit, a single variable control unit, and a log generation unit; The color attribute recognition unit is used to present color recognition training tasks for six basic color attributes: red, yellow, blue, green, purple, and white. The shape attribute recognition unit is used to present shape recognition training tasks for six basic shape attributes: square, trapezoid, triangle, circle, ellipse and star. The round control unit is used to control independent matching tasks with a fixed number of rounds in each matching stage, and provides a stage transition interface after each stage ends. The stage transition interface includes a button to enter the next level and a button to settle and end the process. The voice guidance unit is used to play guidance voice when the task is presented, play success voice when the tap is correct, play error prompt voice when the tap is incorrect, and trigger double prompt when the response times out. The single-variable control unit is used to control the shape variable to be exactly the same in the color recognition training task, and to control the color variable to be exactly the same in the shape recognition training task. The log generation unit is used to record stimulus presentation timestamps, response timestamps, and click accuracy in the visual cognitive assessment task in the form of a structured log, and to generate the interaction log based on the stimulus presentation timestamps, response timestamps, and click accuracy.

[0008] Optionally, in the children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion, the interactive feature extraction module includes: an accuracy calculation unit, a reaction time calculation unit, and a coefficient of variation calculation unit; The accuracy calculation unit is used to record the basic matching accuracy of the child at each difficulty level in the color and shape task, and to calculate the cross-stage accuracy difference between adjacent difficulty levels based on the basic matching accuracy. The reaction time calculation unit is used to record the time from the presentation of the options to the child making a valid click, calculate the average reaction time for each difficulty level based on the time, and calculate the cross-stage reaction time difference between adjacent difficulty levels. The coefficient of variation calculation unit is used to calculate the coefficient of variation of reaction time based on the child's reaction time in each round within the same stage.

[0009] Optionally, in the children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion, the viewpoint estimation module includes: a facial feature point localization unit, a head three-dimensional pose calculation unit, a segmentation unit, a three-dimensional viewpoint vector derivation unit, and a viewpoint coordinate mapping unit. The facial feature point localization unit is used to extract the face frame by frame from the facial video and locate the facial feature points. The head three-dimensional pose calculation unit is used to calculate the pitch angle, yaw angle and roll angle of the child's head based on the facial feature points; The segmentation unit is used to segment the center of the eyeball and the pupil region using a convolutional neural network to obtain the position of the center of the eyeball and the position of the center of the pupil. The three-dimensional viewpoint vector derivation unit is used to calculate the three-dimensional coordinates of the eyeball center and the pupil center respectively, based on the iris as the absolute reference, according to the eyeball center position and the pupil center position, and derive the three-dimensional viewpoint vector with the three-dimensional coordinates of the eyeball center as the starting point and the three-dimensional coordinates of the pupil center as the ending point. The viewpoint coordinate mapping unit is used to convert the three-dimensional viewpoint vector into a two-dimensional gaze trajectory on the screen using a multinomial regression algorithm or a homomorphic mapping matrix, based on the pitch angle, the yaw angle, and the roll angle.

[0010] Optionally, in the children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion, the time alignment module includes: a time boundary acquisition unit, a trajectory alignment unit, and a time window interception unit. The time boundary acquisition unit is used to extract stimulus presentation timestamps and response timestamps from the interaction log; The trajectory alignment unit is used to align the time axis of the two-dimensional gaze trajectory with the absolute timestamp reference of the interaction log to obtain the aligned two-dimensional gaze trajectory. The time window extraction unit is used to extract the effective decision time window corresponding to each matching task from the aligned two-dimensional gaze trajectory, with the stimulus presentation timestamp as the starting boundary and the response timestamp as the ending boundary.

[0011] Optionally, in the aforementioned child visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion, the gaze analysis module includes: an interest area division unit, a first gaze point arrival time calculation unit, a duration ratio and frequency calculation unit, and a saccade path analysis unit. The interest area division unit is used to divide the screen into a target stimulation area, a target correct option area, and an interference option area, and to use the target stimulation area, the target correct option area, and the interference option area as preset interest areas of the screen. The first viewpoint arrival time calculation unit is used to measure the first viewpoint arrival time when the line of sight first falls on the target correct option area, with the stimulus presentation timestamp in the interaction log as zero point. The duration ratio and frequency calculation unit is used to count the number of frames in which the viewpoint falls within the preset interest area within the effective decision time window, and to calculate the gaze duration ratio and dwell frequency based on the corresponding number of frames. The scanning path analysis unit is used to count the number of times the gaze trajectory is not in the preset interest area or outside the screen, and to calculate the cumulative value of the spatial distance between consecutive effective viewpoints, and to use the cumulative value of the spatial distance as the total scanning path length.

[0012] Optionally, in the children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion, the assessment result output module includes: an assessment result generation unit and an output and storage unit; The assessment result generation unit is used to average or summarize the accuracy rate, the cross-stage accuracy rate difference, the average reaction time, the cross-stage reaction time difference, the reaction time coefficient of variation, the target first point of view arrival time, the fixation duration ratio, the dwell frequency, the number of gaze wanders, and the total saccadic path length to obtain the child's cognitive assessment result. The output and storage unit is used to generate attention regulation indicators based on the evaluation results, and to generate an evaluation report by combining the evaluation results, the attention regulation indicators, and the internal interaction features and gaze behavior features extracted at each evaluation stage, along with the child's personal dataset and the evaluation time, for printing and storage in the background database.

[0013] Furthermore, to achieve the above objectives, the present invention also provides a visual cognitive ability assessment method for a children's visual cognitive ability assessment system based on desktop touch control and multimodal behavior fusion, wherein the visual cognitive ability assessment method includes: The touch interaction acquisition module presents children with multi-stage visual cognitive assessment tasks and collects the children's tapping operations on the touch screen to generate an interaction log. The video capture module synchronously records the child's facial video during the visual cognition assessment task; The interaction feature extraction module extracts internal interaction features from the interaction log. The internal interaction features include: accuracy, cross-stage accuracy difference, average reaction time, cross-stage reaction time difference, and reaction time coefficient of variation. The viewpoint estimation module performs deep learning processing on the facial video to extract facial feature points and a three-dimensional viewpoint vector, and maps the three-dimensional viewpoint vector to a two-dimensional gaze trajectory on the screen based on the facial feature points. The temporal alignment module uses the stimulus presentation timestamp and response timestamp in the interaction log as boundaries to align the two-dimensional gaze trajectory on the time axis, and extracts the effective decision time window corresponding to each matching task from the aligned two-dimensional gaze trajectory. The gaze analysis module statistically analyzes the gaze behavior characteristics of the gaze falling on the preset interest area of ​​the screen within the effective decision time window. The gaze behavior characteristics include: the arrival time of the first gaze point, the gaze duration ratio, the dwell frequency, the number of gaze wanderings, and the total saccade path length. The assessment result output module generates and outputs the child's cognitive assessment results based on the internal interaction characteristics and the gaze behavior characteristics.

[0014] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a visual cognitive ability assessment program of a children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion, which is stored in the memory and can run on the processor. When the visual cognitive ability assessment program of the children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion is executed by the processor, it implements the steps of the visual cognitive ability assessment method of the children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion as described above.

[0015] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a visual cognitive ability assessment program of a children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion, wherein when the visual cognitive ability assessment program of the children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion is executed by a processor, it implements the steps of the visual cognitive ability assessment method of the children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion as described above.

[0016] In this invention, the system includes: a touch interaction acquisition module for presenting a multi-stage evaluation task and generating an interaction log with timestamps; a video acquisition module for synchronously recording facial videos; an interaction feature extraction module for extracting behavioral indicators such as accuracy and reaction time from the logs; a viewpoint estimation module for mapping the video into a two-dimensional gaze trajectory using deep learning; a temporal alignment module for aligning with the log timestamps and extracting effective decision time windows; a gaze analysis module for statistically analyzing gaze features such as first viewpoint arrival time and gaze duration ratio within the time window; and an evaluation result output module for fusing behavioral indicators and gaze features to generate a cognitive evaluation result. This invention performs deep temporal alignment and multimodal fusion of high-precision system interaction event streams and continuous external video eye-tracking behavior data within the decision time window, thereby providing a quantitative cognitive ability assessment. Attached Figure Description

[0017] Figure 1 This is the overall architecture diagram of the children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion of the present invention; Figure 2 This is a schematic diagram of the principle architecture of the children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion of the present invention; Figure 3 This is a flowchart of a single paradigm in the children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion, as described in this invention. Figure 4 This is a flowchart of a preferred embodiment of the visual cognitive ability assessment method of the children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion of the present invention; Figure 5 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0019] To address the problems in the existing technology, this embodiment provides a children's visual cognitive ability assessment system based on desktop touch control and multimodal behavior fusion, such as... Figure 1 and Figure 2 As shown, the children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion includes: a touch interaction acquisition module, used to present children with multi-stage visual cognitive assessment tasks and collect the children's tapping operations on the touch screen to generate an interaction log; A video capture module is used to synchronously record the child's facial video during the visual cognitive assessment task; An interaction feature extraction module, connected to the touch interaction acquisition module, is used to extract internal interaction features from the interaction log. The internal interaction features include: accuracy, cross-stage accuracy difference, average reaction time, cross-stage reaction time difference, and reaction time variation coefficient. The viewpoint estimation module, connected to the video acquisition module, is used to perform deep learning processing on the facial video, extract facial feature points and three-dimensional viewpoint vectors, and map the three-dimensional viewpoint vectors to a two-dimensional gaze trajectory on the screen based on the facial feature points. The temporal alignment module is connected to the touch interaction acquisition module and the viewpoint estimation module respectively. It is used to align the two-dimensional gaze trajectory on the time axis with the stimulus presentation timestamp and response timestamp in the interaction log as the boundary, and extract the effective decision time window corresponding to each matching task from the aligned two-dimensional gaze trajectory. The gaze analysis module, connected to the time alignment module, is used to statistically analyze the gaze behavior characteristics of the gaze falling on the preset interest area of ​​the screen within the effective decision time window. The gaze behavior characteristics include: the arrival time of the target first viewpoint, the gaze duration ratio, the dwell frequency, the number of gaze wanderings, and the total scanning path length. The assessment result output module is connected to the interaction feature extraction module and the gaze analysis module, respectively, and is used to generate and output the child's cognitive assessment results based on the internal interaction features and the gaze behavior features.

[0020] Understandably, this invention addresses the widely accepted theories of weak central integration and executive dysfunction in developmental psychology and cognitive neuroscience, designing a highly interactive, multi-stage progressive visual cognitive assessment paradigm based on desktop touch control. Specifically, this invention proposes a testing paradigm encompassing color attribute cognition and shape attribute cognition. Under a strict single-variable control mechanism, it prompts children to undergo cognitive transitions through four progressively deeper stages: basic matching "from abstract to abstract," semantic exclusion "from abstract to concrete," feature extraction "from concrete to abstract," and cross-semantic feature generalization "from concrete to concrete." During this paradigm transition, the local features and polymorphic semantics of concrete everyday objects are used as cognitive interference to specifically induce and quantify the underlying cognitive impairments in children with autism.

[0021] Furthermore, the aforementioned children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion performs cognitive assessment based on desktop touch and eye-tracking trajectory multimodal fusion. First, it uses a low-level event recording module to obtain high-precision system interaction logs, synchronously recording events such as stimulus presentation and response at the millisecond level to calculate internal interaction indicators such as cross-stage accuracy difference and reaction time variation coefficient. Then, it designs a high-definition, non-intrusive viewpoint estimation and behavior analysis algorithm based on a single camera, using deep learning facial tracking to accurately map the subject's gaze movement in three-dimensional physical space to a gaze trajectory on a two-dimensional screen. Finally, through a globally unified timestamp, it performs deep temporal alignment and multimodal fusion of high-precision system interaction event streams and continuous external video eye-tracking behavior data within the decision time window, thereby making a quantitative cognitive ability assessment.

[0022] In this embodiment, the children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion can be divided into two parts: an assessment paradigm and an assessment system. The assessment paradigm is a highly interactive, multi-stage progressive visual cognitive assessment method based on desktop touch. It is used to establish a structured and non-intrusive test scenario for interaction between the machine system and the child. Through a cross-semantic matching task "from abstract to concrete," it specifically induces difficulties in the child's underlying cognitive mechanisms. It also collects internal click interaction logs and external videos in the scenario simultaneously for subsequent multimodal assessment. The assessment system is a cognitive assessment system based on desktop touch and eye-tracking trajectory multimodal fusion. It is used to extract multidimensional behavioral information during the test process and designs a high-precision viewpoint estimation and mapping algorithm to obtain two-dimensional gaze trajectory. Finally, based on the underlying absolute timestamp, it deeply integrates internal and external data to calculate quantitative indicators such as reaction time variation coefficient and target first viewpoint arrival time, thereby completing the quantitative assessment of "weak central integration" and "executive dysfunction" in children with autism.

[0023] Furthermore, the touch interaction acquisition module includes: a color attribute recognition unit, a shape attribute recognition unit, a turn control unit, a voice guidance unit, a single variable control unit, and a log generation unit; The color attribute recognition unit is used to present color recognition training tasks for six basic color attributes: red, yellow, blue, green, purple, and white. The shape attribute recognition unit is used to present shape recognition training tasks for six basic shape attributes: square, trapezoid, triangle, circle, ellipse and star. The round control unit is used to control independent matching tasks with a fixed number of rounds in each matching stage, and provides a stage transition interface after each stage ends. The stage transition interface includes a button to enter the next level and a button to settle and end the process. The voice guidance unit is used to play guidance voice when the task is presented, play success voice when the tap is correct, play error prompt voice when the tap is incorrect, and trigger double prompt when the response times out. The single-variable control unit is used to control the shape variable to be exactly the same in the color recognition training task, and to control the color variable to be exactly the same in the shape recognition training task. The log generation unit is used to record stimulus presentation timestamps, response timestamps, and click accuracy in the visual cognitive assessment task in the form of a structured log, and to generate the interaction log based on the stimulus presentation timestamps, response timestamps, and click accuracy.

[0024] In this embodiment, the touch interaction acquisition module constructs a highly immersive and structured testing environment with strict variable control. The system relies on intuitive turn-based touch interaction to precisely decouple complex visual perception testing, executing it through color attribute perception units and shape attribute perception units.

[0025] The color attribute recognition unit includes training in recognizing six basic color attributes: red, yellow, blue, green, purple, and white. The shape attribute recognition unit includes training in recognizing six basic shape attributes: square, trapezoid, triangle, circle, ellipse, and star.

[0026] Furthermore, the entire interaction in this paradigm relies on a desktop touch system, where autistic children advance the process by tapping the correct options on the screen. In the matching task, a virtual cartoon character is positioned on the left or center of the screen, displaying the target stimulus to the child through a thought bubble above its head; the other areas of the screen present six alternatives for the child to tap and select.

[0027] The round control unit is used to implement a round-based system and a feedback mechanism. Each matching phase contains a fixed 6 rounds (i.e., 6 independent matches).

[0028] The voice guidance unit is used to play guiding voice messages. The system has built-in rich multimodal guidance feedback: not only is there clear voice guidance when the question is presented (such as "Little friend, what color is this Chinese knot? Tap the object on the screen that is the same color as it!"), but a success voice message is triggered when the child taps correctly and an error message message is triggered when the child taps incorrectly. If the child's response exceeds the set time, the system will automatically trigger both visual arrows and voice prompts to guide their attention.

[0029] The single-variable control unit is used to implement a single-variable control mechanism. In the color recognition task, the shape variable is controlled to be completely identical (for example, "abstract red attribute" is uniformly represented as a red circle, and "abstract blue attribute" is represented as a blue circle); in the shape recognition task, the color variable is controlled to be completely identical (for example, "abstract triangle" is uniformly represented as a gray triangle, and "abstract circle" is represented as a gray circle).

[0030] Furthermore, the touch interaction acquisition module supports a multi-stage progressive evaluation process, which includes: cognitive guidance stage, abstract to abstract stage, abstract to concrete stage, concrete to abstract stage, and concrete to concrete stage. The cognitive guidance stage is used to guide children to recognize all basic attributes and their characteristic objects in life through voice and visual images in sequence. The abstract-to-abstract stage is used to display the target abstract attribute in the bubble above the cartoon character's head, requiring children to tap out the corresponding same attribute from the options that are all abstract attributes; The abstract-to-concrete stage is used to display the target abstract attribute in the bubble above the cartoon character's head, requiring children to tap out the object containing the target attribute from the options that are all concrete living objects. The concrete-to-abstract stage is used when displaying concrete everyday objects on the bubbles above the heads of cartoon characters, requiring children to tap out the correct abstract color blocks or standard shapes from options that are all abstract. The concrete-to-concrete stage is used to display the target concrete object in the bubble above the cartoon character's head, requiring children to find another concrete object with the same core target attribute as the target concrete object from all options that are concrete objects.

[0031] like Figure 3As shown, this implementation takes a single cognitive paradigm (color cognitive paradigm or shape cognitive paradigm) as an example. Its overall standard execution process includes a preliminary "guidance phase" and four subsequent progressively deeper "matching phases." In the matching task, each phase includes a fixed six rounds of testing to cyclically cover the six basic attributes within that paradigm domain. The specific process is as follows: Step S101, Cognitive Guidance Stage: The system guides children to recognize all six basic attributes of this paradigm and their characteristic objects in daily life through voice and visual images. For example, in the color paradigm, the system will ask children to tap the screen to continuously display characteristic objects with attributes such as red and yellow, accompanied by reinforced voice (such as "This is a red Chinese knot"), thereby establishing a basic attribute mapping for subsequent assessments.

[0032] Step S102, Phase One (Abstract to Abstract Phase): Entering the matching assessment. In this phase, which includes 6 rounds of tasks, each round of tasks features a cartoon character displaying a "target abstract attribute" through a bubble above their head. Children are required to tap and select the corresponding attribute from 6 options that are all "abstract attributes" (1 correct option + 5 distractors).

[0033] Step S103, First Inter-Stage Transition: After all six rounds of testing in Stage One are completed, the system enters the stage completion interface. The interface provides two operation buttons: "Enter Next Level" and "Settle and End Paradigm". Clicking "Enter Next Level" will enter Stage Two; if "Settle and End Paradigm" is clicked, the system will immediately calculate the current score and store it in the underlying database, and then exit the current paradigm prematurely.

[0034] Step S104, Stage Two (From Abstract to Concrete): Entering a higher cognitive load. A speech bubble above a cartoon character's head displays the "target abstract attribute." Children are required to accurately select and tap the object containing the target attribute from six options, all of which are "concrete everyday objects" (including characteristic objects and generalized objects), forcing children to eliminate semantic interference from concrete objects.

[0035] Step S105, Second Inter-Stage Transition: After completing Stage 2, enter the transition interface again. The logic is the same as in S103. The guide or child can choose to enter the next level or exit after settlement.

[0036] Step S106, Stage Three (Concrete to Abstract Stage): Cognitive Reverse Mapping. A bubble above a cartoon character's head displays "concrete everyday objects." Children are asked to extract the core features of the object and tap out the correct abstract color block or standard shape from six options that are all "abstract attributes."

[0037] Step S107, Inter-stage transition flow three: After completing stage three, enter the transition interface. The logic is the same as S103. Choose to enter the next level or exit after settlement.

[0038] Step S108, Stage Four (Concrete to Concrete Stage): The most difficult cross-semantic feature generalization. Cartoon character bubbles display "target concrete object A", and children are required to find "concrete object B" with the same core target attribute as A from 6 distractor options that are all "concrete objects" (for example, looking at red strawberries, select red clothes).

[0039] Step S109, Final Settlement and Data Sealing: After the final round of tasks in Phase Four is completed, the system enters the final paradigm completion interface. At this point, there is no need to manually click "settlement"; the system automatically seals the dual-track data and calculates the global score in the background. The interface indicates that the task has been successfully completed. After the user clicks any key on the screen, the system automatically returns to the main menu.

[0040] Furthermore, the video acquisition module is typically fixed above the touchscreen and uses a single-view high-definition camera to continuously record video streams of the child's face and upper body throughout the entire evaluation paradigm. This video acquisition module works in parallel with the touch interaction acquisition module, without relying on any wearable devices, to acquire external behavioral data of the child during natural interaction in a seamless manner. The recorded raw video is used, on the one hand, by the subsequent viewpoint estimation module to extract facial feature points, eye regions, and 3D viewpoint vectors; on the other hand, it serves as a data source for multimodal depth temporal alignment, strictly synchronized with the millisecond-level timestamps in the interaction log.

[0041] Furthermore, the interactive feature extraction module includes: an accuracy calculation unit, a reaction time calculation unit, and a coefficient of variation calculation unit; The accuracy calculation unit is used to record the basic matching accuracy of the child at each difficulty level in the color and shape task, and to calculate the cross-stage accuracy difference between adjacent difficulty levels based on the basic matching accuracy. The reaction time calculation unit is used to record the time from the presentation of the options to the child making a valid click, calculate the average reaction time for each difficulty level based on the time, and calculate the cross-stage reaction time difference between adjacent difficulty levels. The coefficient of variation calculation unit is used to calculate the coefficient of variation of reaction time based on the child's reaction time in each round within the same stage.

[0042] It is understood that the interaction feature extraction module is connected to the touch interaction acquisition module, and is used to automatically extract and calculate internal interaction features directly from the interaction event log, thereby modeling and analyzing the underlying cognitive features of the child. The internal interaction features include: accuracy, cross-stage accuracy difference, average reaction time, cross-stage reaction time difference, and reaction time coefficient of variation.

[0043] Specifically, the accuracy calculation unit records the baseline matching accuracy of the subjects (children with autism) at each difficulty level in the color and shape task. The baseline accuracy directly reflects the subject's ability to recognize and match visual targets in a specific dimension. Simultaneously, the system calculates the accuracy difference between adjacent difficulty levels (e.g., from abstract matching in the first stage to concrete exclusion in the second stage). Given the weak central integration characteristics of children with autism, comparing the accuracy differences allows for a quantitative assessment of the degree of impairment in feature generalization. A significant increase in the accuracy difference objectively indicates that the subject has severe underlying cognitive difficulties in excluding concrete semantic interference and completing cross-semantic matching.

[0044] The reaction time calculation unit accurately records the time elapsed from the presentation of options to the subject's effective click, calculates the average reaction time at each difficulty level, and the reaction time increment resulting from paradigm shifts. Average reaction time is a direct indicator of basic information processing speed and visual search efficiency; while the reaction time increment signifies an increase in cognitive load. When faced with complex, concrete distractions, autistic children with executive dysfunction will experience a significant increase in their visual search and decision-making hesitation time.

[0045] This embodiment further introduces the reaction time coefficient of variation (RCV) to measure the internal consistency of individual behavioral performance. Children with autism often experience significant fluctuations in the time taken for a single operation due to difficulties in attention control. An abnormally high RCV can effectively quantify the degree of difficulty in attention control. The RCV calculation unit calculates the RCV based on the child's reaction time in each round within the same stage, thereby eliminating the interference of differences in individual baseline reaction speed.

[0046] Furthermore, the viewpoint estimation module includes: a facial feature point localization unit, a head 3D pose calculation unit, a segmentation unit, a 3D viewpoint vector derivation unit, and a viewpoint coordinate mapping unit; The facial feature point localization unit is used to extract the face frame by frame from the facial video and locate the facial feature points. The head three-dimensional pose calculation unit is used to calculate the pitch angle, yaw angle and roll angle of the child's head based on the facial feature points; The segmentation unit is used to segment the center of the eyeball and the pupil region using a convolutional neural network to obtain the position of the center of the eyeball and the position of the center of the pupil. The three-dimensional viewpoint vector derivation unit is used to calculate the three-dimensional coordinates of the eyeball center and the pupil center respectively, based on the iris as the absolute reference, according to the eyeball center position and the pupil center position, and derive the three-dimensional viewpoint vector with the three-dimensional coordinates of the eyeball center as the starting point and the three-dimensional coordinates of the pupil center as the ending point. The viewpoint coordinate mapping unit is used to convert the three-dimensional viewpoint vector into a two-dimensional gaze trajectory on the screen using a multinomial regression algorithm or a homomorphic mapping matrix, based on the pitch angle, the yaw angle, and the roll angle.

[0047] In this embodiment, a high-precision face tracking and viewpoint mapping algorithm based on deep learning is used to accurately map the child's gaze in physical space to screen coordinates, so as to overcome the interference of movement in an unconstrained state.

[0048] First, the facial feature point localization unit extracts the face frame by frame from the high-definition video stream and locks hundreds of facial key points. The head 3D pose estimation unit uses these feature points to calculate the pitch, yaw, and roll angles of the child's head in real time, serving as a spatial reference for gaze mapping. Subsequently, the segmentation unit uses a convolutional neural network to accurately segment the eyeball region and pupil region, outputting the image positions of the eyeball center and pupil center.

[0049] Furthermore, the three-dimensional viewpoint vector derivation unit calculates the coordinates of the eyeball center and pupil center in three-dimensional space, using the constant physical size of the iris as a reference, and constructs a three-dimensional gaze direction vector with the eyeball center as the starting point and the pupil center as the ending point. Finally, the viewpoint coordinate mapping unit, with the help of head pose parameters, converts the three-dimensional vector into a continuous two-dimensional gaze trajectory on the screen through polynomial regression or a homomorphic mapping matrix, thereby achieving high-precision viewpoint tracking in an unconstrained state without relying on a head-mounted device.

[0050] Among them, the three-dimensional pose of the head determines the reference coordinates and the origin of the line of sight; the center of the eyeball is the starting point of the line of sight vector, the pupil determines the local deflection direction of the line of sight vector, the eyeball will rotate in the eye socket, and the pupil is located on the surface of the eyeball and changes position with the rotation of the eyeball.

[0051] Furthermore, the timing alignment module includes: a time boundary acquisition unit, a trajectory alignment unit, and a time window interception unit; The time boundary acquisition unit is used to extract stimulus presentation timestamps and response timestamps from the interaction log; The trajectory alignment unit is used to align the time axis of the two-dimensional gaze trajectory with the absolute timestamp reference of the interaction log to obtain the aligned two-dimensional gaze trajectory. The time window extraction unit is used to extract the effective decision time window corresponding to each matching task from the aligned two-dimensional gaze trajectory, with the stimulus presentation timestamp as the starting boundary and the response timestamp as the ending boundary.

[0052] Understandably, the time boundary acquisition unit parses the absolute timestamp of stimulus presentation (i.e., the moment when the alternatives are fully displayed on the screen) and the absolute timestamp of the subject's effective response (i.e., the moment when the child taps the screen and the system determines it as a valid click) from the structured interaction log output by the touch interaction acquisition module. These two timestamps together constitute the task cycle boundary.

[0053] The trajectory alignment unit uses the high-precision clock (millisecond level) at the system's bottom layer as a unified benchmark. It performs point-by-point registration of the time label of each frame in the two-dimensional gaze trajectory sequence generated by the viewpoint estimation module with the timestamp of the interaction log event. This solves the time offset caused by different acquisition start times and frame rate fluctuations, so that the gaze trajectory can accurately fall within the corresponding task stage and stimulus presentation range.

[0054] Based on the aforementioned registration, the time window extraction unit precisely cuts out the effective decision time window that strictly corresponds to a single matching task from the aligned complete gaze trajectory sequence, using the stimulus presentation timestamp as the starting boundary and the effective response timestamp as the ending boundary. The gaze data of the effective decision time window completely covers the entire process from when the child sees the options, performs visual search and decision-making, to making a tapping response, eliminating interference data from non-decision stages such as task switching, interface loading, feedback prompts, and response completion, thus providing the gaze analysis module with strictly aligned gaze trajectory segments.

[0055] The line-of-sight analysis module includes: an interest area division unit, a first-viewpoint arrival time calculation unit, a duration ratio and frequency calculation unit, and a saccade path analysis unit; The interest area division unit is used to divide the screen into a target stimulation area, a target correct option area, and an interference option area, and to use the target stimulation area, the target correct option area, and the interference option area as preset interest areas of the screen. The first viewpoint arrival time calculation unit is used to measure the first viewpoint arrival time when the line of sight first falls on the target correct option area, with the stimulus presentation timestamp in the interaction log as zero point. The duration ratio and frequency calculation unit is used to count the number of frames in which the viewpoint falls within the preset interest area within the effective decision time window, and to calculate the gaze duration ratio and dwell frequency based on the corresponding number of frames. The scanning path analysis unit is used to count the number of times the gaze trajectory is not in the preset interest area or outside the screen, and to calculate the cumulative value of the spatial distance between consecutive effective viewpoints, and to use the cumulative value of the spatial distance as the total scanning path length.

[0056] In this embodiment, the interest area division unit divides the display area into three interest areas with independent semantics according to the screen layout: the location of the bubble on the cartoon character's head is the target stimulus area (displaying the target attribute or object that needs to be found), and the area where the 6 alternative options are located on the touch screen is marked as the target correct option area (the only correct target to hit) and the interference option area (the other 5 incorrect options).

[0057] The first gaze arrival time calculation unit uses the effective decision time window output by the temporal alignment module as the analysis range, takes the stimulus presentation timestamp in the interaction log as the absolute zero point, scans the first landing point of the gaze trajectory within the time window frame by frame, and records the length of time (in milliseconds) it takes for the landing point to first enter the target correct option area. This indicator directly reflects the subject's top-down visual search efficiency. For children with weak central integration features, this time is often significantly prolonged due to being attracted by the local features of distracting items.

[0058] Within the same effective decision time window, the duration ratio and frequency calculation unit accumulates the total number of frames in which the gaze point falls on each region of interest, and calculates the gaze time ratio of the target stimulus region, the gaze time ratio of the correct target option region, and the gaze time ratio of the interference option region respectively; at the same time, it counts the number of times the gaze repeatedly enters and exits each region of interest as the dwell frequency.

[0059] The saccadic path analysis unit identifies saccadic trajectories formed by continuous eye movements within the effective decision-making time window. On one hand, it counts the number of times the gaze falls on blank areas of the screen other than the three regions of interest (ROIs) or completely leaves the screen (e.g., looking outside the screen). Each time the gaze leaves all ROIs and remains there for more than a set threshold is counted as one saccadic movement. On the other hand, it calculates the distance between adjacent effective viewpoints and sums up the distances between all consecutive viewpoints within the entire time window to obtain the total saccadic path length. The frequent saccadic movements and the lengthy, disordered total saccadic path length together constitute important external behavioral markers for quantifying attentional difficulties in children with autism.

[0060] Furthermore, the evaluation result output module includes: an evaluation result generation unit and an output and storage unit; The assessment result generation unit is used to average or summarize the accuracy rate, the cross-stage accuracy rate difference, the average reaction time, the cross-stage reaction time difference, the reaction time coefficient of variation, the target first point of view arrival time, the fixation duration ratio, the dwell frequency, the number of gaze wanders, and the total saccadic path length to obtain the child's cognitive assessment result. The output and storage unit is used to generate attention regulation indicators based on the evaluation results, and to generate an evaluation report by combining the evaluation results, the attention regulation indicators, and the internal interaction features and gaze behavior features extracted at each evaluation stage, along with the child's personal dataset and the evaluation time, for printing and storage in the background database.

[0061] In this embodiment, the evaluation result generation unit performs statistical processing on the multidimensional quantitative indicators extracted from all matching rounds within a single evaluation stage (such as stage one to stage four of the color paradigm): for each round's independent indicators such as accuracy, average reaction time, target first point of view arrival time, fixation duration ratio, and number of wanderings, the arithmetic mean of these indicators within the stage is calculated; for cross-stage differences and reaction time variation coefficients, the values ​​calculated based on the entire stage rounds are directly retained; finally, the overall cognitive efficacy and attention regulation comprehensive index of the tested children in each evaluation stage is output.

[0062] The output and storage unit will generate a structured assessment report by combining the above-mentioned aggregated multidimensional quantitative indicators (including accuracy, cross-stage accuracy difference, average reaction time, cross-stage reaction time difference, reaction time coefficient of variation, target first point arrival time, fixation duration ratio, dwell frequency, number of gaze wanders, and total saccadic path length) with the basic personal information of the test children (such as age, gender, diagnostic labels), assessment date and time, and the type of paradigm used (color or shape), etc., and output it in paper or electronic document form as an objective clinical reference for this assessment.

[0063] Meanwhile, the output and storage unit will also store such report data and the underlying raw multimodal logs into the system's backend database, which will facilitate subsequent data processing and analysis, clinical follow-up, and archiving.

[0064] like Figure 4 As shown, based on the children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion described in the above embodiments, the present invention also provides a visual cognitive ability assessment method for children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion. The visual cognitive ability assessment method for children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion includes the following steps: Step S10: The touch interaction acquisition module presents a multi-stage visual cognitive assessment task to the child and collects the child's tapping operations on the touch screen to generate an interaction log.

[0065] Specifically, the evaluation paradigm is initiated on a desktop touchscreen device. A cartoon character and a thought bubble above their head are displayed on the left or center of the screen, with the target stimulus (such as a red circle or a real strawberry image) appearing inside the bubble. The remaining area of ​​the screen simultaneously displays six alternatives, and the child advances the process by tapping the correct option. The system backend records the moment each stimulus is presented, the moment the child taps, and whether the tap is correct with millisecond precision, generating a structured interaction log file.

[0066] Step S20: The video acquisition module synchronously records the child's facial video during the visual cognition assessment task.

[0067] In this embodiment, a high-definition camera fixed above the screen begins recording upon paradigm startup, capturing the child's facial expressions, head movements, and upper body posture throughout the process without interfering with natural interaction. The recorded video stream and interaction log share the same system clock reference, ensuring the accuracy of subsequent multimodal alignment. The video data primarily serves the viewpoint estimation module, used to extract eye gaze direction and head spatial pose.

[0068] Step S30: The interaction feature extraction module extracts internal interaction features from the interaction log. The internal interaction features include: accuracy, cross-stage accuracy difference, average reaction time, cross-stage reaction time difference, and reaction time variation coefficient.

[0069] Specifically, the event sequence in the interaction log is analyzed, and the basic matching accuracy of the child in each difficulty stage of the color and shape task is recorded respectively. The cross-stage accuracy difference between adjacent difficulty stages is calculated based on the basic matching accuracy.

[0070] Furthermore, the time taken from the presentation of the options to the child making a valid click is recorded. The average reaction time for each difficulty level is calculated based on the time taken, and the cross-level reaction time difference between adjacent difficulty levels is calculated. The reaction time variation coefficient is calculated based on the child's reaction time in each round within the same level to reflect the stability of the child's attention regulation.

[0071] Step S40: The viewpoint estimation module performs deep learning processing on the facial video, extracts facial feature points and three-dimensional viewpoint vectors, and maps the three-dimensional viewpoint vectors to a two-dimensional gaze trajectory on the screen based on the facial feature points.

[0072] It is understood that this embodiment uses a high-precision face tracking and viewpoint mapping algorithm based on deep learning to accurately map the child's gaze in physical space to screen coordinates in order to overcome motion interference in an unconstrained state.

[0073] Specifically, a deep neural network is used to detect hundreds of key facial points frame by frame to estimate the head's three-dimensional pose: pitch, yaw, and roll angles. Simultaneously, the center of the eyeball and the pupil region are segmented. Based on the prior knowledge that the physical size of the iris is constant, a three-dimensional gaze direction vector passing through the pupil from the center of the eyeball is calculated. Then, using head pose parameters, multinomial regression is used to project the gaze vector onto the screen, obtaining a continuous sequence of two-dimensional gaze coordinates, enabling eye tracking without wearable devices.

[0074] Step S50: The temporal alignment module aligns the two-dimensional gaze trajectory on the time axis using the stimulus presentation timestamp and response timestamp in the interaction log as boundaries, and extracts the effective decision time window corresponding to each matching task from the aligned two-dimensional gaze trajectory.

[0075] Specifically, stimulus presentation timestamps and response timestamps are extracted from the interaction log; the time axis of the two-dimensional gaze trajectory is aligned with the absolute timestamp reference of the interaction log to obtain the aligned two-dimensional gaze trajectory; with the stimulus presentation timestamp as the starting boundary and the response timestamp as the ending boundary, the effective decision time window corresponding to each matching task is extracted from the aligned two-dimensional gaze trajectory.

[0076] The gaze data within the effective decision-making time window fully covers the entire process from when a child sees the options, performs a visual search and decision, to making a tapping response. It eliminates interference data from non-decision-making stages such as task switching, interface loading, feedback prompts, and after the response is completed, thus providing the gaze analysis module with strictly aligned gaze trajectory segments.

[0077] Step S60: Within the effective decision-making time window, the gaze analysis module statistically analyzes the gaze behavior characteristics of the gaze falling on the preset interest area of ​​the screen. The gaze behavior characteristics include: the arrival time of the target first point of view, the gaze duration ratio, the dwell frequency, the number of gaze wanderings, and the total scanning path length.

[0078] Specifically, the screen is divided into a target stimulus area, a target correct option area, and a distraction option area, and the target stimulus area, the target correct option area, and the distraction option area are used as preset interest areas of the screen; the arrival time of the first gaze point when the gaze first falls on the target correct option area is measured with the stimulus presentation timestamp in the interaction log as zero point.

[0079] Furthermore, the number of frames in which the viewpoint falls within the preset interest area within the effective decision time window is counted, and the gaze duration ratio and dwell frequency are calculated based on the corresponding frame number; the number of times the gaze trajectory wanders outside the preset interest area or outside the screen is counted, and the cumulative value of the spatial distance between consecutive effective viewpoints is calculated, and the cumulative value of the spatial distance is used as the total scanning path length.

[0080] Step S70: The assessment result output module generates and outputs the child's cognitive assessment result based on the internal interaction characteristics and the gaze behavior characteristics.

[0081] In this embodiment, the accuracy, reaction time coefficient of variation, target first-look arrival time, and total saccade path length extracted from all rounds within the assessment phase are averaged or summarized within the phase. These metrics, along with the child's personal information, assessment time, and paradigm type, are integrated into a structured assessment report. The report can be printed locally or stored in a backend database. Simultaneously, the original interaction logs and video trajectory files are archived for clinical follow-up or subsequent analysis.

[0082] The beneficial effects of this invention are as follows: (1) This invention proposes a multi-stage progressive assessment paradigm that specializes in underlying cognitive mechanisms: specifically designed to induce color and shape cognition behaviors based on the recognized theories of "weak central integration" and "executive dysfunction" in autism in developmental psychology. By forcing subjects to switch between two progressive stages, "from abstract to abstract" basic matching and "from abstract to concrete" feature generalization, and using concrete everyday objects as visual interference, the invention specifically and quantitatively assesses the degree of impairment in feature generalization in children with autism, demonstrating strong targeting and high granularity of mechanism specialization.

[0083] (2) This invention innovatively introduces a highly immersive and interactive functional game paradigm, seamlessly integrating rigorous multi-stage cognitive testing logic into fun desktop touch games. This highly approachable design effectively reduces the defensiveness of children with the disease, significantly improves test participation and long-term compliance, thus laying an excellent interactive foundation for obtaining real and effective underlying behavioral data.

[0084] (3) This invention adopts a multi-stage progressive paradigm combined with a fully automated flow mechanism. Through multi-stage progressive testing, it specifically and deeply induces the subject's underlying cognitive impairment. At the same time, it relies on computer programs to achieve end-to-end fully automated flow from warm-up guidance to task-stage switching. The progressive paradigm mechanism is highly specialized and specific, while the fully automated paradigm flow completely eliminates physical and emotional interference from human intervention in the testing process. This not only greatly reduces the cost of human guidance but also ensures a high degree of objectivity and consistency in the induction environment.

[0085] (4) This invention introduces a high-precision viewpoint estimation modality. The gaze trajectory, as a key representation for understanding attention allocation and cognitive processing paths, is the core basis for the system to discover abnormal behavioral patterns. Based on this, the system comprehensively extracts multi-dimensional objective clinical indicators such as the arrival time of the target's first gaze point and the number of times the gaze wanders, realizing the dynamic and precise quantification of children's underlying cognitive deficits.

[0086] (5) This invention proposes a timeline alignment method based on the system's underlying interactive event logs. Utilizing the millisecond-level recorded stimulus presentation and the absolute timestamps of the subject's response, the effective decision-making time window of the external video eye-tracking behavior data is automatically and rigorously segmented on the timeline. This method achieves tight integration of internal interactive data and external video frame sequences without manual intervention, significantly reducing labor costs and offering high efficiency and convenience.

[0087] Furthermore, such as Figure 5 As shown, based on the above-mentioned child visual cognitive ability assessment system and method based on desktop touch and multimodal behavior fusion, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 5 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0088] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media array (SMC), secure digital card (SD), flash memory card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code installed on the terminal. The memory 20 may also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a visual cognitive ability assessment program 40 of a children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion. This visual cognitive ability assessment program 40 can be executed by the processor 10, thereby implementing the visual cognitive ability assessment method of the children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion in this application.

[0089] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the visual cognitive ability assessment method of the children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion.

[0090] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-screen liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visualized patient interface. The components of the terminal communicate with each other via a system bus.

[0091] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a visual cognitive ability assessment program of a children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion, wherein when the visual cognitive ability assessment program of the children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion is executed by a processor, it implements the steps of the visual cognitive ability assessment method of the children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion as described above.

[0092] In summary, this invention provides a system, method, terminal, and storage medium for assessing children's visual cognitive abilities based on desktop touch and multimodal behavior fusion. The system includes: a touch interaction acquisition module for presenting multi-stage assessment tasks and generating timestamped interaction logs; a video acquisition module for synchronously recording facial videos; an interaction feature extraction module for extracting behavioral indicators such as accuracy and reaction time from the logs; a viewpoint estimation module for mapping the video into a two-dimensional gaze trajectory using deep learning; a temporal alignment module for aligning with log timestamps and extracting effective decision time windows; a gaze analysis module for statistically analyzing gaze characteristics such as first gaze arrival time and gaze duration ratio within the time window; and an assessment result output module for fusing behavioral indicators and gaze characteristics to generate cognitive assessment results. This invention performs deep temporal alignment and multimodal fusion of high-precision system interaction event streams and continuous external video eye-tracking behavior data within the decision time window, thereby providing a quantitative cognitive ability assessment.

[0093] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0094] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The computer-readable storage medium can be a memory, magnetic disk, optical disk, etc.

[0095] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A system for assessing children's visual cognitive abilities based on desktop touch control and multimodal behavior fusion, characterized in that, The children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion includes: The touch interaction acquisition module is used to present children with multi-stage visual cognitive assessment tasks and collect the children's tapping operations on the touch screen to generate an interaction log. A video capture module is used to synchronously record the child's facial video during the visual cognitive assessment task; An interaction feature extraction module, connected to the touch interaction acquisition module, is used to extract internal interaction features from the interaction log. The internal interaction features include: accuracy, cross-stage accuracy difference, average reaction time, cross-stage reaction time difference, and reaction time variation coefficient. The viewpoint estimation module, connected to the video acquisition module, is used to perform deep learning processing on the facial video, extract facial feature points and three-dimensional viewpoint vectors, and map the three-dimensional viewpoint vectors to a two-dimensional gaze trajectory on the screen based on the facial feature points. The temporal alignment module is connected to the touch interaction acquisition module and the viewpoint estimation module respectively. It is used to align the two-dimensional gaze trajectory on the time axis with the stimulus presentation timestamp and response timestamp in the interaction log as the boundary, and extract the effective decision time window corresponding to each matching task from the aligned two-dimensional gaze trajectory. The gaze analysis module, connected to the time alignment module, is used to statistically analyze the gaze behavior characteristics of the gaze falling on the preset interest area of ​​the screen within the effective decision time window. The gaze behavior characteristics include: the arrival time of the target first viewpoint, the gaze duration ratio, the dwell frequency, the number of gaze wanderings, and the total scanning path length. The assessment result output module is connected to the interaction feature extraction module and the gaze analysis module, respectively, and is used to generate and output the child's cognitive assessment results based on the internal interaction features and the gaze behavior features.

2. The children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion as described in claim 1, characterized in that, The touch interaction acquisition module includes: a color attribute recognition unit, a shape attribute recognition unit, a turn control unit, a voice guidance unit, a single variable control unit, and a log generation unit; The color attribute recognition unit is used to present color recognition training tasks for six basic color attributes: red, yellow, blue, green, purple, and white. The shape attribute recognition unit is used to present shape recognition training tasks for six basic shape attributes: square, trapezoid, triangle, circle, ellipse and star. The round control unit is used to control independent matching tasks with a fixed number of rounds in each matching stage, and provides a stage transition interface after each stage ends. The stage transition interface includes a button to enter the next level and a button to settle and end the process. The voice guidance unit is used to play guidance voice when the task is presented, play success voice when the tap is correct, play error prompt voice when the tap is incorrect, and trigger double prompt when the response times out. The single-variable control unit is used to control the shape variable to be exactly the same in the color recognition training task, and to control the color variable to be exactly the same in the shape recognition training task. The log generation unit is used to record stimulus presentation timestamps, response timestamps, and click accuracy in the visual cognitive assessment task in the form of a structured log, and to generate the interaction log based on the stimulus presentation timestamps, response timestamps, and click accuracy.

3. The children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion as described in claim 1, characterized in that, The interactive feature extraction module includes: an accuracy calculation unit, a reaction time calculation unit, and a coefficient of variation calculation unit; The accuracy calculation unit is used to record the basic matching accuracy of the child at each difficulty level in the color and shape task, and to calculate the cross-stage accuracy difference between adjacent difficulty levels based on the basic matching accuracy. The reaction time calculation unit is used to record the time from the presentation of the options to the child making a valid click, calculate the average reaction time for each difficulty level based on the time, and calculate the cross-stage reaction time difference between adjacent difficulty levels. The coefficient of variation calculation unit is used to calculate the coefficient of variation of reaction time based on the child's reaction time in each round within the same stage.

4. The children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion according to claim 1, characterized in that, The viewpoint estimation module includes: a facial feature point localization unit, a head 3D pose calculation unit, a segmentation unit, a 3D viewpoint vector derivation unit, and a viewpoint coordinate mapping unit; The facial feature point localization unit is used to extract the face frame by frame from the facial video and locate the facial feature points. The head three-dimensional pose calculation unit is used to calculate the pitch angle, yaw angle and roll angle of the child's head based on the facial feature points; The segmentation unit is used to segment the eyeball center and pupil region using a convolutional neural network to obtain the eyeball center position and pupil center position; The three-dimensional viewpoint vector derivation unit is used to calculate the three-dimensional coordinates of the eyeball center and the pupil center respectively, based on the iris as the absolute reference, according to the eyeball center position and the pupil center position, and derive the three-dimensional viewpoint vector with the three-dimensional coordinates of the eyeball center as the starting point and the three-dimensional coordinates of the pupil center as the ending point. The viewpoint coordinate mapping unit is used to convert the three-dimensional viewpoint vector into a two-dimensional gaze trajectory on the screen using a multinomial regression algorithm or a homomorphic mapping matrix, based on the pitch angle, the yaw angle, and the roll angle.

5. The children's visual cognitive ability assessment system based on desktop touch control and multimodal behavior fusion according to claim 1, characterized in that, The timing alignment module includes: a time boundary acquisition unit, a trajectory alignment unit, and a time window extraction unit; The time boundary acquisition unit is used to extract stimulus presentation timestamps and response timestamps from the interaction log; The trajectory alignment unit is used to align the time axis of the two-dimensional gaze trajectory with the absolute timestamp reference of the interaction log to obtain the aligned two-dimensional gaze trajectory. The time window extraction unit is used to extract the effective decision time window corresponding to each matching task from the aligned two-dimensional gaze trajectory, with the stimulus presentation timestamp as the starting boundary and the response timestamp as the ending boundary.

6. The children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion according to claim 1, characterized in that, The line-of-sight analysis module includes: an interest area division unit, a first-viewpoint arrival time calculation unit, a duration ratio and frequency calculation unit, and a saccade path analysis unit; The interest area division unit is used to divide the screen into a target stimulation area, a target correct option area, and an interference option area, and to use the target stimulation area, the target correct option area, and the interference option area as preset interest areas of the screen. The first viewpoint arrival time calculation unit is used to measure the first viewpoint arrival time when the line of sight first falls on the target correct option area, with the stimulus presentation timestamp in the interaction log as zero point. The duration ratio and frequency calculation unit is used to count the number of frames in which the viewpoint falls within the preset interest area within the effective decision time window, and to calculate the gaze duration ratio and dwell frequency based on the corresponding number of frames. The scanning path analysis unit is used to count the number of times the gaze trajectory is not in the preset interest area or outside the screen, and to calculate the cumulative value of the spatial distance between consecutive effective viewpoints, and to use the cumulative value of the spatial distance as the total scanning path length.

7. The children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion according to claim 1, characterized in that, The evaluation result output module includes: an evaluation result generation unit and an output and storage unit; The assessment result generation unit is used to average or summarize the accuracy rate, the cross-stage accuracy rate difference, the average reaction time, the cross-stage reaction time difference, the reaction time coefficient of variation, the target first point of view arrival time, the fixation duration ratio, the dwell frequency, the number of gaze wanders, and the total saccadic path length to obtain the child's cognitive assessment result. The output and storage unit is used to generate attention regulation indicators based on the evaluation results, and to generate an evaluation report by combining the evaluation results, the attention regulation indicators, and the internal interaction features and gaze behavior features extracted at each evaluation stage, along with the child's personal dataset and the evaluation time, for printing and storage in the background database.

8. A visual cognitive ability assessment method based on the child visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion as described in any one of claims 1-7, characterized in that, The visual cognitive ability assessment method includes: The touch interaction acquisition module presents children with multi-stage visual cognitive assessment tasks and collects the children's tapping operations on the touch screen to generate an interaction log. The video capture module synchronously records the child's facial video during the visual cognition assessment task; The interaction feature extraction module extracts internal interaction features from the interaction log. The internal interaction features include: accuracy, cross-stage accuracy difference, average reaction time, cross-stage reaction time difference, and reaction time coefficient of variation. The viewpoint estimation module performs deep learning processing on the facial video to extract facial feature points and a three-dimensional viewpoint vector, and maps the three-dimensional viewpoint vector to a two-dimensional gaze trajectory on the screen based on the facial feature points. The temporal alignment module uses the stimulus presentation timestamp and response timestamp in the interaction log as boundaries to align the two-dimensional gaze trajectory on the time axis, and extracts the effective decision time window corresponding to each matching task from the aligned two-dimensional gaze trajectory. The gaze analysis module statistically analyzes the gaze behavior characteristics of the gaze falling on the preset interest area of ​​the screen within the effective decision time window. The gaze behavior characteristics include: the arrival time of the first gaze point, the gaze duration ratio, the dwell frequency, the number of gaze wanderings, and the total saccade path length. The assessment result output module generates and outputs the child's cognitive assessment results based on the internal interaction characteristics and the gaze behavior characteristics.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a visual cognitive ability assessment program of a children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion, stored in the memory and executable on the processor. When the visual cognitive ability assessment program of the children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion is executed by the processor, it implements the steps of the visual cognitive ability assessment method of the children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a visual cognitive ability assessment program for a children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion. When the visual cognitive ability assessment program for a children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion is executed by a processor, it implements the steps of the visual cognitive ability assessment method for a children's visual cognitive ability assessment system based on desktop touch and multimodal behavior fusion as described in claim 8.

Images