A method and system for assessing cognitive impairment

Abstract

The application discloses a cognitive impairment evaluation method and system, and relates to the technical field of medical detection and artificial intelligence, and comprises the following steps: collecting eye movement data and voice data of a subject under a plurality of preset paradigms; performing feature extraction on the eye movement data to obtain a plurality of eye movement numerical features and an eye movement heat map; inputting an eye movement splicing vector into a pre-trained eye movement feature extraction model to obtain an eye movement feature vector; converting the voice data into a pinyin sequence, performing feature extraction on the pinyin sequence to obtain semantic features; performing feature extraction on the voice data to obtain acoustic features; inputting an audio feature vector into a pre-trained voice feature extraction model to obtain a voice feature vector; and inputting the eye movement feature vector and the voice feature vector into a pre-trained classification model to obtain an evaluation score of cognitive impairment. The application solves the problems of feature independence and fusion difficulty in the prior art, and significantly improves the accuracy of cognitive impairment evaluation.

Description

Technical Field

[0001] This invention relates to the fields of medical testing and artificial intelligence technology, and in particular to a method and system for assessing cognitive impairment. Background Technology

[0002] Cognitive impairment is a common group of neurological dysfunctions, including Alzheimer's disease (AD), mild cognitive impairment (MCI), vascular dementia, and other types of dementia syndromes. These diseases are typically characterized by memory loss, inattention, language impairment, executive function decline, and behavioral abnormalities, which can severely impact a patient's social functioning and daily living abilities. Currently, there is no cure for cognitive impairment; existing pharmacological and non-pharmacological interventions primarily aim to alleviate symptoms, slow disease progression, and improve patients' quality of life. Drug treatment largely relies on regulating neurotransmitter levels, while non-pharmacological interventions include cognitive training, social support, healthy lifestyle choices, and daily care. Therefore, early screening and diagnosis of cognitive impairment are of paramount importance.

[0003] Current assessment methods primarily rely on clinical exclusion, combined with neurological evaluation, biochemical testing (such as cerebrospinal fluid analysis), brain imaging (such as fMRI and PET), and neuropsychological assessments (such as MoCA and MMSE). While these methods are widely used in clinical practice, they generally suffer from drawbacks such as high invasiveness, high cost, and complex operation, limiting their application in large-scale early screening.

[0004] Studies have shown that patients with cognitive impairment often exhibit abnormal eye movement control (such as shortened fixation time, abnormal fixation frequency, and abnormal saccade patterns) and altered language characteristics (such as decreased speech rate, monotone tone, decreased fluency, increased pauses, and reduced accuracy in answering questions). These characteristics are considered important early markers distinguishing normal aging from cognitive impairment. In recent years, with the development of eye-tracking technology and automatic speech recognition technology, academia and industry have been attempting to extract multiple features related to cognitive impairment from eye-tracking and speech data for auxiliary diagnosis and risk prediction. However, these features are independent of each other, and existing methods struggle to efficiently fuse them, failing to fully utilize the information contained in each feature and comprehensively reflect the patient's cognitive abnormalities, resulting in low assessment accuracy. Summary of the Invention

[0005] Based on the shortcomings of the existing technology, the present invention provides a cognitive impairment assessment method and system, which solves the problems that the existing methods cannot efficiently integrate multiple features, fail to make full use of the information contained in each feature, and are difficult to comprehensively reflect the patient's cognitive abnormality.

[0006] The present invention adopts the following technical solution: In a first aspect, the present invention provides a method for assessing cognitive impairment, comprising the following steps: Eye movement and speech data were collected from subjects under multiple pre-defined paradigms; Feature extraction is performed on eye-tracking data to obtain multiple numerical eye-tracking features and an eye-tracking heatmap; deep feature extraction is performed on the multiple numerical eye-tracking features to obtain a one-dimensional feature vector; spatial feature extraction is performed on the eye-tracking heatmap to obtain a spatial feature vector; the one-dimensional feature vector and the spatial feature vector are concatenated to obtain an eye-tracking concatenated vector; the eye-tracking concatenated vector is input into a pre-trained eye-tracking feature extraction model to obtain an eye-tracking feature vector. The speech data is converted into a Pinyin sequence, and features are extracted from the Pinyin sequence to obtain semantic features; features are extracted from the speech data to obtain acoustic features; the semantic features and acoustic features are concatenated to obtain an audio feature vector; the audio feature vector is input into a pre-trained speech feature extraction model to obtain a speech feature vector. Eye-tracking feature vectors and speech feature vectors are input into a pre-trained classification model to obtain an assessment score for cognitive impairment.

[0007] Preferably, the eye-tracking feature extraction model employs a multilayer perceptron, and its pre-training includes the following steps: Collect multiple historical eye movement data and corresponding cognitive function scale scores from different patients under multiple preset paradigms; Feature extraction is performed on each historical eye-tracking data point to obtain the corresponding eye-tracking stitched vector; The multilayer perceptron was pre-trained using different eye-tracking splicing vectors as input and corresponding cognitive function scale scores as output.

[0008] Preferably, the speech feature extraction model adopts the Conv1D-Attention-BiLSTM model, and its pre-training includes the following steps: Collect multiple historical speech data and corresponding cognitive function scale scores from different patients under multiple preset paradigms; Feature extraction is performed on each historical speech data to obtain the corresponding audio feature vector; The Conv1D-Attention-BiLSTM model is pre-trained using different audio splicing vectors as input and corresponding cognitive function scale scores as output.

[0009] Preferably, after inputting the eye-tracking splicing vector into the pre-trained eye-tracking feature extraction model, the feature vector before the classification layer of the multilayer perceptron is used as the eye-tracking feature vector; after inputting the audio feature vector into the pre-trained speech feature extraction model, the feature vector output by the classification head of the Conv1D-Attention-BiLSTM model is used as the speech feature vector.

[0010] Preferably, the classification model includes a gating fusion module and a classifier; the step of inputting eye-tracking feature vectors and speech feature vectors into the pre-trained classification model to obtain an assessment score for cognitive impairment specifically includes the following steps: The gating fusion module generates corresponding gating vector values ​​for eye-tracking feature vectors and speech feature vectors, and fuses the eye-tracking feature vectors and speech feature vectors with the corresponding gating vector values ​​to obtain fused feature vectors; The classifier classifies the fused feature vectors to obtain an assessment score for cognitive impairment.

[0011] Preferably, the preset paradigms include a picture-based memory voice question-and-answer paradigm, an eye saccade paradigm, an eye movement smooth tracking paradigm, a free view paradigm, a sentence reading paradigm, a text reading paradigm, and a picture-based speech voice question-and-answer paradigm.

[0012] Preferably, the eye movement data includes fixation point, eye position, head posture, and head position, and the eye movement numerical features include average number of fixations, average number of saccades, total saccade path, fixation time, average fixation duration, average saccade amplitude, reaction initiation time, reaction execution speed, average minimum gain, number of erroneous saccades, fixation detachment time from stimulus point, speed of follow-up initiation phase, number of fixations during follow-up, speed difference during follow-up, and position difference during follow-up.

[0013] Preferably, the semantic features include modal word count, expression of forgotten words, and expression of hesitant speech, and the acoustic features include Fbank features, PLP features, Bottleneck features, and temporal acoustic features.

[0014] Preferably, before collecting eye movement and speech data from subjects under multiple preset paradigms, head posture calibration, eye movement calibration, and microphone calibration should be performed on the subjects.

[0015] Secondly, the present invention provides a cognitive impairment assessment system, comprising: The acquisition module is used to collect eye movement data and speech data of subjects under multiple preset paradigms; The first extraction module is used to extract features from eye-tracking data to obtain multiple numerical eye-tracking features and an eye-tracking heatmap; to perform deep feature extraction on the multiple numerical eye-tracking features to obtain a one-dimensional feature vector; to perform spatial feature extraction on the eye-tracking heatmap to obtain a spatial feature vector; to concatenate the one-dimensional feature vector and the spatial feature vector to obtain an eye-tracking concatenation vector; and to input the eye-tracking concatenation vector into a pre-trained eye-tracking feature extraction model to obtain an eye-tracking feature vector. The second extraction module is used to convert speech data into a pinyin sequence, extract features from the pinyin sequence to obtain semantic features, extract features from the speech data to obtain acoustic features, concatenate the semantic features and acoustic features to obtain an audio feature vector, and input the audio feature vector into a pre-trained speech feature extraction model to obtain a speech feature vector. The evaluation module is used to input eye-tracking feature vectors and speech feature vectors into a pre-trained classification model to obtain an evaluation score for cognitive impairment.

[0016] Compared with the prior art, the above-mentioned at least one technical solution adopted by the present invention can achieve the following beneficial effects: This invention extracts features from numerical eye movement features and eye movement heatmaps using an eye movement feature extraction model, achieving efficient fusion of eye movement features. It also achieves efficient fusion of semantic and acoustic features using a speech feature extraction model. Furthermore, it fuses eye movement and speech feature vectors using a classification model, realizing feature extraction and fusion across different modalities of eye movement and speech features. This allows for a more comprehensive capture of abnormal behaviors in visual attention and language expression in patients with cognitive impairment. It solves the problems of feature independence and fusion difficulties in existing methods, fully utilizing complementary information between different modalities. This significantly improves the accuracy of cognitive impairment assessment. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a cognitive impairment assessment method according to the present invention; Figure 2 This is a flowchart of a cognitive impairment assessment method according to the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] This invention provides a cognitive impairment assessment method, specifically a multimodal cognitive impairment assessment method based on eye movement and speech fusion, referring to... Figure 1and Figure 2 This includes the following steps: S1: Input and save the subject's personal information.

[0021] Personal information includes the subject's age, gender, education level, and scores on cognitive function scales, all of which are important characteristics for measuring a person's memory ability.

[0022] S2: Perform head posture calibration, eye movement calibration, and microphone calibration on the subject.

[0023] This device, designed as a mobile medical workstation, is a multimodal cognitive assessment system integrating hardware and software algorithms. The monitor stand can rotate 360° and has a vertical movement range of approximately 12.5cm. The monitor's center height ranges from approximately 1.1m to 1.4m, and it includes built-in cable management channels and connection points. The monitor displays various task paradigms and test reports. The eye-tracking data acquisition device, the core hardware module, is installed at the bottom of the monitor and is used to collect raw eye-tracking data. A camera, installed at the top of the monitor, records head posture data. Data acquisition is achieved by adjusting the seat height, selecting a suitable monitor size, and adjusting the camera angle.

[0024] The eye-tracking data acquisition device is based on non-invasive infrared imaging and optical tracking technology. It uses an infrared light source to illuminate and detect corneal reflection points, and combines a pupil center positioning algorithm to track the positions of the pupil and corneal reflection points in real time. It uses optical calculation methods to calculate the direction of gaze and gaze coordinates, thereby achieving high-precision capture of the subject's gaze point position.

[0025] Subject positioning: The subject sits in front of the device, maintaining an upright posture with the head relatively fixed, about 30cm to 50cm away from the display screen.

[0026] Head posture calibration: The device's camera captures facial feature points of the subject in real time to detect the head posture relative to the screen. When the head position is stable and within a preset range, the head posture calibration is considered successful.

[0027] Eye-tracking calibration: Four cue points are presented sequentially on the display screen, distributed in the center and a triangular area. The subject must fixate on each cue point until it disappears. After completing the fixation task, the calibration calculation phase begins. A three-dimensional eye localization algorithm is used to estimate the subject's gaze direction. This algorithm employs the Pupil-Corneal Reflection (PCCR) method, calculating the gaze direction by the relative displacement between the pupil center and the corneal reflection point. A head motion compensation mechanism is introduced during calibration, using a multi-camera configuration to perform triangulation, thereby estimating real-time changes in the subject's head position and dynamically correcting the gaze point coordinates to ensure the accuracy and robustness of the calibration results. Eye-tracking calibration is considered successful when the average gaze error at the calibration verification points is less than 0.5° of visual angle.

[0028] Microphone Calibration: After successful eye-tracking calibration, adjust the microphone to be level with the subject's mouth, at a distance of approximately 10cm to 20cm. Activate the sound level calibrator, outputting a 94dB SPL, 1kHz sine wave signal, which is then acquired by the microphone. The system calculates the effective voltage of the microphone output and compares it with a reference sound pressure level to obtain the microphone's sensitivity coefficient. This sensitivity coefficient is stored in the system and used as a conversion factor for converting subsequent speech data into actual sound pressure levels, thus completing the microphone calibration.

[0029] S3: Data acquisition based on multiple measurement paradigms.

[0030] The acquisition of raw eye-tracking and audio data relies on various paradigm tasks, including image-based memory and speech-based question-and-answer, saccade, smooth eye-tracking, free-view, sentence reading, text reading, and image-based speech-based question-and-answer. Each paradigm includes a transition page containing the test requirements and accompanying audio prompts. Additionally, in the saccade paradigm, different colored dots were replaced with objects of different shapes to avoid color blindness.

[0031] In the picture-based memory and voice-based question-and-answer paradigm, students learn and memorize pictures of fruits, animals, objects, and vehicles in sequence, followed by recalling all the pictures and identifying any that were not previously shown.

[0032] The saccade paradigm includes anterior saccade tasks and posterior saccade tasks. In the anterior saccade task, participants focus on a green dot displayed in the center of an initial screen, followed by red target dots appearing on either side. Participants must quickly and accurately move their gaze to the target dots (without moving their head). In the posterior saccade task, participants focus on a green dot displayed in the center of an initial screen, followed by a yellow target dot appearing on one side. Participants must suppress the reflexive urge to look at the target and instead move their gaze to the area opposite the target.

[0033] In the eye-tracking smoothing paradigm, subjects need to follow the movement of a red target point with a diameter of 22 pixels on the screen, and the target movement pattern is a sine curve.

[0034] In the free-view paradigm, four images are played sequentially, each for four seconds, and the focus of the subject's gaze is recorded when viewing the images.

[0035] In the sentence reading paradigm, participants are required to read sentences (including short and long sentences) that appear on the screen in sequence, extract reading pattern features, and assess language processing, visual cognition, and other functions in patients with neurodegenerative diseases.

[0036] In the reading passage paradigm, a passage is displayed on the device screen and the corresponding audio is played. After the audio is finished, three questions related to the passage are displayed.

[0037] In the picture-based speech-based question-and-answer paradigm, the task requires participants to observe a picture depicting an everyday scene and freely describe its content within one minute, without specific prompts. This assesses cognitive functions such as visual attention, spontaneous language generation, and attention shifting. Analysis of fixation point distribution and saccade patterns reflects the patient's visual exploration and cognitive integration abilities.

[0038] S4: Acquisition of raw eye-tracking data.

[0039] All paradigms of S3 can be used to collect raw eye-tracking data. When entering each paradigm, the data collection begins and a timestamp is recorded. When leaving a paradigm, the data collection ends and a timestamp is recorded.

[0040] The raw eye-tracking data consists of four categories: eye position (GazePoint), eye position (EyePosition), head pose (HeadPose), and head position (HeadPosition).

[0041] The gaze point (GazePoint) is the core data source for subsequent extraction of eye movement features (number of fixations, saccade amplitude, tracking features, etc.); among them, the gaze fixation is the coordinate data of the gaze point when the subject is stationary and gazing at the target, which is extracted from the gaze point (GazePoint).

[0042] EyePosition refers to the two-dimensional planar position of the eyeball in the facial video coordinate system.

[0043] Head pose is the rotational state of the head in three-dimensional space, represented by Euler angles: Pitch (xDeg): head down / head up; Yaw (yDeg): Turn your head left or right; Roll (zDeg): Tilting one's head.

[0044] Head position is the translational position of the head in three-dimensional space.

[0045] EyePosition, HeadPose, and HeadPosition together form the physical and geometric basis for generating GazePoint, used to determine the position of the gaze point and explain whether changes in GazePoint are caused by eye movements, head rotation, or head translation.

[0046] Eye-tracking data is stored in the root directory of the data file, under the format "Collection Number / Tobiidata / Classification / Paradox.txt". GazePoint: Timestamp, x, y; Fixation: "Fixation started at X: {0}, Y: {1}", x, y; "During fixation, currently at: X: {0}, Y: {1}", x, y; "Fixation ended at X: {0}, Y: {1}", x, y; EyePosition: Timestamp, LeftEye.X, LeftEye.Y, LeftEye.Z; Timestamp, LeftEyeNormalized.X,LeftEyeNormalized.Y, LeftEyeNormalized.Z; Timestamp, RightEye.X, RightEye.Y, RightEye.Z; Timestamps, RightEyeNormalized.X, RightEyeNormalized.Y, RightEyeNormalized.Z; HeadPose: Timestamp, xDeg, yDeg, zDeg; HeadPosition: Timestamp, X, Y, Z.

[0047] Eye-tracking data acquisition devices utilize eye-tracking three-dimensional localization algorithms: Pupil-Corneal Reflection (PCCR): This method calculates the direction of vision by the relative displacement between the pupil center and the corneal reflection point. The formula is: ; in: θ Δ is the angle of line of sight deflection. x Δ y This represents the pixel offset between the pupil center and the corneal reflection point. d This is the distance from the center of eye rotation to the apex of the cornea (default 12mm). f This refers to the camera's focal length.

[0048] Head motion compensation: The eye-tracking data acquisition device supports multiple cameras and can calculate head position changes through triangulation to dynamically correct the gaze point coordinates.

[0049] S5: Extract eye-movement-related features based on the acquired eye-movement data to form an eye-movement feature dataset.

[0050] We extracted feature data such as average fixation count, average saccade count, total saccade path, fixation time, average fixation duration, and average saccade amplitude from the raw eye-tracking data obtained from the image-based memory speech question-and-answer paradigm, sentence reading paradigm, and text reading paradigm.

[0051] Feature data such as reaction initiation time, reaction execution speed, average minimum gain, number of erroneous saccades, duration of gaze detachment from stimulus point, speed of follow-up initiation phase, number of gazes during follow-up, speed difference during follow-up, and position difference during follow-up are extracted from the raw eye movement data obtained by the saccade paradigm and the smooth eye movement tracking paradigm.

[0052] In the raw eye-tracking data obtained by the free-view paradigm, feature data such as average number of fixations, fixation time, average fixation duration, and total saccade path are extracted.

[0053] In the raw eye-tracking data obtained from the picture-based speech-based question-and-answer paradigm, eye-tracking heatmap feature data of the subjects are extracted to reflect the distribution of the subjects' visual exploration attention.

[0054] The eye movement characteristics are shown in Table 1.

[0055] Table 1 Eye movement characteristics Eye-tracking heatmaps are images that visualize the location and duration of eye fixations using color intensity. The generation method involves using a background of the experimental interface and color (e.g., from red to darker) to represent the density of fixation duration / frequency; the longer the duration / more the frequency, the darker the color. The significance of this experiment lies in its ability to visually display the hotspots of the subjects' attention, allowing for quick assessment of whether they are focusing on key areas; it is a core tool for qualitative analysis.

[0056] All eye movement feature data were acquired based on both static and dynamic eye movements, primarily considering four eye movement states: saccades, smooth pursuit, blinking, and fixation. Saccades are rapid eye movements from one point to another, typically lasting less than 100 ms and exceeding 30° / s in speed; smooth pursuit is a slow and steady eye movement accompanying the tracking of a stimulus; blinking refers to the rapid closing and opening of the eyelids; fixation is a steady eye movement that keeps the retina focused on a stationary object of interest, typically lasting more than 100 milliseconds.

[0057] Saccade recognition: A threshold of 40 pixels (0.91°) is used to represent saccade distance (TSD), 30° / s represents saccade velocity (TSV), and 6000° / s represents saccade acceleration (TSA). First, regions with velocities higher than TSV are identified to determine initial saccade candidate segments, and consecutive candidate segments with intervals shorter than 20ms are merged. When the acceleration is greater than TSA, the start and end points of the candidate segment are extended. If the distance between the start and end is greater than TSD, it is determined to be a saccade, thus discarding microsaccades and artifacts.

[0058] Smooth tracking segment recognition: A 50-pixel threshold (1.1°) is used to set the lower limit of smooth tracking distance (TPD) and speed (TPV) to 5° / s, and a new scanning speed threshold of 60° / s (TSPV) is set. If the speed is always greater than TPV and less than TSPV during the movement, and the distance between the start and end points is greater than TPD, then it is determined to be smooth tracking movement.

[0059] Blink Segment Recognition: Candidate segments are evaluated by checking whether no eye position is detected, and continuous blinks with an interval of less than 50ms are merged. Since eyelid closure and opening can interfere with gaze position detection, the algorithm classifies and identifies blink segments from the beginning to the end of a sudden change in gaze position around data without eye movement.

[0060] Gaze segment identification: All segments that are not saccades, smooth pursuits, or blinks are marked as gazes. Additionally, if quality checks detect blinks or missed gaze coordinates for more than 20% of the total test time, the test is discarded.

[0061] S6: Eye-tracking feature fusion and model training, the specific steps are as follows: Deep learning models (such as autoencoders) are used to extract deep features from one-dimensional eye-tracking feature data (15 types of numerical features other than eye-tracking heatmaps in Table 1). The encoder maps the high-dimensional original numerical features into low-dimensional, information-condensed one-dimensional eye-tracking deep feature vectors.

[0062] Using convolutional neural networks (such as ResNet) to extract spatial features from eye-tracking heatmaps, capturing local correlations and global hotspot information of gaze point distribution, and obtaining deep feature vectors of the heatmap.

[0063] The one-dimensional eye-tracking deep feature vector and the heatmap deep feature vector are concatenated and fused to construct an eye-tracking feature dataset.

[0064] The patient's cognitive function scale score was used as the true label of the model. The eye movement features were input into a multilayer perceptron (MLP) for training to obtain the optimal eye movement feature extraction model.

[0065] Based on this optimal model, extract the feature vector (denoted as C) before the output classification layer. eye This vector comprehensively encodes the numerical statistical features of eye movements and the spatial features of heatmaps, serving as a standardized intermediate representation for multimodal feature fusion.

[0066] S7: Raw speech data is collected through a specified paradigm task, specifically including: Image-based memory and voice-based question-and-answer paradigm: Subjects are asked to look at objects (fruits, animals, vehicles, etc.) on a screen within a specified time and then recall and say the names of the objects on the screen.

[0067] Sentence reading paradigm: Subjects are asked to read text (including short and long sentences) that appears on the screen within a specified time.

[0068] Picture description and voice questioning paradigm: Participants are asked to observe a picture depicting a scene from daily life and freely describe the content of the picture within 1 minute.

[0069] The raw audio data collected using the above paradigm is saved in WAV format with a sampling rate of 15kHz, a sampling depth of 16bit, and a mono sampling mode. The audio data is saved in the data root directory, and the file naming format is "collection sequence number / Audiodata / paradigm_detection item number.wav".

[0070] S8: Speech transcription and data preprocessing.

[0071] The collected subject speech samples (wav format) were converted to mp3 format using Python's AudioSegment module for use in the speech-to-text function.

[0072] The transcoded audio signal is preprocessed: spectral subtraction is used to remove environmental noise, and energy thresholding is used to trim invalid silent segments.

[0073] The preprocessed audio signal is transcribed into Chinese text.

[0074] The converted Chinese text is then converted into a pinyin sequence using a pinyin conversion tool (such as pypinyin).

[0075] To address the issue of insufficient data size in the original audio dataset, data augmentation techniques are employed to expand the dataset: Pinyin sequence enhancement: Random word replacement, random deletion, and random word repetition (all with a probability of 0.2) are used to simulate speech impediments and forgetfulness in the subjects. Numerical feature enhancement: Gaussian noise (probability set to 0.05) is added to the subsequently extracted audio numerical features to augment the data.

[0076] S9: Extract speech acoustic and semantic features from the collected subject speech samples, fuse all extracted features, and construct an audio fusion feature dataset. This includes the following steps: Fbank features: The pre-processed speech signal is sequentially pre-emphasized, framed, and windowed. The spectral energy is extracted through the Mel filter bank, and the logarithm of the energy value is used to form the Fbank features. PLP features: The speech signal is converted to the Bark frequency range, pre-emphasized using equal loudness curves, and PLP features are formed after calculating the cepstral coefficients. Bottleneck feature: Construct a neural network with a small number of hidden layer nodes, train the model with speech signals as input, and extract the output of the bottleneck layer of the network as the Bottleneck feature; Temporal acoustic features: Statistically extract total duration, silence time, percentage of silence time, number of pauses when answering questions, average interval time, standard deviation of interval time, speech rate, and voice stability; Semantic feature extraction: Based on the Pinyin sequence transcribed by S8, semantic features are extracted through custom dictionary matching and statistical methods: the number of modal words (such as “ah”, “en”, “Ahem”) is counted, the expression of forgotten words (such as “xiang bu qi”, “ji buzhu”, “jiang bu chu”, “ji bu qing”, “wang ji”) and the expression of hesitant words (such as “zhe ge”, “shen me”, “na ge”) are identified and counted.

[0077] All the above-mentioned speech acoustic features (Fbank, PLP, Bottleneck, and temporal acoustic features) and semantic features were standardized (to eliminate dimensional differences); the standardized speech acoustic feature vectors and semantic feature vectors were concatenated to construct the audio feature dataset. The audio features are shown in Table 2.

[0078] Table 2 Audio Features S10: Speech feature fusion and model training, the specific steps are as follows: Use an audio feature dataset as input to train a hybrid temporal feature extraction model (such as Conv1D-Attention-BiLSTM); The patient's cognitive function scale score is used as the true label of the model, and the cognitive function state is used as the training target for model training. A hierarchical hybrid architecture is adopted: a one-dimensional convolutional layer (Conv1D) automatically extracts the local acoustic patterns of the audio feature sequence.

[0079] Temporal attention mechanism dynamically weights the importance of features at different time steps to enhance the contribution of key temporal features; Bidirectional Long Short-Term Memory Network (BiLSTM) captures long-distance forward and backward contextual dependencies in feature sequences; Two fully connected (FC) layers are connected after the BiLSTM output to serve as the classification head.

[0080] Based on this optimal model, extract the feature vector (denoted as C) from the output of the first layer of the classification head. voi This vector comprehensively encodes key information about speech in terms of acoustics, prosody, temporal structure, and semantics, serving as a standardized intermediate representation for multimodal feature fusion and subsequent multimodal model fusion and comprehensive decision-making.

[0081] S11: The feature vector output by the eye-tracking feature extraction model and the speech feature vector output by the speech feature extraction model are fused using feature-level adaptive gating. This is then combined with an XGBoost classifier to complete the final cognitive impairment assessment. The specific process is as follows: First, obtain the 128-dimensional eye-tracking feature vector output by the eye-tracking model (denoted as C). eye ) and the 128-dimensional speech feature vector (denoted as C) output by the speech model voi The core of the gated fusion mechanism is the introduction of a learnable "gate" network to dynamically adjust and select information flows from different modalities. The gate network calculates a gate value vector g (with the same dimension as the feature vector) between 0 and 1 based on the joint context of the current input. The final fused feature vector F is generated by the following formula:

[0082] F=g⊙C eye +(1−g)⊙Cvoi ; Here, ⊙ represents element-wise multiplication, which means that for each dimension of the feature vector, the gating mechanism can autonomously decide to "adopt" more information from eye-tracking or speech modalities, achieving fine-grained, adaptive feature fusion, rather than simple concatenation or fixed weight addition; this gating network will be trained end-to-end together with the subsequent classifier.

[0083] Finally, the XGBoost classifier was adopted as the decision layer. This classifier takes the fused feature vector F as input and focuses on training to integrate complementary information from eye movement and speech modalities to predict the cognitive function status of the subjects. This approach forms a complete technical chain of "raw data acquisition → multi-dimensional feature extraction → feature-level adaptive fusion → decision layer classification," ensuring logical consistency and process integrity from data input to evaluation result output, effectively improving the accuracy and reliability of early screening for cognitive impairment.

[0084] Based on the same concept, the present invention also provides a cognitive impairment assessment system, including a data acquisition module, a first extraction module, a second extraction module, and an assessment module.

[0085] The acquisition module is used to collect eye movement and speech data of subjects under multiple preset paradigms.

[0086] The first extraction module is used to extract features from eye-tracking data, obtaining multiple numerical eye-tracking features and an eye-tracking heatmap; deep feature extraction is performed on the multiple numerical eye-tracking features to obtain a one-dimensional feature vector; spatial feature extraction is performed on the eye-tracking heatmap to obtain a spatial feature vector; the one-dimensional feature vector and the spatial feature vector are concatenated to obtain an eye-tracking concatenated vector; the eye-tracking concatenated vector is input into a pre-trained eye-tracking feature extraction model to obtain an eye-tracking feature vector.

[0087] The second extraction module is used to convert speech data into a pinyin sequence, extract features from the pinyin sequence to obtain semantic features, extract features from the speech data to obtain acoustic features, concatenate the semantic features and acoustic features to obtain an audio feature vector, and input the audio feature vector into a pre-trained speech feature extraction model to obtain a speech feature vector.

[0088] The evaluation module is used to input eye-tracking feature vectors and speech feature vectors into a pre-trained classification model to obtain an evaluation score for cognitive impairment.

[0089] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0090] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A cognitive impairment assessment system, characterized in that, include: The acquisition module is used to collect eye movement data and speech data of subjects under multiple preset paradigms; The first extraction module is used to extract features from eye-tracking data to obtain multiple numerical eye-tracking features and an eye-tracking heatmap; to perform deep feature extraction on the multiple numerical eye-tracking features to obtain a one-dimensional feature vector; to perform spatial feature extraction on the eye-tracking heatmap to obtain a spatial feature vector; to concatenate the one-dimensional feature vector and the spatial feature vector to obtain an eye-tracking concatenation vector; and to input the eye-tracking concatenation vector into a pre-trained eye-tracking feature extraction model to obtain an eye-tracking feature vector. The second extraction module is used to convert speech data into a pinyin sequence, extract features from the pinyin sequence to obtain semantic features, extract features from the speech data to obtain acoustic features, concatenate the semantic features and acoustic features to obtain an audio feature vector, and input the audio feature vector into a pre-trained speech feature extraction model to obtain a speech feature vector. The evaluation module is used to input eye-tracking feature vectors and speech feature vectors into a pre-trained classification model to obtain an evaluation score for cognitive impairment. The eye-tracking feature extraction model employs a multilayer perceptron, and its pre-training includes the following steps: Collect multiple historical eye movement data and corresponding cognitive function scale scores from different patients under multiple preset paradigms; Feature extraction is performed on each historical eye-tracking data point to obtain the corresponding eye-tracking stitched vector; The multilayer perceptron was pre-trained using different eye-tracking splicing vectors as input and corresponding cognitive function scale scores as output. The speech feature extraction model uses the Conv1D-Attention-BiLSTM model, and its pre-training includes the following steps: Collect multiple historical speech data and corresponding cognitive function scale scores from different patients under multiple preset paradigms; Feature extraction is performed on each historical speech data to obtain the corresponding audio feature vector; The Conv1D-Attention-BiLSTM model is pre-trained using different audio splicing vectors as input and corresponding cognitive function scale scores as output. After inputting the eye-tracking splicing vector into the pre-trained eye-tracking feature extraction model, the feature vector before the classification layer of the multilayer perceptron is used as the eye-tracking feature vector; after inputting the audio feature vector into the pre-trained speech feature extraction model, the feature vector output by the classification head of the Conv1D-Attention-BiLSTM model is used as the speech feature vector.

2. The cognitive impairment assessment system as described in claim 1, characterized in that, The classification model includes a gating fusion module and a classifier; the step of inputting eye-tracking feature vectors and speech feature vectors into the pre-trained classification model to obtain an assessment score for cognitive impairment specifically includes the following steps: The gating fusion module generates corresponding gating vector values ​​for eye-tracking feature vectors and speech feature vectors, and fuses the eye-tracking feature vectors and speech feature vectors with the corresponding gating vector values ​​to obtain fused feature vectors; The classifier classifies the fused feature vectors to obtain an assessment score for cognitive impairment.

3. The cognitive impairment assessment system as described in claim 1, characterized in that, The preset paradigms include a picture-based memory and voice-based question-and-answer paradigm, an eye-saccade paradigm, an eye-tracking smoothing paradigm, a free-view paradigm, a sentence-reading paradigm, a text-reading paradigm, and a picture-based speaking and voice-based question-and-answer paradigm.

4. The cognitive impairment assessment system as described in claim 1, characterized in that, The eye-tracking data includes fixation point, eye position, head posture, and head position. The eye-tracking numerical features include average number of fixations, average number of saccades, total saccade path, fixation time, average fixation duration, average saccade amplitude, reaction initiation time, reaction execution speed, average minimum gain, number of erroneous saccades, fixation detachment time from stimulus point, speed of follow-up initiation, number of fixations during follow-up, speed difference during follow-up, and position difference during follow-up.

5. A cognitive impairment assessment system as described in claim 1, characterized in that, The semantic features include modal word count, expression of forgotten words, and expression of hesitant speech. The acoustic features include Fbank features, PLP features, Bottleneck features, and temporal acoustic features.

6. A cognitive impairment assessment system as described in claim 1, characterized in that, Before collecting eye movement and speech data from subjects under multiple preset paradigms, head posture calibration, eye movement calibration, and microphone calibration must be performed on the subjects.

Images