A multi-modal cognitive impairment assisted assessment method and system incorporating eye movement data

Abstract

The application relates to a multi-modal cognitive impairment auxiliary evaluation method and system combined with eye movement data, the method comprising: acquiring eye movement original data and voice original data synchronously collected when a subject performs a preset visual-audio collaborative cognitive evaluation task and performing time axis alignment; performing spatial mapping, heat value calculation and visual processing on the eye movement original data to generate an eye movement fixation heat map representing visual attention distribution; performing acoustic preprocessing on the voice original data, extracting a voiceprint core acoustic feature and a cognitive related rhythm feature to form a voiceprint feature set; respectively extracting spatial features of the eye movement fixation heat map and time sequence acoustic features of the voiceprint feature set, realizing cross-modal feature deep fusion through an attention mechanism after standardization processing to obtain multi-modal fusion feature parameters; inputting the multi-modal fusion feature parameters into a multi-modal deep neural network model to output existence probability of cognitive impairment of the subject, impairment degree classification and defect evaluation results of cognitive function dimensions.

Description

Technical Field

[0001] This application belongs to the fields of multimodal biometric processing, computer vision, voiceprint recognition and cognitive impairment auxiliary assessment technology, and in particular relates to a multimodal cognitive impairment auxiliary assessment method and system that combines eye movement data. Background Technology

[0002] Cognitive impairment is a syndrome of cognitive decline caused by factors such as neurodegenerative diseases of the brain and cerebrovascular diseases, primarily affecting dimensions such as attention, language, visuospatial perception, and executive function. Early and accurate assessment is key to slowing disease progression and improving intervention effectiveness. Currently, commonly used clinical cognitive assessment methods mostly rely on scale scoring, which has problems such as strong subjectivity, susceptibility to the influence of the subject's language ability, and insufficient early specificity.

[0003] Eye-tracking data is an objective biometric reflecting visual cognitive function. Its visualization product, eye-tracking gaze heatmap, can intuitively present the visual attention distribution, visuospatial search strategy, and executive function status of the subject. Voiceprint data is a set of physiological acoustic features of speech, reflecting the physiological state of the vocal organs and the regulatory function of the brain's language center. Brain neuropathies in patients with cognitive impairment can lead to specific changes in voiceprint features.

[0004] In existing technologies, cognitive impairment assessment methods based on single biomodal data have been applied, but they have the following drawbacks: single eye-tracking data can only reflect visual-related cognitive functions and cannot assess the functional state of the language center; single speech data often focuses on semantics or fluency and does not fully explore the specific correlation between the physiological acoustic features of voiceprints and cognitive impairment; at the same time, existing methods do not deeply integrate the spatial cognitive features of eye-tracking gaze heatmaps with the physiological acoustic cognitive features of voiceprint data, and cannot fully explore the complementary information between multimodal biodata, resulting in insufficient accuracy and specificity of the assessment.

[0005] Furthermore, the design of existing cognitive assessment tasks lacks synergy between vision and speech, and the multi-source data collected has poor temporal correlation and low coupling degree of cognitive features, which further affects the effectiveness of multimodal feature fusion.

[0006] There are currently no effective solutions to the technical problems of existing cognitive impairment assessment methods, such as high subjectivity, one-sided assessment of single-modal data, and insufficient integration of eye movement and voiceprint features. Summary of the Invention

[0007] In view of the shortcomings of the existing technology, the purpose of the invention is to provide a multimodal cognitive impairment auxiliary assessment method and system that combines eye movement data. By designing a visual-audio-speech co-coordinated cognitive assessment task and simultaneously collecting the subject's raw eye movement data and raw speech data, an eye movement gaze heatmap and a voiceprint feature set are generated respectively. Cross-modal feature deep fusion is achieved through the attention mechanism, and multi-dimensional assessment results are output based on a multimodal deep neural network model. This realizes the synchronous acquisition of multimodal biological data, deep feature fusion, and accurate and non-invasive assessment of cognitive impairment, effectively improving the accuracy, specificity and robustness of cognitive impairment assessment.

[0008] The first aspect of this application proposes a multimodal cognitive impairment auxiliary assessment method combining eye-tracking data, comprising: acquiring raw eye-tracking data and raw speech data simultaneously collected by the subject when performing a preset visual-audio-sound co-coordinated cognitive assessment task, and aligning the raw eye-tracking data and the raw speech data along the time axis; performing spatial mapping, heat value calculation, and visualization processing on the time-aligned raw eye-tracking data to generate an eye-tracking gaze heatmap representing the distribution of visual attention; performing acoustic preprocessing on the time-aligned raw speech data to extract core acoustic features of the voiceprint and cognitive prosodic features to form a voiceprint feature set; extracting the spatial features of the eye-tracking gaze heatmap and the temporal acoustic features of the voiceprint feature set respectively, and after standardization processing, achieving deep fusion of cross-modal features through an attention mechanism to obtain multimodal fusion feature parameters; inputting the multimodal fusion feature parameters into a trained multimodal deep neural network model to output the probability of the subject's cognitive impairment, the degree of impairment, and the assessment results of the cognitive function dimension deficit.

[0009] According to a second aspect of the present disclosure, a storage medium is provided, the storage medium including a stored program, wherein, when the program is executed, a processor performs any of the methods described above.

[0010] According to a third aspect of the present disclosure, a multimodal cognitive impairment auxiliary assessment system incorporating eye-tracking data is provided, comprising: a data acquisition and alignment module, configured to acquire raw eye-tracking data and raw speech data synchronously collected by a subject performing a preset visual-audio-speech coordinating cognitive assessment task, and to align the raw eye-tracking data and the raw speech data along a time axis; an eye-tracking gaze heatmap generation module, configured to perform spatial mapping, heat value calculation, and visualization processing on the time-aligned raw eye-tracking data to generate an eye-tracking gaze heatmap representing the distribution of visual attention; and a voiceprint feature set generation module, configured to perform spatial mapping, heat value calculation, and visualization processing on the time-aligned raw eye-tracking data. The raw speech data of the subject undergoes acoustic preprocessing to extract core acoustic features and cognitive prosodic features, forming a voiceprint feature set. A multimodal feature fusion module is used to extract the spatial features of the eye-tracking gaze heatmap and the temporal acoustic features of the voiceprint feature set, respectively. After standardization, cross-modal feature deep fusion is achieved through an attention mechanism to obtain multimodal fusion feature parameters. An intelligent assessment module is used to input the multimodal fusion feature parameters into a trained multimodal deep neural network model and output the probability of the subject's cognitive impairment, the degree of impairment, and the assessment results of the cognitive function dimension deficit.

[0011] According to a fourth aspect of the present disclosure, a multimodal cognitive impairment auxiliary assessment system incorporating eye-tracking data is provided, comprising: a processor; and a memory connected to the processor, configured to provide the processor with instructions to perform the following processing steps: acquiring raw eye-tracking data and raw speech data synchronously collected by a subject performing a preset visual-audio-sound co-coordinated cognitive assessment task, and aligning the raw eye-tracking data and the raw speech data along a time axis; performing spatial mapping, heat value calculation, and visualization processing on the time-aligned raw eye-tracking data to generate an eye-tracking gaze heatmap representing the distribution of visual attention; performing acoustic preprocessing on the time-aligned raw speech data to extract core acoustic features of the voiceprint and cognitive prosodic features to form a voiceprint feature set; extracting the spatial features of the eye-tracking gaze heatmap and the temporal acoustic features of the voiceprint feature set respectively, and after standardization processing, achieving deep fusion of cross-modal features through an attention mechanism to obtain multimodal fusion feature parameters; inputting the multimodal fusion feature parameters into a trained multimodal deep neural network model to output the probability of the subject's cognitive impairment, the degree of impairment, and the assessment results of the cognitive function dimension deficit.

[0012] The beneficial effects of this application are as follows: (1) By deeply fusing the spatial features of eye-tracking gaze heatmaps with the temporal acoustic features of voiceprint feature sets across modalities, the complementary information of the two types of biological data is fully explored. This covers both visual-related cognitive functions and the physiological functional state of the language center. Compared with single-modal assessment methods, multimodal fusion features can more comprehensively and accurately characterize the overall cognitive state of the subjects, significantly improving the accuracy and specificity of cognitive impairment assessment.

[0013] (2) By designing three types of visual-audio-sound co-cognition assessment tasks, namely, picture-reading and comprehension task, text-reading and comprehension task, and visual tracking and broadcasting task, the subjects are required to complete visual operations and speech expression simultaneously, so that the collected eye movement raw data and speech raw data have strong cognitive coupling. Furthermore, by aligning the time axis, the temporal correlation of multi-source data is ensured, thus guaranteeing the cognitive correlation of multi-source data and laying a high-quality data foundation for multimodal feature fusion.

[0014] (3) By dividing voiceprint features into core acoustic features that reflect the physiological state of the vocal organs and cognitive prosodic features that are related to cognitive function, the physiological features of speech are preserved and the acoustic features that are highly related to cognitive impairment are strengthened, thus realizing the precise correlation between voiceprint physiological features and cognitive function and improving the specificity of early assessment.

[0015] (4) Through the multi-output layer structure of the multimodal deep neural network model, the probability of the existence of cognitive impairment, the grade of impairment, and the assessment results of the deficit in cognitive functions such as attention, language, visuospatial, and executive functions are output simultaneously, providing multi-dimensional reference for the development of personalized intervention plans, meeting diverse clinical needs, and having greater clinical guidance value than traditional scale assessments.

[0016] (5) Through modular system structure design, the entire process of data collection, feature extraction, feature fusion to evaluation result output is automated. It is easy to operate, time-saving, and non-invasive. It is suitable for large-scale early screening of cognitive impairment and various clinical evaluation scenarios, and has broad clinical application and market promotion prospects. Attached Figure Description

[0017] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Throughout the drawings, the same reference numerals denote the same components. Obviously, the drawings described below are merely some embodiments described in this application, and those skilled in the art can obtain other drawings based on these drawings.

[0018] Figure 1 This is a hardware structure block diagram of a computing device for implementing the method described in Embodiment 1 of this disclosure; Figure 2This is a flowchart of the multimodal cognitive impairment auxiliary assessment method combining eye movement data according to Embodiment 1 of this application; Figure 3 This is a schematic diagram of the eye-tracking gaze heatmap generation process according to Embodiment 1 of this application; Figure 4 This is a schematic diagram of the multimodal feature fusion process according to Embodiment 1 of this application; Figure 5 This is a schematic diagram of the structure of the multimodal cognitive impairment auxiliary assessment system combining eye movement data according to Embodiment 2 of this application; Figure 6 This is a schematic diagram of a multimodal cognitive impairment auxiliary assessment system that incorporates eye-tracking data, as described in Embodiment 3 of this application. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this disclosure.

[0020] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0021] Example 1

[0022] According to this embodiment, a method embodiment of a multimodal cognitive impairment auxiliary assessment method combining eye movement data is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0023] The method embodiments provided in this example can be executed on a server or similar computing device. Figure 1 A hardware block diagram of a computing device for implementing a multimodal cognitive impairment assistive assessment method incorporating eye-tracking data is shown. Figure 1 As shown, a computing device may include one or more processors (processors may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory for storing data, a transmission device for communication functions, and an input / output interface. The memory, transmission device, and input / output interface are connected to the processor via a bus. In addition, it may also include a display, keyboard, and cursor control device connected to the input / output interface. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, a computing device may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0024] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element in a computing device. As involved in the embodiments of this disclosure, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0025] The memory can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the multimodal cognitive impairment auxiliary assessment method combining eye movement data in the embodiments of this disclosure. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned application of the multimodal cognitive impairment auxiliary assessment method combining eye movement data. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the computing device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0026] The transmission device is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the computing device's communication provider. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0027] The display can be, for example, a touchscreen liquid crystal display (LCD), which allows users to interact with the user interface of the computing device.

[0028] It should be noted here that, in some optional embodiments, the above... Figure 1 The computing device shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computing devices.

[0029] Under the above operating environment, according to the first aspect of this embodiment, a multimodal cognitive impairment auxiliary assessment method combining eye movement data is provided. Figure 2 A flowchart illustrating the method is shown below. (Refer to...) Figure 2 As shown, the method includes: S1: Acquire raw eye movement data and raw speech data that are collected synchronously when the subject performs a preset visual-audio-sound co-cognition assessment task, and align the raw eye movement data and the raw speech data on the time axis. In this embodiment of the application, during the clinical application phase, when a cognitive impairment assessment is required for a subject, a professional first guides the subject to sit in front of an assessment system equipped with eye-tracking and speech acquisition devices. The system displays a pre-designed visual-audio-speech co-application cognitive assessment task to the subject through a high-definition display interface. This type of task requires the subject to verbally express themselves while observing visual content, thereby establishing a cognitive correlation between the acquired eye-tracking and speech data. Simultaneously with the task presentation, the system sends a hardware synchronization trigger signal to activate the eye-tracking and speech acquisition devices for synchronous data acquisition.

[0030] The eye-tracking acquisition device preferably uses an eye tracker with a sampling frequency of not less than 100Hz to collect raw eye-tracking data of the subject during task execution, including the two-dimensional coordinates (x, y) of the fixation point, fixation duration, saccade speed, saccade direction (angle), number of fixations per unit area, number of blinks, and blink duration. The sampling frequency is set to 100Hz by default to ensure the temporal resolution of the eye-tracking data. The speech acquisition device preferably uses a noise-canceling microphone with a sampling rate of not less than 16kHz to collect raw speech data of the subject, including speech temporal waveform, frequency domain characteristics, total speech duration, speech pause interval and duration, speech fundamental frequency (F0), and speech energy. During the acquisition process, a high-precision timestamp is added to each frame of eye-tracking data and each segment of speech data based on hardware timestamp technology. The timestamp accuracy is not greater than 1ms, eliminating the time difference in data acquisition and ensuring the temporal correlation and cognitive coupling of multi-source biological data, thereby achieving precise alignment of the time axis of the raw eye-tracking data and the raw speech data.

[0031] Thus, temporally correlated raw eye-tracking data and raw speech data were obtained, laying a data synchronization foundation for subsequent multimodal feature fusion.

[0032] S2: Perform spatial mapping, heat value calculation and visualization processing on the time-aligned raw eye-tracking data to generate an eye-tracking gaze heatmap representing the distribution of visual attention; In this embodiment, the time-aligned raw eye-tracking data obtained in step S1 is visualized. The fixation point location information in the raw eye-tracking data is mapped to the corresponding spatial location on the task display interface. Based on parameters such as the duration and frequency of fixation points, the fixation heat value for each area on the interface is calculated. Based on the calculated heat values, an eye-tracking fixation heatmap is generated using a visual encoding method. This heatmap visually presents the spatial distribution of the subject's attention during the execution of the audiovisual coordinating cognitive assessment task. Through this method, discrete eye-tracking trajectory data can be transformed into a standardized, easily extractable graphical representation.

[0033] S3: Perform acoustic preprocessing on the time-aligned raw speech data to extract the core acoustic features of the voiceprint and cognitive prosodic features to form a voiceprint feature set; In this embodiment, the time-aligned raw speech data obtained in step S1 undergoes acoustic processing. First, the raw speech data is preprocessed acoustically to remove environmental noise and invalid speech segments, obtaining valid speech data. Then, two types of features are extracted from the valid speech data: one type is the core acoustic features of the voiceprint reflecting the physiological state of the vocal organs, and the other type is the prosodic features related to cognitive function, serving as cognitive-related prosodic features. These two types of features are combined to form a standardized voiceprint feature set, which simultaneously covers the physiological acoustic attributes of speech and information related to cognitive regulation. Through this method, multi-dimensional voiceprint feature representations can be extracted from the speech data, providing an input basis for subsequent temporal feature encoding.

[0034] S4: Extract the spatial features of the eye-tracking gaze heatmap and the temporal acoustic features of the voiceprint feature set respectively. After standardization, achieve deep fusion of cross-modal features through an attention mechanism to obtain multimodal fusion feature parameters. In this embodiment, feature extraction is performed on the eye-tracking gaze heatmap generated in step S2 and the voiceprint feature set formed in step S3. In the eye-tracking modality branch, spatial features reflecting the spatial distribution of visual attention are extracted from the eye-tracking gaze heatmap using a convolutional neural network. In the voiceprint modality branch, temporal acoustic features reflecting the temporal dynamic correlation of acoustic features are extracted from the voiceprint feature set using a temporal network. After standardization of both types of features, they are input into an attention fusion network. The network automatically assigns attention weights based on the contribution of each modality feature to the cognitive impairment assessment and performs cross-modal feature fusion based on these weights to obtain multimodal fusion feature parameters. Through this method, deep fusion of eye-tracking spatial features and voiceprint temporal features is achieved, fully leveraging the complementary information between different modalities.

[0035] S5: Input the multimodal fusion feature parameters into the trained multimodal deep neural network model, and output the probability of the existence of cognitive impairment, the degree of impairment, and the assessment results of the cognitive function dimension deficit of the subject.

[0036] In this embodiment, the multimodal fusion feature parameters obtained in step S4 are input into a pre-trained multimodal deep neural network model. This model is pre-trained using a large amount of sample data annotated by professional physicians and possesses the ability to identify cognitive impairment-related patterns from multimodal fusion features. The model performs in-depth analysis on the input multimodal fusion feature parameters, integrating the fusion information from eye-tracking and voiceprint modalities, and simultaneously outputs three assessment results: first, the probability of the subject having cognitive impairment; second, the current level of cognitive impairment experienced by the subject; and third, the specific deficits in each cognitive function dimension. This achieves a comprehensive, multi-dimensional assessment of the subject's cognitive state, providing an objective basis for subsequently developing personalized intervention plans.

[0037] As described in the background section, cognitive impairment assessment methods based on single biomodal data have been applied, but they suffer from the following limitations: single eye-tracking data can only reflect visual-related cognitive functions and cannot assess the functional state of the language center; single speech data often focuses on semantics or fluency, failing to fully explore the specific correlation between the physiological acoustic features of voiceprints and cognitive impairment; furthermore, existing methods do not deeply integrate the spatial cognitive features of eye-tracking gaze heatmaps with the physiological acoustic cognitive features of voiceprint data, failing to fully explore the complementary information between multimodal biodata, resulting in insufficient accuracy and specificity in the assessment. Moreover, the design of existing cognitive assessment tasks lacks synergy between vision and speech, and the collected multi-source data exhibits poor temporal correlation and low coupling of cognitive features, further affecting the effectiveness of multimodal feature fusion.

[0038] In view of this, this application acquires raw eye movement data and raw speech data simultaneously collected by subjects when performing a pre-set visual-audio-sound co-coordinated cognitive assessment task, and aligns them along the time axis to ensure strong cognitive coupling and temporal consistency of the multi-source data. It generates an eye movement gaze heatmap by spatial mapping, heat value calculation, and visualization processing of the raw eye movement data, transforming discrete eye movement trajectories into an image representation that intuitively represents the distribution of visual attention. It performs acoustic preprocessing on the raw speech data and extracts core acoustic features of the voiceprint and cognitively related prosodic features to form a voiceprint feature set that simultaneously encompasses physiological acoustic attributes and cognitive regulation information. It extracts the spatial features of the eye movement gaze heatmap and the temporal acoustic features of the voiceprint feature set, respectively, and after standardization, uses an attention mechanism to achieve deep fusion of cross-modal features, fully exploring the complementary information of the two types of biological data. By inputting the multimodal fusion feature parameters into a trained multimodal deep neural network model, it simultaneously outputs assessment results of the probability of cognitive impairment, the degree of impairment, and the dimensions of cognitive function deficit, achieving a multi-dimensional objective assessment of visuospatial cognitive function and language center function, thus improving the accuracy and specificity of cognitive impairment assessment. It effectively solves the technical problems of existing cognitive impairment assessment methods, such as strong subjectivity, one-sided assessment of single-modal data, and insufficient integration of eye movement and voiceprint features.

[0039] Optionally, the preset audiovisual coordinating cognitive assessment tasks include: a picture-based descriptive task to assess the coordination ability between visual spatial perception and speech expression; a text-based reading comprehension task to assess language processing and attention maintenance abilities; and a visual tracking and broadcasting task to assess visual executive function and speech timing control abilities. The picture-based descriptive task, the text-based reading comprehension task, and the visual tracking and broadcasting task require the subject to simultaneously complete visual operations and speech expression.

[0040] In this embodiment, the audiovisual co-cognition assessment task is designed in three categories, which are used to assess different core dimensions affected by cognitive impairment.

[0041] The first category is the image-based descriptive task, designed to assess the subject's ability to coordinate visuospatial perception and verbal expression. This task uses, but is not limited to, images of everyday objects such as "living room, sofa, television, coffee table," with a resolution of 1920×1080 pixels, displayed on a high-definition screen. Subjects are required to observe the image content and verbally describe what they see. During this process, subjects need to visually recognize the objects in the image and understand their spatial relationships, while simultaneously organizing their language for verbal expression, thus simultaneously engaging visuospatial cognition and language expression functions. The system simultaneously collects eye movement data while the subject views the image and verbal data during the verbal description. It also records visual reference information for this task, including the coordinates of the visual target area, the pixel positions of the text sequence, reference image feature points, and a standard speech template, including standard speech rate, fundamental frequency, and pause intervals, providing a reference benchmark for subsequent data analysis.

[0042] The second category is a text-based reading comprehension task, used to assess subjects' language processing and attention maintenance abilities. This task presents, for example but not limited to, a short text sequence such as "Spring is here, the flowers in the park are blooming," and records the pixel coordinates of each character on the display interface. Subjects are required to read aloud the displayed text and verbally explain its meaning. During this process, subjects need to maintain their attention on the text content, complete visual recognition and semantic processing of the text, and simultaneously produce accurate verbal output, thereby simultaneously mobilizing attention maintenance and language processing functions. The system simultaneously collects eye-tracking data when subjects read the text and verbal data when they explain it.

[0043] The third category is the visual tracking and broadcasting task, used to assess the subject's visual executive function and speech timing control ability. This task presents, for example but not limited to, a dynamic visual tracking path of "straight line → curve → broken line," with a total trajectory length of approximately 800 pixels. Subjects are required to track the path with their eyes and simultaneously verbally announce the direction of travel. During this process, subjects need to continuously visually track the dynamic target and determine its direction of travel, while simultaneously broadcasting the path changes in sequence, thus simultaneously engaging their visual executive function and speech timing control. The system simultaneously collects eye movement saccades data while the subject tracks the path and speech data while announcing the direction.

[0044] All three types of tasks require parameter configuration. Based on clinical cognitive assessment needs, core parameters such as display duration, eye-tracking acquisition frequency, speech acquisition sampling rate, and data synchronization timestamp accuracy should be set for each task. Specifically, the task display duration is set according to the task type, generally controlled between 10 and 30 seconds per task; the eye-tracking acquisition frequency is preferably set to 100Hz to 200Hz to ensure the temporal resolution of the eye-tracking data; the speech acquisition sampling rate is preferably set to 16kHz to 44.1kHz to meet the acoustic requirements of voiceprint feature extraction; and the data synchronization timestamp accuracy is set to no more than 1ms to ensure the temporal consistency of multi-source data. The speech playback duration threshold is 80%-90% of the task display duration.

[0045] In practice, the task display triggering unit synchronously displays the designed audiovisual co-cognition cognitive assessment task content through a high-definition display interface with a resolution of no less than 1920×1080. Simultaneously with the task display, a hardware synchronization trigger signal is sent to the data acquisition device to initiate the synchronous acquisition of eye-tracking and speech data, ensuring the consistency of the data acquisition sequence.

[0046] Therefore, through the design and demonstration of the above three types of visual-audio-speech co-operational cognitive assessment tasks, subjects must simultaneously complete visual operations and verbal expression during task execution. This ensures a strong cognitive coupling and temporal correspondence between the collected raw eye-tracking data and raw verbal data, providing a high-quality data foundation for subsequent multimodal feature fusion. Furthermore, the three types of tasks correspond to the assessment of different cognitive dimensions, such as visuospatial perception, language processing, attention maintenance, executive function, and temporal control. This allows the extracted eye-tracking gaze heatmaps and voiceprint feature sets to jointly characterize the subject's cognitive function status from multiple dimensions, providing a rich data source for comprehensive assessment of cognitive impairment.

[0047] Optionally, the operation of spatial mapping, heat value calculation, and visualization processing of time-aligned raw eye-tracking data to generate an eye-tracking gaze heatmap representing the distribution of visual attention includes: linearly mapping the physical gaze point coordinates in the time-aligned raw eye-tracking data to the pixel coordinate space of the task display interface; calculating a pixel heat value for each pixel in the pixel coordinate space based on the corresponding cumulative gaze duration, gaze frequency, and a preset blink interference coefficient, wherein the pixel heat value is positively correlated with the gaze duration and gaze frequency, and negatively correlated with the blink interference coefficient; and generating a standardized eye-tracking gaze heatmap by using color gradient encoding to represent the level of the pixel heat value with color depth.

[0048] In this embodiment of the application, the time-aligned raw eye-tracking data obtained in step S1 needs to be processed to generate an eye-tracking fixation heatmap, such as... Figure 3 As shown, the specific operations include: First, coordinate space mapping is performed. The two-dimensional physical coordinates of the gaze point recorded in the raw eye-tracking data are converted into pixel coordinates of the task display interface using a proportional linear mapping method, achieving precise spatial matching between the eye-tracking data and the visual task interface. The specific mapping formula is as follows: , ,in, and These are the mapped pixel coordinates. and These are the original physical coordinates. and These are the maximum values ​​of the physical coordinates. and These are the pixel width and pixel height of the task display interface, for example, 1920×1080 pixels.

[0049] Next, pixel heat value calculation is performed. For each pixel in the pixel coordinate space, the heat value of that pixel is calculated based on its corresponding cumulative gaze duration, gaze frequency, and preset blink interference coefficient. The specific calculation formula is as follows: ,in, Let (i,j) be the heat value of pixel (i,j). This represents the cumulative gaze duration corresponding to that pixel. For the video recording corresponding to this pixel, The preset blink interference coefficient is used to eliminate the interference of the subject's blinking action on the fixation characteristics; , , The preset weighting coefficients are used, and they satisfy the following conditions: Default settings , , The calculation formula shows that pixel heat value is positively correlated with cumulative fixation duration and fixation frequency, and negatively correlated with blink interference coefficient. Areas with longer fixation time and higher fixation frequency have higher heat values; areas affected by blinking have correspondingly lower heat values.

[0050] Finally, heatmap visualization encoding is performed. An eye-tracking fixation heatmap is generated using a four-color gradient encoding method (red, yellow, green, and blue), with color depth representing pixel heat values. Specifically, red areas represent high-heat areas, corresponding to the subject's core fixation point; yellow areas represent medium-heat areas, corresponding to secondary fixation points; green areas represent low-heat areas, corresponding to saccade transition zones; and blue areas represent no-heat areas, corresponding to unfixed areas. The generated heatmap preserves the spatial contours of saccade trajectories, visually presenting the distribution of visual attention during the subject's performance of the audiovisual co-cognition assessment task. The final output is a standardized pixel-size eye-tracking fixation heatmap, such as 224×224 pixels, ensuring consistent dimensionality across different subjects and facilitating subsequent feature extraction.

[0051] Thus, spatial correspondence between eye-tracking data and visual task interface was achieved through coordinate space mapping. The degree of attention of subjects to each area of ​​the interface was quantified by the heat value calculation formula and blink interference was eliminated. A standardized eye-tracking gaze heatmap was generated by color gradient encoding, transforming discrete raw eye-tracking data into an intuitive and quantifiable representation of visual attention distribution, laying a data foundation for subsequent convolutional neural network extraction of spatial features.

[0052] Optionally, the operation of performing acoustic preprocessing on the time-aligned raw speech data to extract core acoustic features and cognitive prosodic features to form a voiceprint feature set includes: pre-emphasizing, framing, windowing, denoising, and endpoint detection on the time-aligned raw speech data to remove invalid speech segments and obtain preprocessed effective speech data; based on the preprocessed effective speech data, extracting core acoustic features reflecting the physiological characteristics of speech, wherein the core acoustic features include at least Mel-frequency cepstral coefficients, linearly predicted cepstral coefficients, fundamental frequency profile, formant frequency, and bandwidth; based on the preprocessed effective speech data, extracting speech prosodic features related to cognitive function as cognitive prosodic features, wherein the cognitive prosodic features include at least the speech rate variation coefficient, fundamental frequency standard deviation, pause interval variation coefficient, and speech energy fluctuation; and combining the core acoustic features and the cognitive prosodic features to form a standardized voiceprint feature set.

[0053] In this embodiment, the time-aligned raw speech data obtained in step S1 undergoes acoustic processing, which can be automated throughout using an acoustic feature extraction toolbox to form a standardized voiceprint feature set. Specific steps include: First, the raw speech data undergoes multi-stage acoustic preprocessing to obtain preprocessed effective speech data. Specifically, this includes the following steps: First, pre-emphasis: a first-order high-pass filter with a pre-emphasis coefficient of 0.97 is used to boost the high frequencies of the speech signal, eliminating spectral distortion generated during transmission and acquisition. Second, frame-by-frame windowing: a frame length of 25ms and a frame shift of 10ms are set, and a Hamming window is used to window each frame to reduce inter-frame spectral distortion. Third, denoising: spectral subtraction is used to reduce noise in the speech signal, suppressing environmental noise interference and improving the signal-to-noise ratio by at least 10dB. Fourth, endpoint detection: a dual-threshold method is used to detect the start and end points of the speech, thereby eliminating invalid speech segments such as silences and stutters, retaining only the portions containing valid speech content. After these preprocessing steps, effective speech data with optimized audio quality is obtained.

[0054] Then, based on the preprocessed effective speech data, core acoustic features of the voiceprint were extracted. These features reflect the physiological state of the vocal organs, are individual-specific, and are highly correlated with cognitive impairment. The extracted core acoustic features specifically include: 12-dimensional Mel-frequency cepstral coefficients and their first and second difference coefficients, 12-dimensional linear predicted cepstral coefficients, fundamental frequency profile, the frequency and bandwidth of the first five formants, spectral flatness, and speech spectrum entropy. A total of 68 core acoustic features of the voiceprint were extracted, comprehensively characterizing the physiological acoustic properties of the subject's speech.

[0055] Simultaneously, based on the preprocessed effective speech data, cognitively relevant prosodic features were extracted. These features reflect the brain's language center's regulatory function on speech and are closely related to cognitive function. The extracted cognitively relevant prosodic features specifically include: speech rate variation coefficient, calculated by dividing the speech rate standard deviation by the speech rate mean; fundamental frequency standard deviation, reflecting the fluctuation amplitude of the speech fundamental frequency; pause interval variation coefficient, reflecting the irregularity of pause intervals in speech; speech energy fluctuation, reflecting the fluctuation state of speech energy in the time dimension; syllable duration variation coefficient; and average formant bandwidth. A total of six dimensions of cognitively relevant prosodic features were extracted.

[0056] Finally, the two types of features are combined: the 68-dimensional core acoustic features of the voiceprint are integrated with the 6-dimensional cognitive prosodic features to form a 74-dimensional standardized voiceprint feature set. This voiceprint feature set simultaneously encompasses the physiological acoustic information of the subject's vocal organs and the cognitive regulatory function information of the brain, providing multi-dimensional voiceprint feature representation for subsequent temporal coding networks.

[0057] Thus, multi-stage acoustic preprocessing effectively improved the quality of speech data and eliminated interference from environmental noise and invalid segments. By extracting core acoustic features of the voiceprint and cognitive prosodic features respectively, dual acoustic representation of the subject's pronunciation physiological state and cognitive regulation function was achieved. By combining the two types of features to form a standardized voiceprint feature set, a high-quality data foundation was laid for the subsequent temporal coding of one-dimensional convolutional neural networks and long short-term memory networks.

[0058] Optionally, the operations of extracting the spatial features of the eye-tracking gaze heatmap and the temporal acoustic features of the voiceprint feature set respectively include: using a lightweight convolutional neural network as the backbone network to extract spatial features from the eye-tracking gaze heatmap, extracting texture features, attention hotspot distribution features, and saccade trajectory morphology features layer by layer through depthwise separable convolutional layers and pooling layers, and outputting the spatial features of the eye-tracking gaze heatmap after passing through a global average pooling layer; using a dual-branch network combining a one-dimensional convolutional neural network and a long short-term memory network to perform temporal encoding on the voiceprint feature set, concatenating the features output by the dual-branch network, and outputting the voiceprint temporal features of the voiceprint feature set through a fully connected layer; wherein, the dual-branch network includes a one-dimensional convolutional neural network branch and a long short-term memory network branch, the one-dimensional convolutional neural network branch extracts local acoustic association features of the voiceprint features through convolutional layers, and the long short-term memory network branch captures the long-term temporal dynamic association of the voiceprint features through a gating mechanism.

[0059] In this embodiment, feature extraction is required for the eye-tracking gaze heatmap generated in step S2 and the voiceprint feature set formed in step S3. In the eye-tracking modality branch, a lightweight convolutional neural network, MobileNetV3-Small, is used as the backbone network to extract spatial features from the standardized pixel-size eye-tracking gaze heatmap. This network uses a stacked structure of depthwise separable convolutional layers and pooling layers to extract features layer by layer from the input eye-tracking gaze heatmap: in the shallow network, low-dimensional texture features are extracted, including basic visual features such as the color distribution and edge contours of the heatmap; in the middle network, mid-dimensional attention hotspot distribution features are extracted, including attention-related features such as the spatial location, distribution range, and density of gaze hotspots; in the deep network, high-dimensional saccade trajectory morphology features are extracted, including features related to the spatial direction, curvature, continuity, and visual search strategy of the saccade path. After feature extraction at each layer, the feature map is compressed into a fixed-dimensional feature vector through a global average pooling layer, outputting a 256-dimensional eye-tracking heatmap feature vector. This feature vector fully exploits the visual cognitive information of the eye-tracking gaze heatmap.

[0060] In the voiceprint modality branch, a dual-branch network combining a one-dimensional convolutional neural network and a long short-term memory network is used to temporally encode the 74-dimensional voiceprint feature set. The specific structure of this dual-branch network is as follows: the first branch is a one-dimensional convolutional neural network branch, which performs convolution operations along the time dimension on the voiceprint feature sequence through multiple one-dimensional convolutional layers to extract local acoustic correlation features between adjacent time frames of the voiceprint features, capturing the change patterns of voiceprint features within a short time range; the second branch is a long short-term memory network branch, which performs cyclic processing on the voiceprint feature sequence through a gating mechanism. The forget gate controls the degree of retention of historical information, the input gate controls the degree of input of current information, and the output gate controls the degree of mapping from hidden states to output, thereby effectively capturing the temporal dynamic correlation of voiceprint features over a long time range and avoiding the gradient vanishing problem in traditional recurrent networks. The features extracted by the two branches are concatenated after output, fusing the local acoustic correlation features with the long-term temporal dynamic correlation features. After mapping through a fully connected layer, a 128-dimensional voiceprint temporal feature vector is output. This feature vector fully exploits the temporal acoustic cognitive information of the voiceprint data.

[0061] Thus, a lightweight convolutional neural network was used to extract multi-level spatial features from eye-tracking gaze heatmaps, obtaining eye-tracking heatmap feature vectors covering texture features, attention distribution features, and saccade trajectory morphology features. A dual-branch network combining a one-dimensional convolutional neural network and a long short-term memory network was used to jointly encode the local acoustic correlation features and long-term temporal dynamic correlation features of the voiceprint feature set, obtaining a voiceprint temporal feature vector. These two types of feature vectors provide high-quality feature input for subsequent cross-modal attention fusion.

[0062] Optionally, the operation of achieving deep fusion of cross-modal features through an attention mechanism to obtain multimodal fusion feature parameters includes: inputting standardized spatial features into a first feature mapping layer and outputting a first mapped feature; inputting standardized temporal acoustic features into a second feature mapping layer and outputting a second mapped feature; inputting the first mapped feature and the second mapped feature into an attention weight calculation layer and outputting eye-tracking modal weights and voiceprint modal weights, wherein the sum of the eye-tracking modal weights and the voiceprint modal weights is equal to 1; performing weighted processing on the first mapped feature and the second mapped feature based on the eye-tracking modal weights and the voiceprint modal weights respectively; and concatenating and nonlinearly mapping the weighted first mapped feature and the second mapped feature through a feature fusion layer to output the multimodal fusion feature parameters.

[0063] In this embodiment of the application, cross-modal attention fusion is required on the 256-dimensional eye-tracking heatmap feature vector and the 128-dimensional voiceprint temporal feature vector obtained through feature extraction in step S4. For example... Figure 4 As shown, the specific steps include: First, the two types of feature vectors are standardized separately. Since the eye-tracking heatmap feature vectors and the voiceprint temporal feature vectors originate from different feature extraction networks, their feature values ​​have different dimensions and scale distributions. Direct fusion can lead to problems such as unstable model training and a single modality dominating the fusion result. Therefore, Z-score standardization is applied to both types of feature vectors separately, ensuring that the standardized feature values ​​follow a standard normal distribution with a mean of 0 and a variance of 1. This eliminates the differences in dimensions and scale between different modal features, laying the foundation for the reasonable allocation of attention weights in the future.

[0064] Then, a dual-branch attention fusion network is used to achieve deep fusion of eye movement and voiceprint features. The fusion process of this network consists of the following four stages: The first stage involves feature mapping. The standardized 256-dimensional eye-tracking heatmap spatial features are input into the first feature mapping layer, and the standardized 128-dimensional voiceprint temporal features are input into the second feature mapping layer. Each of the two feature mapping layers performs high-dimensional spatial mapping on its respective input standardized feature vectors, transforming the original features from the original feature space to a higher-dimensional cognitive feature space to enhance the expressive power and separability of the features. After mapping by the first feature mapping layer, the first mapped features are output; after mapping by the second feature mapping layer, the second mapped features are output.

[0065] The second stage involves calculating attention weights. The first and second mapped features are input into the attention weight calculation layer. Based on the learning results from the training set samples, this layer automatically evaluates the contribution of eye-tracking and voiceprint modal features to cognitive impairment assessment, dynamically outputting the eye-tracking modal weight α and the voiceprint modal weight β, satisfying the constraint α + β = 1, meaning the sum of the eye-tracking and voiceprint modal weights equals 1. This constraint ensures the normalization of the two modal weights, allowing the weight values ​​to directly reflect the relative importance of the corresponding modality feature in cognitive impairment assessment. In typical application scenarios, the eye-tracking modal weight α is approximately 0.55, and the voiceprint modal weight β is approximately 0.45. These close values ​​indicate that both eye-tracking spatial features and voiceprint temporal features contribute significantly to cognitive impairment assessment, with the eye-tracking modality contributing slightly more. The weight values ​​can be dynamically adjusted based on the characteristic performance of different subjects, adapting to the specific correlation between each modality feature and cognitive impairment.

[0066] The third stage involves weighted processing. Based on the eye-tracking modality weights α and voiceprint modality weights β output from the attention weight calculation layer, the first and second mapped features are weighted respectively. Specifically, each dimension value of the first mapped feature is multiplied by α, and each dimension value of the second mapped feature is multiplied by β, resulting in weighted first and second mapped features. Through weighted processing, modal features with higher contributions are assigned greater weights, and their information is enhanced in subsequent fusion; modal features with lower contributions are assigned smaller weights, and their irrelevant or noisy information is suppressed, thereby achieving adaptive filtering and optimization of intermodal information.

[0067] The fourth stage involves feature fusion. Through a feature fusion layer, the weighted features after the first and second mappings are concatenated, forming a combined feature vector. Then, a non-linear mapping is applied to the concatenated feature vector, and a fully connected layer further transforms and interacts with the features, fully exploring the cognitive correlation between the two types of features. This achieves deep fusion and complementary advantages between eye-tracking spatial features and voiceprint temporal features. The final output is a 384-dimensional multimodal fusion feature parameter, which integrates the visual-spatial cognitive information of the eye-tracking gaze heatmap and the temporal acoustic cognitive information of the voiceprint data, providing a unified feature representation for subsequent evaluation and analysis of the multimodal deep neural network model.

[0068] Thus, the dimensional differences of cross-modal features were eliminated through Z-score normalization, the feature mapping layer of the dual-branch attention fusion network was used to realize the dimensional transformation and expressive power enhancement of features, and the attention weight calculation layer adaptively learned and allocated attention weights for each modality, thereby strengthening the modal information that contributes more to the assessment of cognitive impairment and suppressing irrelevant information. The concatenation and nonlinear mapping of the feature fusion layer fully explored the cognitive correlation information between eye movement features and voiceprint features, and finally obtained multimodal fusion feature parameters that combine spatial cognitive features and temporal acoustic features, providing high-quality feature input for the multidimensional evaluation of subsequent models.

[0069] Optionally, the multimodal deep neural network model includes a feature mapping layer, a CNN deep feature extraction layer, an LSTM temporal feature extraction layer, an attention weighting layer, a fully connected layer, and a multi-output layer. The multi-output layer includes a first output branch, a second output branch, and a third output branch. The operation of inputting the multimodal fusion feature parameters into the trained multimodal deep neural network model and outputting the probability of the subject's cognitive impairment, the severity level of the impairment, and the assessment results of cognitive function dimension deficits includes: inputting the multimodal fusion feature parameters into the feature mapping layer for feature mapping, then sequentially extracting joint spatial and temporal features through the CNN deep feature extraction layer and the LSTM temporal feature extraction layer, and then strengthening the feature information related to cognitive impairment through the attention weighting layer; inputting the output features of the attention weighting layer into the fully connected layer, and inputting the output of the fully connected layer into the first output branch, the second output branch, and the third output branch respectively, so as to obtain the probability of the existence of the cognitive impairment through the first output branch, the severity level of the impairment through the second output branch, and the assessment results of the cognitive function dimension deficits through the third output branch.

[0070] In this embodiment, the multimodal deep neural network model is based on CNN-LSTM architecture, combined with attention mechanism and Dropout regularization, and designed with a multi-output layer structure to meet the multi-dimensional assessment needs of cognitive impairment.

[0071] The overall structure of the model is as follows: Input layer → Feature mapping layer → CNN deep feature extraction layer → LSTM temporal feature extraction layer → Attention weighting layer → Dropout regularization layer (dropout rate = 0.2) → Fully connected layer → Multi-output layer.

[0072] The input layer contains 384 neurons, consistent with the 384-dimensional multimodal fusion feature parameters output in step S4, achieving precise matching between the feature parameters and the model's input layer. Following the input layer are sequentially connected a feature mapping layer, a CNN deep feature extraction layer, an LSTM temporal feature extraction layer, an attention weighting layer, a Dropout regularization layer, a fully connected layer, and a multi-output layer.

[0073] The feature mapping layer contains 768 neurons. Through fully connected mapping, it maps the input 384-dimensional multimodal fusion feature parameters to a higher-dimensional cognitive feature space, enhancing the expressive power of the features and providing richer feature representations for subsequent deep feature extraction.

[0074] The CNN deep feature extraction layer extracts deep spatial features from the mapped features through multiple convolutional operations, capturing local correlation patterns and spatial structural information in the multimodal fusion feature parameters.

[0075] The LSTM temporal feature extraction layer uses a gating mechanism to perform temporal modeling on the feature sequences output by the CNN layers, capturing the dynamic changes of features in the temporal dimension and extracting joint spatial and temporal features. This layer fully utilizes the forget gate, input gate, and output gate structure of LSTM to effectively handle long-range dependencies in the feature sequences.

[0076] The attention-weighted layer adaptively weights the features output by the LSTM layer. By automatically learning the contribution of each feature dimension to the assessment of cognitive impairment, it strengthens the feature information that is highly related to cognitive impairment and suppresses the interference of redundant or noisy features that are unrelated to cognitive impairment.

[0077] The Dropout regularization layer is placed after the attention-weighted layer. During model training and inference, it randomly discards the outputs of a portion of neurons with a dropout rate of 0.2, meaning 20% ​​of neurons are randomly dropped. This layer effectively suppresses overfitting during training and improves the model's generalization ability.

[0078] The fully connected layer performs further feature integration and nonlinear transformation on the output of the Dropout layer, providing a unified feature input for each branch of the multi-output layer.

[0079] The multi-output layer contains three output branches, each corresponding to one of the three dimensions of cognitive impairment assessment. The first output branch uses the sigmoid activation function, outputting a probability value between 0 and 1. The closer the probability value is to 1, the higher the likelihood of cognitive impairment in the subject. The default threshold is 0.5; a probability value greater than this threshold indicates cognitive impairment. The second output branch uses the softmax activation function, outputting four-level classification results: Level 0 (no cognitive impairment), Level 1 (mild cognitive impairment), Level 2 (moderate cognitive impairment), and Level 3 (severe cognitive impairment). The third output branch uses the sigmoid activation function to output the probability of deficits in the attention, language, visuospatial, and executive function dimensions. The probability value for each dimension is between 0 and 1; a probability value of 0.5 or higher indicates cognitive impairment in that dimension.

[0080] The training process of the model is as follows: First, multimodal fusion feature parameters of healthy subjects and patients with different degrees of cognitive impairment are collected as training samples. Each sample is manually labeled by a neurologist, including labels for the presence of cognitive impairment (0 for no cognitive impairment, 1 for cognitive impairment), severity of impairment (0 for no cognitive impairment, 1 for mild, 2 for moderate, 3 for severe), and core cognitive deficit dimension labels (labeling the affected dimension as one or more of attention, language, visuospatial, or executive function). After constructing the training dataset, it is divided into training and test sets in an 8:2 ratio. Data augmentation techniques are used to improve the model's generalization ability, specifically including rotation enhancement of eye-tracking gaze heatmaps and noise enhancement of voiceprint features. The model training uses the cross-entropy loss function as the total loss function, and the Adam optimizer is used to optimize the model parameters with a learning rate set to 1e-4. Iteratively train the model on the training set until it converges, requiring the test set accuracy to be no less than 93%, specificity to be no less than 90%, and sensitivity to be no less than 91%. Once the above performance indicators are achieved, save it as a trained multimodal deep neural network model for subsequent application inference stages.

[0081] During the model application inference phase, the 384-dimensional multimodal fusion feature parameters obtained in step S4 are input into the trained multimodal deep neural network model. The model first maps the input features to a high-dimensional cognitive feature space through a feature mapping layer; then, it sequentially passes through a CNN deep feature extraction layer and an LSTM temporal feature extraction layer to extract joint spatial and temporal features; next, it uses an attention weighting layer to enhance features related to cognitive impairment and suppress irrelevant feature interference; then, it uses a Dropout regularization layer to randomly discard some neuron outputs to prevent overfitting; finally, it uses a fully connected layer for feature integration; and finally, the output of the fully connected layer is input into three output branches.

[0082] The first output branch outputs the probability of the presence of cognitive impairment through the sigmoid activation function. This probability is a value between 0 and 1. The closer the probability value is to 1, the higher the likelihood that the subject has cognitive impairment. The default threshold is 0.5. Values ​​greater than the threshold are considered to indicate the presence of cognitive impairment.

[0083] The second output branch outputs a four-level disability rating result through the softmax activation function, namely Level 0 (no cognitive impairment), Level 1 (mild cognitive impairment), Level 2 (moderate cognitive impairment), and Level 3 (severe cognitive impairment).

[0084] The third output branch uses the sigmoid activation function to output the probability of deficits in four dimensions: attention, language, visuospatial, and executive function. A probability value ≥ 0.5 indicates a cognitive deficit in that dimension, thus obtaining the assessment result of the cognitive function dimension deficit.

[0085] Thus, by using the CNN-LSTM architecture and attention mechanism of the multimodal deep neural network model, spatial-temporal joint cognitive correlation information in the multimodal fusion feature parameters was effectively extracted, and key features related to cognitive impairment were strengthened through attention weighting. The generalization ability of the model was improved through the Dropout regularization mechanism, avoiding overfitting. Through the design of three output branches in the multi-output layer, the probability of the presence of cognitive impairment, the degree of impairment classification, and the assessment results of the deficit in each cognitive function dimension were output simultaneously, realizing a multi-dimensional and fine-grained assessment of the cognitive state of the subjects, and providing specific and objective reference for the formulation of personalized intervention programs.

[0086] In addition, refer to Figure 1 As shown, according to a second aspect of this embodiment, a storage medium is provided. The storage medium includes a stored program, wherein, when the program is executed, a processor performs any of the methods described above.

[0087] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0088] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0089] Example 2

[0090] Figure 5 A multimodal cognitive impairment assistive assessment system incorporating eye-tracking data, according to this embodiment, is illustrated. This system corresponds to the method described in Embodiment 1. (Reference) Figure 5As shown, the system includes: a data acquisition and alignment module, used to acquire raw eye movement data and raw speech data synchronously collected by the subject when performing a preset visual-audio-sound co-cooperative cognitive assessment task, and to align the raw eye movement data and the raw speech data along the time axis; an eye movement gaze heatmap generation module, used to perform spatial mapping, heat value calculation and visualization processing on the time-aligned raw eye movement data to generate an eye movement gaze heatmap representing the distribution of visual attention; a voiceprint feature set generation module, used to perform acoustic preprocessing on the time-aligned raw speech data, extract core acoustic features of the voiceprint and cognitive prosodic features to form a voiceprint feature set; a multimodal feature fusion module, used to extract the spatial features of the eye movement gaze heatmap and the temporal acoustic features of the voiceprint feature set respectively, and after standardization processing, to achieve deep fusion of cross-modal features through an attention mechanism to obtain multimodal fusion feature parameters; and an intelligent assessment module, used to input the multimodal fusion feature parameters into a trained multimodal deep neural network model, and output the probability of the subject's cognitive impairment, the degree of impairment, and the assessment results of the cognitive function dimension deficit.

[0091] Optionally, the data acquisition and alignment module includes a collaborative cognitive task design submodule and a multi-source biological data synchronous acquisition submodule; the collaborative cognitive task design submodule includes an audiovisual task design unit, a parameter configuration unit, and a task display triggering unit; the multi-source biological data synchronous acquisition submodule includes a high-precision eye-tracking acquisition unit, a high-fidelity voice acquisition unit, and a hardware synchronization unit.

[0092] The audiovisual task design unit is used to design three types of audiovisual collaborative cognitive assessment tasks: image-based descriptive tasks, text-based reading comprehension tasks, and visual tracking and broadcasting tasks. It records information such as the coordinates of the visual target area, the pixel positions of the text sequence, feature points of the reference image, and standard speech templates. The image-based descriptive task assesses the coordination ability between visual spatial perception and speech expression; the text-based reading comprehension task assesses language processing and attention maintenance abilities; and the visual tracking and broadcasting task assesses visual executive function and speech temporal control abilities. The image-based descriptive task, the text-based reading comprehension task, and the visual tracking and broadcasting task require the subject to simultaneously complete visual operations and speech expression.

[0093] The parameter configuration unit is used to configure the task display duration, eye-tracking acquisition frequency, voice acquisition sampling rate, data synchronization timestamp accuracy, and voice broadcast duration threshold; the task display trigger unit is used to synchronously display the task content on the high-definition display interface, send a synchronization trigger signal, and start the synchronous acquisition of eye-tracking and voice data.

[0094] Optionally, the high-precision eye-tracking acquisition unit is used to acquire raw eye-tracking data (fixation point coordinates, duration, saccade speed, etc.) of the subject performing three types of tasks using a desktop non-contact eye tracker; the high-fidelity speech acquisition unit is used to acquire raw speech data (time-domain waveform, fundamental frequency, energy, etc.) of the subject using a noise-reduced lavalier microphone; and the hardware synchronization unit is used to precisely align eye-tracking and speech data based on FPGA timestamp technology, with a time difference of ≤1ms, to ensure the temporal correlation of the data.

[0095] Optionally, the eye-tracking gaze heatmap generation module includes a coordinate space mapping unit, a pixel heat value calculation unit, and a heatmap visualization encoding unit. The coordinate space mapping unit is used to linearly map the physical gaze coordinates in the time-aligned raw eye-tracking data to the pixel coordinate space of the task display interface. The pixel heat value calculation unit is used to calculate the pixel heat value for each pixel in the pixel coordinate space based on the corresponding cumulative gaze duration, gaze frequency, and a preset blink interference coefficient. The pixel heat value is positively correlated with the gaze duration and gaze frequency, and negatively correlated with the blink interference coefficient. The heatmap visualization encoding unit is used to generate a standardized eye-tracking gaze heatmap by using a color gradient encoding method to characterize the pixel heat value with color depth.

[0096] Optionally, the voiceprint feature set generation module includes a speech acoustic preprocessing unit, a voiceprint core feature extraction unit, and a cognitive-related prosodic feature extraction unit. The speech acoustic preprocessing unit performs pre-emphasis, framing, windowing, denoising, and endpoint detection on the time-aligned raw speech data, removing invalid speech segments to obtain preprocessed valid speech data. The voiceprint core feature extraction unit extracts voiceprint core acoustic features reflecting the physiological characteristics of speech based on the preprocessed valid speech data. These voiceprint core acoustic features include at least Mel-frequency cepstral coefficients, linearly predicted cepstral coefficients, fundamental frequency profile, formant frequency, and bandwidth. The cognitive-related prosodic feature extraction unit extracts speech prosodic features related to cognitive function based on the preprocessed valid speech data, serving as cognitive-related prosodic features. These cognitive-related prosodic features include at least the speech rate variation coefficient, fundamental frequency standard deviation, pause interval variation coefficient, and speech energy fluctuation. The voiceprint core acoustic features are then combined with the cognitive-related prosodic features to form a standardized voiceprint feature set.

[0097] Optionally, the multimodal feature fusion module includes an eye-tracking heatmap spatial feature extraction unit, a voiceprint temporal feature encoding unit, a feature normalization unit, and an attention cross-modal fusion unit. The eye-tracking heatmap spatial feature extraction unit uses a lightweight convolutional neural network as the backbone network to extract spatial features from the eye-tracking gaze heatmap. It extracts texture features, attention hotspot distribution features, and saccade trajectory morphology features layer by layer through depthwise separable convolutional layers and pooling layers. After passing through a global average pooling layer, it outputs the spatial features of the eye-tracking gaze heatmap. The voiceprint temporal feature encoding unit uses a dual-branch network combining a one-dimensional convolutional neural network and a long short-term memory network to perform temporal encoding on the voiceprint feature set. After concatenating the features output by the dual-branch network, it outputs the voiceprint temporal features of the voiceprint feature set through a fully connected layer. The dual-branch network includes a one-dimensional convolutional neural network branch and a long short-term memory network branch. The one-dimensional convolutional neural network branch extracts local acoustic association features of the voiceprint features through convolutional layers, while the long short-term memory network branch captures long-term temporal dynamic associations of the voiceprint features through a gating mechanism.

[0098] The feature standardization unit is used to perform Z-score standardization on the two types of feature vectors respectively, eliminating the difference in dimensions.

[0099] The attention cross-modal fusion unit performs the following steps: inputting standardized spatial features into a first feature mapping layer and outputting a first mapped feature; inputting standardized temporal acoustic features into a second feature mapping layer and outputting a second mapped feature; inputting the first mapped feature and the second mapped feature into an attention weight calculation layer and outputting eye-tracking modal weights and voiceprint modal weights, wherein the sum of the eye-tracking modal weights and the voiceprint modal weights is equal to 1; weighting the first mapped feature and the second mapped feature based on the eye-tracking modal weights and the voiceprint modal weights respectively; and concatenating and nonlinearly mapping the weighted first mapped feature and the second mapped feature through a feature fusion layer to output the multimodal fusion feature parameters.

[0100] Optionally, the intelligent evaluation module includes a feature input unit, a multimodal deep neural network model, and an evaluation result output unit. The feature input unit is used to input 384-dimensional multimodal fused feature parameters into the trained multimodal deep neural network model. The multimodal deep neural network model includes a feature mapping layer, a CNN deep feature extraction layer, an LSTM temporal feature extraction layer, an attention weighting layer, a fully connected layer, and a multi-output layer. The multi-output layer includes a first output branch, a second output branch, and a third output branch. The operation of inputting the multimodal fusion feature parameters into the trained multimodal deep neural network model and outputting the probability of the subject's cognitive impairment, the severity level of the impairment, and the assessment results of cognitive function dimension deficits includes: inputting the multimodal fusion feature parameters into the feature mapping layer for feature mapping; sequentially extracting joint spatial and temporal features through the CNN deep feature extraction layer and the LSTM temporal feature extraction layer; then strengthening the feature information related to cognitive impairment through the attention weighting layer; inputting the output features of the attention weighting layer into the fully connected layer; and inputting the output of the fully connected layer into the first output branch, the second output branch, and the third output branch, respectively, to obtain the probability of the presence of cognitive impairment through the first output branch, the severity level of the impairment through the second output branch, and the assessment results of cognitive function dimension deficits through the third output branch.

[0101] The evaluation result output unit is used to output and present the probability of the existence of the cognitive impairment, the classification result of the degree of impairment, and the assessment result of the deficit in the cognitive function dimension, which are output by the multimodal deep neural network model.

[0102] Furthermore, in clinical practice, the system in this embodiment is entirely non-invasive. Subjects only need to complete three audiovisual cognition tasks under the guidance of professionals, with a total time of less than 10 minutes. The entire process of data collection, processing, and evaluation is automated, and the evaluation results can be output instantly. It is suitable for various scenarios such as hospital neurology departments, geriatric departments, community health screening, and cognitive assessment in elderly care institutions. It is easy to operate, highly efficient, and highly specific.

[0103] It should be noted that the multimodal cognitive impairment auxiliary assessment system combined with eye movement data provided in this embodiment can realize all the functions and steps in the above method embodiments, solve the same technical problems, and achieve the same technical effects. The similarities will not be repeated here.

[0104] Example 3

[0105] Figure 6 A multimodal cognitive impairment assistive assessment system incorporating eye-tracking data, according to this embodiment, is illustrated. This system corresponds to the method described in Embodiment 1. (Reference) Figure 6As shown, the system includes: a processor 610; and a memory 620 connected to the processor 610, used to provide the processor 610 with instructions to process the following steps: acquiring raw eye movement data and raw speech data synchronously collected when the subject performs a preset visual-audio-sound co-cognition cognitive assessment task, and aligning the raw eye movement data and the raw speech data on the time axis; performing spatial mapping, heat value calculation and visualization processing on the time-aligned raw eye movement data to generate an eye movement gaze heatmap representing the distribution of visual attention; performing acoustic preprocessing on the time-aligned raw speech data to extract core acoustic features of the voiceprint and cognitive prosodic features to form a voiceprint feature set; extracting the spatial features of the eye movement gaze heatmap and the temporal acoustic features of the voiceprint feature set respectively, and after standardization processing, achieving deep fusion of cross-modal features through an attention mechanism to obtain multimodal fusion feature parameters; inputting the multimodal fusion feature parameters into a trained multimodal deep neural network model to output the probability of the subject's cognitive impairment, the degree of impairment, and the assessment results of the cognitive function dimension deficit.

[0106] It should be noted that the multimodal cognitive impairment auxiliary assessment system combined with eye movement data provided in this embodiment can realize all the functions and steps in the above method embodiments, solve the same technical problems, and achieve the same technical effects. The similarities will not be repeated here.

[0107] It should be noted that the method and system provided in this application obtain assessment results of the probability of cognitive impairment, the degree of impairment, and the dimensions of cognitive function deficit in subjects through techniques such as collecting eye-tracking and speech data, time-axis alignment, image processing, feature extraction, feature fusion, and information processing based on neural network models. Essentially, it is an information processing method that processes biometric data and performs computer-aided analysis. The aforementioned assessment results are not directly used to determine the final diagnosis of the subject's disease, but rather serve as intermediate or supplementary reference information for professional physicians to use in conjunction with other examination methods during clinical decision-making. Therefore, the method in this application does not constitute a disease diagnosis method in the sense of patent law.

[0108] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0109] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0110] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0111] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0112] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0113] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0114] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A multimodal cognitive impairment auxiliary assessment method combining eye-tracking data, characterized in that, include: Acquire raw eye movement data and raw speech data that are collected synchronously when the subject performs a preset visual-audio-sound co-cognition assessment task, and align the raw eye movement data and raw speech data on the time axis. Spatial mapping, heat value calculation and visualization processing are performed on the time-aligned raw eye-tracking data to generate an eye-tracking gaze heatmap representing the distribution of visual attention; Acoustic preprocessing is performed on the time-aligned raw speech data to extract the core acoustic features of the voiceprint and cognitive prosodic features, forming a voiceprint feature set; The spatial features of the eye-tracking gaze heatmap and the temporal acoustic features of the voiceprint feature set are extracted respectively. After standardization, cross-modal feature deep fusion is achieved through an attention mechanism to obtain multimodal fusion feature parameters. The multimodal fusion feature parameters are input into a trained multimodal deep neural network model, which outputs the probability of the subject's cognitive impairment, the severity of the impairment, and the assessment results of the cognitive function dimension deficit.

2. The method according to claim 1, characterized in that, The preset audiovisual co-cognition assessment task includes: The image-based descriptive task is used to assess the coordination between visual-spatial perception and verbal expression. The text reading comprehension task is used to assess language processing and attention maintenance abilities; Visual tracking and broadcasting tasks are used to evaluate visual execution capabilities and voice timing control capabilities; The tasks of describing meaning from pictures, reading and understanding text, and visual tracking and broadcasting require the subjects to simultaneously perform visual operations and verbal expressions.

3. The method according to claim 1, characterized in that, The process of spatial mapping, heat value calculation, and visualization of time-aligned raw eye-tracking data to generate an eye-tracking gaze heatmap representing the distribution of visual attention includes: The physical gaze coordinates in the time-aligned raw eye-tracking data are linearly mapped proportionally to the pixel coordinate space of the task display interface. For each pixel in the pixel coordinate space, a pixel heat value is calculated based on the corresponding cumulative gaze duration, number of gazes, and a preset blink interference coefficient. The pixel heat value is positively correlated with the gaze duration and the number of gazes, and negatively correlated with the blink interference coefficient. A standardized eye-tracking heatmap is generated by using color gradient encoding to characterize the pixel heat value with color depth.

4. The method according to claim 1, characterized in that, The process of performing acoustic preprocessing on time-aligned raw speech data to extract core acoustic features and cognitively relevant prosodic features to form a voiceprint feature set includes: The time-aligned raw speech data is pre-emphasized, framed, windowed, denoised, and endpoint detected to remove invalid speech segments and obtain pre-processed valid speech data. Based on the preprocessed effective speech data, core acoustic features of the voiceprint reflecting the physiological characteristics of speech are extracted. The core acoustic features of the voiceprint include at least Mel-frequency cepstral coefficients, linearly predicted cepstral coefficients, fundamental frequency profile, formant frequency, and bandwidth. Based on the preprocessed effective speech data, speech prosodic features related to cognitive function are extracted as cognitive prosodic features. The cognitive prosodic features include at least speech rate variation coefficient, fundamental frequency standard deviation, pause interval variation coefficient, and speech energy fluctuation. The core acoustic features of the voiceprint are combined with the cognitive prosodic features to form a standardized voiceprint feature set.

5. The method according to claim 1, characterized in that, The operations of extracting the spatial features of the eye-tracking gaze heatmap and the temporal acoustic features of the voiceprint feature set respectively include: A lightweight convolutional neural network is used as the backbone network to extract spatial features from the eye-tracking gaze heatmap. Texture features, attention hotspot distribution features, and saccade trajectory morphology features are extracted layer by layer through depthwise separable convolutional layers and pooling layers. After passing through a global average pooling layer, the spatial features of the eye-tracking gaze heatmap are output. A dual-branch network combining a one-dimensional convolutional neural network and a long short-term memory network is used to perform temporal encoding on the voiceprint feature set. After the features output by the dual-branch network are concatenated, the voiceprint temporal features of the voiceprint feature set are output through a fully connected layer. The dual-branch network includes a one-dimensional convolutional neural network branch and a long short-term memory network branch. The one-dimensional convolutional neural network branch extracts local acoustic association features of voiceprint features through convolutional layers, while the long short-term memory network branch captures long-term temporal dynamic associations of voiceprint features through a gating mechanism.

6. The method according to claim 1, characterized in that, The operation of achieving deep fusion of cross-modal features through an attention mechanism to obtain multimodal fusion feature parameters includes: The standardized spatial features are input into the first feature mapping layer, and the first mapped features are output. The standardized temporal acoustic features are input into the second feature mapping layer, and the second mapped features are output. The first mapped feature and the second mapped feature are input into the attention weight calculation layer, and the eye-tracking modal weight and the voiceprint modal weight are output. The sum of the eye-tracking modal weight and the voiceprint modal weight is equal to 1. Based on the eye-tracking modal weights and the voiceprint modal weights, the first mapped features and the second mapped features are weighted respectively; The feature fusion layer concatenates and performs nonlinear mapping on the weighted first and second mapped features to output the multimodal fusion feature parameters.

7. The method according to claim 1, characterized in that, The multimodal deep neural network model includes a feature mapping layer, a CNN deep feature extraction layer, an LSTM temporal feature extraction layer, an attention weighting layer, a fully connected layer, and a multi-output layer. The multi-output layer includes a first output branch, a second output branch, and a third output branch. Furthermore, the operation of inputting the multimodal fusion feature parameters into a trained multimodal deep neural network model and outputting the probability of the subject's cognitive impairment, the severity level of the impairment, and the assessment results of the cognitive function dimension deficit includes: After the multimodal fusion feature parameters are input into the feature mapping layer for feature mapping, the joint spatial and temporal features are extracted sequentially by the CNN deep feature extraction layer and the LSTM temporal feature extraction layer, and then the feature information related to cognitive impairment is enhanced by the attention weighting layer. The output features of the attention-weighted layer are input into the fully connected layer, and the output of the fully connected layer is input into the first output branch, the second output branch, and the third output branch, respectively, so as to obtain the probability of the existence of the cognitive impairment through the first output branch, obtain the impairment severity classification result through the second output branch, and obtain the deficit assessment result of the cognitive function dimension through the third output branch.

8. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, the method described in any one of claims 1 to 7 is performed by a processor.

9. A multimodal cognitive impairment auxiliary assessment system incorporating eye-tracking data, characterized in that, include: The data acquisition and alignment module is used to acquire raw eye movement data and raw speech data collected synchronously by the subject when performing a preset visual-audio-sound co-cognition assessment task, and to align the raw eye movement data and the raw speech data along the time axis. The eye-tracking gaze heatmap generation module is used to perform spatial mapping, heat value calculation and visualization processing on the time-aligned raw eye-tracking data to generate an eye-tracking gaze heatmap that represents the distribution of visual attention. The voiceprint feature set generation module is used to perform acoustic preprocessing on the time-aligned raw speech data, extract the core acoustic features of the voiceprint and cognitive prosodic features, and form a voiceprint feature set. The multimodal feature fusion module is used to extract the spatial features of the eye-tracking gaze heatmap and the temporal acoustic features of the voiceprint feature set, respectively. After standardization, the cross-modal feature deep fusion is achieved through an attention mechanism to obtain multimodal fusion feature parameters. The intelligent assessment module is used to input the multimodal fusion feature parameters into a trained multimodal deep neural network model and output the probability of the subject's cognitive impairment, the degree of impairment, and the assessment results of the cognitive function dimension deficit.

10. A multimodal cognitive impairment auxiliary assessment system incorporating eye-tracking data, characterized in that, include: processor; A memory, connected to the processor, for providing the processor with instructions to perform the following processing steps: Acquire raw eye movement data and raw speech data that are collected synchronously when the subject performs a preset visual-audio-sound co-cognition assessment task, and align the raw eye movement data and raw speech data on the time axis. Spatial mapping, heat value calculation and visualization processing are performed on the time-aligned raw eye-tracking data to generate an eye-tracking gaze heatmap representing the distribution of visual attention; Acoustic preprocessing is performed on the time-aligned raw speech data to extract the core acoustic features of the voiceprint and cognitive prosodic features, forming a voiceprint feature set; The spatial features of the eye-tracking gaze heatmap and the temporal acoustic features of the voiceprint feature set are extracted respectively. After standardization, cross-modal feature deep fusion is achieved through an attention mechanism to obtain multimodal fusion feature parameters. The multimodal fusion feature parameters are input into a trained multimodal deep neural network model, which outputs the probability of the subject's cognitive impairment, the severity of the impairment, and the assessment results of the cognitive function dimension deficit.

Images