Multi-modal cognitive impairment evaluation system based on multi-dimensional cognitive function hypergraph

Abstract

The invention discloses a multi-modal cognitive impairment evaluation system based on a multi-dimensional cognitive function hypergraph. The multi-modal cognitive impairment evaluation system comprises a multi-modal signal acquisition system, a cognitive function evaluation normal form design, a universal feature extraction framework of each modal and online hypergraph learning based on a multi-dimensional cognitive function. The system constructs a dimensionality evaluation normal form containing the cognitive function based on multi-modal physiological and behavior signals, so that comprehensive evaluation of the cognitive function of the elderly is realized. Firstly, physiological and behavior responses of a user under different tasks and situations are recorded and analyzed in real time by fusing multi-mode signals such as electroencephalogram, electrocardio, voice and facial expressions. And then, mining fusion representation among the multi-modal signals, generating unified feature representation, introducing an online hypergraph of clinical knowledge, capturing a complex multi-modal data relationship, and providing an analysis result containing cognitive function evaluation. Finally, in combination with the technical means, the system can realize early detection of cognitive impairment.

Description

[0001] The field

[0002] The present invention relates to the field of cognitive impairment assessment and multimodal fusion technology, and in particular to a multimodal cognitive impairment assessment system based on a multidimensional cognitive function hypergraph. Background Art

[0003] As the global aging trend intensifies, cognitive health issues in the elderly are increasingly receiving widespread attention. According to the World Health Organization, there are currently more than 55 million people suffering from dementia worldwide, of which more than 60% live in low- and middle-income countries. In addition, about 10 million new cases appear each year, and it is estimated that by 2050, the number of dementia patients worldwide will increase to 152.8 million. In China, there are about 38 million patients with mild cognitive impairment (MCI) and about 15 million patients with dementia among people over 65 years old. In particular, Alzheimer's disease has become the main cause of cognitive impairment, with more than 10 million patients. Alzheimer's disease is characterized by a variety of cognitive impairments, often accompanied by symptoms such as memory impairment, aphasia, apraxia, agnosia, visual-spatial skill impairment, executive dysfunction and behavioral changes.

[0004] Early warning and intervention are widely considered to be important measures to prevent the occurrence of cognitive impairment, especially at the community level, which is the primary place for identification. As a transitional state between normal aging and early dementia, the early identification of mild cognitive impairment (MCI) has become a public health issue that needs to be addressed urgently. However, due to the significant heterogeneity of individuals, the decline of cognitive function often manifests itself in a variety of different dimensions and forms, and a single assessment method is difficult to fully reflect the actual cognitive status of patients. Therefore, it is particularly important to conduct detailed assessments of different cognitive function dimensions (such as visual-spatial and executive functions, naming, attention, language, abstract thinking, delayed recall, and orientation).

[0005] To address this challenge, we proposed a wearable multimodal cognitive impairment assessment system. The system takes full advantage of the low cost, portability, and real-time monitoring capabilities of wearable devices, making it an ideal early warning tool. By fusing multimodal data, the system can not only capture subtle changes that may be overlooked by traditional detection methods, but also achieve accurate identification and analysis of cognitive impairment. Studies have shown that machine learning combined with a single wearable sensor can achieve high-precision step length estimation in the elderly and patients with neurological diseases. This comprehensive assessment method provides valuable information for early detection and timely intervention, greatly improving the quality of life of elderly patients. Summary of the invention

[0006] The multimodal cognitive impairment assessment system of the present invention uses a variety of physiological and behavioral signals in a comprehensive manner, and realizes a comprehensive and dynamic cognitive ability assessment of elderly users through an innovative cognitive function assessment paradigm. The system designs multi-dimensional cognitive function assessment tasks, focusing on key cognitive function areas such as spatial cognition, understanding decision-making, verbal communication, attention execution and episodic memory. During the assessment process, the system combines multimodal signals such as electrocardiogram (ECG), electroencephalogram (EEG), speech and facial expressions, and captures the physiological reactions and behavioral performance of users when performing cognitive tasks in real time through specific task stimulation. In terms of signal encoding, the system constructs a special encoder for the signal characteristics of different modalities. ECG signals provide important information for emotional state and cognitive function by reflecting the activity state of the autonomic nervous system; EEG signals use changes in electrical activity to capture the neural activity patterns of individuals when performing different cognitive tasks; speech signals contain acoustic features and semantic information, which can deeply analyze the individual's language expression ability and its relationship with cognitive state; facial expressions reveal the user's performance in cognitive tasks by analyzing facial action units. The encoder of each modality uses a machine learning algorithm to perform in-depth analysis and feature extraction of the signal, providing a high-dimensional, interpretable feature vector for subsequent fusion. In terms of modal fusion methods, the system adopts a post-encoding feature fusion strategy to effectively integrate signal features from different modalities. The system constructs a dynamic knowledge graph and uses a hypergraph structure to effectively associate the inherent logical relationship between multimodal signal features and cognitive function symptoms. Through the deep learning framework of the hypergraph, the system can learn the potential associations between different tasks and modalities, and then build a comprehensive cognitive function assessment model. This fusion not only improves the accuracy of the assessment, but also enhances the model's ability to adapt to individual heterogeneity, making the assessment results more comprehensive and reliable. Finally, the system combines clinical diagnostic knowledge to provide strong support for the early identification and intervention of cognitive impairment, thereby promoting the scientific management and promotion of cognitive health.

[0007] In order to achieve the above purpose, the technical solution adopted by the present invention is: a multimodal cognitive impairment assessment system based on a multidimensional cognitive function hypergraph, characterized in that a cognitive function assessment paradigm is constructed through a multimodal signal acquisition system, interpretable features matching cognitive functions in physiological and behavioral signals are simultaneously mined, physiological and behavioral signal feature encoders are designed, and finally multimodal signal feature fusion is realized to construct an online hypergraph assessment strategy for multidimensional cognitive functions. Specifically including:

[0008] Multimodal cognitive assessment module: The multimodal cognitive assessment system includes multimodal signal acquisition and multidimensional cognitive function assessment paradigm. This module integrates a real-time acquisition system for multiple physiological and behavioral signals, aiming to comprehensively assess the individual's cognitive function status. By combining multimodal signals such as electrocardiogram, electroencephalogram, speech and facial expressions, the system can effectively capture and analyze multidimensional features related to cognitive processes. In the cognitive assessment paradigm, experimental tasks are designed for multiple cognitive dimensions such as spatial cognition, decision understanding and decision making, verbal communication, attention execution and situational memory. Through interactive games and actual task scenarios, the performance of subjects in different cognitive functions is obtained. This module not only realizes the precise acquisition and processing of signals, but also improves the comprehensiveness and accuracy of the assessment through multimodal fusion technology, providing a solid data foundation for the early detection and intervention of cognitive impairment.

[0009] Multimodal signal fusion module: The multimodal signal fusion module includes physiological and behavioral signal feature encoders and hypergraph fusion based on multidimensional cognitive functions. This module integrates multimodal signal encoders and hypergraph models to achieve deep analysis and fusion of ECG, EEG, speech and facial expression signals. Through the feature encoders of each modality, the system can extract the physiological and behavioral information contained in each signal, and represent and summarize the spatiotemporal features. ECG signals are used to evaluate emotional states and changes in the activity of the autonomic nervous system, EEG signals are used to capture the electrical activity of the brain in cognitive tasks, speech signals reflect emotions and cognitive abilities through acoustic and semantic features, and facial expressions reveal the emotional state of individuals. Subsequently, the constructed hypergraph model associates the features of each modality with the clinical symptoms of cognitive impairment through a dynamic knowledge graph, achieving deep mapping between multidimensional cognitive functions and emotional states. This comprehensive approach not only enhances the understanding of individual cognitive states, but also provides an important basis for formulating personalized intervention strategies.

[0010] As a further improvement of the present invention, the multimodal signal acquisition system is designed to acquire and integrate multidimensional data from different physiological and behavioral signals in real time to comprehensively evaluate the cognitive function status of an individual. The system combines a variety of sensors, including electrocardiogram (ECG), electroencephalogram (EEG), speech recognition and facial expression analysis equipment. Through these sensors, the physiological reactions and behavioral performances of individuals can be monitored simultaneously, providing rich information. The system is responsible for collecting ECG, EEG, speech and facial expression signals from the user. Through wearable devices and sensors, the user's physiological changes and emotional reactions are monitored in real time. The collected signals are preprocessed, including denoising, signal enhancement, etc., to improve data quality. After processing, the signal will be transmitted to the feature encoder for further analysis.

[0011] As a further improvement of the present invention, the multidimensional cognitive function evaluation paradigm aims to comprehensively analyze the cognitive abilities of elderly users from multiple perspectives by designing a comprehensive multidimensional cognitive function evaluation paradigm. This paradigm is based on the principle of combining theory with practice and revolves around seven main cognitive function dimensions, including visual-spatial and executive functions, naming, attention, language, abstract thinking, delayed recall, and orientation. The evaluation of each dimension is achieved through corresponding cognitive tasks to ensure that the user's cognitive performance can be accurately captured:

[0012] (1) During the assessment process, electrocardiogram (ECG) signals are used to monitor the user's physiological state, reflect the activity of the autonomic nervous system, and provide the user's emotional response and stress level during the performance of cognitive tasks.

[0013] (2) Electroencephalogram (EEG) signals record brain activity through an electrode array, analyze brain wave patterns under different cognitive tasks, identify electrical activity characteristics related to specific cognitive functions, and thus reveal the user's cognitive state.

[0014] (3) The analysis of speech signals is divided into two levels: acoustic and semantic. Acoustic information includes pitch, volume, speech rate and rhythm. By monitoring the vocalization process, it reflects the user's emotional changes and cognitive ability. Semantic information focuses on the user's vocabulary selection and sentence structure in language communication, helping to evaluate their cognitive functions such as language fluency, attention and memory.

[0015] (4) Facial expressions are important behavioral signals. By analyzing facial action units (AUs), we can capture the user’s emotional expressions and their changes during cognitive tasks. Patients with cognitive impairment often show stiffness and slow movements in their facial expressions. By identifying these subtle changes, the system can provide valuable information for the assessment of cognitive status.

[0016] As a further improvement of the present invention, the physiological and behavioral signal feature encoder is used to perform feature encoding on the collected signals. In-depth analysis and feature extraction of physiological and behavioral signals from different modalities are performed to achieve more accurate cognitive function assessment. The module includes multiple dedicated encoders for processing electrocardiogram (ECG), electroencephalogram (EEG), speech signals and facial expressions respectively.

[0017] In terms of ECG and EEG signal processing, the encoder reveals the activity of the individual's autonomic nervous system by analyzing heart rate variability and other ECG features, and extracts signal patterns related to emotional states through specific algorithms. The EEG signal encoder uses a multi-head self-attention mechanism and a convolutional neural network to extract the characteristics of brain wave frequency bands such as α, β, and γ from the time-frequency domain to capture the changes in the brain's electrical activity under different cognitive tasks. The physiological signal encoder consists of a convolutional Sinc filter and a multi-head self-attention mechanism, where the Sinc filter is:

[0018] g[n,f1,f2]=2f2sinc(2πf2n)-2f1sinc(2πf1n)(1)

[0019] The low cutoff frequency and high cutoff frequency are f1, f2, n is the sampling point index, the sinc function is defined as sinc(x) = sin(x) / x, and the cutoff frequency can be in [0, f s / 2] randomly initialized within the range, where f s Represents the sampling frequency of the input signal. The Sinc filter is implemented by a one-dimensional convolutional neural network. After obtaining the time series features divided by frequency bands, it is flipped along the time dimension and outputs the physiological signal feature vector x through the multi-head self-attention mechanism layer. MSA :

[0020] x MSA =MSA(LN(x sinc ))(2)

[0021] In the formula, LN is the LayerNorm layer, x sinc is the frequency band temporal feature of the Sinc filter, where MSA is the multi-head self-attention mechanism. The Sinc adaptive convolution can interpretably extract the frequency band features, and by flipping the time and channel dimensions, the attention score is calculated in the temporal direction to more effectively integrate the global information.

[0022] The speech signal encoder is divided into two parts: acoustic and semantic. Acoustic features are extracted through Mel spectrograms and modeled using the Transformer architecture. This model can directly capture long-distance global context features in speech signals, providing an important basis for emotion and cognitive function analysis. Semantic features are processed using DistilBERT to analyze the user's vocabulary selection and sentence structure to evaluate their language fluency and cognitive ability. The combination of the two enables the speech signal encoder to fully reflect the user's language performance and emotional state.

[0023] The expression signal encoder can detect the emotional changes of users in cognitive tasks by analyzing facial action units (AU). The encoder adopts a deep learning architecture based on U-net, combined with a residual learning mechanism, to improve the ability to capture subtle changes in facial expressions. The residual with mask is calculated as:

[0024]

[0025] In the formula, F R is the feature map transformed by the residual layer; F M is the feature map after passing through the U-net network; F N is the final feature map. By focusing on the dynamic changes of facial features, the system can accurately assess the user's emotional state and cognitive function, especially the facial expressions during task execution.

[0026] As a further improvement of the present invention, the hypergraph fusion based on multidimensional cognitive function constructs and updates the knowledge graph dynamically, with each node representing the multimodal features of each cognitive task and each edge representing a different cognitive function, thereby constructing a hypergraph model to deeply mine the complex relationship between multimodal signals and cognitive functions. This hypergraph model not only integrates the feature representation of each modality, but also can adapt to the changes of different tasks and emotional states, thereby achieving a comprehensive assessment of mild cognitive impairment.

[0027] The hypergraph model consists of vertices (nodes) S = S1, S2, ..., S m and hyperedges (edges) D = D1, D2, ..., D n The vertices represent the multimodal features of users under different cognitive tasks, while the hyperedges associate different dimensions of emotions and cognitive functions. In this system, the vertices not only cover traditional physiological signal features (such as ECG and EEG), but also behavioral signals (such as voice and facial expressions). These features are encoded through deep learning models to form high-dimensional feature vectors. The establishment of hyperedges is based on the relationship between the clinical symptoms of each cognitive function and multimodal signals, and weights are assigned through expert knowledge and statistical analysis, so as to achieve dynamic mapping between task status and emotional cognitive dimensions.

[0028] In the hypergraph model, feature vectors from different modalities are first merged through a multimodal fusion algorithm. Through an adaptive aggregation strategy, the model can dynamically adjust the fusion weights of features according to the current task and emotional state. This process enables the system to extract deep features related to cognitive impairment while maintaining sensitivity to emotional changes. Subsequently, by defining the association matrix of the hypergraph, vertices under different tasks are associated with hyperedges. The update process is as follows: based on online data, when new symptom manifestations are detected, they are added to the hypergraph as new nodes. New hyperedges are constructed, and new symptom combinations are discovered through an incremental clustering algorithm, and hyperedges are added or deleted accordingly. new , which consists of symptoms S1, S2, …, S k , add it as a new hyperedge to the hypergraph, and calculate its weight:

[0029]

[0030] where φ(i) is the embedding vector of symptom i, |D new | is a super edge D new The number of symptoms in . Then update the disease-symptom association matrix B. For the newly identified disease-symptom association, update the B matrix:

[0031]

[0032] Then, by optimizing the corresponding objective function, the association matrix B and the new hyperedge set are trained and updated, and the edge weights are dynamically updated according to the association strength between the nodes in the hypergraph, and the hypergraph embedding model is updated. This can be achieved by optimizing the following objective function:

[0033]

[0034] Among them, φ(d) is the embedding representation of the edge, f(d) is the hyperedge representation reconstructed from the vertex embedding, θ is the parameter, and λ is the normalization term. Finally, the nodes and hyperedges on the graph are dynamically updated, and the model reasoning process can be traced back to display the decision basis containing the clinical knowledge graph, so as to improve the auxiliary diagnosis efficiency and interpretability of cognitive impairment.

[0035] Compared with the prior art, the beneficial effects of the present invention are:

[0036] 1. Application of multimodal wearable devices: This invention effectively integrates multimodal signals such as ECG, EEG, speech and facial expressions in the assessment of cognitive impairment, and utilizes the low cost, portability and real-time monitoring capabilities of wearable devices to achieve efficient, convenient and accurate cognitive function assessment, overcoming the limitations of traditional assessment methods.

[0037] 2. Multidimensional cognitive function assessment paradigm: The seven-dimensional cognitive function assessment paradigm of the present invention covers seven dimensions, including visual-spatial and executive functions, naming, attention, language, abstract thinking, delayed recall and orientation. Through real-time monitoring of multimodal signals, this paradigm provides a comprehensive and dynamic assessment of cognitive status, providing a scientific basis for personalized intervention and clinical decision-making. This paradigm effectively addresses individual heterogeneity and improves the accuracy and practicality of cognitive assessment.

[0038] 3. Focus on individual heterogeneity and interpretability: This system uses a multidimensional cognitive function assessment paradigm to deeply explore the differences in cognitive function performance among different individuals, focusing on the mapping relationship between different modal signals and specific cognitive functions, significantly improving the personalization and interpretability of the assessment results and meeting the specific needs of different patients.

[0039] 4. Efficient signal feature coding and fusion method: By designing a specific signal encoder, efficient spatiotemporal feature extraction and fusion are performed based on the characteristics of different modal signals, which improves the information expression ability and analysis depth, provides technical support for the comprehensive evaluation of multimodal signals, and enhances the overall performance of the system.

[0040] 5. Construction of dynamic knowledge graph: This invention constructs a dynamic hypergraph knowledge graph based on clinical diagnosis, integrating cognitive functions and emotional characteristics to form an internal logical relationship. This innovation not only improves the interpretability of data, but also provides new perspectives and methods for future research. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the overall multimodal signal acquisition system of the present invention.

[0042] Figure 2 A flow chart of the cognitive assessment process of the multimodal system of the present invention.

[0043] Figure 3 It is a schematic diagram of the structure of the physiological signal encoder of the present invention.

[0044] Figure 4 It is a schematic diagram of the structure of the speech acoustic and semantic encoder of the present invention.

[0045] Figure 5 This is a schematic diagram of the structure of the facial expression encoder of the present invention.

[0046] Figure 6 It is a schematic diagram of the hypergraph fusion structure based on multi-dimensional cognitive function of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be described in more detail below with reference to the accompanying drawings and implementation examples.

[0048] The overall multimodal acquisition system is Figure 1 As shown in the figure, the architecture of the multimodal signal acquisition system is shown in detail. The system realizes the synchronous collection of multiple physiological and behavioral signals through wearable devices, including electrocardiogram (ECG), electroencephalogram (EEG), voice signals, and facial expressions. The collection of these signals can reflect the user's physiological state and emotional changes during the cognitive assessment process in real time. After the signal is collected, the data will be processed and encoded in real time by the mobile device, and the signal will be analyzed in multiple dimensions to extract key features related to cognitive function. The processed data will then be transmitted to the evaluation output module, combined with the standards and models of cognitive assessment, to generate the corresponding cognitive assessment results. Figure 2 The specific process of cognitive assessment for users is demonstrated, including real-time monitoring of signals and various steps of the assessment task. The process is designed to ensure that each link can operate efficiently so as to accurately reflect the user's cognitive status and functional level, and provide a scientific basis for subsequent intervention and treatment.

[0049] Table 1 lists the assessment items for different cognitive function dimensions, including seven cognitive function dimensions: visual-spatial ability, executive ability, attention, language ability, memory, abstract thinking, and orientation.

[0050] Table 1 Multidimensional cognitive function assessment paradigm

[0051]

[0052] In order to comprehensively evaluate the individual's performance in these cognitive dimensions, each project has designed corresponding tasks and scoring criteria. These tasks are presented in the form of games, aiming to increase the enthusiasm and participation of participants. In various cognitive function test games, participants will perform corresponding task operations in specific situations, which can stimulate and reflect different cognitive functions. For example, in the visual-spatial ability test, participants may need to complete spatial positioning tasks in a virtual environment, while in the execution ability test, it may involve the sequential execution of multi-step tasks. Through these designs, the system can monitor the changes in the physiological and behavioral signals of participants during the execution of tasks in real time, including electrocardiogram signals, brain waves, voice characteristics and facial expressions.

[0053] During the task execution, the wearable device will synchronously collect these signals and capture the participants' physiological reactions and emotional changes under different cognitive activities. These data can not only reflect the participants' cognitive performance, but also provide multi-dimensional support and basis for the evaluation report by analyzing their physiological signals and behavioral reactions, and ultimately form a comprehensive and accurate evaluation result. This comprehensive evaluation method provides a reliable basis for subsequent intervention measures and helps to effectively intervene and support individuals according to their specific needs.

[0054] Figure 3The structure and working principle of the physiological signal encoder are demonstrated, mainly including the processing part of ECG and EEG signals. The encoder uses convolutional neural network and self-attention mechanism to extract features for different frequency bands, thereby realizing real-time analysis of emotional state and cognitive function in ECG and EEG signals. Figure 4 The Transformer architecture based on Mel-spectrogram was demonstrated. The encoder can analyze an individual's language ability, emotional expression, and cognitive state by extracting acoustic and semantic features. By converting speech signals into feature vectors, the system can evaluate cognitive functions such as language fluency and memory. Figure 5 The design of the expression encoder is presented, which uses a deep learning network based on a residual structure to analyze facial action units. The encoder can capture subtle changes in the facial expressions of MCI patients, thereby assisting in identifying their emotional state and degree of cognitive impairment, providing important data support for overall assessment.

[0055] Figure 6 This is a schematic diagram of the hypergraph fusion structure based on multidimensional cognitive functions. The figure shows how to integrate multiple physiological and behavioral signal features into cognitive function assessment through the hypergraph model. The design of the hypergraph aims to effectively capture the intrinsic relationship between different cognitive tasks and cognitive functions, and provide rich semantic information for cognitive impairment assessment. In this figure, the vertices of the hypergraph represent the multimodal fusion features of individuals under specific tasks. Each vertex corresponds to a specific cognitive function dimension, such as visuospatial ability, executive ability, attention, etc., which can be evaluated through specific task design. For example, for visuospatial ability, tasks may include identifying and memorizing spatial locations, while the evaluation of executive ability may involve the ability to solve multi-step problems. Hyperedges are used to represent the association between different tasks. In the specific implementation process, when the participants perform the tasks, the system will collect relevant physiological signals in real time and evaluate the performance of each cognitive function through the preset hyperedge weights. This structure enables the hypergraph to be dynamically updated and adjusted to adapt to the changes in the individual's cognitive state, thereby providing more accurate evaluation results. Overall, the hypergraph fusion structure based on multidimensional cognitive functions not only enhances the flexibility and adaptability of cognitive assessment, but also provides strong support for individualized cognitive intervention.

[0056] The combination of these figures will effectively illustrate the specific embodiments of the present invention and its practical application, and demonstrate its innovation and practicality in the assessment of cognitive impairment.

Claims

1. A multimodal cognitive impairment assessment system based on a multidimensional cognitive function hypergraph, characterized in that: It includes a multimodal cognitive assessment module and a multimodal signal fusion module; the multimodal cognitive assessment module includes a multimodal signal acquisition system and a multidimensional cognitive function assessment paradigm, and the multimodal signal fusion module includes a physiological and behavioral signal feature encoder and a hypergraph fusion based on multidimensional cognitive function; through the multimodal signal acquisition system, a cognitive function assessment paradigm is constructed, and interpretable features matching cognitive functions in physiological and behavioral signals are simultaneously mined, and physiological and behavioral signal feature encoders are designed, and finally multimodal signal feature fusion is realized, and an online hypergraph assessment strategy for multidimensional cognitive function is constructed.

2. The multimodal cognitive impairment assessment system based on a multidimensional cognitive function hypergraph as claimed in claim 1, characterized in that: The multimodal signal acquisition system is used to acquire and integrate multidimensional data from different physiological and behavioral signals in real time to comprehensively evaluate the cognitive function status of an individual; It collects electrocardiogram, electroencephalogram, voice and facial expression signals from the user, and monitors the user's physiological changes and emotional reactions in real time through wearable devices and sensors. The collected signals are pre-processed and then transmitted to the signal feature encoder for further analysis.

3. The multimodal cognitive impairment assessment system based on a multidimensional cognitive function hypergraph as claimed in claim 2, characterized in that: The multidimensional cognitive function assessment paradigm is used to comprehensively analyze the cognitive abilities of elderly users from multiple perspectives. The paradigm revolves around seven main cognitive function dimensions, including visual-spatial and executive functions, naming, attention, language, abstract thinking, delayed recall, and orientation. The assessment of each dimension is achieved through corresponding cognitive tasks to ensure that the user's cognitive function performance can be captured.

4. The multimodal cognitive impairment assessment system based on a multidimensional cognitive function hypergraph as claimed in claim 3, characterized in that: The multidimensional cognitive function assessment paradigm is used during the assessment process: (1) ECG signals are used to monitor the user's physiological state, reflect the activity of the autonomic nervous system, and provide the user's emotional response and stress level during the performance of cognitive tasks; (2) EEG signals record brain activity through an electrode array, analyze brain wave patterns under different cognitive tasks, identify electrical activity characteristics related to specific cognitive functions, and thus reveal the user's cognitive state; (3) The analysis of speech signals is divided into two levels: acoustic and semantic. Acoustic information includes pitch, intensity, speech rate and rhythm. By monitoring the vocalization process, it reflects the user's emotional changes and cognitive ability. Semantic information focuses on the user's vocabulary selection and sentence structure in language communication, helping to evaluate their cognitive functions such as language fluency, attention and memory. (4) Facial expression signals: by analyzing facial action units, the user’s emotional expressions and changes in facial expressions and movements during cognitive tasks can be captured.

5. The multimodal cognitive impairment assessment system based on a multidimensional cognitive function hypergraph as claimed in claim 4, characterized in that: The physiological and behavioral signal feature encoder is used to perform feature encoding on the collected signals, and to perform in-depth analysis and feature extraction on physiological and behavioral signals from different modalities, so as to achieve more accurate cognitive function assessment; it includes multiple dedicated encoders for processing electrocardiogram, electroencephalogram, speech signals and facial expressions respectively.

6. The multimodal cognitive impairment assessment system based on a multidimensional cognitive function hypergraph as claimed in claim 5, characterized in that: (1) In terms of ECG and EEG signal processing, the physiological signal encoder is composed of a convolutional neural network and a multi-head self-attention mechanism. It analyzes heart rate variability and other ECG features to reveal the individual's autonomic nervous system activity. For EEG signals, the characteristics of brain wave frequency bands are extracted from the time-frequency domain to capture the changes in the brain's electrical activity under different cognitive tasks and provide in-depth information about cognitive activities. The physiological signal encoder is composed of a convolutional Sinc filter and a multi-head self-attention mechanism, where the Sinc filter is: g[n,f1,f2]=2f2sinc(2πf2n)-2f1sinc(2πf1n)(1) The low cutoff frequency and the high cutoff frequency are f1 and f2, n is the sampling point index, and the sinc function is defined as sinc(x) = sin(x) / x. The cutoff frequency is in the range [0, f s / 2] randomly initialized within the range, f s Represents the sampling frequency of the input signal; the Sinc filter is implemented by a one-dimensional convolutional neural network. After obtaining the time series features divided by frequency bands, it is flipped along the time dimension and outputs the physiological signal feature vector x through the multi-head self-attention mechanism layer. MSA : x MSA =MSA(LN(x sinc ))(2) Where LN is the LayerNorm layer, x sinc is the frequency band temporal feature of Sinc filter, where MSA is the multi-head self-attention mechanism; Sinc adaptive convolution can interpretably extract frequency band features, and by flipping the time and channel dimensions, the attention score is calculated in the temporal direction to more effectively integrate global information; (2) The speech signal encoder is divided into two parts: acoustic and semantic. Acoustic features are extracted through Mel spectrograms and modeled using the Transformer architecture. Semantic features are processed using DistilBERT to analyze the user's vocabulary choice and sentence structure to assess their language fluency and cognitive ability. The combination of the two enables the speech signal encoder to fully reflect the user's language performance and emotional state. (3) The expression signal encoder can detect the emotional changes of users in cognitive tasks by analyzing facial action units. The encoder adopts a deep learning architecture based on U-net and combines it with a residual learning mechanism to improve the ability to capture subtle changes in facial expressions. The residual with mask is calculated as: In the formula, F R is the feature map transformed by the residual layer; F M It is the feature map after passing through the U-net network; F N is the final feature map; By focusing on the dynamic changes of facial features, it is possible to accurately assess the user's emotional state and cognitive function, especially the facial expressions during task execution.

7. The multimodal cognitive impairment assessment system based on a multidimensional cognitive function hypergraph as claimed in claim 6, characterized in that: The hypergraph fusion based on multidimensional cognitive functions constructs a hypergraph model to mine the complex relationship between multimodal signals and cognitive functions through dynamically constructed and updated knowledge graphs, in which each node represents the multimodal features of each cognitive task and each edge represents a different cognitive function. It not only integrates the feature representations of each modality, but also can adapt to changes in different tasks and emotional states, thereby achieving a comprehensive assessment of mild cognitive impairment.

8. The multimodal cognitive impairment assessment system based on a multidimensional cognitive function hypergraph as claimed in claim 7, characterized in that: The hypergraph model consists of vertices and hyperedges, where vertices represent multimodal features of users under different cognitive tasks S = S1, S2, ..., S m , while hyperedges are associated with different cognitive functions D=D1,D2,...,D n Among them, vertices not only cover traditional physiological signal features, but also behavioral signals. These features are encoded through deep learning models to form high-dimensional feature vectors; the establishment of hyperedges is based on the relationship between the clinical symptoms of various cognitive functions and multimodal signals, and weights are assigned through expert knowledge and statistical analysis, thereby realizing the dynamic mapping of task status and emotional cognitive dimensions.

9. The multimodal cognitive impairment assessment system based on a multidimensional cognitive function hypergraph as claimed in claim 8, characterized in that: In the hypergraph model, the feature vectors from different modalities are first merged through a multimodal fusion algorithm. Through an adaptive aggregation strategy, the model can dynamically adjust the fusion weight of the features according to the current task and emotional state. Then, by defining the association matrix of the hypergraph, the vertices under different tasks are associated with the hyperedges; the update process is: when new symptoms are detected based on online data, they are added as new nodes to the hypergraph; and new hyperedges are constructed, and new symptom combinations are discovered through the incremental clustering algorithm, and hyperedges are added or deleted accordingly. new , which consists of symptoms S1, S2, …, S k , add it as a new hyperedge to the hypergraph, and calculate its weight: where φ(i) is the embedding vector of symptom i, |D new | is a super edge D new The number of symptoms in ; then update the disease-symptom association matrix B; for the newly identified disease-symptom association, update the B matrix: Then, by optimizing the corresponding objective function, the association matrix B and the new hyperedge set are trained and updated, and the edge weights are dynamically updated according to the association strength between the nodes in the hypergraph, and the hypergraph embedding model is updated; this is achieved by optimizing the following objective function: Among them, φ(d) is the embedded representation of the edge, f(d) is the hyperedge representation reconstructed from the vertex embedding, θ is the parameter, and λ is the normalization term; finally, each node and hyperedge on the graph is dynamically updated, and the model reasoning process is traced back to display the decision-making basis containing the clinical knowledge graph, thereby improving the auxiliary diagnosis efficiency and interpretability of cognitive impairment.

Images