A multi-modal cognitive impairment evaluation system based on multi-dimensional cognitive function hypergraph

Abstract

The patent discloses a kind of multi-modal cognitive impairment evaluation system based on multidimensional cognitive function hypergraph, including multi-modal signal acquisition system, cognitive function evaluation paradigm design, each modal general feature extraction framework and online hypergraph learning based on multidimensional cognitive function.System based on multi-modal physiological and behavioral signals, constructs the cognitive function dimension evaluation paradigm contained, to realize the overall assessment of old cognitive function.Firstly, by fusing electroencephalogram, electrocardiogram, speech and facial expression and other multi-modal signals, the physiological and behavioral responses of users under different tasks and situations are recorded and analyzed in real time.Then, the fusion representation between multi-modal signals is mined, a unified feature representation is generated, and an online hypergraph of clinical knowledge is introduced to capture complex multi-modal data relationships and provide analysis results containing cognitive function evaluation.Finally, combined with the above technical means, the system can realize early detection of cognitive impairment.

Description

[0001] The field

[0002] The present invention relates to the field of cognitive impairment assessment and multimodal fusion technology, and in particular to a multimodal cognitive impairment assessment system based on a multidimensional cognitive function hypergraph. Background Art

[0003] With the intensification of the global aging population, cognitive health issues in the elderly are receiving increasing attention. According to the World Health Organization, over 55 million people worldwide currently suffer from dementia, with over 60% of these patients living in low- and middle-income countries. Furthermore, approximately 10 million new cases occur annually, and the number of people with dementia worldwide is projected to reach 152.8 million by 2050. In China, there are approximately 38 million people aged 65 and over with mild cognitive impairment (MCI) and approximately 15 million with dementia. Alzheimer's disease, in particular, has become the leading cause of cognitive impairment, with over 10 million patients. Dementia is characterized by a decline in multiple cognitive functions, often accompanied by symptoms such as memory impairment, aphasia, apraxia, agnosia, impaired visual-spatial skills, executive dysfunction, and behavioral changes.

[0004] Early warning and intervention are widely considered to be important measures to prevent the occurrence of cognitive impairment, especially at the community level, which is the primary place for identification. As a transitional state between normal aging and early dementia, the early identification of mild cognitive impairment (MCI) has become a public health issue that needs to be addressed urgently. However, due to the significant heterogeneity among individuals, the decline of cognitive function often manifests itself in multiple different dimensions and forms, and a single assessment method is difficult to fully reflect the patient's actual cognitive status. Therefore, it is particularly important to conduct detailed assessments of different cognitive function dimensions (such as visual-spatial and executive function, naming, attention, language, abstract thinking, delayed recall, and orientation).

[0005] To address this challenge, we propose a wearable multimodal cognitive impairment assessment system. This system leverages the low cost, portability, and real-time monitoring capabilities of wearable devices, making them an ideal early warning tool. By fusing multimodal data, the system not only captures subtle changes that traditional detection methods may overlook, but also enables accurate identification and analysis of cognitive impairment. Research has shown that machine learning combined with a single wearable sensor can achieve high-precision stride length estimation in elderly people and patients with neurological diseases. This comprehensive assessment method provides valuable information for early detection and timely intervention, greatly improving the quality of life of elderly patients. Summary of the Invention

[0006] The multimodal cognitive impairment assessment system of this invention utilizes a comprehensive and dynamic cognitive function assessment paradigm by integrating multiple physiological and behavioral signals. The system incorporates multidimensional cognitive function assessment tasks, focusing on key cognitive function areas such as spatial cognition, comprehension and decision-making, verbal communication, attention execution, and episodic memory. During the assessment process, the system combines multimodal signals such as electrocardiogram (ECG), electroencephalogram (EEG), speech, and facial expressions, using specific task stimuli to capture the user's physiological responses and behavioral performance in real time while performing cognitive tasks. Regarding signal encoding, the system constructs specialized encoders tailored to the signal characteristics of different modalities. ECG signals, by reflecting the activity of the autonomic nervous system, provide important information on emotional state and cognitive function. EEG signals, by utilizing changes in electrical activity, capture patterns of neural activity during different cognitive tasks. Speech signals, containing both acoustic and semantic information, enable in-depth analysis of an individual's language expression ability and its relationship to cognitive status. Facial expressions, by analyzing facial action units, reveal the user's performance on cognitive tasks. The encoder of each modality uses a machine learning algorithm to perform in-depth analysis and feature extraction of the signal, providing a high-dimensional, interpretable feature vector for subsequent fusion. In terms of modal fusion methods, the system adopts a post-encoding feature fusion strategy to effectively integrate signal features from different modalities. By constructing a dynamic knowledge graph, the system uses a hypergraph structure to effectively associate the inherent logical relationship between multimodal signal features and cognitive function symptoms. Through the deep learning framework of the hypergraph, the system is able to learn the potential associations between different tasks and modalities, and then build a comprehensive cognitive function assessment model. This fusion not only improves the accuracy of the assessment, but also enhances the model's adaptability to individual heterogeneity, making the assessment results more comprehensive and reliable. Ultimately, the system combines clinical diagnostic knowledge to provide strong support for the early identification and intervention of cognitive impairment, thereby promoting the scientific management and promotion of cognitive health.

[0007] To achieve the above objectives, the present invention adopts a technical solution: a multimodal cognitive impairment assessment system based on a multidimensional cognitive function hypergraph. This system is characterized by constructing a cognitive function assessment paradigm through a multimodal signal acquisition system, simultaneously mining interpretable features in physiological and behavioral signals that match cognitive function, designing physiological and behavioral signal feature encoders, and ultimately achieving multimodal signal feature fusion to construct an online hypergraph assessment strategy for multidimensional cognitive function. Specifically, this system includes:

[0008] Multimodal cognitive assessment module: The multimodal cognitive assessment system includes multimodal signal acquisition and multidimensional cognitive function assessment paradigm. This module integrates a real-time acquisition system for multiple physiological and behavioral signals, aiming to comprehensively assess the individual's cognitive function status. By combining multimodal signals such as electrocardiogram, electroencephalogram, voice and facial expressions, the system can effectively capture and analyze multidimensional features related to cognitive processes. In the cognitive assessment paradigm, experimental tasks are designed for multiple cognitive dimensions such as spatial cognition, decision-making understanding, verbal communication, attention execution and situational memory. Through interactive games and actual task scenarios, the performance of subjects in different cognitive functions is obtained. This module not only realizes the precise acquisition and processing of signals, but also improves the comprehensiveness and accuracy of the assessment through multimodal fusion technology, providing a solid data foundation for the early detection and intervention of cognitive disorders.

[0009] Multimodal Signal Fusion Module: This module includes physiological and behavioral signal feature encoders and a hypergraph fusion model based on multidimensional cognitive functions. This module integrates a multimodal signal encoder and a hypergraph model to enable in-depth analysis and fusion of ECG, EEG, speech, and facial expression signals. Through the feature encoders of each modality, the system can extract the physiological and behavioral information contained in each signal and represent and summarize its spatiotemporal features. ECG signals are used to assess emotional state and changes in autonomic nervous system activity, EEG signals are used to capture the brain's electrical activity during cognitive tasks, speech signals reflect emotions and cognitive abilities through acoustic and semantic features, and facial expressions reveal an individual's emotional state. Subsequently, the constructed hypergraph model uses a dynamic knowledge graph to associate the features of each modality with the clinical symptoms of cognitive impairment, achieving a deep mapping between multidimensional cognitive functions and emotional states. This comprehensive approach not only enhances understanding of an individual's cognitive state but also provides an important basis for developing personalized intervention strategies.

[0010] As a further improvement of the present invention, the multimodal signal acquisition system is designed to acquire and integrate multidimensional data from different physiological and behavioral signals in real time to comprehensively evaluate the cognitive function status of an individual. The system combines a variety of sensors, including electrocardiogram (ECG), electroencephalogram (EEG), speech recognition and facial expression analysis equipment. Through these sensors, the physiological reactions and behavioral performances of an individual can be monitored simultaneously, providing rich information. The system is responsible for collecting ECG, EEG, speech and facial expression signals from the user. Through wearable devices and sensors, the user's physiological changes and emotional reactions are monitored in real time. The collected signals are preprocessed, including denoising, signal enhancement, etc., to improve data quality. After processing, the signal will be transmitted to the feature encoder for further analysis.

[0011] As a further improvement of the present invention, the multidimensional cognitive function assessment paradigm aims to comprehensively analyze the cognitive abilities of elderly users from multiple perspectives by designing a comprehensive multidimensional cognitive function assessment paradigm. This paradigm is based on the principle of combining theory with practice and revolves around seven main cognitive function dimensions, specifically including visual-spatial and executive function, naming, attention, language, abstract thinking, delayed recall, and orientation. The assessment of each dimension is achieved through corresponding cognitive tasks to ensure that the user's cognitive performance can be accurately captured:

[0012] (1) During the assessment process, electrocardiogram (ECG) signals are used to monitor the user's physiological state, reflect the activity of the autonomic nervous system, and provide the user's emotional response and stress level during the performance of cognitive tasks.

[0013] (2) Electroencephalogram (EEG) signals record brain activity through an electrode array, analyze brain wave patterns under different cognitive tasks, identify electrical activity characteristics related to specific cognitive functions, and thus reveal the user's cognitive state.

[0014] (3) Speech signal analysis is divided into two levels: acoustic and semantic. Acoustic information includes pitch, intensity, speaking rate, and prosody. By monitoring the vocalization process, it reflects the user's emotional changes and cognitive ability. Semantic information focuses on the user's vocabulary choice and sentence structure in language communication, helping to assess their cognitive functions such as language fluency, attention, and memory.

[0015] (4) Facial expressions are important behavioral signals. By analyzing facial action units (AUs), we can capture the user's emotional expression and its changes during cognitive tasks. Patients with cognitive impairment often show stiffness in facial expressions and slow movements. By identifying these subtle changes, the system can provide valuable information for the assessment of cognitive status.

[0016] As a further improvement to the present invention, the physiological and behavioral signal feature encoder is used to perform feature encoding on the collected signals. This module conducts in-depth analysis and feature extraction of physiological and behavioral signals from different modalities to achieve more accurate cognitive function assessment. This module includes multiple dedicated encoders for processing electrocardiogram (ECG), electroencephalogram (EEG), speech signals, and facial expressions.

[0017] In terms of ECG and EEG signal processing, the encoder analyzes heart rate variability and other ECG features to reveal an individual's autonomic nervous system activity and uses a specific algorithm to extract signal patterns related to emotional state. The EEG signal encoder uses a multi-head self-attention mechanism and a convolutional neural network to extract features of brain wave frequency bands such as α, β, and γ from the time-frequency domain, capturing changes in the brain's electrical activity under different cognitive tasks. The physiological signal encoder is composed of a convolutional Sinc filter and a multi-head self-attention mechanism, where the Sinc filter is:

[0018] g[n,f1,f2]=2f2sinc(2πf2n)-2f1sinc(2πf1n)(1)

[0019] The low cutoff frequency and high cutoff frequency are f1, f2, n is the sampling point index, the sinc function is defined as sinc(x) = sin(x) / x, and the cutoff frequency can be in [0, f s / 2] randomly initialized within the range, where f s Represents the sampling frequency of the input signal. The Sinc filter is implemented by a one-dimensional convolutional neural network. After obtaining the time series features divided by frequency band, it is flipped along the time dimension and outputs the physiological signal feature vector x through the multi-head self-attention mechanism layer. MSA :

[0020] x MSA =MSA(LN(x sinc ))(2)

[0021] Where LN is the LayerNorm layer, x sinc is the Sinc filter frequency band temporal feature, where MSA is the multi-head self-attention mechanism. Sinc adaptive convolution can interpretably extract frequency band features, and by flipping the time and channel dimensions, the attention score is calculated in the temporal direction to more effectively integrate global information.

[0022] The speech signal encoder is divided into two major components: acoustic and semantic. Acoustic features are extracted using Mel-spectrograms and modeled using the Transformer architecture. This model directly captures long-range, global contextual features in speech signals, providing an important basis for analyzing emotional and cognitive functions. Semantic features are processed using DistilBERT, analyzing the user's vocabulary and sentence structure to assess their language fluency and cognitive ability. The combination of these two components enables the speech signal encoder to comprehensively reflect the user's language performance and emotional state.

[0023] The facial expression encoder can detect the user's emotional changes during cognitive tasks by analyzing facial action units (AUs). The encoder uses a U-net-based deep learning architecture combined with a residual learning mechanism to improve the ability to capture subtle changes in facial expressions. The masked residual is calculated as:

[0024]

[0025] Where, F R is the feature map transformed by the residual layer; F M is the feature map after passing through the U-net network; F N is the final feature map. By focusing on the dynamic changes of facial features, the system can accurately assess the user's emotional state and cognitive function, especially the facial expressions during task execution.

[0026] As a further improvement to the present invention, the hypergraph fusion based on multidimensional cognitive functions uses a dynamically constructed and updated knowledge graph, with each node representing the multimodal characteristics of each cognitive task and each edge representing a different cognitive function. This constructs a hypergraph model to deeply explore the complex relationship between multimodal signals and cognitive functions. This hypergraph model not only integrates the feature representations of each modality but also adapts to changes in different tasks and emotional states, thereby achieving a comprehensive assessment of mild cognitive impairment.

[0027] The hypergraph model consists of vertices (nodes) S = S1, S2, ..., S m and hyperedges (edges) D = D1, D2, ..., D n The system consists of vertices representing multimodal features of users under different cognitive tasks, while hyperedges connect different dimensions of emotion and cognitive function. In this system, vertices encompass not only traditional physiological signal features (such as electrocardiogram and electroencephalogram) but also behavioral signals (such as speech and facial expressions). These features are encoded using a deep learning model to form high-dimensional feature vectors. Hyperedges are established based on the relationship between clinical symptoms of various cognitive functions and multimodal signals. Weights are assigned through expert knowledge and statistical analysis, enabling a dynamic mapping between task status and emotional cognitive dimensions.

[0028] In the hypergraph model, the feature vectors from different modalities are first merged through a multimodal fusion algorithm. Through an adaptive aggregation strategy, the model can dynamically adjust the fusion weight of features according to the current task and emotional state. This process enables the system to extract deep features related to cognitive impairment while maintaining sensitivity to emotional changes. Subsequently, by defining the association matrix of the hypergraph, the vertices under different tasks are associated with hyperedges. The update process is: based on online data, when new symptom manifestations are detected, they are added as new nodes to the hypergraph. New hyperedges are also constructed, and new symptom combinations are discovered through an incremental clustering algorithm, and hyperedges are added or deleted accordingly. new , which consists of symptoms S1, S2, …, S k , add it as a new hyperedge to the hypergraph, and calculate its weight:

[0029]

[0030] Where φ(i) is the embedding vector of symptom i, |D new |It is a super edge D new The number of symptoms in . Then update the disease-symptom association matrix B. For the newly identified disease-symptom association, update the B matrix:

[0031]

[0032] Then, by optimizing the corresponding objective function, we train and update the association matrix B, the new set of hyperedges, and dynamically update the edge weights according to the association strength between nodes in the hypergraph, thus updating the hypergraph embedding model. This can be achieved by optimizing the following objective function:

[0033]

[0034] Here, φ(d) represents the edge embedding, f(d) represents the hyperedge representation reconstructed from vertex embeddings, θ is a parameter, and λ is a regularization term. Finally, the nodes and hyperedges on the graph are dynamically updated, and the model's reasoning process can be backtracked to display the decision-making basis contained in the clinical knowledge graph, improving the efficacy and interpretability of auxiliary diagnosis for cognitive impairment.

[0035] Compared with the existing technology, the beneficial effects of the present invention are:

[0036] 1. Application of multimodal wearable devices: This invention effectively integrates multimodal signals such as electrocardiogram (ECG), electroencephalogram (EEG), speech, and facial expressions in the assessment of cognitive impairment. By leveraging the low cost, portability, and real-time monitoring capabilities of wearable devices, it achieves efficient, convenient, and accurate cognitive function assessment, overcoming the limitations of traditional assessment methods.

[0037] 2. Multidimensional Cognitive Function Assessment Paradigm: This seven-dimensional cognitive function assessment paradigm encompasses seven dimensions: visuospatial and executive function, naming, attention, language, abstract thinking, delayed recall, and orientation. Through real-time monitoring of multimodal signals, this paradigm provides a comprehensive and dynamic assessment of cognitive status, providing a scientific basis for personalized intervention and clinical decision-making. This paradigm effectively addresses individual heterogeneity and enhances the accuracy and practicality of cognitive assessment.

[0038] 3. Focus on individual heterogeneity and interpretability: This system uses a multidimensional cognitive function assessment paradigm to deeply explore the differences in cognitive function performance among different individuals, focusing on the mapping relationship between different modal signals and specific cognitive functions, significantly improving the personalization and interpretability of the assessment results and meeting the specific needs of different patients.

[0039] 4. Efficient signal feature encoding and fusion method: By designing a specific signal encoder, efficient spatiotemporal feature extraction and fusion are performed based on the characteristics of different modal signals, which improves the information expression capability and the depth of analysis, provides technical support for the comprehensive evaluation of multimodal signals, and enhances the overall performance of the system.

[0040] 5. Construction of a Dynamic Knowledge Graph: This paper constructs a dynamic hypergraph knowledge graph based on clinical diagnosis, integrating cognitive functions and emotional characteristics to form internal logical relationships. This innovation not only improves the interpretability of data but also provides new perspectives and methods for future research. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 Schematic diagram of the overall multimodal signal acquisition system of the present invention.

[0042] Figure 2 Flowchart of the cognitive assessment process of the multimodal system of the present invention.

[0043] Figure 3 This is a schematic diagram of the structure of the physiological signal encoder of the present invention.

[0044] Figure 4 Schematic diagram of the structure of the speech acoustic and semantic encoder of the present invention.

[0045] Figure 5 This is a structural diagram of the facial expression encoder of the present invention.

[0046] Figure 6 This is a schematic diagram of the hypergraph fusion structure based on multi-dimensional cognitive functions of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be described in detail below with reference to the accompanying drawings and implementation examples.

[0048] The overall multimodal acquisition system is as follows Figure 1 As shown in the figure, the architecture of the multimodal signal acquisition system is detailed. The system uses wearable devices to simultaneously collect multiple physiological and behavioral signals, including electrocardiogram (ECG), electroencephalogram (EEG), voice signals, and facial expressions. The collection of these signals can reflect the user's physiological state and emotional changes during the cognitive assessment process in real time. After signal acquisition, the data will be processed and encoded in real time by the mobile device, and the signals will be analyzed in multiple dimensions to extract key features related to cognitive function. The processed data will then be transmitted to the evaluation output module, which will generate the corresponding cognitive assessment results based on the standards and models of cognitive assessment. Figure 2 The detailed process for users to conduct cognitive assessments is demonstrated, including real-time signal monitoring and each step of the assessment task. This process is designed to ensure that each link operates efficiently to accurately reflect the user's cognitive status and functional level, providing a scientific basis for subsequent intervention and treatment.

[0049] Table 1 lists the assessment items for different cognitive function dimensions, including seven cognitive function dimensions: visual-spatial ability, executive ability, attention, language ability, memory, abstract thinking and orientation.

[0050] Table 1 Multidimensional cognitive function assessment paradigm

[0051]

[0052] In order to comprehensively assess an individual's performance in these cognitive dimensions, corresponding tasks and scoring criteria are designed for each project. These tasks are presented in the form of games, aiming to enhance the participants' enthusiasm and sense of participation. In various cognitive function test games, participants will perform corresponding task operations in specific situations. These operations can stimulate and reflect different cognitive functions. For example, in a visual-spatial ability test, participants may need to complete spatial positioning tasks in a virtual environment, while an executive ability test may involve the sequential execution of multi-step tasks. Through these designs, the system can monitor in real time the changes in the participants' physiological and behavioral signals during the task execution process, including electrocardiogram signals, brain waves, voice characteristics, and facial expressions.

[0053] During the task, wearable devices will simultaneously collect these signals, capturing participants' physiological responses and emotional changes during different cognitive activities. This data not only reflects participants' cognitive performance but also provides multi-dimensional support and basis for assessment reports by analyzing their physiological signals and behavioral responses, ultimately forming comprehensive and accurate assessment results. This comprehensive assessment approach provides a reliable basis for subsequent intervention measures, helping to effectively implement interventions and support tailored to individual needs.

[0054] Figure 3The structure and working principle of the physiological signal encoder are demonstrated, mainly including the processing of ECG and EEG signals. The encoder uses a convolutional neural network and self-attention mechanism to extract features from different frequency bands, thereby enabling real-time analysis of emotional states and cognitive functions in ECG and EEG signals. Figure 4 The company showcased its Mel-spectrogram-based Transformer architecture. This encoder extracts acoustic and semantic features to analyze an individual's language ability, emotional expression, and cognitive state. By converting speech signals into feature vectors, the system can assess cognitive functions such as language fluency and memory. Figure 5 The design of an expression encoder was demonstrated, using a residual-based deep learning network to analyze facial action units. This encoder can capture subtle changes in facial expressions in MCI patients, helping to identify their emotional state and degree of cognitive impairment, providing important data support for overall assessment.

[0055] Figure 6 This diagram illustrates a hypergraph fusion architecture based on multidimensional cognitive functions. It demonstrates how a hypergraph model can be used to integrate multiple physiological and behavioral signal features into cognitive function assessment. The hypergraph is designed to effectively capture the intrinsic relationships between different cognitive tasks and cognitive functions, providing rich semantic information for cognitive impairment assessment. In this diagram, the vertices of the hypergraph represent the multimodal fusion features of an individual during a specific task. Each vertex corresponds to a specific cognitive function dimension, such as visuospatial ability, executive function, and attention, which can be assessed through specific task design. For example, for visuospatial ability, tasks may include identifying and remembering spatial locations, while executive function assessment may involve the ability to solve multi-step problems. Hyperedges represent the relationships between different tasks. In implementation, as participants perform tasks, the system collects relevant physiological signals in real time and evaluates the performance of each cognitive function using pre-set hyperedge weights. This structure enables the hypergraph to dynamically update and adapt to changes in the individual's cognitive state, thereby providing more accurate assessment results. Overall, the hypergraph fusion architecture based on multidimensional cognitive functions not only enhances the flexibility and adaptability of cognitive assessment but also provides strong support for personalized cognitive intervention.

[0056] The combination of these figures will effectively illustrate the specific embodiments of the present invention and its practical applications, and demonstrate its innovation and practicality in the assessment of cognitive impairment.

Claims

1. A multimodal cognitive impairment assessment system based on a multidimensional cognitive function hypergraph, characterized in that: The system comprises a multimodal cognitive assessment module and a multimodal signal fusion module; the multimodal cognitive assessment module comprises a multimodal signal acquisition system and a multidimensional cognitive function assessment paradigm; the multimodal signal fusion module comprises a physiological and behavioral signal feature encoder and a hypergraph fusion based on multidimensional cognitive function; through the multimodal signal acquisition system, a cognitive function assessment paradigm is constructed, interpretable features matching cognitive function in physiological and behavioral signals are simultaneously mined, physiological and behavioral signal feature encoders are designed, and ultimately multimodal signal feature fusion is achieved, thereby constructing an online hypergraph assessment strategy for multidimensional cognitive function; The hypergraph fusion based on multi-dimensional cognitive functions constructs and updates knowledge graphs dynamically, with each node representing the multimodal features of each cognitive task and each edge representing a different cognitive function, thereby constructing a hypergraph model to mine the complex relationship between multimodal signals and cognitive functions; it not only integrates the feature representations of each modality, but also can adapt to changes in different tasks and emotional states, and realize a comprehensive assessment of mild cognitive impairment; the hypergraph model consists of vertices and hyperedges, and vertices represent the multimodal features of users under different cognitive tasks , while hyperedges are associated with different cognitive functions ; Among them, vertices not only cover traditional physiological signal features, but also behavioral signals. These features are encoded through deep learning models to form high-dimensional feature vectors; the establishment of hyperedges is based on the relationship between the clinical symptoms of each cognitive function and multimodal signals, and weights are assigned through expert knowledge and statistical analysis, so as to realize the dynamic mapping of task status and emotional cognitive dimensions; in the hypergraph model, the feature vectors from different modalities are first merged through the multimodal fusion algorithm, and the model can dynamically adjust the fusion weight of the features according to the current task and emotional state through the adaptive aggregation strategy; then, by defining the association matrix of the hypergraph, the vertices under different tasks are associated with the hyperedges; the update process is: according to the online data, when a new symptom is detected, it is added as a new node to the hypergraph; and new hyperedges are constructed, and new symptom combinations are discovered through the incremental clustering algorithm, and hyperedges are added or deleted accordingly. , which is caused by symptoms , add it as a new hyperedge to the hypergraph, and calculate its weight: （4） in, It's a symptom The embedding vector of It is a super edge The number of symptoms in the disease-symptom association matrix is ​​then updated ; Update for newly identified disease-symptom associations matrix: （5） Then, by optimizing the corresponding objective function, the training update association matrix , a new set of hyperedges, and dynamically updates the edge weights according to the strength of the association between nodes in the hypergraph, and updates the hypergraph embedding model; this is achieved by optimizing the following objective function: （6） in, The edge embedding representation, is the hyperedge representation reconstructed by vertex embedding, Type parameters, Finally, the nodes and hyperedges on the graph are dynamically updated, and the model reasoning process is backtracked to display the decision basis containing the clinical knowledge graph, thereby improving the auxiliary diagnosis efficiency and interpretability of cognitive impairment.

2. The multimodal cognitive impairment assessment system based on a multidimensional cognitive function hypergraph according to claim 1, characterized in that: The multimodal signal acquisition system is used to acquire and integrate multidimensional data from different physiological and behavioral signals in real time to comprehensively assess the individual's cognitive function status; It collects electrocardiogram, electroencephalogram, voice and facial expression signals from the user, and monitors the user's physiological changes and emotional reactions in real time through wearable devices and sensors. The collected signals are pre-processed and then transmitted to the signal feature encoder for further analysis.

3. The multimodal cognitive impairment assessment system based on a multidimensional cognitive function hypergraph according to claim 2, characterized in that: The multidimensional cognitive function assessment paradigm is used to comprehensively analyze the cognitive abilities of elderly users from multiple perspectives. The paradigm revolves around seven main cognitive function dimensions, including visual-spatial and executive functions, naming, attention, language, abstract thinking, delayed recall, and orientation. The assessment of each dimension is achieved through corresponding cognitive tasks to ensure that the user's cognitive function performance can be captured.

4. The multimodal cognitive impairment assessment system based on a multidimensional cognitive function hypergraph according to claim 3, characterized in that: The multidimensional cognitive function assessment paradigm is used during the assessment process: (1) ECG signals are used to monitor the user's physiological state, reflect the activity of the autonomic nervous system, and provide the user's emotional response and stress level during the performance of cognitive tasks; (2) EEG signals record brain activity through an electrode array, analyze brain wave patterns under different cognitive tasks, identify electrical activity characteristics related to specific cognitive functions, and thus reveal the user's cognitive state; (3) The analysis of speech signals is divided into two levels: acoustic and semantic. Acoustic information includes pitch, intensity, speaking speed and rhythm. By monitoring the vocalization process, it reflects the user's emotional changes and cognitive ability. Semantic information focuses on the user's vocabulary selection and sentence structure in language communication, helping to assess their cognitive functions such as language fluency, attention and memory. (4) Facial expression signals: by analyzing facial action units, the user’s emotional expression and changes in facial movements during cognitive tasks can be captured.

5. The multimodal cognitive impairment assessment system based on a multidimensional cognitive function hypergraph according to claim 4, characterized in that: The physiological and behavioral signal feature encoder is used to perform feature encoding on the collected signals, and conduct in-depth analysis and feature extraction on physiological and behavioral signals from different modalities to achieve more accurate cognitive function assessment; it includes multiple dedicated encoders for processing electrocardiograms, electroencephalograms, voice signals and facial expressions respectively.

6. The multimodal cognitive impairment assessment system based on a multidimensional cognitive function hypergraph according to claim 5, characterized in that: (1) In terms of ECG and EEG signal processing, the physiological signal encoder is composed of a convolutional neural network and a multi-head self-attention mechanism. It analyzes heart rate variability and other ECG characteristics to reveal the individual's autonomic nervous system activity; For EEG signals, the characteristics of brain wave frequency bands are extracted from the time-frequency domain to capture the changes in the brain's electrical activity under different cognitive tasks, providing in-depth information about cognitive activities. The physiological signal encoder is composed of a convolutional Sinc filter and a multi-head self-attention mechanism, where the Sinc filter is: （1） The low cutoff frequency and high cutoff frequency are f1, f2, n is the sampling point index, the sinc function is defined as sinc(x) =sin(x) / x, and the cutoff frequency is in [0, f s / 2] Randomly initialized within the range, f s Represents the sampling frequency of the input signal; the Sinc filter is implemented by a one-dimensional convolutional neural network. After obtaining the time series features divided by frequency bands, it is flipped along the time dimension and outputs the physiological signal feature vector through the multi-head self-attention mechanism layer. : （2） Where LN is the LayerNorm layer, x sinc is the frequency band temporal feature of the Sinc filter, where MSA is the multi-head self-attention mechanism; Sinc adaptive convolution can interpretably extract frequency band features, and by flipping the time and channel dimensions, the attention score is calculated in the temporal direction to more effectively integrate global information; (2) The speech signal encoder is divided into two parts: acoustic and semantic. Acoustic features are extracted through Mel spectrograms and modeled using the Transformer architecture. Semantic features are processed using DistilBERT to analyze the user's vocabulary and sentence structure to assess their language fluency and cognitive ability. The combination of these two allows the speech signal encoder to fully reflect the user's language performance and emotional state. (3) The facial expression signal encoder can detect the user's emotional changes during cognitive tasks by analyzing facial action units. The encoder adopts a deep learning architecture based on U-net and combines it with a residual learning mechanism to improve the ability to capture subtle changes in facial expressions. The residual with mask is calculated as: （3） Where, F R is the feature map transformed by the residual layer; F M It is the feature map after passing through the U-net network; F N is the final feature map; By focusing on the dynamic changes of facial features, it is possible to accurately assess the user's emotional state and cognitive function, especially the facial expressions during task execution.

Images