An integrated multi-modal physiological data cognitive impairment risk identification apparatus and method

Abstract

The application relates to the fields of artificial intelligence and biomedical sensing technology, and discloses a cognitive disorder risk identification device and method integrated with multi-modal physiological data, wherein the device comprises an electroencephalogram (EEG) module, an eye movement module, an fNIRS module, an edge computing module and an acceleration sensor; the EEG module collects EEG data, the eye movement module collects eye-tracking data, the fNIRS module collects near-infrared spectrum (fNIRS) data, and the data are input to the edge computing module; the edge computing module runs a multi-modal space-time attention fusion model to output a cognitive disorder risk assessment result by using the model; and the acceleration sensor runs a dynamic filtering algorithm based on a motion accelerometer to suppress signal drift caused by head movement. By using a modular head-mounted multi-modal edge device, physiological data are collected, edge computing modules are used for edge processing and cognitive disorder identification screening, and neurodegenerative disease early identification screening can be conveniently and effectively carried out at low cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and biomedical sensing technology, in particular to a cognitive impairment risk identification device and method integrating multi-modal physiological data. BACKGROUND

[0002] Cognitive screening technology is a key link for early diagnosis and intervention of senile dementia (Alzheimer's disease and other types of dementia), and through early identification and screening, early warning, and effective intervention measures, the incidence of Alzheimer's disease can be significantly delayed. The existing related screening technologies include:

[0003] (1) Traditional cognitive screening methods: Mini-Mental State Examination (MMSE), Montreal Cognitive Assessment Scale (MoCA), etc. Due to cultural and educational bias of the test population, traditional scales are prone to false positives in low education populations.

[0004] (2) Biomarker screening methods: blood biomarker detection of Aβ, tau protein and other indicators, brain imaging technology (PET / MRI) to detect brain atrophy, amyloid deposition and other pathological changes. However, biomarker screening is costly and has low popularity, and is mainly used for diagnosis rather than screening.

[0005] In view of the problems of the existing related screening technologies, a new device and method are needed to realize low-cost, convenient and effective early identification and screening of Alzheimer's disease (AD), mild cognitive impairment (MCI) and other neurodegenerative diseases in hospitals, communities and families. SUMMARY

[0006] The present application provides a cognitive impairment risk identification device and method integrating multi-modal physiological data, which collects electroencephalogram (EEG), eye-tracking, near-infrared spectroscopy (fNIRS) and other physiological data through a modular head-mounted multi-modal edge device, and combines an edge computing module to process the collected physiological data on the edge and identify and screen cognitive impairment, thereby enabling low-cost, convenient and effective early identification and screening of neurodegenerative diseases.

[0007] The present application provides a cognitive impairment risk identification device integrating multi-modal physiological data, comprising a brain electroencephalogram (EEG) module, an eye movement module, a near-infrared spectroscopy (fNIRS) module, an edge computing module and an acceleration sensor connected to each other.

[0008] The brain electroencephalogram (EEG) module is used to collect EEG data and input it to the edge computing module.

[0009] The eye movement module is used to collect eye-tracking data and input it to the edge computing module.

[0010] The fNIRS module is configured to collect near-infrared spectroscopy (fNIRS) data and input the fNIRS data to the edge computing module.

[0011] The edge computing module is configured to run a multi-modal spatio-temporal attention fusion model to output a cognitive impairment risk assessment result by using the multi-modal spatio-temporal attention fusion model.

[0012] The acceleration sensor runs a motion accelerometer-based running dynamic filtering algorithm to suppress signal drift caused by head movement.

[0013] Further, the master chip of the edge computing module includes an ARM multi-core CPU processor and an NPU processor, and runs a customized Linux kernel; in the edge computing module, the multi-modal spatio-temporal attention fusion model includes: (1) a time attention layer configured to process the electroencephalogram (EEG) data, encode an EEG time-frequency matrix by using an LSTM, and generate a time weight vector ; specifically, the time attention layer includes:

[0014] The pre-processed EEG data is input as a time-frequency matrix , and a dimension of the time-frequency matrix is TxF, where T represents a time step, and F represents a frequency domain feature quantity of the EEG signal.

[0015] The time-frequency matrix is input into an LSTM network in sequence according to a time step t, and the LSTM network can capture long-term dependencies in a time sequence; at each time point t, a hidden state vector is generated according to a current input and a hidden state at a previous time point, and the vector is regarded as a condensed representation of the EEG signal features up to the time point t.

[0016] An attention mechanism calculates an attention score for each hidden state vector , and the attention score is calculated by using a feedforward neural network, and a calculation formula of the attention score is as follows: wherein, is a hidden state output of the LSTM at the time point t, is a learnable weight matrix of the attention network, is a learnable bias term of the attention network, is a hyperbolic tangent activation function, and is configured to perform nonlinear transformation on the calculated score.

[0017] All of the attention scores at the time points are normalized by using a Softmax function, so as to obtain a final time weight vector : wherein, is a vector of length T, where each element has a value between 0 and 1, and the sum of all elements is 1; The value of represents the importance of the EEG data at the jth time point for the final judgment, the greater the value, the more critical the brain electrical activity pattern at this time point, (2) a spatial attention layer for processing the near-infrared spectrum fNIRS data, performing graph convolution network processing on the fNIRS functional connection matrix, and outputting brain region weights

[0018] ; specifically comprising: inputting a functional connection matrix A calculated according to the fNIRS signal, the functional connection matrix A being an SxS square matrix, wherein S is a preset number of brain regions, and an element A(i,j) in the matrix represents the functional connection strength between the ith brain region and the jth brain region, and also needing an initial feature matrix H of each brain region;

[0019] The GCN takes the functional connection matrix A as the adjacency matrix of the graph, and performs convolution operation on the brain region node features, and the GCN aggregates the information of each node neighbor node, thereby learning the high-order representation of the node in the whole brain network, and the propagation rule of the GCN layer is: wherein, is the node feature matrix of the mth layer; , the adjacency matrix with a self-loop is added, and I is a unit matrix, indicating that the information of each brain region is also considered; is the degree matrix of , used for normalization; is the trainable weight matrix of the mth layer; is an activation function; by stacking multiple GCN layers, the complex interaction mode between different brain regions is captured; the final output of the GCN is a highly abstract feature representation of each brain region, and the feature vector output by the GCN is passed through a global average pooling layer or a fully connected layer, and finally a weight score is generated for each of the S brain regions;

[0020] The weight score is normalized through a Softmax function, and finally a brain region weight vector is obtained: wherein, is a vector of length S, where each element represents the importance of the ith brain region for the cognitive impairment risk assessment, and the greater the value, the more attention should be paid to the functional state of the brain region; (3) a cross-modal fusion layer for fusing the time weight vector and the brain region weight Perform tensor outer product to generate a spatiotemporal attention map, concatenate it with the eye-tracking data, and then input it into the fully connected layer classifier; specifically, it includes:

[0021] Receive time weight vector and brain region weights , and perform tensor outer product operation to generate a T×S matrix, which is the spatiotemporal attention map. The calculation formula is: Spatiotemporal Attention Map Each element M(t,s) is composed of time weight (t) and spatial weights (s) is multiplied together, so the value of M(t,s) represents the comprehensive importance of brain area s at time t; (4) A fully connected layer classifier is used to receive the spliced ​​data and determine the value of the cognitive impairment risk assessment level, and output the cognitive impairment risk assessment result; specifically including:

[0022] Data splicing: First, the T×S dimensional spatiotemporal attention map Perform a flattening operation to form a one-dimensional long vector, and then concatenate this long vector with the preprocessed eye-tracking data feature vector: The concatenated long vector is input into one or more fully connected layers, where a series of nonlinear weighted summation operations are performed to deeply integrate all input features. The last layer of the classifier uses a Softmax activation function. If the risk level is divided into N categories, the last layer will have N neurons. The output of the Softmax layer contains a vector of N probability values, each value corresponding to the probability of a risk level, and the sum of all probabilities is 1. The model ultimately outputs the level with the highest probability as the cognitive impairment risk assessment result.

[0023] Furthermore, a flexible printed circuit board (FPCB) is used to realize three-dimensional stacking of the EEG module, the fNIRS module, and the eye movement module, and an optical isolation layer is used to reduce crosstalk between the EEG module and the fNIRS module.

[0024] Furthermore, the EEG module uses a 16-channel flexible dry electrode array with an integrated adaptive impedance matching circuit to support signal stability during motion. The flexible dry electrode array is coated with Ag / AgCl and has a contact impedance of less than 5kΩ.

[0025] The fNIRS module uses a 24-channel light source-detector layout, covering the frontal and parietal brain regions, and uses time-division multiplexing technology to eliminate crosstalk;

[0026] The eye tracking module uses a miniature infrared camera and an 850nm ring light source to achieve a pupil positioning accuracy of 0.1°.

[0027] The present invention also provides a method for identifying cognitive impairment risk by integrating multimodal physiological data. Based on the device for identifying cognitive impairment risk by integrating multimodal physiological data as described above, the method includes:

[0028] The raw data are collected and synchronized through the EEG module, eye-tracking module, and fNIRS module; wherein the raw data includes EEG data, eye-tracking data, and fNIRS data;

[0029] Preprocessing the EEG data, eye-tracking data, and fNIRS data;

[0030] The preprocessed raw data is input into the multimodal spatiotemporal attention fusion model trained in the edge computing module to output the cognitive impairment risk assessment result.

[0031] Furthermore, the step of collecting and synchronizing raw data through the EEG module, eye movement module, and fNIRS module further includes:

[0032] After the user wears the cognitive impairment risk identification device integrating multimodal physiological data, automatic calibration of EEG, eye movement, and fNIRS is performed;

[0033] The motion accelerometer-based dynamic filtering algorithm is run through the acceleration sensor to suppress signal drift caused by head movement, and the raw data is synchronously captured through a hardware-level timestamp marking method. The motion accelerometer-based dynamic filtering algorithm specifically includes:

[0034] Step 1: Integrate and synchronize data acquisition. A triaxial accelerometer is integrated into the headset used to collect EEG and fNIRS signals, positioned close to the fNIRS / EEG sensors. This ensures that the accelerometer data is collected strictly simultaneously with the raw EEG and fNIRS data, as guaranteed by hardware-level timestamps.

[0035] Step 2: Signal preprocessing: Process the acceleration signal itself to generate a reference signal that can represent motion artifacts. This is done by calculating the magnitude of the acceleration vector (Magnitude = sqrt(X² + Y² + Z²). This single time series can comprehensively reflect the intensity of motion in any direction.

[0036] Step 3: Apply dynamic filtering algorithm,

[0037] The input signal is defined as the main input signal and the reference input signal, the main input signal is represented as d(t)=s(t)+n(t), wherein s(t) is the real brain signal, and n(t) is the motion artifact noise; the reference input signal is the preprocessed accelerometer signal, which is assumed to be x(t), and it is highly correlated with the noise n(t) but not correlated with the real signal s(t);

[0038] An adaptive filter is established: an LMS least mean square algorithm adaptive filtering algorithm is selected, the reference input x(t) is received, and an output y(t) is generated, and the goal of the filter is to continuously adjust its internal parameters so that its output y(t) is as close as possible to the noise part n(t) in the main input;

[0039] Iterative optimization and noise cancellation: at each time point t, the current output of the filter is calculated: y(t)=W(t)×x(t), wherein W(t) is the weight of the filter at time t; the error signal is calculated: e(t)=d(t)-y(t), e(t) is the best estimate of the brain signal s(t) obtained after removing the noise; the filter weight is updated: the weight W(t) is adjusted according to the error e(t) so that it can predict the noise at the next time point t+1, and the update rule is: W(t+1)=W(t)+μ×e(t)×x(t), wherein μ is the step size, which controls the learning speed and stability of the filter;

[0040] Output result: after the above iterative process, the output error signal e(t) is the relatively clean fNIRS or EEG signal with motion artifacts suppressed, and the process is repeated for each channel that needs to be processed.

[0041] Further, in the first step, the hardware-level time stamp marking synchronization method comprises:

[0042] A master clock system is established, a central synchronization controller is used as an accurate microcontroller, which generates a stable and high-frequency square wave pulse signal, and the pulse signal stream serves as a common time reference for all devices;

[0043] Physical connection and signal distribution: physical cables are drawn from the master clock controller and connected to the trigger / synchronization port of each data acquisition device, including the electroencephalogram (EEG) module, the eye movement module, the fNIRS module, the edge computing module and the acceleration sensor, so that each pulse generated by the master clock is sent to all devices at the same time;

[0044] At the hardware level, the firmware of each data acquisition device is programmed to: when its internal circuit completes a data sampling, it immediately checks the status or count value of the master clock signal received on its trigger / synchronization port, and directly packages this external, unified timestamp information with the voltage value just collected to form a data frame, which is then sent to the computer for recording;

[0045] Data Fusion and Alignment,During the data analysis phase, multiple data files from different devices are received,,and each row of data in each file contains a hardware timestamp generated by the same,master clock. These timestamps are read to perfectly align all data streams on the,timeline.

[0046] Furthermore, after the user wears the cognitive impairment risk identification device integrating multimodal physiological data, the steps of automatically calibrating EEG, eye movement, and fNIRS are performed, including:

[0047] EEG impedance detection: When performing impedance detection, a 10Hz sinusoidal signal with an amplitude of 50μA is injected through the electrodes. The icon of each electrode is set with a color and value to display its impedance status in real time. When the contact impedance values ​​between all EEG electrodes involved in the calculation and the scalp are lower than the set value, the EEG impedance detection calibration is determined to be successful;

[0048] fNIRS light intensity adjustment: When performing fNIRS light intensity adjustment, the laser power of the light source is automatically adjusted within the range of 1-5 mW for each channel consisting of a light source and a detector. After obtaining a stable light intensity reading, continuous fNIRS signals are collected for a set time and compared with the intensity of the background noise to calculate the signal-to-noise ratio. When the signal-to-noise ratio of all channels exceeds 70 dB, the fNIRS calibration is confirmed to be successful.

[0049] Eye movement calibration: The user looks at nine targets that appear on the screen in sequence. The corresponding pupil center-corneal reflection vector is recorded when the user looks at each target at a known position. When the user looks at the verification target, the target's real coordinates and the line of sight coordinates estimated by the eye tracker are recorded. The Euclidean distance between the two is calculated and converted into visual angle error. When the calculated average error and maximum error are both lower than the system's preset accuracy threshold, the calibration is successful.

[0050] Furthermore, the step of synchronously capturing the original data using a hardware-level timestamp marking method includes:

[0051] The N450 component related to conflict monitoring was recorded as EEG data; the increase in HbO concentration in the dorsolateral prefrontal cortex was monitored as fNIRS data; and the number of glances back during incorrect responses was captured as eye-tracking data.

[0052] Furthermore, the step of preprocessing the EEG data, eye-tracking data, and fNIRS data includes:

[0053] EEG data preprocessing, including:

[0054] 1) Bandpass filtering: The digital filter retains the signal between 0.5Hz and 40Hz; Notch filtering: removes 50Hz or 60Hz power frequency interference;

[0055] 2) Remove bad segments: An automated algorithm is used to scan the entire continuous EEG data. If the amplitude of a certain segment exceeds the threshold, it is considered that this segment cannot be repaired, marked as a "bad segment", and removed from subsequent analysis;

[0056] 3) Artifact removal: Using independent component analysis, the multi-channel EEG signal is decomposed into N statistically independent source signal components. The topography and temporal waveforms of these independent components are examined. After identifying components representing eye movements, blinks, or heartbeats, the weights of these components are set to zero. Finally, the remaining components representing pure brain activity are remixed to restore the clean multi-channel EEG signal.

[0057] 4) Segmented extraction: The data is divided into epochs with the stimulus presentation moment in the task as time point 0;

[0058] 5) Baseline correction: For each epoch, calculate the average voltage value before stimulus presentation, and then subtract this average value from each data point of the epoch;

[0059] 6) Calculation of frequency band energy proportion: Apply fast Fourier transform or wavelet transform to each clean epoch to convert it from the time domain to the frequency domain to obtain the power spectral density. According to the preset frequency band ranges (δ: 1-4 Hz, θ: 4-8 Hz, α: 8-13 Hz, β: 13-30 Hz), calculate the absolute power in each frequency band. The total power is calculated as the sum of the absolute powers of all frequency bands. The final data is calculated as (absolute power of a frequency band / total power) * 100%. This calculation is performed for each frequency band to obtain a set of energy proportion values. The energy proportions of δ, θ, α, and β in the frequency bands of 1-4 Hz, 4-8 Hz, 8-13 Hz, and 13-30 Hz are obtained.

[0060] Near infrared spectroscopy fNIRS data preprocessing, specifically including:

[0061] 1) Conversion: Applying the modified Lambert-Beer law to calculate the relative concentration changes of HbO and HbR using the difference in the absorbance of two different wavelengths of near-infrared light on oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR);

[0062] 2) Filtering: Apply a low-pass filter or a band-pass filter to remove noise, including interference from heartbeat and breathing;

[0063] 3) Feature extraction:

[0064] HbO / HbR concentration change slope: In the task-related fNIRS signal segment, the stage of rising HbO concentration is identified. By calculating the first-order derivative of the signal segment or performing a linear regression fit on the data segment, the slope of the line is obtained. This slope value is ΔHbO / Δt, which represents the speed of change in blood oxygen concentration. A similar calculation is performed for HbR.

[0065] Functional connectivity matrix: All fNIRS channels were divided into different brain regions according to their location. The average HbO or HbR time series signal of each ROI was extracted. The Pearson correlation coefficient of the time series between each pair of ROIs was calculated. The value of this correlation coefficient represents the strength and direction of the functional connection between the two brain regions. The calculation results of all pairs were organized into an N×N matrix as the functional connectivity matrix.

[0066] Eye-tracking data processing, including:

[0067] 1) Event Recognition: Apply an eye movement event recognition algorithm to traverse the raw data and automatically classify it into: fixations, saccades, and smooth pursuits;

[0068] 2) Feature extraction: Gaze entropy: Divide the screen into a virtual grid, count the number of fixations in each grid that all fixations fall into during a complete task trial, calculate the fixation probability p(i) for each grid i = (number of fixations in grid i / total number of fixations), and apply the Shannon entropy formula , sum up all grids to get the gaze point entropy;

[0069] Glance peak velocity: For each identified saccade event, obtain its complete position and time information from start to end. By deriving the position information over time, we obtain the instantaneous velocity curve during the saccade process. The saccade peak velocity is the maximum value on the velocity curve.

[0070] The present invention also provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0071] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0072] The beneficial effects of the present invention are:

[0073] This method collects physiological data such as EEG, eye-tracking, and fNIRS, and uses edge computing devices to process this data using edge-side artificial intelligence algorithms to identify and screen for cognitive impairment. Multimodal data is integrated into artificial intelligence models to improve the accuracy of these models in identifying and screening for cognitive impairment. The use of multimodal head-mounted devices offers convenient hardware and significantly reduces the time required for user cognitive impairment assessment and screening compared to traditional methods. Edge-side models significantly reduce feedback latency compared to cloud-based models, and can effectively protect individual user data. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 This is a schematic diagram of the structure of the cognitive impairment risk identification device that integrates multimodal physiological data according to the present invention.

[0075] Figure 2 Schematic diagram of the process of the cognitive impairment risk identification method integrating multimodal physiological data according to the present invention.

[0076] Figure 3 Schematic diagram of the internal structure of a computer device according to an embodiment of the present invention.

[0077] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0078] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0079] This invention is applicable to the early identification and screening of neurodegenerative diseases such as Alzheimer's disease (AD) and mild cognitive impairment (MCI). It is compatible with deployment in multiple scenarios, including hospitals, communities, and homes. Specific technical areas involved include:

[0080] 1. Biomedical sensing technology: A multimodal signal synchronization acquisition system integrating electroencephalogram (EEG, sampling rate ≥512Hz), functional near-infrared spectroscopy (fNIRS, wavelength 730 / 850nm), and eye tracking (infrared camera, 120Hz) solves the problem of spatiotemporal alignment of heterogeneous sensors.

[0081] 2. Embedded edge computing: A real-time signal processing architecture based on a multi-core heterogeneous processor (ARM+NPU) enables localized noise reduction, feature extraction, and model inference of physiological signals, breaking through the latency bottleneck of traditional cloud processing.

[0082] 3. Artificial Intelligence Algorithm: Build an artificial intelligence spatiotemporal attention fusion network, dynamically weighted fuse the heterogeneous features of EEG and fNIRS through a hierarchical attention mechanism, and combine multimodal data such as eye movement parameters to improve the accuracy of cognitive decline model recognition.

[0083] This invention belongs to the fields of EEG, eye-tracking, fNIRS, and cognitive impairment. It provides an apparatus and method for edge-side cognitive impairment risk identification by using a modular, head-mounted, multimodal edge device to collect physiological data such as EEG, eye-tracking, and fNIRS. This data is then processed using an edge computing module, and an artificial intelligence spatiotemporal attention fusion algorithm is constructed to perform edge-side cognitive impairment risk identification. The basic principle is to construct a joint feature matrix combining EEG (temporal high resolution) and fNIRS (spatial high resolution) using EEG, eye-tracking, and fNIRS data. This is then combined with the eye-tracking data to input a hierarchical attention mechanism model, thereby identifying a user's cognitive impairment risk index.

[0084] like Figure 1 As shown, the present invention provides a cognitive impairment risk identification device that integrates multimodal physiological data, including an interconnected EEG module, an eye movement module, an fNIRS module, an edge computing module, and an accelerometer. A flexible printed circuit board (FPCB) is used to achieve three-dimensional stacking of the EEG, fNIRS, and eye movement modules, and an optical isolation layer is used to reduce crosstalk between the EEG and fNIRS modules.

[0085] (1) EEG Module 101

[0086] The EEG module is used to collect EEG data and input it into the edge computing module; the EEG module adopts a 16-channel flexible dry electrode array (Ag / AgCl coating, contact impedance <5kΩ) and integrates an adaptive impedance matching circuit to support signal stability in motion.

[0087] (2) Eye Movement Module 103

[0088] The eye-tracking module is used to collect eye-tracking data and input it into the edge computing module. The eye-tracking module uses a miniature infrared camera (OVM9284, 1 / 4-inch CMOS) with an 850nm ring light source, and the pupil positioning accuracy reaches 0.1°.

[0089] (3) fNIRS module 102

[0090] The fNIRS module is used to collect near-infrared spectral fNIRS data and input it into the edge computing module; the fNIRS module adopts a 24-channel (12 sources × 12 detectors) light source-detector layout, covers the frontal and parietal brain regions (based on a 10-20 system), and uses time division multiplexing (TDM) technology to eliminate crosstalk.

[0091] (4) Edge computing module

[0092] The edge computing module is used to run a multimodal spatiotemporal attention fusion model to output cognitive impairment risk assessment results. By providing the model's inferred results in real time, feedback latency can be significantly improved and personal data can be effectively protected. The edge computing module's main control chip includes an ARM multi-core CPU processor and an NPU processor, running a customized Linux kernel.

[0093] The multimodal spatiotemporal attention fusion model is an artificial intelligence model that runs in the edge computing module and includes the following neural network layers:

[0094] 1) Temporal Attention Layer

[0095] Temporal Attention Layer: This layer is specifically used to process EEG data. Its core goal is to identify which time periods of EEG signals are most critical for determining the risk of cognitive impairment during cognitive tasks. This is achieved through a long short-term memory network (LSTM) and attention mechanism. It is used to process the EEG data, perform LSTM encoding on the EEG time-frequency matrix, and generate a temporal weight vector. ; Specifically including the following steps:

[0096] ① Input data: According to the pre-processed EEG data, it is expressed as a time-frequency matrix , the time-frequency matrix The dimension is T×F, where T represents the time step and F represents the number of frequency domain features of the EEG signal.

[0097] ②LSTM encoding: transform the time-frequency matrix The LSTM network is input into the time step t (from 1 to T) in sequence. The LSTM network can capture the long-term dependencies in the time series. At each time point t, a hidden state vector is generated based on the current input and the hidden state of the previous moment. ,vector It is regarded as a condensed representation of the EEG signal features up to time t;

[0098] ③Calculate attention score: In order to evaluate the importance of each time point t, the attention mechanism is used for each hidden state vector Calculate an attention score , which is realized by feedforward neural network, and its calculation formula is: in, is the hidden state output of LSTM at time point t, is the learnable weight matrix of the attention network, is the learnable bias term of the attention network, is the hyperbolic tangent activation function, which is used to perform nonlinear transformation on the calculated score;

[0099] ④Generate time weight vector : The calculated attention scores for all time points Normalized by a Softmax function to obtain the final time weight vector : in, is a vector of length T, where each element The value of is between 0 and 1, and the sum of all elements is 1; The value of represents the importance of the EEG data at the jth time point to the final judgment. The larger the value, the more critical the EEG activity pattern at that time point is.

[0100] 2) Spatial Attention Layer

[0101] Spatial Attention Layer: This layer processes fNIRS data specifically to identify which functional brain regions (brain areas) have activity patterns most closely associated with cognitive impairment risk. The brain is viewed as a network (graph) and analyzed using a graph convolutional network (GCN). The fNIRS data is processed, and the fNIRS functional connectivity matrix is ​​processed using a graph convolutional network to output brain region weights. ; Specifically, the following steps are included: ① Input data: Input the functional connectivity matrix A calculated based on the fNIRS signal. The functional connectivity matrix A is an S×S square matrix, where S is the preset number of brain regions. The element A(i,j) in the matrix represents the functional connectivity strength between the i-th brain region and the j-th brain region (for example, the correlation of the signal). At the same time, the initial feature matrix H of each brain region is also required; ② Graph convolutional network processing: GCN uses the functional connectivity matrix A as the adjacency matrix of the graph and performs convolution operations on the features of the brain region nodes. GCN aggregates the information of the neighboring nodes of each node (brain region) to learn the high-order representation of the node in the entire brain network. The propagation rule of the GCN layer is: in, For the The node feature matrix of the layer; , the adjacency matrix with self-loops is added, and I is the identity matrix, indicating that each brain region also considers its own information; for The degree matrix of is used for normalization; For the The trainable weight matrix of the layer; is the activation function; by stacking multiple layers of GCN, the complex interaction patterns between different brain regions are captured;

[0102] ③Generate brain area weights : The final output of GCN is a highly abstract feature representation of each brain region. The feature vector output by GCN is passed through a global average pooling layer or a fully connected layer to generate a weight score for each of the S brain regions.

[0103] ④ Normalization: Similar to the temporal attention layer, the weight scores are normalized through a Softmax function to obtain the brain region weight vector : in, is a vector of length S, where each element It represents the importance of the i-th brain region for the risk assessment of cognitive impairment. The larger the value, the more attention the functional status of the brain region deserves.

[0104] 3) Cross-modal fusion layer and risk level assessment

[0105] Cross-modal Fusion Layer: Effectively fuse the “time” importance information extracted from EEG and the “space” importance information extracted from fNIRS. and brain region weights Perform tensor outer product to generate a spatiotemporal attention map and splice it with the eye-tracking data before inputting it into the fully connected layer classifier; specifically, the following steps are included: ① Tensor Outer Product: Receive the time weight vector (dimension is T×1) and brain region weights (dimension is 1×S), and perform tensor outer product operation to generate a T×S matrix, which is the spatiotemporal attention map, where the calculation formula is ② Spatiotemporal Attention Map Each element M(t,s) is composed of time weight (t) and spatial weights (s), so the value of M(t,s) represents the comprehensive importance of brain area s at time t; if a certain time point and a certain brain area are very critical, then the corresponding value in the figure will be very high.

[0106] 4) Fully connected layer classifier

[0107] The fully connected layer classifier is the final part of the model, responsible for making the final judgment based on the fused features and eye movement data. It receives the spliced ​​data, determines the cognitive impairment risk assessment level, and outputs the cognitive impairment risk assessment result. The specific steps include:

[0108] ① Data splicing: First, the T×S dimensional spatiotemporal attention map Perform a flattening operation to form a one-dimensional long vector, and then concatenate this long vector with the preprocessed eye-tracking data feature vector:

[0109] Eye movement data (such as fixation duration, saccade path, etc.) as independent biomarkers provide the model with valuable information about the subject's visual attention and information processing ability, further enhancing the model's judgment basis.

[0110] ② Classification and output: The concatenated long vector is input into one or more fully connected layers (also known as multilayer perceptrons), which perform a series of nonlinear weighted summations to deeply integrate all input features. The final layer of the classifier uses a Softmax activation function. If the risk level is divided into N categories, the final layer will have N neurons. The output of the Softmax layer contains a vector of N probability values, each corresponding to the probability of a risk level, and the sum of all probabilities is 1.

[0111] ③ Get the risk level value: the model outputs the highest probability of one level as the cognitive impairment risk assessment result. For example, if the output probability is [0.1, 0.3, 0.6], corresponding to "low, medium, high" three risk levels, the final assessment result is "high risk". This result is the "cognitive impairment risk assessment level value" of the application.

[0112] (5) Acceleration sensor

[0113] The acceleration sensor runs a running dynamic filtering algorithm based on a motion accelerometer to suppress signal drift caused by head movement.

[0114] The core innovation of the application is:

[0115] 1) Modular head-mounted hardware device: flexible printed circuit board (FPCB) is used to realize three-dimensional stacking of EEG electrodes, fNIRS light source / detector and eye movement camera, and optical isolation layer (black light shielding film with a thickness of 0.2mm) is used to reduce the crosstalk between EEG and fNIRS, so that the signal-to-noise ratio (SNR) is improved to 78dB.

[0116] 2) Active noise reduction technology built-in hardware device: based on the dynamic filtering algorithm of the motion accelerometer on the modular head-mounted edge hardware device, the signal drift caused by head movement is suppressed, and the quality of the brain electrical data is improved.

[0117] 3) Hardware device integrated edge computing module: an end-side multi-modal artificial intelligence model is deployed on the modular head-mounted edge hardware device, which can give the model prediction result through real-time processing of the collected data. Compared with the cloud model feedback delay, it is greatly improved, and can effectively protect personal data.

[0118] 4) Artificial intelligence model constructs a spatio-temporal attention fusion network, which dynamically weights and fuses the heterogeneous features of EEG (time resolution 10ms level) and fNIRS (spatial resolution 5mm level) through hierarchical attention mechanism, and combines eye movement behavior parameters (saccade speed, fixation point entropy) to improve the specificity and accuracy of cognitive decline identification.

[0119] 5) Based on the hardware level timestamp marking method, the clock alignment of multi-modal EEG signal, near-infrared spectrum fNIRS and eye movement signal is realized, the problem of inconsistent correlation between multiple types of signal data is solved, and the accuracy of model judgment is improved.

[0120] As Figure 2 shown, the application also provides a cognitive impairment risk identification method integrating multi-modal physiological data, based on the cognitive impairment risk identification device integrating multi-modal physiological data as described above, the method comprises:

[0121] S1, collecting and synchronizing raw data through the electroencephalogram (EEG) module, eye movement module, and functional near-infrared spectroscopy (fNIRS) module; wherein the raw data includes EEG data, eye-tracking data, and fNIRS data.

[0122] S101, after the user wears the integrated multi-modal physiological data cognitive impairment risk identification device, performing automatic calibration of electroencephalogram (EEG), eye movement, and functional near-infrared spectroscopy (fNIRS); wherein the automatic calibration step includes:

[0123] 1) EEG impedance detection

[0124] The success criteria for EEG impedance detection: the contact impedance values between all participating EEG electrodes and the scalp are below a set value (such as 20 kΩ). EEG signals are very weak bioelectric signals (microvolt level), and if the electrodes do not have good contact with the scalp, the resistance will be too high. High impedance can severely weaken the strength of the true EEG signal, so low impedance is a prerequisite for ensuring high signal-to-noise ratio EEG signals.

[0125] When performing impedance detection, the icon of each electrode will display its impedance state in real time through color and numerical value.

[0126] a. Red: indicates that the impedance is much higher than the set threshold, the signal quality is unacceptable, and the prompt "please adjust the XX electrode position" is clear.

[0127] b. Yellow / orange: indicates that the impedance is close to or slightly higher than the set threshold, and is in a critical state.

[0128] c. Green: indicates that the impedance is much lower than 20 kΩ (for example, lower than 5 kΩ or 10 kΩ), and the contact state is very ideal.

[0129] By adjusting the tightness of the head-mounted device, brushing the hair, and other methods, until all the indicator lights of the key electrodes on the interface turn green. When it is confirmed that the impedance values of all the electrodes are stable below the pre-set threshold (such as 20 kΩ), the calibration (or preparation work) of the EEG is successful, and the next step can be entered.

[0130] 2) fNIRS light intensity adjustment

[0131] The success criteria for fNIRS light intensity adjustment: the signal-to-noise ratio (SNR) of all fNIRS channels is stably greater than 70 dB, and the signal intensity received by the detector is in an ideal range that is neither saturated nor too weak.

[0132] fNIRS infers blood oxygen levels by measuring changes in light absorption. If the emitted laser power is too low, the signal received by the detector will be weak and easily overwhelmed by electronic noise in addition to physiological noise (heartbeat, breathing). If the power is too high, the detector may saturate, making it impossible to distinguish subtle signal changes. Therefore, it is necessary to automatically find the "just right" transmission power.

[0133] Judgment process:

[0134] The first stage (power regulation) automatically adjusts the laser power of the light source (in the range of 1-5mW) for each channel (consisting of a light source and a detector). It tries different powers until the detector of the channel reports an optimal raw light intensity reading.

[0135] Phase 2 (Signal-to-Noise Ratio Verification): After obtaining a stable light intensity reading, a short period (a few seconds) of continuous data is collected. A typical feature of the fNIRS signal is the regular pulsation of the heartbeat. The intensity of this heartbeat signal is identified and compared with the intensity of the background noise to calculate the signal-to-noise ratio (SNR).

[0136] Confirming Success: Each fNIRS channel will have a status indicator. After completing the automatic adjustment, the final signal-to-noise ratio value for each channel will be displayed. Only when the signal-to-noise ratio of all channels exceeds the hard target of 70dB is the fNIRS calibration confirmed to be successful. If a channel does not meet the standard, it will prompt you to check whether the probe in that position is blocked by hair or whether the contact with the scalp is firm.

[0137] 3) Eye movement calibration

[0138] Success criterion for eye tracking calibration: During an independent validation phase, the calculated average gaze tracking error is smaller than a pre-set threshold (usually within 1° of visual angle).

[0139] The eye tracker uses a camera to capture pupil and corneal reflections, but it doesn't know which "world coordinates" on the screen these "in-image coordinates" correspond to. The goal of calibration is to establish an accurate mathematical mapping from the "eye image" to "screen coordinates," which is referred to as the "mapping matrix" in this invention.

[0140] ① Judgment process: divided into two stages: "calibration" and "verification".

[0141] During the calibration phase, the user is instructed to look at nine targets on the screen in sequence. During this phase, the corresponding pupil center-corneal reflection vector is recorded when the user looks at each target at a known location. These data points are then used to calculate the optimal mapping matrix.

[0142] Verification phase:

[0143] a. Qualitative Verification (Visual Feedback): After calibration is complete, the system enters verification mode, where a new target or a smoothly moving object, previously unseen by the user, may appear on the screen. The user's gaze point, calculated based on the mapping matrix, is superimposed on the screen (for example, as a cross or circle). The operator and user can visually observe whether the cross accurately tracks the target. If the cross accurately matches the target, the calibration quality is high.

[0144] b. Quantitative Verification (Error Calculation): When the user gazes at a verification target, the system records the target's true coordinates (X_target, Y_target) and the gaze coordinates estimated by the eye tracker (X_gaze, Y_gaze). The Euclidean distance between the two is calculated and converted into visual angle error. The errors across multiple verification points are averaged.

[0145] ② Confirmed Success: Calibration is successful when both the calculated average and maximum errors are below the system's preset accuracy thresholds (e.g., average error <1.0°, maximum error <1.5°). If the error is too large, a message "Calibration failed, please try again" will be displayed, and the user will be advised to keep their head steady and focus more closely on the target during the next calibration.

[0146] S102. Run a dynamic filtering algorithm based on a motion accelerometer through an acceleration sensor to suppress signal drift caused by head movement, and synchronously capture the raw data through a hardware-level timestamp marking method while the user performs standardized cognitive tasks. The dynamic filtering algorithm based on a motion accelerometer accurately separates and removes the noise (signal drift) caused by head movement from the original fNIRS / EEG signal containing real brain activity signals and head movement noise. The present invention uses an independent sensor (accelerometer) to directly measure the noise source (head movement), and then uses the measurement data of this noise source to dynamically and specifically subtract the noise component from the mixed signal through an adaptive algorithm. Specifically comprising the following steps:

[0147] Step 1: Integrate and synchronize data collection

[0148] 1) Hardware Integration: A three-axis accelerometer should be integrated into the headset used to collect EEG and fNIRS signals. The sensor should be positioned as close as possible to the fNIRS / EEG sensors to ensure accurate capture of head movements that could affect signal acquisition.

[0149] 2) Synchronous acquisition: Ensure that the data from the acceleration sensor (e.g., X, Y, Z axis acceleration values) and the raw data from EEG, fNIRS are strictly collected at the same time. This is guaranteed by the "hardware-level timestamp".

[0150] The hardware-level timestamp marking synchronization method ensures that all data points from different devices (EEG, fNIRS, accelerometer, eye tracker) can be accurately aligned in time, with an error of sub-millisecond level. Hardware-level synchronization is achieved by a unified, high-precision physical clock signal, which "imprints" a time on each data point at the source of data generation (hardware level), thereby bypassing the uncertainty of the software layer. The specific steps include:

[0151] ① Establish a master clock system: Use a central synchronization controller as an accurate microcontroller, which generates a stable and high-frequency square wave pulse signal (e.g., a pulse every 1 ms or less), which is the "heartbeat" of the entire system and serves as the common time reference for all devices.

[0152] ② Physical connection and signal distribution: Physical cables (usually BNC or TTL cables) are connected from the master clock controller to the trigger / synchronization port of each data acquisition device, including EEG module, eye tracking module, fNIRS module, edge computing module, and acceleration sensor, so that each pulse generated by the master clock is sent to all devices at the same time.

[0153] ③ Mark data at the hardware level: The firmware (Firmware) of each data acquisition device (e.g., EEG amplifier) is programmed to check the status or count value of the master clock signal received at its trigger / synchronization port when its internal circuit completes a data sampling (e.g., acquires a voltage value), and directly packages this external, unified timestamp information with the just-acquired voltage value to form a data frame (Data Frame), for example, a data frame may contain [sample point value, hardware timestamp]. Finally, the data frame is then sent to the computer for recording.

[0154] ④ Data fusion and alignment: In the data analysis stage, multiple data files from different devices are received, each line of data in each file contains a hardware timestamp generated by the same master clock, and these timestamps are read to perfectly align all data streams on the time axis. For example, the software can easily find the value of EEG at time point 150.3214s, and the value of fNIRS and accelerometer at the same time point 150.3214s, thereby realizing subsequent dynamic filtering and multi-modal fusion analysis.

[0155] Step 2: Signal Preprocessing

[0156] Before applying the filtering algorithm, the acceleration signal itself is processed to generate a reference signal that can represent the motion artifact. That is, by calculating the magnitude of the acceleration vector Magnitude = sqrt(X²+Y²+Z²), this single time series can comprehensively reflect the intensity of motion in any direction.

[0157] Step 3: Apply dynamic filtering algorithm (taking adaptive filtering as an example)

[0158] Steps to use Adaptive Noise Cancellation:

[0159] ①Define input signal

[0160] Primary Input: The signal to be cleaned is the data from a single raw fNIRS or EEG channel. This signal is represented as d(t) = s(t) + n(t), where s(t) is the true brain signal and n(t) is the motion artifact noise.

[0161] Reference Input: The pre-processed accelerometer signal (e.g., the modulus signal mentioned above). This signal is assumed to be x(t), which is highly correlated with the noise n(t) but uncorrelated with the true signal s(t).

[0162] ② Establish an adaptive filter: Select the LMS (Least Mean Squares) adaptive filtering algorithm, receive the reference input x(t), and generate an output y(t). The goal of the filter is to continuously adjust its internal parameters so that its output y(t) is as close as possible to the noise part n(t) in the main input.

[0163] ③Iterative optimization and noise elimination

[0164] At each time point t, the following calculation is performed:

[0165] a. Calculate the current output of the filter: y(t) = W(t) × x(t), where W(t) is the weight of the filter at time t;

[0166] b. Calculate the error signal (i.e., the cleaned signal): e(t) = d(t) - y(t), where e(t) is the best estimate of the final, noise-free brain signal s(t);

[0167] c. Update filter weights: adjust the weights W(t) according to the error e(t) to predict the noise at the next time point t+1, the update rule is: W(t+1) = W(t) + μ × e(t) × x(t), where μ is the step size, which controls the learning speed and stability of the filter.

[0168] ④ Output results: After the above iteration process, the output error signal e(t) is the relatively clean fNIRS or EEG signal with motion artifacts suppressed. Repeat this process for each channel that needs to be processed.

[0169] The present application first uses hardware-level synchronization to ensure that the motion signal (from the accelerometer) and the brain signal (from the EEG / fNIRS) are perfectly matched in time, which is a prerequisite for effective dynamic filtering. Then use the dynamic filtering algorithm, take the accurately aligned motion signal as the "noise template", subtract the corresponding artifacts from the brain signal, and obtain high-quality pure data that can be used for subsequent cognitive impairment risk assessment.

[0170] The user performs a standardized cognitive task (such as the Stroop color-word test), and the device synchronously collects the following data: records the N450 component related to conflict monitoring as EEG data; monitors the increase in HbO concentration in the dorsolateral prefrontal cortex as fNIRS data; captures the number of back looks at the error reaction time as eye-tracking data.

[0171] 1) EEG data

[0172] N450 is an event-related potential (ERP), which is a negative wave peak that appears about 450 milliseconds after the stimulus is presented. It is widely recognized as a neural indicator of conflict monitoring. When the brain needs to suppress an automated, incorrect response and choose a correct response that requires more cognitive effort, the amplitude of N450 will significantly increase. The specific collection process is as follows:

[0173] ① Design and perform conflict task:

[0174] Ask the subject to perform a cognitive task that includes "conflict" conditions. Stroop task: ask the subject to say the color of the word, not the meaning of the word itself (for example, when seeing the word "red" written in blue ink, the correct answer is "blue"). "Inconsistent" trials (word meaning and color conflict) are conflict conditions, and EEG data is recorded synchronously when performing the task.

[0175] ② Data segmentation (Epoching / Segmentation):

[0176] The continuous EEG data stream is segmented according to the events in the task, forming many short data segments called "epochs." An event is the moment the conflicting stimulus is presented (time point 0). Each epoch typically includes a short period before the stimulus (e.g., -200ms, for baseline correction) and a period after the stimulus (e.g., +1000ms, for observing the full EEG response).

[0177] ③Baseline Correction:

[0178] For each epoch, the mean voltage value of the pre-stimulus period (-200ms to 0ms) was calculated and then subtracted from the data of the entire epoch. This can eliminate low-frequency drift between trials.

[0179] ④Averaging:

[0180] The most critical step in forming the ERP is to add up all the epochs belonging to the "conflict condition" and then calculate their average. Because the EEG signal of a single trial is very noisy, by averaging the signals of a large number of similar trials, the random noise will cancel each other out, and the weak neural responses time-locked to the stimulus (i.e., ERP, including N450) will stand out.

[0181] ⑤Component Quantification:

[0182] On the superimposed and averaged waveforms, specifically at electrodes in the central prefrontal cortex (e.g., Fz, FCz, and Cz), look for a clear negative peak between 400 and 500 ms. The resulting data is the amplitude of this peak, or the magnitude of its negative voltage. For example, -5 μV. A more negative N450 amplitude (e.g., -8 μV vs. -4 μV) typically indicates that the brain is detecting a stronger conflict and devoting more cognitive resources to monitoring it.

[0183] 2) Near-infrared spectroscopy fNIRS data

[0184] The dorsolateral prefrontal cortex (DLPFC) is a core center for executive functions in the brain, such as working memory, planning, and cognitive flexibility. When performing challenging cognitive tasks, neuronal activity in this region increases, leading to increased local blood flow and oxygen consumption, which manifests as an increase in oxygenated hemoglobin (HbO) concentration and a decrease in deoxygenated hemoglobin (HbR) concentration. The magnitude of the increase in HbO is a reliable indicator of the level of neural activation in this brain region. The specific collection process is as follows:

[0185] ① Channel Localization:

[0186] Before data analysis, the placement of the fNIRS probe on the scalp and the standard brain atlas were used to determine which fNIRS channels' measurement areas covered the left and right DLPFC.

[0187] ②Block Averaging:

[0188] Task block average: The task was designed in blocks (e.g., a 20-second conflict task followed by a 20-second break), and the average HbO concentration of all relevant channels in the DLPFC region during the entire conflict task block was calculated.

[0189] ③Response amplitude calculation:

[0190] In the averaged HbO time series curve, the blood flow response typically peaks 5-8 seconds after stimulation. The final data is the maximum increase in HbO concentration. This increase is calculated as: peak HbO concentration minus baseline HbO concentration before the task. A greater increase in HbO indicates greater activation of the DLPFC to complete the task. Individuals with cognitive impairment may have reduced activation of this region.

[0191] 3) Eye-tracking data

[0192] When people make an incorrect choice during a task and realize their mistake, they often react very quickly, even unconsciously, by moving their eyes back to the stimulus or option they just selected to confirm or check their response. This saccade, where the eyes return to the stimulus after a response, is called a regressive saccade. The specific process for collecting eye-tracking data is as follows:

[0193] ①Define the Area of ​​Interest (AOI):

[0194] Before you begin, identify the critical areas on the screen. For example, in the Stroop task, the area where the text is displayed is a core AOI.

[0195] ②Record responses and accuracy:

[0196] The time and content of each response (such as the keys pressed) of the subject are accurately recorded, and the response is immediately judged as "correct" or "incorrect".

[0197] ③ Screening error trials (Error Trials):

[0198] When analyzing the data, we first screen out those trials in which the subjects responded incorrectly from all the trials.

[0199] ④Analyze eye movement trajectory:

[0200] For each error trial, analyze the eye movement trajectory within a short period of time after the incorrect response (e.g., the next 1-2 seconds).

[0201] ⑤Counting the number of times you look back:

[0202] The algorithm checks whether, during this time, there is a saccade from elsewhere on the screen (e.g., gaze on the response button area) back to the stimulus's AOI. Each time such a "return" saccade is detected, a count is incremented, resulting in the total number of return gazes detected across all error trials. Frequent return gazes after errors generally indicate a high level of error detection. On the other hand, frequent return gazes after errors may indicate a deficit in self-monitoring.

[0203] S2. Preprocessing the EEG data, eye-tracking data, and fNIRS data, specifically including:

[0204] S201, EEG data preprocessing, specifically including:

[0205] 1) Filtering: Remove frequency components and noise that are not related to brain activity.

[0206] Band-pass filtering: The digital filter retains the signal between 0.5Hz and 40Hz.

[0207] Notch Filtering: A very narrow filter specifically designed to remove 50Hz or 60Hz power frequency interference.

[0208] 2) Bad Segment Rejection: Remove data segments that are severely contaminated by short, intense non-brain-related activities (such as body shaking and momentary electrode detachment).

[0209] Using an automated algorithm, the system scans the entire continuous EEG data. If the amplitude of a certain segment of data exceeds a very high threshold (for example, ±100 microvolts), it is considered that this segment of data is beyond repair, marked as a "bad segment" and removed from subsequent analysis.

[0210] 3) Artifact Removal: Separate and remove regular artifacts mixed in EEG signals, which are generated by specific physiological activities (mainly eye movement and blinking).

[0211] Independent Component Analysis (ICA) is used.a. Decompose the multi-channel EEG signal (assuming there are N electrodes) into N statistically independent "source signal" components.b. Check the topography and time waveform of these independent components. After identifying the components representing eye movement, blinking or heartbeat, set the weights of these components to zero.c. Finally, mix the remaining components representing pure brain activity back into a clean multi-channel EEG signal.

[0212] 4) Epoching / Segmentation: Cut the continuous, clean EEG data into short segments associated with specific cognitive events.

[0213] Take the moment of stimulus presentation in the task as "time point 0", cut the data into individual "Epochs", for example, from 200 milliseconds before the stimulus to 1000 milliseconds after the stimulus.

[0214] 5) Baseline Correction: Eliminate the DC offset between different trials (Epochs) to make all trials comparable.

[0215] For each Epoch, calculate the average voltage value of the pre-stimulus period (e.g. -200ms to 0ms), and then subtract this average value from each data point in the Epoch.

[0216] 6) Band Energy Proportion Calculation: Calculate the final features from the processed Epochs.

[0217] a. Apply Fast Fourier Transform (FFT) or Wavelet Transform to each clean Epoch to convert it from time domain to frequency domain and get the Power Spectral Density.

[0218] b. According to the pre-set frequency band range (δ: 1-4Hz, θ: 4-8Hz, α: 8-13Hz, β: 13-30Hz), calculate the absolute power in each frequency band (i.e. the area on the power spectrum curve within the frequency range).

[0219] c. Calculate the total power (sum of absolute power in all frequency bands).

[0220] d. Final data (energy percentage) = (absolute power in a frequency band / total power) * 100%. This calculation is performed for each frequency band to obtain a set of energy percentage values. The resulting set of energy percentages includes the energy percentages of δ, θ, α, and β in the 1-4 Hz, 4-8 Hz, 8-13 Hz, and 13-30 Hz frequency bands, reflecting the dominant oscillatory rhythm in the brain during a specific cognitive state.

[0221] S202. Preprocess fNIRS data to obtain two key features: a. HbO / HbR concentration change slope, reflecting the speed of neural activation; b. Functional connectivity matrix, reflecting the collaborative working mode between brain regions. Specifically, it includes:

[0222] 1) Conversion: Convert the raw optical density data collected by the device into physiologically meaningful hemoglobin concentration change data.

[0223] The modified Beer-Lambert Law (MBLL) is applied to calculate the relative concentration changes of HbO and HbR by utilizing the difference in the absorbance of two different wavelengths of near-infrared light on oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR).

[0224] 2) Filtering: Remove high-frequency physiological noise from fNIRS signals.

[0225] Apply a low-pass filter or band-pass filter. Since the hemodynamic response measured by fNIRS is a slow process (usually below 0.1 Hz), it is necessary to filter out higher-frequency noise, mainly including interference from heartbeat (about 1 Hz) and respiration (about 0.2-0.3 Hz).

[0226] 3) Feature extraction:

[0227] a. HbO / HbR concentration change slope (ΔHbO / Δt): quantifies the rate of neural activation.

[0228] Within the task-related fNIRS signal segment, the phase in which HbO concentration rises is identified (typically 2-8 seconds after stimulation). The slope of the line is then calculated by calculating the first-order derivative of the signal or performing a linear regression fit on the data. This slope is ΔHbO / Δt, which represents the speed of change in blood oxygen concentration. A similar calculation is performed for HbR.

[0229] b. Functional Connectivity Matrix: describes the temporal synchronization of activities in different brain regions.

[0230] All fNIRS channels were divided into different brain regions (ROIs) based on their location. The mean HbO (or HbR) time series signal was extracted for each ROI. The Pearson Correlation Coefficient (PEC) between the time series of each pair of ROIs was calculated. This correlation coefficient (between -1 and 1) represents the strength and direction of the functional connection between the two brain regions. The calculated results of all pairings were organized into an N×N matrix (N is the number of ROIs). This matrix is ​​the functional connectivity matrix.

[0231] S203: Eye-tracking data processing to obtain two advanced eye movement metrics: a. fixation entropy, reflecting the complexity of visual search patterns; b. peak scan velocity, reflecting the speed and efficiency of cognitive processing. Specifically, it includes:

[0232] 1) Event Detection: Convert the raw, continuous eye position coordinate stream (gaze data) into meaningful eye movement events.

[0233] Apply an eye movement event recognition algorithm (such as one based on velocity or dispersion) to iterate over the raw data and automatically classify it into:

[0234] Fixation: The eyes remain relatively still for a short period of time to acquire information.

[0235] Saccade: A rapid movement of the eyes between two fixations.

[0236] Smooth Pursuit, Blink, and other events.

[0237] 2) Feature extraction:

[0238] ① Fixation Entropy: quantifies the randomness and exploration range of visual search patterns.

[0239] a. Divide the screen into a virtual grid (e.g. 10x10 grid).

[0240] b. Count the grids where all fixations fell during a complete task trial, and calculate the number of fixations in each grid.

[0241] c. Calculate the fixation probability p(i) of each grid i = (number of fixations in grid i / total number of fixations) and apply the Shannon entropy formula , summing up all grids to get the gaze point entropy; high entropy means that the gaze is scattered and exploratory; low entropy means that the gaze is concentrated and patterned.

[0242] ②Saccade Peak Velocity: quantifies the dynamic characteristics of eye movements.

[0243] a. For each identified glance event, the system has complete position and time information from its start to end.

[0244] b. By differentiating the position information over time, the instantaneous velocity curve during the scanning process can be obtained.

[0245] c. The peak scanning velocity is the maximum value on this velocity curve. This value is strongly correlated with the amplitude (distance) of the scanning and may also be affected by the subject's alertness, fatigue, and other conditions.

[0246] S3. Input the preprocessed raw data into the multimodal spatiotemporal attention fusion model trained in the edge computing module, output the cognitive impairment risk assessment results, feed them back to the user and provide reference for other tasks.

[0247] The multimodal spatiotemporal attention fusion model includes different temporal and spatial deep learning AI model network layers, specifically:

[0248] The temporal attention layer is used to process the EEG data, perform LSTM encoding on the EEG time-frequency matrix, and generate a temporal weight vector α_t;

[0249] A spatial attention layer is used to process the fNIRS data, perform graph convolutional network processing on the fNIRS functional connectivity matrix, and output brain region weights β_s;

[0250] The cross-modal fusion layer is used to perform a tensor outer product of the time weight vector α_t and the brain region weight β_s to generate a spatiotemporal attention map, which is then spliced ​​with the eye-tracking data and input into the fully connected layer classifier.

[0251] The fully connected layer classifier is used to receive the spliced ​​data and determine the value of the cognitive impairment risk assessment level, and output the cognitive impairment risk assessment result.

[0252] like Figure 3 As shown, the present invention also provides a computer device, which can be a server, and its internal structure can be as follows Figure 3As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer design is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store all data required for the process of the method for identifying the risk of cognitive impairment by integrating multimodal physiological data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the method for identifying the risk of cognitive impairment by integrating multimodal physiological data is implemented.

[0253] Those skilled in the art will understand that Figure 3 The structure shown in is merely a block diagram of a portion of the structure related to the present application solution and does not constitute a limitation on the computer device to which the present application solution is applied.

[0254] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, any one of the above-mentioned methods for identifying cognitive impairment risks by integrating multimodal physiological data is implemented.

[0255] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAM bus dynamic RAM (RDRAM).

[0256] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, apparatus, article, or method comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, apparatus, article, or method. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, apparatus, article, or method comprising the element.

[0257] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A device for identifying cognitive impairment risk by integrating multimodal physiological data, characterized in that: Includes interconnected EEG module, eye movement module, fNIRS module, edge computing module and acceleration sensor; The EEG module is used to collect EEG data and input it into the edge computing module; The eye-tracking module is used to collect eye-tracking data and input it into the edge computing module; The fNIRS module is used to collect near-infrared spectroscopy fNIRS data and input it into the edge computing module; The acceleration sensor runs a dynamic filtering algorithm based on a motion accelerometer to suppress signal drift caused by head movement; The edge computing module is used to run a multimodal spatiotemporal attention fusion model to output a cognitive impairment risk assessment result using the multimodal spatiotemporal attention fusion model; specifically, it includes: The main control chip of the edge computing module includes an ARM multi-core CPU processor and an NPU processor, running a customized Linux kernel; in the edge computing module, the multimodal spatiotemporal attention fusion model includes: (1) Temporal attention layer, used to process the EEG data, perform LSTM encoding on the EEG time-frequency matrix, and generate a temporal weight vector ; Specifically include: Input the pre-processed EEG data, which is expressed as a time-frequency matrix , the time-frequency matrix The dimension is T×F, where T represents the time step and F represents the number of frequency domain features of the EEG signal; The time-frequency matrix The LSTM network is input into the LSTM network in sequence according to the time step t. The LSTM network can capture the long-term dependencies in the time series. At each time point t, a hidden state vector is generated based on the current input and the hidden state of the previous moment. ,vector It is regarded as a condensed representation of the EEG signal features up to time t; The attention mechanism is used for each hidden state vector Calculate an attention score , which is realized by feedforward neural network, and its calculation formula is: in, is the hidden state output of LSTM at time point t, is the learnable weight matrix of the attention network, is the learnable bias term of the attention network, is the hyperbolic tangent activation function, which is used to perform nonlinear transformation on the calculated score; Calculated attention scores at all time points Normalized by a Softmax function to obtain the final time weight vector : in, is a vector of length T, where each element The value of is between 0 and 1, and the sum of all elements is 1; The value of represents the importance of the EEG data at the jth time point to the final judgment. The larger the value, the more critical the EEG activity pattern at that time point is. (2) Spatial attention layer, used to process the fNIRS data, perform graph convolutional network processing on the fNIRS functional connectivity matrix, and output brain region weights. ; Specifically include: Input the functional connectivity matrix A calculated based on the fNIRS signal. The functional connectivity matrix A is an S×S square matrix, where S is the preset number of brain regions. The element A(i,j) in the matrix represents the functional connectivity strength between the i-th brain region and the j-th brain region. The initial feature matrix H of each brain region is also required. GCN uses the functional connectivity matrix A as the adjacency matrix of the graph and performs convolution operations on the features of the brain region nodes. GCN aggregates the information of each node's neighboring nodes to learn the high-order representation of the node in the entire brain network. The propagation rule of the GCN layer is: in, For the The node feature matrix of the layer; , the adjacency matrix with self-loops is added, and I is the identity matrix, indicating that each brain region also considers its own information; for The degree matrix of is used for normalization; For the The trainable weight matrix of the layer; is the activation function; by stacking multiple layers of GCN, the complex interaction patterns between different brain regions are captured; The final output of GCN is a highly abstract feature representation of each brain region. The feature vector output by GCN is passed through a global average pooling layer or a fully connected layer to generate a weight score for each of the S brain regions. The weight scores are normalized by a Softmax function to finally obtain the brain region weight vector : in, is a vector of length S, where each element represents the importance of the i-th brain region for the risk assessment of cognitive impairment. The larger the value, the more attention the functional status of the brain region deserves; (3) Cross-modal fusion layer, used to transform the temporal weight vector and brain region weights Perform tensor outer product to generate a spatiotemporal attention map, concatenate it with the eye-tracking data, and then input it into the fully connected layer classifier; specifically, it includes: Receive time weight vector and brain region weights , and perform tensor outer product operation to generate a T×S matrix, which is the spatiotemporal attention map. The calculation formula is: Spatiotemporal Attention Map Each element M(t,s) is composed of time weight (t) and spatial weights (s), so the value of M(t,s) represents the comprehensive importance of brain area s at time t; (4) A fully connected layer classifier is used to receive the spliced ​​data and determine the cognitive impairment risk assessment level value, and output the cognitive impairment risk assessment result; specifically, it includes: Data splicing: First, the T×S dimensional spatiotemporal attention map Perform a flattening operation to form a one-dimensional long vector, and then concatenate this long vector with the preprocessed eye-tracking data feature vector: The concatenated long vector is fed into one or more fully connected layers, where a series of nonlinear weighted sum operations are performed to deeply integrate all input features. The final layer of the classifier uses a Softmax activation function. If the risk level is divided into N categories, the final layer will have N neurons. The output of the Softmax layer contains a vector of N probability values, each corresponding to the probability of a risk level, and the sum of all probabilities is 1. The model ultimately outputs the level with the highest probability as the cognitive impairment risk assessment result.

2. The device for identifying cognitive impairment risk by integrating multimodal physiological data according to claim 1, characterized in that: A flexible printed circuit board (FPCB) is used to achieve three-dimensional stacking of the EEG module, the fNIRS module, and the eye movement module, and an optical isolation layer is used to reduce crosstalk between the EEG module and the fNIRS module.

3. The device for identifying cognitive impairment risk by integrating multimodal physiological data according to claim 1, characterized in that: The EEG module uses a 16-channel flexible dry electrode array with an integrated adaptive impedance matching circuit to support signal stability during motion. The flexible dry electrode array is coated with Ag / AgCl and has a contact impedance of less than 5kΩ. The fNIRS module uses a 24-channel light source-detector layout, covering the frontal and parietal brain regions, and uses time-division multiplexing technology to eliminate crosstalk; The eye tracking module uses a miniature infrared camera and an 850nm ring light source to achieve a pupil positioning accuracy of 0.1°.

4. A method for identifying cognitive impairment risk by integrating multimodal physiological data, characterized in that: The device for identifying cognitive impairment risk based on integrated multimodal physiological data according to any one of claims 1 to 3, the method comprising: The raw data are collected and synchronized through the EEG module, eye-tracking module, and fNIRS module; wherein the raw data includes EEG data, eye-tracking data, and fNIRS data; Preprocessing the EEG data, eye-tracking data, and fNIRS data; The preprocessed raw data is input into the multimodal spatiotemporal attention fusion model trained in the edge computing module to output the cognitive impairment risk assessment result.

5. The method for identifying cognitive impairment risk by integrating multimodal physiological data according to claim 4, characterized in that: The step of collecting and synchronizing raw data through the EEG module, eye movement module, and fNIRS module further includes: After the user wears the cognitive impairment risk identification device integrating multimodal physiological data, automatic calibration of EEG, eye movement, and fNIRS is performed; The motion accelerometer-based dynamic filtering algorithm is run through the acceleration sensor to suppress signal drift caused by head movement, and the raw data is synchronously captured through a hardware-level timestamp marking method. The motion accelerometer-based dynamic filtering algorithm specifically includes: Step 1: Integrate and synchronize data acquisition. A triaxial accelerometer is integrated into the headset used to collect EEG and fNIRS signals, positioned close to the fNIRS / EEG sensors. This ensures that the accelerometer data is collected strictly simultaneously with the raw EEG and fNIRS data, as guaranteed by hardware-level timestamps. Step 2: Signal preprocessing: Process the acceleration signal itself to generate a reference signal that can represent motion artifacts. This is done by calculating the magnitude of the acceleration vector (Magnitude = sqrt(X² + Y² + Z²). This single time series can comprehensively reflect the intensity of motion in any direction. Step 3: Apply dynamic filtering algorithm, The input signals are defined as the main input signal and the reference input signal. The main input signal is expressed as d(t)=s(t)+n(t), where s(t) is the true brain signal and n(t) is the motion artifact noise. The reference input signal is the preprocessed accelerometer signal, assuming it is x(t), which is highly correlated with the noise n(t) but uncorrelated with the true signal s(t). Establish an adaptive filter: Select the LMS least mean square algorithm adaptive filtering algorithm, receive the reference input x(t), and produce an output y(t). The goal of the filter is to continuously adjust its internal parameters so that its output y(t) is as close as possible to the noise part n(t) in the main input; Iterative optimization and noise elimination: At each time point t, calculate the current output of the filter: y(t) = W(t) × x(t), where W(t) is the weight of the filter at time t; calculate the error signal: e(t) = d(t) - y(t), where e(t) is the best estimate of the final brain signal s(t) without noise; update the filter weight: adjust the weight W(t) according to the error e(t) so that it can predict the noise at the next time point t+1. The update rule is: W(t+1) = W(t) + μ×e(t) × x(t), where μ is the step size, which controls the learning speed and stability of the filter; Output results: After the above iterative process, the output error signal e(t) is a relatively clean fNIRS or EEG signal with motion artifacts suppressed. This process is repeated for each channel that needs to be processed.

6. The method for identifying cognitive impairment risk by integrating multimodal physiological data according to claim 5, characterized in that: In the first step, the tag synchronization method of hardware-level timestamp includes: Establish a master clock system, using a central synchronization controller as a precise microcontroller, which generates a stable and high-frequency square wave pulse signal. The pulse signal stream serves as a common time reference for all devices; Physical connection and signal distribution: Physical cables are drawn from the master clock controller and connected to the trigger / synchronization ports of each data acquisition device, including the EEG module, eye movement module, fNIRS module, edge computing module, and accelerometer, so that every pulse generated by the master clock is sent to all devices simultaneously; At the hardware level, the firmware of each data acquisition device is programmed to: when its internal circuit completes a data sampling, it immediately checks the status or count value of the master clock signal received on its trigger / synchronization port, and directly packages this external, unified timestamp information with the voltage value just collected to form a data frame, which is then sent to the computer for recording; Data Fusion and Alignment,During the data analysis phase, multiple data files from different devices are received,,and each row of data in each file contains a hardware timestamp generated by the same,master clock. These timestamps are read to perfectly align all data streams on the,timeline.

7. The method for identifying cognitive impairment risk by integrating multimodal physiological data according to claim 6, characterized in that: After the user wears the cognitive impairment risk identification device integrating multimodal physiological data, the steps of automatically calibrating EEG, eye movement, and fNIRS are performed, including: EEG impedance detection: When performing impedance detection, a 10Hz sinusoidal signal with an amplitude of 50μA is injected through the electrodes. The icon of each electrode is set with a color and value to display its impedance status in real time. When the contact impedance values ​​between all EEG electrodes involved in the calculation and the scalp are lower than the set value, the EEG impedance detection calibration is determined to be successful; fNIRS light intensity adjustment: When performing fNIRS light intensity adjustment, the laser power of the light source is automatically adjusted within the range of 1-5 mW for each channel consisting of a light source and a detector. After obtaining a stable light intensity reading, continuous fNIRS signals are collected for a set time and compared with the intensity of the background noise to calculate the signal-to-noise ratio. When the signal-to-noise ratio of all channels exceeds 70 dB, the fNIRS calibration is confirmed to be successful. Eye movement calibration: The user looks at nine targets that appear on the screen in sequence. The corresponding pupil center-corneal reflection vector is recorded when the user looks at each target at a known position. When the user looks at the verification target, the target's real coordinates and the line of sight coordinates estimated by the eye tracker are recorded. The Euclidean distance between the two is calculated and converted into visual angle error. When the calculated average error and maximum error are both lower than the system's preset accuracy threshold, the calibration is successful.

8. The method for identifying cognitive impairment risk by integrating multimodal physiological data according to claim 6, characterized in that: The step of synchronously capturing the original data using a hardware-level timestamp marking method includes: The N450 component related to conflict monitoring was recorded as EEG data; the increase in HbO concentration in the dorsolateral prefrontal cortex was monitored as fNIRS data; and the number of glances back during incorrect responses was captured as eye-tracking data.

9. The method for identifying cognitive impairment risk by integrating multimodal physiological data according to claim 6, characterized in that: The step of preprocessing the EEG data, eye-tracking data, and fNIRS data includes: EEG data preprocessing, including: 1) Bandpass filtering: The digital filter retains the signal between 0.5Hz and 40Hz; Notch filtering: removes 50Hz or 60Hz power frequency interference; 2) Bad segment removal: An automated algorithm is used to scan the entire continuous EEG data. If the amplitude of a segment exceeds a threshold, the segment is considered to be beyond repair and marked as a "bad segment" and removed from subsequent analysis. 3) Artifact removal: Using independent component analysis, the multi-channel EEG signal is decomposed into N statistically independent source signal components. The topography and temporal waveforms of these independent components are examined. After identifying components representing eye movements, blinks, or heartbeats, the weights of these components are set to zero. Finally, the remaining components representing pure brain activity are remixed to restore the clean multi-channel EEG signal. 4) Segmented extraction: The data is divided into epochs with the stimulus presentation moment in the task as time point 0; 5) Baseline correction: For each epoch, calculate the average voltage value before stimulus presentation, and then subtract this average value from each data point of the epoch; 6) Calculation of frequency band energy proportion: Apply fast Fourier transform or wavelet transform to each clean epoch to convert it from the time domain to the frequency domain to obtain the power spectral density. According to the preset frequency band ranges (δ: 1-4 Hz, θ: 4-8 Hz, α: 8-13 Hz, β: 13-30 Hz), calculate the absolute power in each frequency band. The total power is calculated as the sum of the absolute powers of all frequency bands. The final data is calculated as (absolute power of a frequency band / total power) * 100%. This calculation is performed for each frequency band to obtain a set of energy proportion values. The energy proportions of δ, θ, α, and β in the frequency bands of 1-4 Hz, 4-8 Hz, 8-13 Hz, and 13-30 Hz are obtained. Near infrared spectroscopy fNIRS data preprocessing, specifically including: 1) Conversion: Applying the modified Lambert-Beer law to calculate the relative concentration changes of HbO and HbR using the difference in the absorbance of two different wavelengths of near-infrared light on oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR); 2) Filtering: Apply a low-pass filter or a band-pass filter to remove noise, including interference from heartbeat and breathing; 3) Feature extraction: HbO / HbR concentration change slope: In the task-related fNIRS signal segment, the stage of rising HbO concentration is identified. By calculating the first-order derivative of the signal segment or performing a linear regression fit on the data segment, the slope of the line is obtained. This slope value is ΔHbO / Δt, which represents the speed of change in blood oxygen concentration. A similar calculation is performed for HbR. Functional connectivity matrix: All fNIRS channels were divided into different brain regions according to their location. The average HbO or HbR time series signal of each ROI was extracted. The Pearson correlation coefficient of the time series between each pair of ROIs was calculated. The value of this correlation coefficient represents the strength and direction of the functional connection between the two brain regions. The calculation results of all pairs were organized into an N×N matrix as the functional connectivity matrix. Eye-tracking data processing, including: 1) Event Recognition: Apply an eye movement event recognition algorithm to traverse the raw data and automatically classify it into: fixations, saccades, and smooth pursuits; 2) Feature extraction: Gaze entropy: Divide the screen into a virtual grid, count the number of fixations in each grid that all fixations fall into during a complete task trial, calculate the fixation probability p(i) = (number of fixations in grid i / total number of fixations) for each grid i, and apply the Shannon entropy formula , sum up all grids to get the gaze point entropy; Glance peak velocity: For each identified saccade event, obtain its complete position and time information from start to end. By deriving the position information over time, we obtain the instantaneous velocity curve during the saccade process. The saccade peak velocity is the maximum value on the velocity curve.

Images