A dual modality signal processing method for mild cognitive impairment intelligent screening
By aligning and fusing EEG and eye-tracking signals, the problem of inaccurate screening caused by modal signal differences has been solved, achieving efficient and accurate screening for mild cognitive impairment, which is applicable to community and primary healthcare institutions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NORTHWESTERN POLYTECHNICAL UNIV
- Filing Date
- 2025-08-27
- Publication Date
- 2026-07-21
AI Technical Summary
In existing technologies, mild cognitive impairment screening methods based on single-modality EEG signals or combining EEG and eye movement signals have differences in acquisition methods, time scales, and data structures, which leads to the inability to effectively fuse signals and affect the accuracy of screening.
By acquiring the EEG and eye movement signals of the target object under cognitive task-driven conditions, registration techniques such as dynamic temporal warping algorithm and fully convolutional network are used to achieve temporal series alignment of EEG and eye movement signals. Furthermore, the temporal correlation is maximized by minimizing statistical differences, and feature fusion is performed in conjunction with attention mechanisms.
It achieves efficient synchronization and feature fusion of EEG and eye movement signals, improving the accuracy and efficiency of screening for mild cognitive impairment, reducing costs, and making it suitable for large-scale promotion.
Smart Images
Figure CN120763485B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of early screening technology for mild cognitive impairment, and in particular to a dual-modal signal processing method, system, device and medium for intelligent screening of mild cognitive impairment. Background Technology
[0002] Mild cognitive impairment (MCI) is an early stage of dementia. People with MCI experience poorer memory or thinking compared to their peers. Approximately 60% of Alzheimer's disease (AD)-related MCI cases will eventually develop into dementia. MCI represents a golden window for dementia intervention, but it is often overlooked. Without effective interventions, the number of people with dementia will continue to rise.
[0003] Traditional screening for mild cognitive impairment relies on subjective scales, such as the Mini-Mental State Examination (MMSE) and the Montreal Cognitive Assessment (MoCA). However, characterizing a patient's cognitive characteristics through the results of subjective scales is highly subjective and has low accuracy in representing the patient's cognitive characteristics, which is not conducive to assisting doctors in screening patients for MCI. Alternatively, screening based on positron emission tomography (PET), magnetic resonance imaging (MRI), or cerebrospinal fluid (CSF) is invasive and costly, which is not conducive to large-scale promotion and is also not conducive to assisting doctors in screening patients for MCI.
[0004] Based on this, existing technologies have proposed using patients' EEG signals to assess MCI risk and achieve MCI screening. However, this screening method based on a single modality is not accurate enough. Some technologies combine patients' bimodal signals, such as EEG signals and eye movement signals. However, the signals of different modalities have significant differences in acquisition methods, time scales and data structures, and cannot be effectively integrated, which also affects the screening of MCI. Summary of the Invention
[0005] The purpose of this invention is to provide a dual-modal signal processing method, system, device, and medium for intelligent screening of mild cognitive impairment, which can solve the problem that signals from different modalities have significant differences in acquisition methods, time scales, and data structures, and cannot be effectively fused, thus affecting MCI screening.
[0006] To address the aforementioned technical problems, embodiments of the present invention provide a dual-modal signal processing method for intelligent screening of mild cognitive impairment, comprising the following steps: Acquire the electroencephalogram (EEG) and eye movement signals of the target subject under cognitive task-driven conditions; EEG and eye movement signals are registered according to a cognitive task to obtain EEG and eye movement signals with the same time series length; the time series of the registered EEG and eye movement signals are each composed of multiple time windows; By minimizing the statistical differences between the registered EEG and eye-tracking signals in each corresponding time window and maximizing the temporal correlation between the registered EEG and eye-tracking signals in each corresponding time window, target EEG and target eye-tracking signals with consistent distribution attributes can be obtained; where statistical differences include mean differences, variance / standard deviation differences, and distribution shape differences.
[0007] Optionally, the cognitive task consists of multiple key events, and the registration of EEG signals and eye movement signals according to the cognitive task to obtain EEG signals and eye movement signals of the same time series length includes: Both EEG signals and eye movement signals were divided into multiple EEG signal segments and multiple eye movement signal segments related to multiple key events; The dynamic time warping algorithm is used to register EEG signal segments and eye movement signal segments corresponding to the same key event based on morphological similarity, in order to compensate for the reaction time difference of the target object's EEG signal segments and eye movement signal segments driven by the same key event, so as to obtain EEG signal segments and eye movement signal segments with the same time series length corresponding to the same key event.
[0008] Optionally, after obtaining the target EEG signal and target eye movement signal with consistent distribution properties, the method further includes: The target EEG signal and the target eye movement signal are input into the trained fully convolutional network in parallel. Multi-scale convolutional kernels are used to extract features at different time scales of the target EEG signal and the target eye movement features, which are then used as the target EEG features and the target eye movement features. After mapping the target EEG features and target eye movement features to a common representation space, the target EEG features and target eye movement features are weighted separately through an attention mechanism, and the weighted target EEG features and target eye movement features are fused to obtain the target fused features.
[0009] Optionally, the step of inputting the target EEG signal and the target eye movement signal into the trained fully convolutional network in parallel includes: The target EEG signal is preprocessed to extract the spatiotemporal features of the target EEG signal in different frequency bands. The spatiotemporal features include the power spectrum, phase lock value and microstate features of the target EEG signal. Preprocessing of the target eye movement signal is performed to extract the pupil diameter, saccade speed, fixation position, and blink frequency of the target object as dynamic behavioral features of the target eye movement signal; The spatiotemporal features of the target EEG signal and the dynamic behavioral features of the target eye movement signal are input in parallel into the trained fully convolutional network.
[0010] Optionally, the spatiotemporal features of the target EEG signal are obtained through independent component analysis or empirical mode decomposition, and the dynamic behavioral features of the target eye movement signal are obtained through multinomial fitting or principal component analysis.
[0011] Embodiments of the present invention also provide a dual-modal signal processing system for intelligent screening of mild cognitive impairment, comprising: The data acquisition module is used to acquire the electroencephalogram (EEG) and eye movement signals of the target object under cognitive task-driven conditions. The first data processing module is used to register EEG signals and eye movement signals according to the cognitive task to obtain EEG signals and eye movement signals with the same time series length; wherein, the time series of the registered EEG signals and eye movement signals are composed of multiple time windows; The second data processing module is used to obtain target EEG signals and target eye movement signals with consistent distribution attributes by minimizing the statistical differences between the registered EEG signals and eye movement signals in each corresponding time window and maximizing the temporal correlation between the registered EEG signals and eye movement signals in each corresponding time window; wherein, the statistical differences include mean differences, variance / standard deviation differences and distribution shape differences.
[0012] Embodiments of the present invention also provide a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described bimodal signal processing method for intelligent screening of mild cognitive impairment.
[0013] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described dual-modal signal processing method for intelligent screening of mild cognitive impairment.
[0014] The dual-modal signal processing method for intelligent screening of mild cognitive impairment provided by this invention has at least the following beneficial effects: By stimulating the target subject through cognitive tasks, the system obtains the target subject's EEG and eye-tracking signals driven by the cognitive tasks. This invention not only performs registration operations on the bimodal signals (EEG and eye-tracking signals) to achieve synchronous alignment of the bimodal data, but also minimizes intermodal statistical differences and maximizes intermodal correlations, enabling the bimodal signals to achieve consistency at a higher level (statistical distribution level), thus obtaining the target EEG and target eye-tracking signals. In other words, this invention considers the differences in time series length and distribution attributes between EEG and eye-tracking signals, and processes the bimodal signals accordingly. The resulting bimodal signals can be used in MCI screening, overcoming the limitation that the ineffective fusion of bimodal signals affects MCI screening. Attached Figure Description
[0015] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:
[0016] Figure 1 This is a flowchart illustrating a dual-modal signal processing method for intelligent screening of mild cognitive impairment provided by the present invention. Figure 2 This is a schematic diagram of the framework of a dual-modal signal processing method for intelligent screening of mild cognitive impairment provided by the present invention. Detailed Implementation
[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.
[0018] The technical solutions provided by the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0019] One embodiment of the present invention relates to a bimodal signal processing method for intelligent screening of mild cognitive impairment. The implementation details of the bimodal feature fusion intelligent screening method for mild cognitive impairment in this embodiment are described below. The following implementation details are provided for ease of understanding and are not necessary for implementing this solution.
[0020] The specific process of the dual-modal signal processing method for intelligent screening of mild cognitive impairment in this embodiment can be described as follows: Figure 1 As shown, it includes: Step 101: Obtain the EEG and eye movement signals of the target object under cognitive task-driven conditions.
[0021] Specifically, this embodiment aims to stimulate brain activity through structured cognitive interactive tasks while simultaneously acquiring neural electrical signals (EEG signals) and eye movement signals reflecting visual attention and behavioral responses. This step includes the following operations:
[0022] 1. Multi-task cognitive interaction design: Multiple cognitive task modules are pre-installed in mobile terminals (such as tablets), such as episodic memory tasks, continuous attention tests, and spatial reaction tasks, covering cognitive dimensions such as memory, attention, and reaction. The system automatically executes tasks and generates event tags to ensure standardization and repeatability.
[0023] 2. Task event marking and synchronization mechanism: In each task, the system automatically generates timestamped event tags to align EEG signals with eye movement signals.
[0024] 3. Mechanism for simultaneous acquisition of EEG and eye movements: The target subject wears an EEG acquisition device and an eye-tracking device to simultaneously record EEG and eye movement signals (such as gaze position, pupil diameter, saccade speed, blink frequency, etc.) during task execution. All signals are recorded synchronously through a unified clock to construct structured bimodal sequence data.
[0025] Compared to traditional questionnaire-based scales, digitally driven tasks are more objective and repeatable; the strict synchronization between EEG and interaction time points can form structured and labeled neural response samples, which facilitates subsequent modeling.
[0026] Step 102: Register the EEG signal and eye movement signal according to the cognitive task to obtain EEG signal and eye movement signal with the same time series length; wherein, the time series of the registered EEG signal and eye movement signal are composed of multiple time windows.
[0027] Specifically, the cognitive task consists of multiple key events. Both EEG and eye movement signals are divided into multiple EEG signal segments and multiple eye movement signal segments corresponding to multiple key events. The Dynamic Time Warping (DTW) algorithm is used to register the EEG signal segments and eye movement signal segments corresponding to the same key event based on morphological similarity. This is to compensate for the reaction time difference of the target object's EEG signal segments and eye movement signal segments driven by the same key event, so as to obtain the EEG signal segments and eye movement signal segments corresponding to the same key event with the same time series length.
[0028] The DTW algorithm is used to measure and align two time series that may have different lengths, so as to find an "optimal mapping path" that minimizes the overall distance between corresponding points of the two series (by calculating the Euclidean distance) while preserving their temporal structure.
[0029] Step 103: By minimizing the statistical differences between the registered EEG signals and eye movement signals in each corresponding time window and maximizing the temporal correlation between the registered EEG signals and eye movement signals in each corresponding time window, target EEG signals and target eye movement signals with consistent distribution attributes are obtained; wherein, the statistical differences include mean differences, variance / standard deviation differences and distribution shape differences.
[0030] In practical implementation, even after temporal alignment, the aligned multimodal features may still exhibit distributional biases. The Cross-Modal Consistency Calibration (CMCC) method is employed to minimize statistical differences or maximize correlations between modalities, ensuring consistent distributional properties of each modality's features within the same shared space. This improves the convergence and discriminative capabilities of the subsequent fusion network. Aligning the peak times between statistical features further eliminates temporal biases from multi-sensor acquisition. After alignment, the data from each modality precisely correspond on a common time reference, achieving synchronization at the level of minute events.
[0031] For example, the CMCC method employs statistical alignment: minimizing the difference in the covariance matrix between the features of the source modality (e.g., EEG) and the target modality (e.g., eye-tracking signal).
[0032] After obtaining the target EEG signal and target eye movement signal with consistent distribution properties, the target EEG signal and target eye movement signal are input in parallel into the trained fully convolutional network for feature extraction.
[0033] In one example, the target EEG signal is first preprocessed to extract its spatiotemporal features in different frequency bands. These features include the power spectrum, phase lock value, and microstate features of the target EEG signal. The target eye movement signal is then preprocessed to extract the pupil diameter, saccade speed, fixation position, and blink frequency of the target object, which are used as the dynamic behavioral features of the target eye movement signal. The spatiotemporal features of the target EEG signal and the dynamic behavioral features of the target eye movement signal are then input in parallel into a pre-trained fully convolutional network.
[0034] In the specific implementation, EEG signals undergo preprocessing such as filtering, Independent Component Correlation Algorithm (ICA), and Empirical Mode Decomposition (EMD) to extract spatiotemporal features (such as power spectrum, phase lock value, and microstate features) for each frequency band. Eye-tracking signals are then normalized and subjected to dimensionality reduction using polynomial fitting or principal component analysis to obtain statistical feature vectors. Both types of modalities possess comparable feature representations after processing.
[0035] In one example, during training, the aforementioned multimodal dataset is grouped according to the needs of the input network: for instance, group 1 contains spatiotemporal feature segments of EEG, and group 2 contains dynamic behavioral features of eye-tracking signals. Each group of data employs a different slicing strategy, such as segmenting the EEG into pre-stimulation and post-stimulation segments according to task stages, or taking overlapping sliding windows for signals within a time window. To increase data diversity and prevent overfitting, data augmentation mechanisms are introduced: temporal random jitter, signal mirroring, and frequency band shifting are applied to the EEG signals; for eye-tracking signals, additional samples are synthesized using a generative adversarial network (GAN). The augmented bimodal data is still grouped after uniform temporal alignment and together with the original data to form the training set.
[0036] In a specific example, during the training process, the aforementioned bimodal dataset of EEG and eye movement is preprocessed and enhanced according to the requirements of the neural network structure: Data slicing strategy: Divide EEG data into pre-stimulation and post-stimulation segments according to task event fragments, or use an overlapping sliding window method for temporal slicing; divide eye movement signals into event-related sequences according to fixation segments or saccade segments.
[0037] Data augmentation mechanism: To improve model robustness, a multimodal data augmentation strategy is introduced: EEG signals are subjected to time-domain random jitter, mirror flip, and frequency band perturbation; eye movement trajectories are augmented with random perturbation, noise insertion, and linear pupil diameter scaling to increase sample diversity.
[0038] The enhanced bimodal data still needs to be aligned using dynamic time warping (DTW) to ensure that each pair of EEG and eye-tracking samples strictly corresponds in the time dimension, thus constructing a structured training set.
[0039] This embodiment introduces advanced temporal algorithms such as DTW for the first time in early screening for cognitive impairment, and performs cross-modal accurate alignment of EEG and eye movement indicators; it adopts cross-modal statistical calibration methods to correct acquisition delay differences and construct a truly strictly synchronized multi-source dataset; it proposes a multi-level data augmentation strategy, which expands sample capacity and diversity through time-frequency perturbation and generative models, thereby significantly improving alignment quality and model robustness.
[0040] After inputting the target EEG signal and target eye movement signal into the trained fully convolutional network in parallel, multi-scale convolutional kernels are used to extract features at different time scales of the target EEG signal and target eye movement features, which are then used as target EEG features and target eye movement features. After mapping the target EEG features and target eye movement features to a common representation space, the target EEG features and target eye movement features are weighted separately through an attention mechanism, and the weighted target EEG features and target eye movement features are fused to obtain the target fusion feature. Finally, by feeding the target fusion feature into a classifier, the cognitive impairment risk level of the target object can be output.
[0041] In its implementation, the network includes the following processing flow: 1. Parallel feature extraction: Each set of data has its spatiotemporal features extracted using a customized, improved Fully Convolutional Network (FCN) network.
[0042] To address the characteristics of bimodal data, multi-scale convolutional kernels (including 3×3, 5×5 and dilated convolutions) are used on the basis of traditional FCN to capture features at different time scales. Skip connections and residual units are added in the middle of the network to ensure the full integration of low-level details and high-level semantic features.
[0043] 2. Feature Fusion Layer: Map the output feature maps or vectors of each sub-network to a common representation space; Map the eye-tracking feature modality vectors to the same dimension of the EEG signal: ; In the formula, To unify dimensions; The original mode vector, when hour, Represents the event embedding vector; For a trainable mapping matrix, the i-th mode is transformed from the original dimension. (Right now , or Linear projection to a unified dimension ; For the bias vector, and Same dimension; These are the modal vectors that have been mapped and unified to the same dimension.
[0044] Expand the feature vectors to the same temporal dimension as the EEG feature vectors: , ; In the formula, The number of time steps is consistent with the time length of the EEG features (e.g., the EEG is divided into T time segments). "Copy" or broadcast into a time series matrix of the same length as the EEG features. This allows it to be multiplied with EEG features at each time point during subsequent fusion.
[0045] Deep fusion using attention mechanisms: EEG feature F EEG After convolution, it is compressed into a two-dimensional matrix. ; For each modality embedding, a separate attention network is designed. The attention calculation function is:
[0046] ; In the formula, This represents the embedding of the EEG and the i-th modality in the time dimension. It is a shared multilayer perceptron, and its output is attention weights of length T.
[0047] Attention weights are applied to each modality feature to calculate the weighted total feature: ; In the formula, ⊙ represents the Hadamard product (dot product) at each time point, which allows the embedding of each mode to dynamically adjust its contribution at different time points.
[0048] 3. Classification and early screening result output: The fused features are fed into a subsequent classifier to output the risk level of Mild Cognitive Impairment (MCI) and the early screening conclusion of cognitive impairment. Meanwhile, the model is paired with the results of traditional gold standards (MMSE / MoCA, PET / CSF) during the training phase for iterative optimization to continuously improve sensitivity and specificity.
[0049] This network framework breaks through the limitations of the traditional "single-modal input + single-path convolution" and achieves deep semantic fusion and collaborative discrimination between different modalities, which greatly improves the sensitivity and specificity of early screening for cognitive impairment. At the same time, it has self-learning and adaptive optimization capabilities.
[0050] In one embodiment, the bimodal signal processing method of the present invention for intelligent early screening of cognitive impairment using bimodal feature fusion for intelligent screening of mild cognitive impairment is as follows: Figure 2The framework diagram shown illustrates the implementation of this method, which employs a "three-step closed-loop" process: ①: Present multi-task digital-driven cognitive tests (such as episodic memory, number tapping, classification prompts, etc.) to the subjects (i.e. the target objects) on mobile terminals. The system automatically generates coded incentive events and records timestamps when the task is started, key stimuli are triggered, and the task is completed. At the same time, 16-channel EEG signals are collected simultaneously to accurately capture the spatiotemporal dynamic response of the brain under digital stimulation. ②: Strict Time-Synchronized Bimodal Data Construction Module—This module performs filtering, EMD, and self-supervised denoising on EEG signals to extract spatiotemporal features; encodes event markers into fixed-length embedding vectors; normalizes and reduces the dimensionality of eye-tracking feature signals; aligns the three types of data on a millisecond-level benchmark using Dynamic Time Warping (DTW) and cross-modal consistency calibration techniques; and constructs a high-quality, rich, and diverse time-synchronized bimodal dataset through sliding window segmentation and GAN-based enhancement strategies. ③: The improved fully convolutional network (FCN) inputs data from each subgroup in parallel. The network first extracts features from each subgroup, and then deeply fuses the features between modalities through an attention mechanism in the fusion layer. After passing through a classifier, it finally outputs the subject's mild cognitive impairment (MCI) risk assessment and diagnostic label. At the same time, it is paired with the results of traditional MMSE / MoCA and PET / MRI / CSF clinical gold standard for training to continuously optimize the model performance.
[0051] In one example, to ensure the model's clinical usability, this embodiment was trained on the gold standard dataset under the IWG-2021 framework. Clinicians used the IWG-2021 standard (the gold standard for clinical MCI examination), including scales such as MMSE / MOCA, molecular imaging such as PET / MRI, and cerebrospinal fluid (CSF), to detect mild cognitive impairment (MCI). This dataset included two groups of 180 participants each aged 60 and above: one group comprised patients with mild cognitive impairment (MCI) meeting the IWG-2021 definition, and the other group comprised cognitively normal individuals meeting the IWG-2021 definition, forming a control group. The bimodal EEG-Eye signal fusion model trained on this dataset consistently mapped to the mild cognitive impairment discriminant labels such as MMSE / MoCA, PET / MRI, and CSF, giving the model high clinical discriminative ability and generalization performance.
[0052] The dual-modal signal processing method for intelligent screening of mild cognitive impairment of the present invention has the following beneficial effects: 1. Objective and efficient task incentives: It adopts a multi-task digital-driven cognitive test, which automatically generates coded incentive events and accurately records timestamps, avoiding the bias of subjective scoring in traditional scales and improving screening efficiency and repeatability.
[0053] 2. Strictly synchronized time alignment, with complementary and mutual information between the two modalities: By employing techniques such as DTW and cross-modal calibration, millisecond-level precise alignment of EEG signals and eye-tracking signals is achieved, constructing a high-precision time-synchronized bimodal dataset. This dataset fully leverages the complementary advantages of dynamic EEG responses and stable physiological indicators, reducing information loss caused by data mismatches and thus significantly improving the model's sensitivity in recognizing MCI.
[0054] 3. Improved Fully Convolutional Network (FCN): Multi-scale convolution, skip connections, and upsampling design effectively ensure the complete extraction of spatiotemporal features of dual-modal signals. The grouped parallel deep learning architecture inputs different modal data into the FCN deep network in parallel, and after each extracts features, they interact deeply in the attention fusion layer, which significantly improves the ability to capture weak signals of mild cognitive impairment (MCI) and the classification accuracy.
[0055] 4. Non-invasive, low-cost, and large-scale deployment: All tests are non-invasive and can be performed with standardized software. The required equipment cost is far lower than that of PET / MRI, making it suitable for large-scale screening in community and primary healthcare institutions.
[0056] 5. Iterative optimization and clinical integration: The model can be trained by pairing with gold standard data such as MMSE / MoCA, PET / MRI / CSF, and continuously iterated and optimized; it can also output interpretable attention weights and eye movement signal contributions, providing auxiliary support for clinical decision-making.
[0057] In one specific embodiment, the dual-modal signal processing method for intelligent screening of mild cognitive impairment according to the present invention can achieve the following community-based early screening for MCI, as exemplified below: One hundred volunteers aged 60 and above wore EEG caps and completed a 10-minute multi-task digital interaction test while eye-tracking signals were collected. The DTW algorithm was used for strict time synchronization to construct a time-synchronized bimodal dataset, which was then input into a deep network according to the above steps. The system automatically generated an MCI risk report, with the overall screening time being less than 30 minutes.
[0058] In one specific embodiment, the dual-modal signal processing method for intelligent screening of mild cognitive impairment according to the present invention is validated through clinical trials, as shown in the following example: The same procedure was performed on the MCI patient group and the cognitively normal group, and the classification accuracy of the AI model was compared with the results of PET / MRI / CSF detection to verify that this method is superior to a single scale or a single EEG method in terms of sensitivity and specificity.
[0059] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of this invention. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the protection scope of this invention.
[0060] Another embodiment of the present invention relates to a bimodal signal processing system for intelligent screening of mild cognitive impairment. The implementation details of this bimodal signal processing system for intelligent screening of mild cognitive impairment are described below. The following implementation details are provided for ease of understanding and are not essential for implementing this solution. The bimodal signal processing system for intelligent screening of mild cognitive impairment in this embodiment includes: The data acquisition module is used to acquire the electroencephalogram (EEG) and eye movement signals of the target object under cognitive task-driven conditions. The first data processing module is used to register EEG signals and eye movement signals according to the cognitive task to obtain EEG signals and eye movement signals with the same time series length; wherein, the time series of the registered EEG signals and eye movement signals are composed of multiple time windows; The second data processing module is used to obtain target EEG signals and target eye movement signals with consistent distribution attributes by minimizing the statistical differences between the registered EEG signals and eye movement signals in each corresponding time window and maximizing the temporal correlation between the registered EEG signals and eye movement signals in each corresponding time window; wherein, the statistical differences include mean differences, variance / standard deviation differences and distribution shape differences.
[0061] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.
[0062] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by this invention; however, this does not mean that other units are absent from this embodiment.
[0063] Another embodiment of the present invention relates to a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the bimodal signal processing method for intelligent screening of mild cognitive impairment as described in the above embodiments.
[0064] The memory and processor are connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and memories. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.
[0065] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.
[0066] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.
[0067] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0068] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing the present invention, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of the present invention.
Claims
1. A dual-modal signal processing method for intelligent screening of mild cognitive impairment, characterized in that, The method includes: Acquire the electroencephalogram (EEG) and eye movement signals of the target subject under cognitive task-driven conditions; EEG and eye movement signals are registered according to a cognitive task to obtain EEG and eye movement signals with the same time series length; the time series of the registered EEG and eye movement signals are each composed of multiple time windows; By minimizing the statistical differences between the registered EEG and eye-tracking signals in each corresponding time window and maximizing the temporal correlation between the registered EEG and eye-tracking signals in each corresponding time window, target EEG and target eye-tracking signals with consistent distribution attributes can be obtained; where statistical differences include differences in mean, variance / standard deviation, and distribution shape. The cognitive task consists of multiple key events. The registration of EEG and eye movement signals according to the cognitive task to obtain EEG and eye movement signals of the same time series length includes: Both EEG and eye movement signals are divided into multiple EEG signal segments and multiple eye movement signal segments corresponding to multiple key events. A dynamic time warping algorithm is used to register the EEG signal segments and eye movement signal segments corresponding to the same key event based on morphological similarity, so as to compensate for the reaction time difference of the target object's EEG signal segments and eye movement signal segments driven by the same key event, thus obtaining EEG signal segments and eye movement signal segments corresponding to the same key event with the same time series length. After obtaining the target EEG signal and target eye movement signal with consistent distribution properties, the method further includes: The target EEG signal and target eye movement signal are input into a pre-trained fully convolutional network in parallel. Multi-scale convolutional kernels are used to extract features at different time scales of the target EEG signal and target eye movement features, which are then used as target EEG features and target eye movement features. After mapping the target EEG features and target eye movement features to a common representation space, the target EEG features and target eye movement features are weighted separately through an attention mechanism. The weighted target EEG features and target eye movement features are then fused to obtain the target fusion feature.
2. The dual-modal signal processing method for intelligent screening of mild cognitive impairment according to claim 1, characterized in that, The method of inputting the target EEG signal and the target eye movement signal into the trained fully convolutional network in parallel includes: The target EEG signal is preprocessed to extract the spatiotemporal features of the target EEG signal in different frequency bands. The spatiotemporal features include the power spectrum, phase lock value and microstate features of the target EEG signal. Preprocessing of the target eye movement signal is performed to extract the pupil diameter, saccade speed, fixation position, and blink frequency of the target object as dynamic behavioral features of the target eye movement signal; The spatiotemporal features of the target EEG signal and the dynamic behavioral features of the target eye movement signal are input in parallel into the trained fully convolutional network.
3. The dual-modal signal processing method for intelligent screening of mild cognitive impairment according to claim 2, characterized in that, The spatiotemporal characteristics of the target EEG signal are obtained through independent component analysis or empirical mode decomposition, and the dynamic behavioral characteristics of the target eye movement signal are obtained through polynomial fitting or principal component analysis.
4. A dual-modal signal processing system for intelligent screening of mild cognitive impairment, characterized in that, The system includes: The data acquisition module is used to acquire the electroencephalogram (EEG) and eye movement signals of the target object under cognitive task-driven conditions. The first data processing module is used to register EEG signals and eye movement signals according to the cognitive task to obtain EEG signals and eye movement signals with the same time series length; wherein, the time series of the registered EEG signals and eye movement signals are composed of multiple time windows; The second data processing module is used to obtain target EEG signals and target eye movement signals with consistent distribution attributes by minimizing the statistical differences between the registered EEG signals and eye movement signals in each corresponding time window and maximizing the temporal correlation between the registered EEG signals and eye movement signals in each corresponding time window; wherein, the statistical differences include mean differences, variance / standard deviation differences and distribution shape differences. The cognitive task consists of multiple key events, and the first data processing module is further used for: Both EEG and eye movement signals are divided into multiple EEG signal segments and multiple eye movement signal segments corresponding to multiple key events. A dynamic time warping algorithm is used to register the EEG signal segments and eye movement signal segments corresponding to the same key event based on morphological similarity, so as to compensate for the reaction time difference of the target object's EEG signal segments and eye movement signal segments driven by the same key event, thus obtaining EEG signal segments and eye movement signal segments corresponding to the same key event with the same time series length. The target EEG signal and the target eye movement signal are input in parallel into a pre-trained fully convolutional network. Multi-scale convolutional kernels are used to extract features at different time scales of the target EEG signal and the target eye movement feature, which are then used as the target EEG feature and the target eye movement feature. After the target EEG feature and the target eye movement feature are mapped to a common representation space, the target EEG feature and the target eye movement feature are weighted separately through an attention mechanism. The weighted target EEG feature and the target eye movement feature are then fused to obtain the target fusion feature.
5. A computer device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the bimodal signal processing method for intelligent screening of mild cognitive impairment as described in any one of claims 1 to 3.
6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the dual-modal signal processing method for intelligent screening of mild cognitive impairment as described in any one of claims 1 to 3.