Bimodal signal processing method for intelligent screening of mild cognitive impairment

Abstract

The invention relates to the technical field of early screening of mild cognitive impairment, and discloses a bimodal signal processing method for intelligent screening of mild cognitive impairment, comprising: acquiring an electroencephalogram signal and an eye movement signal of a target object driven by a cognitive task; registering the electroencephalogram signal and the eye movement signal according to the cognitive task to obtain the electroencephalogram signal and the eye movement signal with the same time sequence length; a target electroencephalogram signal and a target eye movement signal which are consistent in distribution attribute are obtained by minimizing the statistical difference of the registered electroencephalogram signal and eye movement signal in each corresponding time window and maximizing the time sequence correlation of the registered electroencephalogram signal and eye movement signal in each corresponding time window; wherein the statistical difference comprises a mean value difference, a variance / standard deviation difference and a distribution shape difference. According to the bimodal signal processing method for intelligent screening of mild cognitive impairment, the data acquisition cost is relatively low, and the representation accuracy of the finally obtained signal on cognitive features is relatively high.

Description

Technical Field

[0001] The present invention relates to the technical field of early screening for mild cognitive impairment, and in particular to a dual-modal signal processing method, system, device and medium for intelligent screening for mild cognitive impairment. Background Art

[0002] Mild cognitive impairment (MCI) is an early stage of dementia, characterized by poorer memory and thinking compared to age-matched peers. Approximately 60% of MCI cases due to Alzheimer's disease (AD) will eventually progress to dementia. MCI represents a golden window for dementia intervention, yet it is often overlooked. Without effective interventions, the number of people with dementia will continue to rise.

[0003] Traditional screening for mild cognitive impairment relies on subjective scales, such as the Mini-Mental State Examination (MMSE) and the Montreal Cognitive Assessment (MoCA). However, using the results of subjective scales to characterize patients' cognitive characteristics is highly subjective and inaccurate, making it difficult for doctors to assist in screening patients for MCI. Alternatively, screening based on positron emission tomography (PET) / magnetic resonance imaging (MRI) / cerebrospinal fluid (CSF) testing is invasive and costly, making it difficult for large-scale promotion and assisting doctors in screening patients for MCI.

[0004] Based on this, existing technologies have proposed using patients' EEG signals to assess MCI risk and achieve MCI screening, but this screening method based on a single modality is not accurate enough; some technologies combine patients' bimodal signals, such as EEG signals and eye movement signals, but signals of different modalities have significant differences in acquisition methods, time scales, and data structures, and cannot be effectively integrated, which will also affect MCI screening. Summary of the Invention

[0005] The purpose of the present invention is to provide a dual-modal signal processing method, system, device and medium for intelligent screening of mild cognitive impairment, which can solve the problem that signals of different modalities have significant differences in acquisition methods, time scales and data structures, cannot be effectively integrated, and thus affect MCI screening.

[0006] To solve the above technical problems, embodiments of the present application provide a dual-mode signal processing method for mild cognitive impairment intelligent screening, comprising the following steps: obtaining the electroencephalogram signal and the eye movement signal of the target object under the driving of the cognitive task; registering the electroencephalogram signal and the eye movement signal according to the cognitive task to obtain the electroencephalogram signal and the eye movement signal with the same time sequence length; wherein the time sequence of the registered electroencephalogram signal and the eye movement signal is composed of a plurality of time windows; minimizing the statistical difference of the registered electroencephalogram signal and the eye movement signal in each corresponding time window, and maximizing the time sequence correlation of the registered electroencephalogram signal and the eye movement signal in each corresponding time window, to obtain the target electroencephalogram signal and the target eye movement signal with consistent distribution attributes; wherein the statistical difference includes mean difference, variance / standard deviation difference and distribution shape difference.

[0007] Optionally, the cognitive task is composed of a plurality of key events, and the registration of the electroencephalogram signal and the eye movement signal according to the cognitive task to obtain the electroencephalogram signal and the eye movement signal with the same time sequence length comprises: dividing the electroencephalogram signal and the eye movement signal into a plurality of electroencephalogram signal segments and a plurality of eye movement signal segments corresponding to a plurality of key events; adopting a dynamic time warping algorithm to register the electroencephalogram signal segment and the eye movement signal segment corresponding to the same key event according to the morphological similarity, to compensate for the reaction time difference of the electroencephalogram signal segment and the eye movement signal segment of the target object driven by the same key event, and obtain the electroencephalogram signal segment and the eye movement signal segment corresponding to the same key event with the same time sequence length.

[0008] Optionally, after obtaining the target electroencephalogram signal and the target eye movement signal with consistent distribution attributes, the method further comprises: inputting the target electroencephalogram signal and the target eye movement signal into a trained full convolution network in parallel, and adopting a multi-scale convolution kernel to extract features of different time scales of the target electroencephalogram signal and the target eye movement signal as target electroencephalogram features and target eye movement features; after mapping the target electroencephalogram features and the target eye movement features to a common representation space, weighting the target electroencephalogram features and the target eye movement features through an attention mechanism, and fusing the weighted target electroencephalogram features and the target eye movement features to obtain target fusion features.

[0009] Optionally, the inputting the target electroencephalogram signal and the target eye movement signal into a trained full convolution network in parallel comprises: preprocessing the target electroencephalogram signal to extract the spatiotemporal features of the target electroencephalogram signal in different frequency bands, wherein the spatiotemporal features include power spectrum, phase locking value and microstate features of the target electroencephalogram signal; The target eye movement signal is preprocessed to extract a pupil diameter, a saccade speed, a fixation position and a blink frequency of the target object as dynamic behavior characteristics of the target eye movement signal. The spatio-temporal characteristics of the target electroencephalogram signal and the dynamic behavior characteristics of the target eye movement signal are input into the trained full convolution network in parallel.

[0010] Optionally, the spatio-temporal characteristics of the target electroencephalogram signal are obtained through independent component analysis or empirical mode decomposition processing, and the dynamic behavior characteristics of the target eye movement signal are obtained through polynomial fitting or principal component analysis processing.

[0011] Embodiments of the present application also provide a dual-modality signal processing system for mild cognitive impairment intelligent screening, comprising: A data acquisition module is configured to acquire an electroencephalogram signal and an eye movement signal of a target object under the driving of a cognitive task; A first data processing module is configured to register the electroencephalogram signal and the eye movement signal according to the cognitive task to obtain electroencephalogram signals and eye movement signals with the same time sequence length; wherein the time sequences of the registered electroencephalogram signals and eye movement signals are both composed of a plurality of time windows; A second data processing module is configured to minimize statistical differences of the registered electroencephalogram signal and the eye movement signal in each corresponding time window and maximize time sequence correlations of the registered electroencephalogram signal and the eye movement signal in each corresponding time window to obtain target electroencephalogram signals and target eye movement signals with consistent distribution attributes; wherein the statistical differences include mean difference, variance / standard deviation difference and distribution shape difference.

[0012] Embodiments of the present application also provide a computer device, comprising: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the dual-modality signal processing method for mild cognitive impairment intelligent screening.

[0013] Embodiments of the present application also provide a computer readable storage medium storing a computer program, and the computer program is executed by a processor to implement the dual-modality signal processing method for mild cognitive impairment intelligent screening.

[0014] The dual-modality signal processing method for mild cognitive impairment intelligent screening provided by the present application has at least the following beneficial effects: The target object is stimulated through a cognitive task, and the electroencephalogram signal and the eye movement signal of the target object under the driving of the cognitive task are obtained. Not only is a registration operation performed on the bimodal signals (the electroencephalogram signal and the eye movement signal) to realize the synchronization alignment of the bimodal data, but also the consistency of the bimodal signals is achieved at a higher level (the statistical distribution level) by minimizing the inter-modal statistical difference and maximizing the inter-modal correlation, so that the target electroencephalogram signal and the target eye movement signal are obtained. That is, the bimodal signals obtained by the present application can be used for MCI screening, so as to overcome the defect that the MCI screening is affected due to the fact that the bimodal signals cannot be effectively fused. BRIEF DESCRIPTION OF DRAWINGS

[0015] The accompanying drawings, which are included to provide a further understanding of the present application, form a part of the present application and are included to further explain the present application and, together with the specification, serve to explain the present application. In the drawings:

[0016] Figure 1 A flowchart of a bimodal signal processing method for mild cognitive impairment intelligent screening provided by the present application is shown in the figure. Figure 2 A framework diagram of a bimodal signal processing method for mild cognitive impairment intelligent screening provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0017] To make the objects, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in detail with reference to the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0018] The technical solutions provided by the embodiments of the present application will be described in detail below with reference to the drawings.

[0019] One embodiment of the present application relates to a bimodal signal processing method for mild cognitive impairment intelligent screening. The implementation details of the cognitive mild impairment intelligent screening method of the bimodal feature fusion of the present embodiment will be described in detail below. The following content is only provided for the implementation details for the convenience of understanding, and is not necessary for implementing the present solution.

[0020] The specific process of the bimodal signal processing method for mild cognitive impairment intelligent screening of the present embodiment can be as shown in the figure, which includes: Figure 1 ​Step 101: Obtain EEG signals and eye movement signals of a target subject driven by a cognitive task.

[0021] Specifically, this embodiment aims to stimulate brain activity through structured cognitive interaction tasks while simultaneously collecting neural electrical signals (EEG signals) and eye movement signals reflecting visual attention and behavioral responses. This step includes the following operations:

[0022] 1. Multi-task cognitive interaction design: Multiple cognitive task modules are pre-installed on mobile devices (such as tablets), including episodic memory tasks, continuous attention tests, and spatial reaction tasks, covering cognitive dimensions such as memory, attention, and reaction. The system automatically executes tasks and generates event labels to ensure standardization and repeatability.

[0023] 2. Task event marking and synchronization mechanism: In each task, the system automatically generates event labels with timestamps to align EEG signals with eye movement signals.

[0024] 3. Synchronous EEG and eye movement acquisition mechanism: The target subject wears an EEG acquisition device and an eye tracking device, and simultaneously records EEG and eye movement signals (such as gaze position, pupil diameter, saccade speed, blink frequency, etc.) during the task execution. All signals are recorded synchronously through a unified clock to construct structured dual-modal sequence data.

[0025] Compared with traditional questionnaire-based scales, digitally driven tasks are more objective and repeatable; EEG is strictly synchronized with interaction time points, which can form structured and labeled neural response samples, facilitating subsequent modeling.

[0026] Step 102 , aligning the EEG signal and the eye movement signal according to the cognitive task to obtain EEG signals and eye movement signals with the same time series length; wherein the time series of the aligned EEG signal and eye movement signal are both composed of multiple time windows.

[0027] Specifically, the cognitive task consists of multiple key events, and the EEG signals and eye movement signals are divided into multiple EEG signal segments and multiple eye movement signal segments corresponding to the multiple key events; the Dynamic Time Warping (DTW) algorithm is used to align the EEG signal segments and eye movement signal segments corresponding to the same key event according to morphological similarity to compensate for the reaction time difference of the target object's EEG signal segments and eye movement signal segments driven by the same key event, and obtain EEG signal segments and eye movement signal segments corresponding to the same key event with the same time series length.

[0028] The DTW algorithm is used to measure and align two time series of different lengths, so as to find an "optimal mapping path" that minimizes the total distance between corresponding points of the two sequences (calculating the Euclidean distance) while preserving the time structure of the sequences.

[0029] In step 103, the target electroencephalogram signal and the target eye movement signal with consistent distribution attributes are obtained by minimizing the statistical difference of the registered electroencephalogram signal and the eye movement signal in each corresponding time window and maximizing the time correlation of the registered electroencephalogram signal and the eye movement signal in each corresponding time window; wherein the statistical difference includes mean difference, variance / standard deviation difference and distribution shape difference.

[0030] In a specific implementation, after time alignment, the aligned multi-modal features may still have deviations in the distribution level. The Cross-Modal Consistency Calibration (CMCC) method is used to minimize the statistical difference between modalities or maximize the correlation between modalities, so that the modal features have consistent distribution attributes in the same shared space, thereby improving the convergence and discriminability of the subsequent fusion network. The peak time between the aligned statistical features is further eliminated to eliminate the time deviation of multi-sensor acquisition. After alignment, the modal data accurately corresponds on the common time reference, realizing the synchronization at the micro-event level.

[0031] For example, the CMCC method uses statistical quantity alignment: minimizing the difference in covariance matrix of source modalities (such as EEG) and target modalities (such as eye movement signal) features.

[0032] After obtaining the target electroencephalogram signal and the target eye movement signal with consistent distribution attributes, the target electroencephalogram signal and the target eye movement signal are input into the trained full convolutional network in parallel for feature extraction.

[0033] In one example, the target electroencephalogram signal is first preprocessed to extract the spatiotemporal features of the target electroencephalogram signal in different frequency bands, including the power spectrum, phase locking value and microstate features of the target electroencephalogram signal; the target eye movement signal is preprocessed to extract the pupil diameter, saccade speed, fixation position and blink frequency of the target object as dynamic behavior features of the target eye movement signal; and the spatiotemporal features of the target electroencephalogram signal and the dynamic behavior features of the target eye movement signal are input into the trained full convolutional network in parallel.

[0034] In a specific implementation, the electroencephalogram signal is preprocessed by filtering, independent component analysis (ICA), empirical mode decomposition (EMD), etc., and the spatiotemporal features (such as power spectrum, phase locking value, microstate feature, etc.) of each frequency band are extracted. The eye movement signal is normalized, polynomial fitted or principal component analyzed to obtain a statistical feature vector. After processing, the two modalities have comparable feature representations.

[0035] In one example, during the training process, the above multi-modal data set is grouped according to the requirements of the input network: for example, group 1 is the EEG spatiotemporal feature segment, and group 2 is the dynamic behavior feature of the eye movement signal. Different slicing strategies are used for each group of data, such as dividing the EEG according to the task stage to pre-stimulus and post-stimulus segments, or taking overlapping sliding windows for the signal in the time window. In order to increase the diversity of data and prevent overfitting, a data enhancement mechanism is introduced: the EEG signal is subjected to time domain random jitter, signal mirroring, frequency band time shift, etc.; for the eye movement signal, additional samples are synthesized using a generative adversarial network (GAN). The enhanced dual modal data is still grouped after being aligned in time sequence, and together with the original data forms the training set.

[0036] In one specific example, during the training process, the above dual modal data set of electroencephalogram and eye movement is preprocessed and enhanced according to the requirements of the neural network structure: Data slicing strategy: the EEG data is divided into pre-stimulus and post-stimulus segments according to the task event segments, or time slicing is performed using overlapping sliding windows; the eye movement signal is divided into event-related sequences according to the fixation segment or saccade segment.

[0037] Data enhancement mechanism: in order to improve the robustness of the model, a multi-modal data enhancement strategy is introduced: the EEG signal is subjected to time domain random jitter, mirror flip, frequency band disturbance, etc.; the eye movement trajectory is subjected to random disturbance, noise insertion, pupil diameter linear scaling, etc. to increase the diversity of samples.

[0038] The enhanced dual modal data still needs to be aligned by dynamic time warping (DTW) to make each pair of electroencephalogram and eye movement samples strictly correspond in the time dimension, and to construct a structured training set.

[0039] This embodiment first introduces advanced time sequence algorithms such as DTW in cognitive impairment early screening, accurately aligns the electroencephalogram and eye movement indicators across modalities, uses cross-modal statistical calibration methods to correct the acquisition time delay difference, and constructs a truly strictly synchronized multi-source data set. A multi-level data enhancement strategy is proposed to expand the sample capacity and diversity through time-frequency disturbance and generative model, thereby significantly improving the alignment quality and the robustness of the model.

[0040] After the target electroencephalogram signal and the target eye movement signal are input into the trained full convolutional network in parallel, different time scales of the target electroencephalogram signal and the target eye movement feature are extracted by using multi-scale convolution kernels as the target electroencephalogram feature and the target eye movement feature; after the target electroencephalogram feature and the target eye movement feature are mapped to a common representation space, the target electroencephalogram feature and the target eye movement feature are weighted respectively through an attention mechanism, and the weighted target electroencephalogram feature and the target eye movement feature are fused to obtain a target fusion feature; finally, by sending the target fusion feature into a classifier, a cognitive impairment risk level of a target object can be output.

[0041] In a specific implementation, the network includes the following processing procedures: 1. Parallel feature extraction: Each group of data extracts spatiotemporal features through a customized improved full convolutional network (FCN).

[0042] In view of the characteristics of the dual-modal data, multi-scale convolution kernels (including 3x3, 5x5 and empty convolution) are used to capture features of different time scales on the basis of the traditional FCN, and a skip connection and a residual unit are added in the middle of the network to ensure full fusion of low-level details and high-level semantic features.

[0043] 2. Feature fusion layer: Map the feature maps or vectors output by each sub-network to a common representation space; Map the eye movement feature modal vector to the same dimension of the EEG signal: ; In the formula, is the unified dimension; is the original modal vector, when , denotes an event embedding vector; is a trainable mapping matrix, which linearly projects the i-th modal from the original dimension (i.e. , or ) to the unified dimension ; is a bias vector, which has the same dimension as ; is the modal vector after mapping and unified to the same dimension.

[0044] The feature vector is expanded to the same time sequence dimension as the EEG feature vector: , ; Where, is the number of time steps, which is consistent with the time length of the EEG feature (for example, EEG is cut into T time series segments). "Copy" or broadcast into a time series matrix of the same length as the EEG feature , so that it can be multiplied with the EEG features at each time point in the subsequent fusion.

[0045] Using attention mechanism for deep fusion: The EEG feature F EEG After convolution, it is compressed into a two-dimensional matrix ; For each modality embedding, a separate attention network is designed. The attention calculation function is:

[0046] ; Where, represents the embedding of splicing EEG and the i-th modality in the time dimension, It is a shared multi-layer perceptron whose output is the attention weight of length T.

[0047] Apply the attention weight to each modal feature and calculate the weighted total feature: ; Where ⊙ represents the Hadamard product (point product) at each time point, so that the embedding of each modality dynamically adjusts its contribution at different time points.

[0048] 3. Classification and early screening result output: The fused features are fed into the subsequent classifier to output the risk level of Mild Cognitive Impairment (MCI) and the conclusion of early screening for cognitive impairment. At the same time, during the training phase, the model is paired with traditional gold standard (MMSE / MoCA, PET / CSF) results for iterative optimization to continuously improve sensitivity and specificity.

[0049] This network framework breaks the limitations of the traditional "single-modal input + single-channel convolution", realizes deep semantic fusion and collaborative discrimination between different modalities, greatly improves the sensitivity and specificity of early screening for cognitive impairment, and at the same time has self-learning and adaptive optimization capabilities.

[0050] In one embodiment, the dual-modal feature fusion intelligent early screening for cognitive impairment of the present invention is used for the dual-modal signal processing method of intelligent screening for mild cognitive impairment by Figure 2The framework schematic diagram is implemented, and the method adopts a "three-step closed loop" process: ①: On the mobile terminal, a multi-task digital-driven cognitive test (such as scene memory, digital tapping, classification prompt, etc.) is presented to the subject (i.e. target object), and the system automatically generates coded incentive events and records time stamps at the task start, key stimulus and completion, while synchronously collecting 16-channel EEG signals to accurately capture the spatiotemporal dynamic response of the brain under digital incentive; ②: Strict time-synchronized dual-mode data construction module - process the EEG signal for filtering, EMD, self-supervised denoising, etc., extract the spatiotemporal features; encode the event markers into fixed-length embedding vectors; normalize and reduce the dimension of the eye movement feature signal; use dynamic time warping (DTW) and cross-modal consistency calibration technology to align the three types of data under the millisecond level benchmark, and through sliding window segmentation and GAN-based enhancement strategy, construct a high-quality, diverse and time-synchronized dual-mode dataset; ③: The improved fully convolutional network (FCN) inputs the data of each sub-group in parallel, extracts features in the network, then deep-fuses the inter-modal features through the attention mechanism in the fusion layer, and finally outputs the mild cognitive impairment (MCI) risk assessment and diagnosis label of the subject through the classifier; at the same time, paired training with traditional MMSE / MoCA and PET / MRI / CSF clinical gold standard results is performed to continuously optimize the model performance.

[0051] In one example, to ensure the clinical usability of the model, the embodiment is trained based on the gold standard dataset under the IWG-2021 framework, and the clinician detects mild cognitive impairment (MCI) through the IWG-2021 standard (clinical MCI examination gold standard) and scales such as MMSE / MOCA, molecular images such as PET / MRI, and CSF cerebrospinal fluid. The dataset includes two groups of 180 subjects over 60 years old, one group is the MCI group consisting of patients with mild cognitive impairment (MCI) defined by IWG-2021, and the other group is the control group consisting of normal cognitive people defined by IWG-2021. The dual-mode EEG-Eye signal fusion model trained on this dataset can consistently map on the mild cognitive impairment discrimination labels of MMSE / MoCA, PET / MRI, CSF, etc., so that the model has high clinical discrimination ability and generalization performance.

[0052] The dual-mode signal processing method for intelligent screening of mild cognitive impairment of the present application has the following beneficial effects: 1. Objective and efficient task incentive: Multi-task digital-driven cognitive test is adopted, coded incentive events are automatically generated, and time stamps are accurately recorded, which avoids the bias of traditional scale subjective scoring, improves screening efficiency and repeatability.

[0053] 2. Strictly synchronized time alignment, complementary mutual information of dual-modal information: Through techniques such as DTW and cross-modal calibration, millisecond-level accurate alignment of EEG signals and eye movement signals is achieved, and a high-precision time-synchronized dual-modal dataset is constructed. This dataset fully combines the complementary advantages of dynamic EEG response and stable physiological indicators, reduces information loss caused by data mismatch, and significantly improves the sensitivity of the model in identifying MCI.

[0054] 3. Improved fully convolutional (FCN) network: Multi-scale convolution, skip connection, and up-sampling design effectively ensure the complete extraction of the spatial and temporal features of dual-modal signals. The deep learning architecture divides different modal data into groups and inputs them into the FCN deep network in parallel. After extracting features, they are deeply interacted in the attention fusion layer, significantly improving the ability to capture weak signals of mild cognitive impairment (MCI) and classification accuracy.

[0055] 4. Non-invasive, low-cost, and large-scale promotion: All detections are non-invasive sampling, and can be implemented with standardized software. The required equipment cost is much lower than that of PET / MRI, making it suitable for large-scale screening in communities and primary medical institutions.

[0056] 5. Iterative optimization and clinical interface: The model can be paired with MMSE / MoCA, PET / MRI / CSF, and other gold standard data for training, continuous iterative optimization, and output of interpretable attention weights and eye movement signal contribution, providing auxiliary support for clinical decision-making.

[0057] In one specific embodiment, the dual-modal signal processing method for mild cognitive impairment intelligent screening of the present application can achieve the following community MCI early screening, for example: 100 volunteers over the age of 60 wear EEG caps and complete a 10-minute multi-task digital interaction test while collecting eye movement signals. DTW algorithm is used for strict time synchronization to construct a time-synchronized dual-modal dataset, which is input into the deep network according to the above steps. The system automatically generates an MCI risk report, and the overall screening time is < 30 minutes.

[0058] In one specific embodiment, the dual-modal signal processing method for mild cognitive impairment intelligent screening of the present application is verified through clinical trials, for example: The MCI patient group and the cognitively normal group perform the same procedure, compare the AI model classification accuracy, and compare it with the PET / MRI / CSF detection results to verify that the method is superior to single scale or single EEG method in terms of sensitivity and specificity.

[0059] The step division of the above various methods is only for the purpose of clear description, and can be combined into one step or split into multiple steps in implementation, as long as the same logical relationship is included, and all are within the protection scope of the present application; adding irrelevant modifications or introducing irrelevant designs in the algorithm or flow, but not changing the core design of the algorithm and flow, are within the protection scope of the present application.

[0060] Another embodiment of the present application relates to a dual-mode signal processing system for mild cognitive impairment intelligent screening, and the implementation details of the dual-mode signal processing system for mild cognitive impairment intelligent screening in the present embodiment will be specifically described below. The following content is only provided for the implementation details for the convenience of understanding, and is not essential for implementing the present solution. The dual-mode signal processing system for mild cognitive impairment intelligent screening in the present embodiment comprises: A data acquisition module configured to acquire electroencephalogram signals and eye movement signals of a target object under the driving of a cognitive task; A first data processing module configured to register the electroencephalogram signals and the eye movement signals according to the cognitive task to obtain electroencephalogram signals and eye movement signals with the same time sequence length; wherein the time sequences of the registered electroencephalogram signals and the eye movement signals are both composed of multiple time windows; A second data processing module configured to minimize the statistical difference of the registered electroencephalogram signals and the eye movement signals in each corresponding time window and maximize the time sequence correlation of the registered electroencephalogram signals and the eye movement signals in each corresponding time window to obtain target electroencephalogram signals and target eye movement signals with consistent distribution attributes; wherein the statistical difference includes mean difference, variance / standard deviation difference and distribution shape difference.

[0061] It can be found that the present embodiment is a system embodiment corresponding to the above method embodiment, and the present embodiment can be implemented in cooperation with the above method embodiment. The related technical details and technical effects mentioned in the above embodiments are still effective in the present embodiment. In order to reduce repetition, they will not be described here. Correspondingly, the related technical details mentioned in the present embodiment can also be applied in the above embodiments.

[0062] It is worth mentioning that each module involved in the present embodiment is a logical module. In actual application, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of the present application, units not closely related to solving the technical problems proposed in the present application are not introduced in the present embodiment, but this does not mean that there are no other units in the present embodiment.

[0063] Another embodiment of the present application relates to a computer device, comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the dual-modality signal processing method for mild cognitive impairment intelligent screening in the above embodiments.

[0064] The memory and the processor are connected by a bus in a bus mode, the bus can include any number of interconnected buses and bridges, and the bus connects various circuits of the one or more processors and the memory together. The bus can also connect various other circuits such as peripheral devices, voltage stabilizers and power management circuits together, which are well known in the art, and thus, further description thereof will not be given herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements such as multiple receivers and transmitters, which provide a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium through the antenna, and further, the antenna also receives data and transmits the data to the processor.

[0065] The processor is responsible for managing the bus and general processing, and can also provide various functions including timing, peripheral interface, voltage regulation, power management and other control functions. The memory can be used to store data used by the processor in performing operations.

[0066] Another embodiment of the present application relates to a computer readable storage medium, which stores a computer program. The computer program is executed by the processor to implement the method embodiments described above.

[0067] That is, those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by programs instructing related hardware, the programs are stored in a storage medium, and the storage medium includes a plurality of instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various storage medium capable of storing program codes.

[0068] Those skilled in the art can understand that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes can be made in form and details without departing from the spirit and scope of the present application.

Claims

1. A dual-modal signal processing method for intelligent screening of mild cognitive impairment, characterized in that: The method comprises: Obtaining EEG signals and eye movement signals of the target subject driven by cognitive tasks; The EEG signals and eye movement signals are registered according to the cognitive task to obtain EEG signals and eye movement signals with the same time series length; wherein the time series of the registered EEG signals and eye movement signals are each composed of multiple time windows; By minimizing the statistical differences between the registered EEG signals and eye movement signals in each corresponding time window and maximizing the temporal correlation between the registered EEG signals and eye movement signals in each corresponding time window, target EEG signals and target eye movement signals with consistent distribution properties are obtained; the statistical differences include mean differences, variance / standard deviation differences, and distribution shape differences.

2. The dual-modal signal processing method for intelligent screening of mild cognitive impairment according to claim 1, characterized in that: The cognitive task is composed of multiple key events, and the registration of EEG signals and eye movement signals according to the cognitive task to obtain EEG signals and eye movement signals with the same time series length includes: Dividing the EEG signal and the eye movement signal into multiple EEG signal segments and multiple eye movement signal segments corresponding to multiple key events; The dynamic time warping algorithm is used to align the EEG signal segments and eye movement signal segments corresponding to the same key event according to morphological similarity to compensate for the reaction time difference of the target object's EEG signal segments and eye movement signal segments driven by the same key event, and obtain the EEG signal segments and eye movement signal segments corresponding to the same key event with the same time series length.

3. The dual-modal signal processing method for intelligent screening of mild cognitive impairment according to claim 1, characterized in that: After obtaining the target EEG signal and the target eye movement signal with consistent distribution properties, the method further includes: The target EEG signal and target eye movement signal are input into the trained fully convolutional network in parallel, and multi-scale convolution kernels are used to extract the features of the target EEG signal and target eye movement features at different time scales as the target EEG features and target eye movement features respectively. After mapping the target EEG features and target eye movement features into a common representation space, the target EEG features and target eye movement features are weighted separately through the attention mechanism, and the weighted target EEG features and target eye movement features are fused to obtain the target fusion features.

4. The dual-modal signal processing method for intelligent screening of mild cognitive impairment according to claim 3, characterized in that: The step of inputting the target EEG signal and the target eye movement signal into the trained fully convolutional network in parallel includes: Preprocessing the target EEG signal to extract the spatiotemporal features of the target EEG signal in different frequency bands, wherein the spatiotemporal features include the power spectrum, phase locking value, and microstate features of the target EEG signal; Preprocessing the target eye movement signal to extract the target object's pupil diameter, saccade speed, gaze position, and blink frequency as the dynamic behavior characteristics of the target eye movement signal; The spatiotemporal features of the target EEG signal and the dynamic behavioral features of the target eye movement signal are input into the trained fully convolutional network in parallel.

5. The dual-modal signal processing method for intelligent screening of mild cognitive impairment according to claim 4, characterized in that: The spatiotemporal characteristics of the target EEG signal are obtained through independent component analysis or empirical mode decomposition, and the dynamic behavior characteristics of the target eye movement signal are obtained through polynomial fitting or principal component analysis.

6. A dual-modal signal processing system for intelligent screening of mild cognitive impairment, characterized in that: The system comprises: A data acquisition module is used to obtain the target subject's EEG signals and eye movement signals driven by cognitive tasks; a first data processing module, configured to register the EEG signal and the eye movement signal according to a cognitive task, so as to obtain EEG signals and eye movement signals having the same time series length; wherein the time series of the registered EEG signal and eye movement signal are each composed of multiple time windows; The second data processing module is used to obtain target EEG signals and target eye movement signals with consistent distribution properties by minimizing the statistical differences of the aligned EEG signals and eye movement signals in each corresponding time window and maximizing the temporal correlation of the aligned EEG signals and eye movement signals in each corresponding time window; wherein the statistical differences include mean differences, variance / standard deviation differences, and distribution shape differences.

7. A computer device, characterized in that: include: at least one processor; And, a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the dual-modal signal processing method for intelligent screening of mild cognitive impairment as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the dual-modal signal processing method for intelligent screening of mild cognitive impairment according to any one of claims 1 to 5 is implemented.

Images