Intelligent system and method for early screening of depression based on electroencephalogram-eye movement multi-modal data fusion

Abstract

The invention discloses an intelligent system and method for early screening of depression based on electroencephalogram-eye movement multi-modal data fusion. According to the system, in a virtual reality situation, emotion and cognitive response of a subject are stimulated through an elaborately-designed cognitive task, and brain electrical activity and sight behavior data are synchronously collected by adopting a wearable EEG device and a high-precision eye tracker. After signal preprocessing and feature extraction, deep fusion of multi-modal data is realized by using a weighted fusion formula or a graph neural network, the weight of each modal is automatically adjusted, interaction features between electroencephalogram and eye movement are extracted, and then the depression risk of a subject is accurately judged through a classifier. Experiments show that the method can effectively capture weak physiological abnormalities of mild depression patients, has the advantages of high sensitivity, high accuracy and real-time online screening, and meets the requirements of non-invasive, portable and intelligent clinical application.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent mental health screening, and particularly relates to an intelligent system and method for early screening of depression by fusing electroencephalogram-eye movement multimodal data. Background Art

[0002] Depression is a common mental illness that seriously affects the social function of patients, and its core symptoms include low mood, decreased interest, etc. Depression has a high prevalence and high harmfulness, and has become a public health problem that needs to be addressed urgently globally. Especially mild depression is in the early stage of depression. If it can be detected and intervened early, it is of great significance for preventing the aggravation of the condition. However, due to the often non-obvious early symptoms of mild depression and the low medical treatment rate, traditional diagnosis mainly relies on the patient's self-report and clinical interview, and it is easy to miss or misdiagnose. Some existing scale assessment methods (such as the PHQ-9 questionnaire) are subjective, and the specific detection effects of auxiliary means such as brain function imaging and speech analysis on mild symptoms are limited, resulting in insufficient attention and recognition of mild depression.

[0003] Electroencephalogram (EEG), as a non-invasive neurophysiological signal, is very sensitive to the slightest changes in brain activity and has a high temporal resolution of milliseconds, and can objectively reflect the brain function state. In the study of depression, there have been literatures reporting that EEG signals such as α-wave asymmetry and certain event-related potential (ERP) components are associated with depressive symptoms. However, relying solely on the single modality of EEG for depression detection faces problems such as large individual differences and insufficient specificity. The differences in EEG characteristics of different subjects may lead to unstable classification results and poor generalization performance.

[0004] Eye movement tracking technology can record behavioral data such as the fixation points, gaze duration, and pupil diameter changes of the subjects. These eye movement characteristics can intuitively reflect the attention allocation and emotional responses of individuals. For example, depressed individuals may have a negative attention bias when viewing emotional stimuli, manifested as longer fixation on negative stimuli or more difficulty in shifting the line of sight from negative information. Using eye movement data alone for depression recognition has made some progress to a certain extent, but there are also problems such as incomplete features and limited accuracy.

[0005] The identification of depression based on multimodal data fusion has become a research hotspot in recent years. As two complementary physiological signal sources, electroencephalogram (EEG) and eye movement are of great potential in depression detection: EEG provides information on internal neural activities, while eye movement reflects external behavioral characteristics. The combination of the two is expected to more comprehensively characterize depression-related states. However, the current research on depression identification based on EEG-eye movement fusion is still in its infancy. Existing methods mostly simply splice the features of the two signals and input them into a classifier, lacking in-depth modeling of the correlation and complementary relationship between multimodal data. In addition, the previous cognitive task paradigms used are relatively single, such as freely viewing emotional faces and static stimulus responses, which may not be sensitive enough to elicit subtle symptoms of mild depression. In terms of technical implementation, there is currently no solution that introduces advanced technologies such as virtual reality, brain-computer interface, and remote sensing signal processing into this field to improve the screening performance.

[0006] In summary, the existing technologies have the following deficiencies in the early objective screening of mild depression: (1) The means of single-modal physiological signals are limited, making it difficult to comprehensively and accurately identify mild depression; (2) The multimodal fusion methods are not yet perfect, and the associated features between different signals have not been fully explored. The fusion strategy is simple, resulting in the need to improve the recognition accuracy and generalization ability; (3) There is a lack of cognitive task stimulation paradigms carefully designed for the characteristics of mild depression, and the existing test sensitivity is insufficient; (4) Advanced technologies in other fields have not been organically combined to achieve a new architecture and performance improvement for depression screening. To solve the above problems, it is necessary to provide a brand-new intelligent screening system and method for early-stage depression that combines EEG and eye movement to improve the detection sensitivity and accuracy of mild depression. Summary of the Invention

[0007] Technical Objectives: Aiming at the deficiencies of the existing technologies in the early objective screening of depression, the present invention discloses an intelligent system and method for early-stage depression screening based on EEG-eye movement multimodal data fusion. By fusing two physiological signals, EEG and eye movement, and designing specific cognitive task paradigms, the present invention can sensitively capture the subtle cognitive abnormalities of mild depression patients and achieve objective and efficient screening of the early risks of depression.

[0008] Technical Solutions: To achieve the above technical objectives, the present invention adopts the following technical solutions:

[0009] An intelligent system for early-stage depression screening based on EEG-eye movement multimodal data fusion, comprising:

[0010] An EEG signal acquisition module, configured to acquire the EEG signals of a subject during the execution of a cognitive task and output multi-channel EEG data;

[0011] An eye movement tracking module, configured to synchronously capture the eye movement information of the subject and generate eye movement data, including the position of the fixation point, the duration of the gaze, and the change in pupil diameter;

[0012] A cognitive task presentation module, configured to provide a pre-designed cognitive task and a stimulation scenario to a subject to induce cognitive processes and emotional responses related to depression, wherein the cognitive task includes an emotional stimulus browsing and a cognitive function test session;

[0013] A data processing and fusion analysis module, configured to receive the electroencephalogram data and eye movement data, preprocess them, extract features, and perform correlation modeling and comprehensive analysis on the electroencephalogram features and eye movement features by using a multimodal data fusion algorithm, and output a depression risk assessment result of the subject;

[0014] A result output module, configured to present or store the assessment result in a human-readable form, including displaying a depression tendency score, a discrimination conclusion and corresponding explanatory information of the subject.

[0015] Preferably, the cognitive task presentation module includes a virtual reality display device, and the cognitive task is carried out in a virtual reality scenario to simulate a real scenario and enhance the immersion of the cognitive task for the subject;

[0016] The cognitive task paradigm includes: presenting positive and negative emotional stimuli simultaneously to detect the attention bias of the subject, measuring his behaviors and physiological responses in a high cognitive load task, and observing his emotional responses in a reward decision scenario, so as to stimulate the specific response patterns of mildly depressed subjects from multiple angles.

[0017] Preferably, the data processing and fusion analysis module includes:

[0018] A signal synchronization unit, configured to perform time alignment on the electroencephalogram data and eye movement data according to a unified time stamp or a trigger signal;

[0019] A preprocessing unit, configured to filter and remove artifacts from the electroencephalogram data, and smooth and calibrate the eye movement data;

[0020] A feature extraction unit, configured to extract frequency domain features, event-related potential features, and brain network connection features from the electroencephalogram data, and extract fixation distribution, gaze sequence, and pupil change features from the eye movement data;

[0021] A multimodal fusion unit, configured to input the extracted electroencephalogram features and eye movement features into a multimodal data fusion algorithm model, establish an association between the two, and output a fused feature representation;

[0022] A decision classification unit, configured to perform pattern recognition or classification on the fused feature representation to generate a depression risk assessment result.

[0023] Preferably, the multimodal fusion unit constructs an association graph containing EEG feature nodes and eye movement feature nodes, runs a graph neural network model on this graph to extract the association pattern between EEG and eye movement data, and the multimodal fusion unit uses the following weighted fusion formula to combine the EEG feature vector HE, the eye movement feature vector HO, and their element-wise product, thereby constructing a fusion feature:

[0024]

[0025] Among them, represents the feature vector extracted from the EEG signal, containing n feature components, represents the feature vector extracted from the eye movement data, containing m feature components, represents the element-wise product of the vectors, that is, multiplying the corresponding elements in HE and HO to generate a vector with a length of min(n,m), representing the interaction between EEG and eye movement features. α, β, and γ are weight coefficients, and their values are in the range of (0,1), which are used to adjust the contribution ratios of EEG unimodal, eye movement unimodal, and their interaction terms to the final fusion feature.

[0026] Preferably, the EEG signal acquisition module is a wearable wireless device, including multiple dry electrode sensors and signal amplifiers, and can transmit EEG data to the data processing and fusion analysis module in real time via Bluetooth;

[0027] The eye movement tracking module includes a high-frame-rate infrared eye tracker or an eye movement sensor integrated in a head-mounted display, and ensures millisecond-level alignment of eye movement data and EEG data through a synchronous triggering mechanism.

[0028] An intelligent method for early screening of depression by EEG-eye movement multimodal data fusion, applied to an intelligent system for early screening of depression by EEG-eye movement multimodal data fusion as described above, includes the following steps:

[0029] Execute an eye movement calibration program through the EEG signal acquisition module and the eye movement tracking module, and record the baseline EEG signal and eye movement signal in the resting state;

[0030] Show a predetermined cognitive task sequence to the subject through the cognitive task presentation module. The cognitive task includes multiple links such as emotional stimulus browsing and cognitive function testing, and key events are marked in real time;

[0031] Synchronously collect the EEG signal and the eye movement signal when the subject is performing the cognitive task, and synchronize the two types of data according to the event markers;

[0032] Filter and denoise the collected EEG signals and process artifacts, smooth filter the eye movement signals and detect fixation events to obtain high-quality EEG data and eye movement data sequences;

[0033] Extract multiple EEG feature indexes representing the subject's neural activities from the preprocessed EEG data, and extract multiple eye movement feature indexes representing the subject's fixation behavior and pupil response from the eye movement data;

[0034] Input the EEG features and eye movement features into a multi-modal fusion analysis model, fuse the information of the two modalities to identify depression-related patterns, and output the depression risk assessment results of the subject;

[0035] Present the discrimination results and relevant information to the user through the result output module or store them in the database, and provide subsequent processing suggestions when needed.

[0036] Preferably, the multi-modal fusion analysis includes: constructing an association map between EEG features and eye movement features, calculating edge weights according to the synchronous activities of EEG channels and regions of interest of the line of sight, and using a graph neural network to extract cross-modal feature patterns; or combining EEG features, eye movement features and their interaction terms through a formula fusion method to form a fusion feature vector, and inputting the fusion feature into a pre-trained machine learning classifier for depression state discrimination.

[0037] Preferably, the cognitive task presentation module uses virtual reality technology to present the cognitive tasks, each task scenario is displayed in an immersive three-dimensional environment, and the subject's perspective and operation behaviors in the virtual environment are recorded; improve the task-induced effect through rich multi-sensory stimuli, so as to amplify the response differences in EEG and eye movement of mildly depressed subjects.

[0038] Preferably, the extracted EEG features include at least one of the following: power spectral intensity in different frequency bands, amplitude and latency of event-related potentials, functional connectivity network indexes;

[0039] The extracted eye movement features include at least one of the following: the proportion of fixation time on negative stimuli, the transfer frequency of fixation points, the maximum dilation amplitude of the pupil in response to stimuli; comprehensively characterize the subject's mental state through the combination of multi-dimensional features.

[0040] Beneficial effects: An intelligent system and method for early screening of depression by multi-modal data fusion of EEG and eye movement provided by the present invention has the following beneficial effects:

[0041] 1. The present invention realizes highly sensitive detection of mild depression symptoms through the deep fusion of EEG and eye movement modality signals and the design of a new cognitive task stimulus. By using the high temporal resolution of EEG to capture subtle electroactivity changes in various brain regions, and simultaneously recording fixation behavior, gaze duration, and pupil dynamic changes through eye movement tracking, the system can extract multi-dimensional features such as specific frequency band power changes, ERP components, and abnormal fixation biases. The proposed fusion unit adopts a multi-modal weighted fusion formula, which introduces an element-wise product interaction term to explicitly model the coupling effect between EEG and eye movement features, thereby helping to capture the feature linkage effect of "EEG response when eye movement is abnormal" and significantly improving the accuracy and stability of the screening results.

[0042] 2. The present invention introduces virtual reality and brain-computer interface technologies in the system design to construct a highly immersive and multi-sensory stimulated cognitive task paradigm. This paradigm can stimulate the emotional and cognitive responses of subjects under different cognitive loads by presenting emotional stimuli, performing working memory or attention tasks, and participating in decision-making feedback, etc., further amplifying the neural and behavioral abnormalities related to mild depression; the virtual reality environment not only improves the realism of the task and user participation, but also realizes real-time closed-loop monitoring of data acquisition and task feedback through the brain-computer interface, thus ensuring high-precision data acquisition and flexible regulation of system response, providing a strong technical guarantee for early diagnosis.

[0043] 3. The present invention uses a wireless wearable device and a high-frame-rate eye movement tracking instrument to form a portable and real-time online monitoring intelligent screening platform; through multi-modal data synchronous acquisition, signal preprocessing, feature extraction, and deep fusion analysis, the system can quickly and non-invasively screen for early depression in a wide range of people. At the same time, the fine-grained physiological and behavioral data obtained by this platform provide an objective basis for subsequent mental health big data analysis, clinical precision intervention, and self-regulation and rehabilitation training, promoting the wide application of intelligent mental health monitoring and intervention technologies in the field of public health. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art.

[0045] Figure 1 It is the overall block diagram of the system of the present invention;

[0046] Figure 2 It is the method flow chart of the present invention;

[0047] Figure 3 It is the flow chart of multi-modal signal preprocessing and feature extraction of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] The present invention will be more clearly and completely described below by way of a preferred embodiment in conjunction with the accompanying drawings, but the present invention is not limited to the scope of the described embodiments.

[0049] As Figure 1 shown, an intelligent system for early screening of depression by electroencephalogram (EEG)-eye movement multimodal data fusion includes:

[0050] An EEG signal acquisition module, configured to acquire the EEG signals of a subject during the execution of a cognitive task and output multi-channel EEG data;

[0051] The EEG signal acquisition module is used to acquire the EEG signals of the subject in real time through a head-mounted EEG device. The EEG device can adopt a wearable wireless headgear, and a plurality of electrode channels are arranged in relevant cortical regions of the brain to record the EEG activities generated during the cognitive task.

[0052] An eye movement tracking module, configured to synchronously capture the eye movement information of the subject and generate eye movement data, including the position of the fixation point, the duration of fixation, and the change in pupil diameter;

[0053] The eye movement tracking module is used to record the eye movement data such as the line-of-sight position and pupil change of the subject through an eye tracker or a camera. Eye movement tracking can adopt a high-precision infrared eye tracker or a tracking device integrated in a VR headset / smart glasses to ensure synchronous acquisition of eye movement information during the cognitive task.

[0054] A cognitive task presentation module, configured to provide the subject with pre-designed cognitive tasks and stimulus scenarios to induce cognitive processes and emotional responses related to depression, and the cognitive tasks include an emotional stimulus browsing and a cognitive function test session;

[0055] The cognitive task presentation module is used to present a customized cognitive task paradigm to stimulate specific cognitive processes and emotional responses. The task content is presented through a computing device screen or a VR virtual reality scene, and the task paradigm includes various stimulus scenarios and interactions, such as free browsing of emotional pictures, scenario memory / attention tasks, decision feedback tasks, etc., to comprehensively induce cognitive behavioral characteristics related to depression such as attentional bias, memory processing, and decision-making response. Preferably, virtual reality technology is used to create an immersive environment to improve task participation and ecological validity.

[0056] A data processing and fusion analysis module, configured to receive the EEG data and eye movement data, preprocess and extract features therefrom, and perform correlation modeling and comprehensive analysis on the EEG features and eye movement features by using a multimodal data fusion algorithm, and output the depression risk assessment result of the subject;

[0057] The data processing and fusion analysis module is used to synchronously process, extract features, and perform multi-modal fusion modeling on the collected EEG and eye movement data to achieve depression risk assessment. This module includes a signal preprocessing unit (such as filtering, artifact removal, signal synchronization), a feature extraction unit (extracting frequency domain, time domain, and brain network features from EEG, and features such as gaze duration, fixation sequence, and pupil diameter change from eye movement), and a fusion decision unit. The fusion decision unit, based on the new multi-modal data fusion algorithm architecture proposed in the present invention, deeply fuses and models EEG and eye movement features, mines the correlation patterns between them, and finally gives the depression risk assessment result of the subject through a classifier or prediction model.

[0058] Specifically, the signal synchronization unit is used to align the EEG data and eye movement data according to a unified timestamp or trigger signal;

[0059] The preprocessing unit is used to filter and remove artifacts from the EEG data, and smooth and calibrate the eye movement data;

[0060] The feature extraction unit is used to extract frequency domain features, event-related potential features, and brain network connection features from the EEG data, and fixation distribution, gaze sequence, and pupil change features from the eye movement data;

[0061] The multi-modal fusion unit is used to input the extracted EEG features and eye movement features into the multi-modal data fusion algorithm model, establish the correlation between them, and output the fused feature representation;

[0062] The decision classification unit is used to perform pattern recognition or classification on the fused feature representation to generate a depression risk assessment result.

[0063] The multi-modal fusion unit constructs an association graph containing EEG feature nodes and eye movement feature nodes, and runs a graph neural network model on this graph to extract the correlation pattern between EEG and eye movement data. And the multi-modal fusion unit uses the following weighted fusion formula to combine the EEG feature vector HE, the eye movement feature vector HO, and their element-wise product to construct the fused feature:

[0064]

[0065] Where, represents the feature vector extracted from the EEG signal, containing n feature components, represents the feature vector extracted from the eye movement data, containing m feature components, Denotes the element-wise product of vectors, that is, multiplying the corresponding elements in HE and HO to generate a vector of length min(n,m), representing the interaction between EEG and eye movement features. α, β, and γ are weight coefficients, whose values are in the range of (0,1), and are used to adjust the contribution ratios of EEG unimodal, eye movement unimodal, and their interaction terms to the final fused features.

[0066] The result output module is used to present or store the evaluation results in a human-readable form, including displaying the depression tendency score, discrimination conclusion, and corresponding explanatory information of the subject.

[0067] The result output and feedback module is used to output the screening results and provide a visual feedback or subsequent processing interface. The results can include a quantified depression risk index, a judgment on whether there is mild depression, and relevant feature prompts. The output form can be displayed on the mobile App interface, reported on the computer screen, or uploaded to the clinical system for doctors' reference.

[0068] As Figure 2 shown, the present invention also provides an intelligent method for early screening of depression by EEG-eye movement multimodal data fusion, which is applied to an intelligent system for early screening of depression by EEG-eye movement multimodal data fusion as described above, and includes the following steps:

[0069] S1. Through the EEG signal acquisition module and the eye movement tracking module, execute the eye movement calibration program and record the baseline EEG signal and eye movement signal in the resting state;

[0070] S2. Show a predetermined cognitive task sequence to the subject through the cognitive task presentation module. The cognitive task includes multiple links such as emotional stimulus browsing and cognitive function testing, and at the same time, key events are marked in real time;

[0071] Present a pre-designed cognitive task sequence to the subject in a computing device or VR environment. The task paradigm includes multiple stages, for example: emotional stimulus browsing (showing positive and negative pictures or faces to elicit attention bias and emotional responses), working memory or attention tasks (such as the N-Back test, Stroop color-word task, to elicit cognitive load and reaction time changes), decision-making and feedback tasks (such as a simple risk decision-making game, observing decision-making behaviors under the influence of emotions). Each stage is accompanied by standardized instructions, and event markers in the task scenario are continuously recorded.

[0072] S3. Synchronously collect the EEG signal and eye movement signal when the subject is performing the cognitive task, and synchronize the two kinds of data according to the event markers;

[0073] During the entire process of the subject performing the task, the EEG module and the eye movement module collect data synchronously. The time alignment of EEG and eye movement data is achieved through a unified timestamp or trigger signal to ensure the correspondence of the two modalities of data for each stimulus event. In this process, the present invention uses brain-computer interface technology to achieve efficient data transmission and real-time monitoring, ensuring the stable and reliable data collection.

[0074] S4. Filter and denoise the collected EEG signals and process artifacts, smooth-filter the eye movement signals and detect fixation events to obtain high-quality EEG data and eye movement data sequences;

[0075] Preprocess the original EEG and eye movement data respectively. EEG preprocessing includes filtering (such as 0.5 - 45Hz band-pass filtering to remove power frequency and myoelectric noise), eye movement artifact correction (such as removing the interference of blink artifacts on EEG), segmentation and baseline correction (dividing EEG segments according to task events and subtracting the baseline). Eye movement data preprocessing includes filtering and smoothing of line-of-sight coordinates, identifying and removing blink / fixation null segments, and calculating basic eye movement metrics (such as the number of fixations per second, average fixation duration, etc.).

[0076] S5. Extract multiple EEG feature metrics representing the subject's neural activities from the preprocessed EEG data, and extract multiple eye movement feature metrics representing the subject's fixation behavior and pupil response from the eye movement data;

[0077] Calculate the EEG features of the subject under various task conditions, such as conventional frequency domain features (power spectral intensity of each frequency band, such as the power of α wave, β wave, etc.), time domain features (such as the amplitude and latency of event-related potentials such as P300, N200, etc.), non-linear dynamics features (such as approximate entropy, Lyapunov exponent), and brain network features (calculating functional connectivity or synchrony metrics according to different brain region channels, such as coherence, Pearson correlation, etc.). These features comprehensively reflect the state differences in the brain's cognitive processing and emotional responses.

[0078] Calculate the eye movement behavior features corresponding to the task scenarios, such as fixation distribution features (the proportion of fixation time in different regions of interest, especially the difference in attention to emotional stimulus regions, used to measure negative or positive attention bias), fixation sequence and transfer pattern (using the scan path to measure attention flexibility, such as individuals with depression may have a single and slow fixation path), pupil diameter change (pupil response caused by emotion or cognitive load, used to measure emotional arousal and concentration), etc. When necessary, the multi-scale feature extraction method in remote sensing image analysis can be combined to perform multi-time scale analysis on the eye movement trajectory to capture information on both short instantaneous fixations and long-term trends.

[0079] Such as Figure 3The figure shows the flowchart of multi-modal signal preprocessing and feature extraction, which demonstrates the entire data preprocessing process starting from the acquisition of raw EEG and eye movement data, to signal filtering, artifact removal, data segmentation, and finally feature extraction. In the figure, the EEG signal and eye movement signal are processed separately, and then fused into a feature extraction node to further extract frequency domain features, time domain features, non-linear features, and behavioral indicators. This process provides clean and structured feature data for the subsequent multi-modal fusion analysis module and is the basic link of the entire system.

[0080] S6. Input the EEG features and eye movement features into the multi-modal fusion analysis model, fuse the information of the two modalities to identify depression-related patterns, and output the depression risk assessment results of the subject.

[0081] Input the extracted EEG features and eye movement features into the multi-modal fusion analysis model of the present invention for correlation modeling and depression state discrimination. Different from the simple feature splicing method, the present invention designs a new multi-modal fusion algorithm architecture, including the following innovative steps:

[0082] Feature normalization and dimensionality reduction: First, perform normalization processing (such as Z-score normalization) on the EEG and eye movement features respectively, and use dimensionality reduction techniques (such as principal component analysis PCA or autoencoder) to reduce redundancy and extract the key information subspace representation of each modality, denoted as H E (EEG feature vector) and H O (eye movement feature vector).

[0083] Construction of correlation graph: Refer to the methods of brain network analysis and remote sensing multi-source data fusion to establish a correlation graph model between EEG features and eye movement features. Specifically, take each channel / feature of EEG and each type of feature of eye movement as nodes, and establish edge connections according to their correlation degree within the same task event window. The edge weight can be calculated by the statistical correlation or mutual information between the two signals. For example, for the i-th dimension of EEG feature and the j-th dimension of eye movement feature, define the initial correlation weight: where E i (t) represents the value of the i-th component of the EEG feature at time t, O j (t) represents the value of the j-th component of the eye movement feature at time t, T is the total number of time frames of the task process, and ω(t) is the time weight function (used to highlight the contribution of specific event moments, and ω(t) = 1 can be taken to represent equal weights). The original correlation degree between the i and j features is obtained by accumulating the product of the two-modal signals at the same time through the above formula. Subsequently, w ij is normalized to obtain the correlation matrix W, which is used as the adjacency matrix of the multi-modal graph. This graph intuitively represents the coupling strength between EEG and eye movement features during the entire task.

[0084] Multi-modal Graph Neural Network Fusion: Input the above-mentioned associated graph into the graph neural network model for feature fusion and extraction. Through iterative propagation and aggregation of node neighborhood information, the graph neural network can extract deep cross-modal association patterns. For example, at each layer of the graph neural network, the feature representations of EEG nodes and eye movement nodes will be updated and influenced by each other according to the adjacency relationship. Denote the feature vector of node v in the k-th layer of the graph neural network as The update formula is:

[0085]

[0086] where N(v) represents the set of neighbor nodes of node v (determined by the associated graph), d v and d u are node degrees for normalization, W (k) and V (k) are trainable weight matrices, and σ(·) is a non-linear activation function. Through multi-layer propagation, the information of EEG and eye movement features cross-fuses in the graph structure, gradually forming a comprehensive representation that includes both EEG features and the influence of eye movement features.

[0087] Fusion Feature Representation and Discrimination: After several layers of graph neural networks, the high-level representation vectors of each node are finally obtained. One can choose to aggregate all node representations (such as taking the average or maximum value) to form a global fusion feature vector, or connect the graph network to a readout layer to obtain a direct task discrimination output. In the preferred implementation of the present invention, for the depression recognition task, we further input the fused EEG-eye movement comprehensive feature vector into a fully connected neural network or a support vector machine (SVM) classifier to output the prediction result. For example, the classifier outputs the discrimination category of whether the subject is mildly depressed, or a depression risk probability value.

[0088] In addition, to improve the robustness and interpretability of the model, a multi-evidence decision fusion mechanism is introduced in the algorithm architecture of the present invention: sub-models are trained separately for different types of stimulus tasks to obtain discrimination results (such as an emotion preference sub-model, a cognitive performance sub-model, etc.), and then these sub-results are fused through weighted voting or a secondary classifier to obtain the final comprehensive determination. This multi-evidence fusion method ensures the utilization of specific diagnostic information provided by each cognitive task, improving the overall recognition accuracy.

[0089] S7. Present the discrimination result and related information to the user through the result output module or store it in the database, and provide subsequent processing suggestions when needed.

[0090] The data processing module sends the results of the analysis to the result output module. The system generates a screening report for the subject, including a quantitative depression tendency score, corresponding conclusions (such as "suspected mild depression" or "no obvious abnormalities found in the screening"), and abnormal behavior / physiological indicator prompts observed during the task. The report can be displayed to the user on the application interface or for further review by professionals. If combined with clinical use, the report can be uploaded to the medical database to assist doctors in making diagnostic decisions.

[0091] Based on the screening results, the system can provide follow-up suggestions or guide the user to the corresponding mental health services. For example, when a depressive tendency is detected, it is recommended that the user undergo professional consultation or further medical diagnosis. Since the system of the present invention has brain-computer interface capabilities, it can also be expanded to implement simple feedback functions, such as adjusting the intensity of stimulation in real time during the task (increasing the fun of the task based on the decrease in concentration monitored by EEG), or providing a relaxation training module to relieve the stress of the subject after screening. However, these are not core steps and can be implemented according to application requirements.

[0092] Example 1

[0093] The depression early screening system provided in this embodiment includes a wearable EEG EEG acquisition headgear, a desktop eye tracking camera, a virtual reality task presentation terminal, a back-end data processing and analysis server, and a client result display device. The subject wears an EEG EEG acquisition headgear, on which a number of electrodes are evenly arranged (for example, 32 leads are placed according to the international 10-20 system) for collecting EEG signals from various brain regions. The desktop eye tracking camera is fixed below the display screen to capture the pupil and corneal reflections of the subject's eyes in real time to calculate the sight point. The virtual reality task presentation terminal consists of a high-performance computer and a VR head-mounted display, which is used to present a three-dimensional virtual task environment and record user behavior. The back-end data processing and analysis server is connected to the EEG EEG acquisition headgear and the desktop eye tracking camera through a wireless network, receives the collected data stream and executes the method flow of the present invention. The final analysis results are sent to the client result display device (which can be a tablet or mobile phone, etc.) through the network, and are displayed to the user or doctor in the form of a report in a dedicated application App.

[0094] In this embodiment, the EEG headgear uses dry electrode technology to improve wearing comfort, and has a built-in amplifier and Bluetooth module to achieve wireless signal transmission; the desktop eye tracking camera is an infrared eye tracker with a sampling rate of 120Hz, which is placed about 60cm away from the user; the VR head-mounted display provides immersive task scenes, including multi-channel stimulation such as sound and tactile feedback. The overall architecture of the system ensures a good balance between synchronization of data collection and user experience, and can be applied to scenes such as hospital outpatient clinics and school psychology laboratories to screen and evaluate subjects for mild depression.

[0095] Example 2

[0096] This example details the process of the early screening method for depression of the present invention. The steps are as follows.

[0097] Step S1: The subject connects the device and calibrates. The subject first wears an EEG cap and sits in front of a desktop eye-tracking camera (or wears a VR headset to replace the camera, depending on the implementation environment). Start the system software, check the connection of each EEG channel lead, and adjust the electrode impedance to an appropriate range (such as <10 kΩ). Start the eye-tracking calibration program, require the subject to fixate on the calibration points that appear on the screen in sequence, and calculate the accurate eye movement mapping parameters to ensure the accuracy of subsequent eye movement data. After calibration, record the EEG in the resting state for 1 minute and the eye movement data in the natural fixation state as the individual baseline.

[0098] Step S2: Presentation of the task paradigm. After calibration, the system enters the formal test. According to the designed cognitive task paradigm, the system gradually presents various task stimuli:

[0099] First, perform the free viewing task of emotional pictures: Display multiple face pictures with different emotions (positive / neutral / negative) simultaneously in a VR scene or on the screen. Instruct the subject to freely view for a certain time (such as 30 seconds). During this process, the system records the fixation order and duration distribution. It is expected that depressed individuals may be more inclined to fixate on negative expressions or pay less attention to positive expressions, thus reflecting negative attentional bias.

[0100] Then, perform the Stroop color-word test (or a similar attention control task): Continuously present several color words, where the meaning of the word and the font color are either consistent or inconsistent. Require the subject to ignore the word meaning and say the font color. This stage examines the cognitive control ability and reaction time. Depressed individuals may show slower reactions and higher error rates, and at the same time, the amplitude of the P300 wave in the prefrontal region of the EEG may decrease, etc.

[0101] Next, perform a working memory task (such as N-Back): Randomly present letters or numbers in a sequence, and the subject needs to judge whether the current item is the same as the item N steps before. This stage induces memory load, and observes the change in the power of the theta wave in the EEG and the degree of eye movement concentration (such as whether the subject frequently gets distracted and shifts the gaze) of depressed individuals under high load.

[0102] Finally, perform a decision-making feedback task: For example, play a simple virtual money reward and punishment game, and the subject makes a choice among several options and gets immediate reward or punishment feedback. This stage aims to elicit the response differences of depressed individuals in reward processing (such as lack of pleasant response to rewards, weakening of reward-related positive waves in EEG, etc.).

[0103] The entire task process is automatically controlled and advanced by the system, and short breaks can be set between stages. Trigger signals are sent for key events (stimulus appearance, subject response) to be recorded in the EEG, so as to achieve precise alignment between the EEG and the task scenario. In the VR environment, the operations of the subject (such as triggering a selection when gazing at an object for more than a threshold time) are also recorded to replace the traditional button press response.

[0104] Step S3: Synchronous acquisition of multimodal data. During Step S2, the EEG cap and the desktop eye-tracking camera are continuously working. Through the system software, a unified timeline is implemented for the EEG data stream and the eye movement data stream to ensure the alignment of the two types of data on the same event basis. For example, when a picture appears, the eye movement records the area where the subject first gazes and the pupil changes, and the EEG records the visual evoked potential, and this information is associated through a common event label. In the case of using a VR headset, the eye tracking is completed by the infrared sensor inside the headset, and the EEG is collected by the cap, and the two ensure data time synchronization through the built-in synchronization module of the VR system. During the entire acquisition process, BCI technology is used to monitor the data quality: if electrode detachment or signal saturation is detected, the system will prompt for adjustment in a timely manner; if the subject closes their eyes for too long or is distracted (judged by the lack of focus in eye movement and a significant increase in EEG alpha waves), the system can pause the task to ensure the data validity.

[0105] Step S4: Data preprocessing and feature extraction. After the task is completed, the backend data processing and analysis server preprocesses and extracts features from the acquired raw data. In this embodiment, MATLAB and Python mixed programming is used to implement signal processing. Among them, for EEG preprocessing, band-pass filtering (1 - 40 Hz) and independent component analysis (ICA) are used to remove electrooculogram artifacts, and the eye movement data identifies the fixation and saccade sequences through a custom algorithm. In terms of feature extraction, some task-specific features are additionally added: for example, in the emotion browsing task, behavioral metrics such as the total gaze time of the subject on each type of emotional face and the first fixation latency are calculated; in the decision-making task, the amplitude of the EEG reward positive component (such as a positive wave with a latency of 200 - 300 ms) after the appearance of the reward stimulus is extracted. These features, together with conventional frequency features, etc., constitute the input for subsequent analysis.

[0106] Step S5: Multimodal fusion analysis and discrimination. The system performs fusion analysis on the extracted EEG and eye movement features. In this embodiment, we select a weighted interaction fusion method to construct the fusion feature Η fusion . Specifically, first, the EEG feature vector H E is set as a vector with dimension n = 50 (obtained by compressing through an autoencoder to a 50-dimensional representation), and the eye movement feature vector H OSet a vector with \(m = 20\), and then extend the shorter vector to the same length (in this example, pad the eye movement vector with zeros to 50 dimensions for element-wise multiplication with the EEG vector). Take the initial weights \(\alpha=\beta = 0.5\), \(\gamma=0.5\), and substitute them into the formula to calculate the fused features. The obtained \(\mathbf{H}\) fusion (50 - dimensional) is then input into a two - layer fully connected neural network classifier. The hidden layer uses ReLU activation, and the output layer is a Softmax binary classification (depression / non - depression). The classifier model has been pre - trained with a large amount of existing labeled data and has good performance. In actual screening, the classifier gives a prediction score for \(\mathbf{H}\) fusion For example, if the output is \([0.82, 0.18]\), it means the model believes the probability of the "depression" category is 82%. Based on this, it is judged that the subject may be at high risk of mild depression. In addition to the neural network, this embodiment also tried a support vector machine (SVM) classifier. By comparison, it was found that the accuracies of the two are similar, but the neural network has better scalability.

[0107] It should be noted that the weights \(\alpha\), \(\beta\), and \(\gamma\) can be automatically adjusted to the optimal values during the model training process. In our offline training, finally, \(\alpha\approx0.4\), \(\beta\approx0.3\), \(\gamma\approx0.8\) are learned, which means the interaction term has a relatively large weight in the decision - making, verifying the importance of the EEG - eye movement feature interaction for depression recognition. In addition, if the graph neural network fusion path is used instead, only the pre - processed features need to be imported into the constructed heterogeneous graph, and the GNN model described above can be used for training and prediction. Both schemes fall within the protection scope of the present invention.

[0108] Step S6: Result output and explanation. Once the discrimination model obtains the result, the system generates a screening report. In this embodiment, the result report of a certain subject shows that his depression risk score is 0.82 (above the threshold of 0.5). Combining with his task performance characteristics, the system judges that "there may be mild depression". The report details some objective indicators supporting this conclusion: for example, "the gaze time ratio of the subject to negative expressions in the emotional face browsing task is as high as 45% (the normal average is about 30%), showing an obvious negative attention bias; at the same time, the EEG alpha wave asymmetry index is 0.2 (it should be negative normally), indicating relatively less activity in the left frontal lobe, which is consistent with the depression pattern". These interpretable feedbacks help users or clinicians understand the screening results. For individuals screened as positive, the system will recommend that they seek professional diagnosis as soon as possible and can provide the contact information of local psychological counseling institutions. In subsequent services, if the user agrees, the system can anonymously upload the multi - modal raw data and results of this screening to the cloud database to accumulate data for the big data research on the objective diagnosis of depression.

[0109] The above embodiments were mainly tested and verified around the college student group. The results show that the discrimination accuracy rate of the system of the present invention for students with mild depression determined by psychological questionnaire screening reaches more than 90%, and the specificity for non-depressed healthy controls also exceeds 85%. The entire screening process of the system takes about 20 minutes, and the user compliance is good.

[0110] The above is only the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. An intelligent system for early screening of depression based on EEG-eye movement multimodal data fusion, characterized in that: include: The EEG signal acquisition module is used to collect the EEG signals of the subjects while they are performing cognitive tasks, and output multi-channel EEG data; An eye tracking module, used to synchronously capture the subject's eye movement information and generate eye movement data, including gaze point position, gaze duration, and pupil diameter changes; A cognitive task presentation module, used to provide the subject with pre-designed cognitive tasks and stimulation scenarios to induce cognitive processes and emotional responses related to depression, wherein the cognitive tasks include emotional stimulation browsing and cognitive function testing sessions; A data processing and fusion analysis module is used to receive the EEG data and eye movement data, perform preprocessing and feature extraction on them, and use a multimodal data fusion algorithm to perform correlation modeling and comprehensive analysis on the EEG features and eye movement features, and output the depression risk assessment results of the subject; The result output module is used to present or store the evaluation results in a human- and machine-readable form, including displaying the subject's depression tendency score, judgment conclusion and corresponding explanation information.

2. According to claim 1, the intelligent system for early screening of depression based on EEG-eye movement multimodal data fusion is characterized in that: The cognitive task presentation module includes a virtual reality display device, and the cognitive task is performed in a virtual reality scene to simulate a real scene and enhance the immersion of the cognitive task to the subject; The cognitive task paradigm includes: presenting positive and negative emotional stimuli simultaneously to detect the subjects' attention bias, measuring their behavioral and physiological responses in high cognitive load tasks, and observing their emotional responses in reward decision-making scenarios, in order to stimulate specific response patterns in mildly depressed subjects from multiple angles.

3. The intelligent system for early screening of depression based on EEG-eye movement multimodal data fusion according to claim 1 is characterized in that: The data processing and fusion analysis module includes: A signal synchronization unit, used to time-align the EEG data and the eye movement data according to a unified timestamp or trigger signal; A preprocessing unit, used to filter and remove artifacts from EEG data, and to smooth and calibrate eye movement data; A feature extraction unit, used to extract frequency domain features, event-related potential features, and brain network connection features from EEG data, and to extract gaze distribution, gaze sequence, and pupil change features from eye movement data; A multimodal fusion unit, used to input the extracted EEG features and eye movement features into the multimodal data fusion algorithm model, establish a correlation between the two and output a fusion feature representation; The decision classification unit is used to perform pattern recognition or classification on the fused feature representation to generate a depression risk assessment result.

4. The intelligent system for early screening of depression based on EEG-eye movement multimodal data fusion according to claim 3 is characterized in that: The multimodal fusion unit constructs an association graph including EEG feature nodes and eye movement feature nodes, and runs a graph neural network model on the graph to extract the association pattern between EEG and eye movement data. The multimodal fusion unit uses the following weighted fusion formula to transform the EEG feature vector H E and the eye movement feature vector H O and its element-by-element product to construct the fusion feature: in, Represents the feature vector extracted from the EEG signal, containing n feature components, Represents the feature vector extracted from the eye movement data, containing m feature components, represents the element-by-element product of vectors, that is, H E and H O The corresponding elements in are multiplied to generate a vector of length min(n,m), which represents the interaction between EEG and eye movement features. α, β and γ are weight coefficients, which are in the range of (0,1) and are used to adjust the contribution ratio of EEG single modality, eye movement single modality and the interaction between the two to the final fusion features.

5. The intelligent system for early screening of depression based on EEG-eye movement multimodal data fusion according to claim 1 is characterized in that: The EEG signal acquisition module is a wearable wireless device, comprising a plurality of dry electrode sensors and a signal amplifier, and can transmit EEG data to the data processing and fusion analysis module in real time via Bluetooth; The eye tracking module includes a high frame rate infrared eye tracker or an eye sensor integrated in a head-mounted display, and ensures millisecond-level alignment of eye movement data and EEG data through a synchronous trigger mechanism.

6. An intelligent method for early screening of depression based on EEG-eye movement multimodal data fusion, characterized in that: An intelligent system for early screening of depression using EEG-eye movement multimodal data fusion as described in any one of claims 1 to 5 comprises the following steps: Through the EEG signal acquisition module and the eye tracking module, the eye movement calibration procedure is performed and the baseline EEG signal and eye movement signal in the resting state are recorded; The cognitive task presentation module presents a predetermined cognitive task sequence to the subject, wherein the cognitive task includes multiple steps of emotional stimulus browsing and cognitive function testing, and key events are marked in real time; When the subjects perform cognitive tasks, EEG signals and eye movement signals are collected synchronously, and the two types of data are synchronized in time according to event markers; Filter and remove noise and artifacts from the collected EEG signals, perform smoothing filtering and gaze event detection on the eye movement signals, and obtain high-quality EEG data and eye movement data sequences; Extracting multiple EEG feature indices representing the subject's neural activity from the preprocessed EEG data, and extracting multiple eye movement feature indices representing the subject's gaze behavior and pupil response from the eye movement data; Input EEG features and eye movement features into a multimodal fusion analysis model, fuse the information of the two modalities to identify depression-related patterns, and output the subject's depression risk assessment results; The identification results and related information are presented to the user through the result output module or stored in the database, and subsequent processing suggestions are provided when necessary.

7. The intelligent method for early screening of depression based on EEG-eye movement multimodal data fusion according to claim 6 is characterized in that: Multimodal fusion analysis includes: constructing a correlation map between EEG features and eye movement features, calculating edge weights based on the synchronous activities of EEG channels and visual interest areas, and extracting cross-modal feature patterns using graph neural networks; or combining EEG features with eye movement features and their interaction terms through a formula fusion method to form a fused feature vector, and inputting the fused feature into a pre-trained machine learning classifier for depressive state discrimination.

8. The intelligent method for early screening of depression by fusion of EEG-eye movement multimodal data according to claim 6 is characterized in that: The cognitive task presentation module uses virtual reality technology to present the cognitive tasks. Each task scene is displayed in an immersive three-dimensional environment, and the subject's perspective and operating behavior in the virtual environment are recorded. The task induction effect is enhanced through rich multi-sensory stimulation, thereby amplifying the differences in EEG and eye movement responses of mildly depressed subjects.

9. The intelligent method for early screening of depression by fusion of EEG-eye movement multimodal data according to claim 6 is characterized in that: The extracted EEG features include at least one of the following: power spectrum intensity in different frequency bands, amplitude and latency of event-related potentials, and functional connectivity network indicators; The extracted eye movement features include at least one of the following: the proportion of fixation time on negative stimuli, the frequency of gaze point shifting, and the maximum dilation amplitude of the pupil to the stimulus; The psychological state of the subject can be more comprehensively characterized by combining multidimensional features.

Images