Multi-modal feature combined depression auxiliary diagnosis system

Abstract

The invention discloses a multi-modal feature combined depression auxiliary diagnosis system. The system comprises a sampling unit which is used for constructing a multi-modal depression data set by acquiring a depression screening scale, an electroencephalogram, a magnetoencephalogram and functional magnetic resonance imaging based on acquisition equipment; the feature extraction unit is used for extracting multi-modal brain features based on the depression data set, and the multi-modal brain features comprise power spectral density obtained by electroencephalogram signals, event-related potential, micro-state, prefrontal lobe gamma frequency band power spectral density obtained by magnetoencephalogram and event-related magnetic field; gray matter volume and resting state functional connection density are obtained through functional magnetic resonance imaging; a data preprocessing unit; the diagnosis model unit is used for constructing a multi-modal depression diagnosis model and training the model on the basis of the multi-modal brain features in combination with a fusion strategy; and an analysis and prediction unit. The extracted features are comprehensive and reasonable, the defect of each mode is overcome by the feature fusion method, and the fused features are advanced.

Description

Technical Field

[0001] The present invention belongs to the field of auxiliary diagnosis of depression, and in particular relates to an auxiliary diagnosis system for depression combined with multimodal features. Background Art

[0002] Single-modality technology is prone to information bias due to data limitations. In the process of disease diagnosis, if data from different modalities can be integrated and complemented, it can effectively reduce information bias and improve accuracy. Data from a single modality collected under non-laboratory conditions (such as clinical settings) is prone to missing or noisy data.

[0003] Depression is a heterogeneous disease, and this heterogeneity can present diagnostic challenges. Recent years have seen a deepening understanding of depression, leading to more specific and specific features that characterize depression. This has significantly helped overcome this heterogeneity. Understanding the patient's pathological changes during the diagnostic process is also crucial for medical professionals to provide targeted treatment.

[0004] CN118888154 discloses a multimodal depression recognition system based on multi-level feature fusion. The fused modalities include text, speech, and vision. Its diagnostic steps include: 1. Acquisition; 2. Extracting the temporal and spatial dimensions of multiple modalities and the high-level depression features and fusing them to obtain spatiotemporal fused depression features; 3. Inputting the spatiotemporal fused depression features into a prediction model to obtain prediction results. First, the patent collects data such as sentences, images, videos, and sounds. While it can capture the described depressive emotional features at different levels and modalities and use them to identify depression, due to personality and cultural factors, different individuals can easily exhibit different emotional characteristics, which can affect the model's generalization ability. Furthermore, conditions such as bipolar disorder and anxiety disorders can sometimes present similar symptoms, making misdiagnosis prone. Second, while this patent can identify depression, the model's complexity prevents it from providing appropriate explanations to doctors. Summary of the Invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a multimodal feature-based auxiliary diagnosis system for depression, improve diagnostic accuracy, quantify the diagnostic process, and provide a diagnostic basis.

[0006] Technical solution: The present invention provides a multimodal feature-based auxiliary diagnosis system for depression, comprising:

[0007] Sampling unit: The same as the acquisition device based on the depression screening scale, electroencephalogram (EEG), magnetoencephalogram (MEG) and functional magnetic resonance imaging (fMRI) to construct a multimodal depression dataset;

[0008] Feature extraction unit: used to extract multimodal brain features based on the depression dataset, including power spectral density (PSD) of delta, theta, alpha, beta, and gamma frequency bands obtained from electroencephalography, event-related potential (ERP), microstates, prefrontal gamma frequency band power spectral density and event-related magnetic field (ERF) obtained from magnetoencephalography, gray matter volume and resting-state functional connectivity density obtained from functional magnetic resonance imaging;

[0009] Data preprocessing unit: used to preprocess the extracted multimodal brain features;

[0010] Diagnostic model unit: used to build and train a multimodal depression diagnosis model based on multimodal brain features and combined with fusion strategies;

[0011] Analysis and prediction unit: used to obtain analysis and prediction of depression based on the collected multimodal data and diagnostic model unit.

[0012] Furthermore, the sampling unit includes:

[0013] Grouping unit: used to divide patients into patient group or healthy group according to Hamilton Depression Rating Scale;

[0014] Experimental acquisition unit: includes resting state and task state. The resting state requires the subjects to maintain 10-15 minutes of closed-eye meditation with their heads still. The resting state experiment is conducted twice, recording the subjects' EEG, MEG data and functional magnetic resonance imaging data respectively; the task state is to conduct a set of emotion induction experiments and synchronously collect EEG and MEG signals.

[0015] Furthermore, in the feature extraction unit, the power spectral density is extracted as follows: for each test signal, band-pass filtering is performed separately to obtain signals in four frequency bands of delta, theta, alpha, and beta under a single channel; the power spectral density under each frequency band is estimated using the Welch method to obtain the power spectral density of all frequency bands of all channels, and the power spectral densities of all channels are averaged to obtain four average power spectral densities.

[0016] Furthermore, in the feature extraction unit, the power spectral density of the gamma frequency band of the frontal lobe obtained by the magnetoencephalogram is specifically as follows: the signal of the frontal lobe region is extracted based on the automatic anatomical labeling atlas AAL; the gamma signal in the region of interest (ROI) is extracted using a bandpass filtering method; the power spectral density of the regional channel is calculated using the Welch method; and the calculated power spectral density is averaged.

[0017] Furthermore, in the feature extraction unit, the extraction of the microstate is specifically as follows: for each subject, the global field power GFP is calculated, and the calculation formula of the global field power is:

[0018]

[0019] Among them, V i (t) is the voltage value of the i-th electrode at time t, is the average voltage of all electrodes;

[0020] The K-means algorithm is used to cluster the topographic map of GFP peak points, identify a specified number of microstate categories, and back-fit to the original curve. The goal of the K-means algorithm is to minimize the sum of the squares of the distances from all data points to the center of their cluster. The objective function is:

[0021]

[0022] Among them, ||x i -μ j || 2 is the data point x i and cluster center μ j The square of the Euclidean distance; r ij is a binary indicator function that ensures that each data point belongs to only one cluster.

[0023] Furthermore, in the feature extraction unit, the extraction of the resting-state functional connectivity density is specifically as follows: screening the region of interest, selecting the limbic system network, frontoparietal control network, and default mode network in the Yeo-7 network; calculating the intra-network connectivity density, that is, the average connectivity density of all voxel pairs in the network, first extracting the time series of the voxels in the network, calculating the Pearson correlation coefficient between all voxels in the network, setting the correlation threshold to r = 0.2, retaining connections above 0.2, obtaining a correlation coefficient matrix, and calculating the functional connectivity density. The calculation formula of the Pearson correlation coefficient is as follows:

[0024]

[0025] Among them, Xi, Yi are the two variables of the i-th observation; is the mean of variables X and Y;

[0026] Then the correlation coefficient matrix is transformed by Fisher so that the result obeys the normal distribution. The formula of Fisher transformation is:

[0027]

[0028] Finally, the connection density between networks is calculated, and the correlation coefficient matrix between each pair of networks is calculated.

[0029] Furthermore, the data preprocessing unit includes:

[0030] EEG data preprocessing unit: Removes useless bad tracks; discards data if the number of bad tracks exceeds 5%; performs filtering, including bandpass filtering and notch filtering. Bandpass filtering limits the frequency to 0.25Hz to 40Hz, and notch filtering is set based on the power spectral density image to eliminate power frequency interference; re-references to the average reference; decides whether to resample based on the original sampling rate; performs independent component analysis (ICA) on the data to remove electrooculogram and electromyography artifacts;

[0031] MEG signal preprocessing unit: Removes useless bad tracks; discards data if the number of bad tracks exceeds 5%. Filters, including bandpass and notch filtering. Bandpass filtering limits the frequency to 0.25Hz to 100Hz. Notch filtering is set based on the power spectral density image to eliminate power frequency interference. Independent component analysis is performed on the data to remove electrooculogram and electromyography artifacts. Resampling is selected based on the original sampling rate.

[0032] Functional magnetic resonance imaging signal preprocessing unit: manually check abnormal data; use the MCFLIRT command in the FSL tool to correct head motion; use the FLIRT command in FSL to standardize to the standard space MNI; and smooth the image with a Gaussian kernel.

[0033] Furthermore, the diagnostic model unit includes:

[0034] The unit analyzes frequency band signals extracted from the EEG and MEG: Based on the power spectral density images of the delta, theta, alpha, beta, and gamma frequency bands of the individual subjects, combined with the subjects' gender, age, and actual depression status, two structured information sets are constructed: the first set is gender, age, and power spectral density; the second set is depression / non-depression. These two sets of data are input into a convolutional neural network (CNN), and training is performed to obtain the collective decision result r1, which is the probability of depression for all training sets.

[0035] Microstate analysis units extracted from magnetoencephalography: Based on the individual microstates of the subjects, combined with the subjects' gender, age, and actual depression status, two structured information sets were constructed. The first set contained gender, age, microstate category, microstate duration, microstate frequency, and microstate time coverage. The second set contained depression / non-depression data. These two sets of data were input into a support vector machine (SVM) and trained to obtain the collective decision result r2, which is the probability of depression for all training sets.

[0036] The unit analyzes event-related potential and event-related magnetic field data extracted from the electroencephalogram (EEG) and magnetoencephalogram (MEG). Based on the obtained ERPM and ERFM data, combined with the subject's gender, age, and actual depression status, two structured data sets are constructed. The first set includes gender, age, event category, ERPM, and ERFM; the second set includes depression / non-depression. These two sets of data are input into a convolutional neural network, and training is performed to obtain the collective decision result r3, which is the probability of depression for all training sets.

[0037] Functional connectivity density analysis unit extracted from functional magnetic resonance imaging: Based on functional connectivity density data, combined with the subject's gender, age, and actual depression status, two structured data sets were constructed. The first set included gender, age, intra-network functional connectivity density, and inter-network functional connectivity density; the second set included depression / non-depression. These two data sets were input into a support vector machine, and training was performed to obtain the collective decision result r4, which is the probability of depression for all training sets.

[0038] Gray matter volume analysis unit extracted from functional magnetic resonance imaging: Based on gray matter volume data, combined with the subject's gender, age, and actual depression status, two structured data sets were constructed. The first set included gender, age, gray matter type, and volume; the second set included depression / non-depression. These two data sets were input into a support vector machine, and training was performed to obtain the collective decision result r5, which is the probability of depression for all training sets.

[0039] Decision fusion unit: Based on the obtained r1, r2, r3, r4, and r5, two sets of structured data are constructed according to the gender, age, and actual depression status of each subject in the training set. The first set includes gender, age, r1, r2, r3, r4, and r5; the second set includes depression / non-depression. These two sets of data are input into the extreme gradient boosting tree XGBoost to train a multimodal diagnostic model.

[0040] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0041] (1) The features extracted by the present invention are comprehensive and reasonable. The feature fusion method can make up for the deficiencies of each modality, and the fused features are advanced. The model of the present invention has strong interpretability. Based on this method, doctors can understand the physiological state of the brain while making a diagnosis, thereby optimizing the treatment plan. The design of the present invention is modular, which is conducive to expansion and new modality data can be added in the future. The present invention takes into account the different importance of different features and improves the generalization ability of the model. The present invention has a high tolerance for missing values and can accept a certain degree of data missing in both the training and prediction stages.

[0042] (2) The features that the present invention focuses on are diverse. When designing the model, we also considered how to better help doctors and avoid some problems. First, we designed an emotion-induced experiment to collect event-related potentials and event-related magnetic fields, focusing on changes in emotions. At the same time, we also captured microstates, power spectral density, gray matter volume, functional connectivity density, etc. in the state without induced emotions, which reflect changes in brain function and structure. Second, we can directly provide characteristic information such as event-related potentials, event-related magnetic fields, microstates, power spectral density, gray matter volume, and functional connectivity density to doctors, so that doctors can intuitively see the pathological changes of patients and provide objective treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 To implement the steps of this system;

[0044] Figure 2 is the sampling unit, i.e., the step of collecting experimental data;

[0045] Figure 3 It is a set of brain topographic maps;

[0046] Figure 4 Schematic diagram of binary mask;

[0047] Figure 5 This is the anatomical section diagram of MNI152. DETAILED DESCRIPTION

[0048] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0049] Example 1: The process of collecting experimental data, such as Figure 2 As shown, the process is as follows

[0050] The subjects were recruited with the following requirements: first, the patient group was not allowed to have other mental illnesses or a history of taking psychotropic drugs, brain trauma, or drug abuse within six months; second, the healthy group needed to have no mental illness or neurological disease, no family history of mental illness, be between 20 and 45 years old, be right-handed, and match the age and education level of the patients.

[0051] Fill out the Hamilton Depression Rating Scale, and then an experienced physician will choose whether to classify the patient into the patient group or the healthy group;

[0052] The experiment was organized into a resting state and a task state. The resting state required the subjects to meditate with their eyes closed for ten minutes, during which they needed to keep their minds clear and sit upright, especially with their heads still. The resting state experiment was conducted twice, recording the subjects' EEG, MEG data, and functional magnetic resonance imaging data (the EEG and MEG data were collected simultaneously, that is, the subjects wore an EEG cap to collect EEG data while the MEG data was collected). The task state experiment involved a set of emotion induction experiments, during which the EEG and MEG data were collected simultaneously. The key points of the task state experiment are as follows:

[0053] From the Chinese Affective Picture System (CAPS), 20 pictures representing happiness, 20 pictures representing neutrality, and 20 pictures representing sadness were selected for the experiment;

[0054] Control the synchronous acquisition of EEG and MEG through photoelectric trigger signals;

[0055] These pictures were randomly presented in the experiment, and the presentation time was 2s;

[0056] The time interval between each presentation was random, ranging from 1 to 2 seconds;

[0057] Example 2: Extraction of power spectrum density, the process is as follows

[0058] For each test signal, band-pass filtering is performed to obtain signals in four frequency bands: delta, theta, alpha, and beta.

[0059] Use the Welch method to estimate the power spectral density in each frequency band;

[0060] Obtain the power spectral density of all frequency bands of all channels according to steps 1 and 2;

[0061] Average the power spectral densities of all channels to obtain four average power spectral densities;

[0062] Example 3: Gamma activity extraction, taking the frontal lobe as an example

[0063] Extracting signals from the prefrontal region based on an automated anatomical labeling atlas;

[0064] The gamma signal in the region of interest is extracted using the bandpass filtering method;

[0065] The power spectral density of the channels in these regions is calculated using the Welch method;

[0066] averaging the calculated power spectral densities;

[0067] Example 4: Extraction of microstates, the process is as follows

[0068] For each subject, calculate the global field power. The formula for calculating the global field power is

[0069]

[0070] Where Vi(t) is the voltage value of the i-th electrode at time t, is the average voltage of all electrodes

[0071] Use the K-means algorithm to identify the topographic map of GFP peak points and perform clustering. Identify a specified number of microstate categories and back-fit them to the original curve. For example, set it to 4 categories. The goal of the K-means algorithm is to minimize the sum of the squares of the distances from all data points to the center of their clusters. The objective function is:

[0072]

[0073] Among them, ||x i -μ j || 2 is the data point x i and cluster center μ j The square of the Euclidean distance.

[0074] r ij is a binary indicator function that ensures that each data point belongs to only one cluster.

[0075] Figure 3 A set of brain topography maps showing four different microstates.

[0076] Example 5: Extraction of event-related potentials (the extraction process of event-related magnetic fields is similar), the process is as follows

[0077] Baseline correction is performed for each channel of each trial. The purpose of baseline correction is to reduce data offset and noise and ensure the stability of data in different trials.

[0078] Define a time window, for example, 200ms before the event to 800ms after the event.

[0079] According to different types of events, the EEG signals within the average time window are superimposed, such as positive stimulus event-related potential, neutral stimulus event-related potential, and negative stimulus event-related potential;

[0080] Example 6: Screening regions of interest, including using the AAL3 template to load regions related to depression, such as the cingulate gyrus, amygdala, hippocampus, etc. The process is as follows

[0081] Load AAL3 template;

[0082] Query the number of the area to be filtered in the AAL3 template;

[0083] According to the number, load the corresponding mask and use the binary mask. Explanation of the binary mask:

[0084] In a binary mask, the values 1 and 0 are defined for each position, indicating "included" and "not included". After applying it to the image, the masked image is obtained, which can be expressed as follows;

[0085] I ROI (x)=I(x)·M binary (x)

[0086] Among them, when x is in the target area, M binary (x)=1, otherwise M binary (x) = 0

[0087] Save and apply the mask, Figure 4 This is a binary mask diagram generated with the help of Python's nibabel toolkit, showing the area where the amygdala is located;

[0088] Example 7: Extraction of functional connectivity density. This method uses the Yeo-7 network template, and the process is as follows:

[0089] Screening the region of interest, the process is shown in Example 6;

[0090] Select the limbic system network (Lymbic_Area), frontoparietal control network (Frontoparietal_Area), and default mode network (Default_Network_Area) in Yeo-7;

[0091] Calculate the intra-network connection density, that is, the average connection density of all voxel pairs in the network. The specific steps are:

[0092] Extracting time series for voxels within the network;

[0093] Calculate the Pearson correlation coefficient between all voxels in the network, set the correlation threshold to r = 0.2, and retain connections above 0.2. You can get a correlation coefficient matrix and calculate the functional connection density. The calculation formula of the Pearson correlation coefficient is as follows

[0094]

[0095] Among them, Xi, Yi are the two variables of the i-th observation value, is the mean of variables X and Y.

[0096] Perform Fisher transformation on the correlation coefficient matrix to make the result obey the normal distribution. The formula of Fisher transformation is

[0097]

[0098] Calculate the inter-network connection density. The specific steps are similar to those within the network, but the correlation coefficient matrix between each pair of networks is calculated.

[0099] Example 8: Extraction of brain gray matter volume. In this process, tools such as FSL and ANTs can be used. Taking FSL as an example, the process is as follows:

[0100] Use the Bet command in FSL tools for skull segmentation and SimpleITK tools for bias field correction;

[0101] Gray matter segmentation was performed using FSL's FAST command;

[0102] Calculate gray matter volume;

[0103] Example 9: EEG preprocessing, the process is as follows

[0104] Eliminate useless bad sectors. If the number of bad sectors is greater than 5%, the data will be discarded.

[0105] Filtering, including bandpass filtering and notch filtering. Bandpass filtering limits the frequency to 0.25Hz to 40Hz, and notch filtering is set according to the power spectrum density image to eliminate power frequency interference;

[0106] The re-reference is the average reference;

[0107] Determine whether to resample based on the original sampling rate. It is recommended to set it to 200Hz.

[0108] Perform independent component analysis on the data to remove electrooculogram and electromyography artifacts. Here, artifacts with a probability of more than 90% are removed to reduce the loss of key information;

[0109] Example 10: Magnetoencephalogram Preprocessing

[0110] Eliminate useless bad sectors. If the number of bad sectors is greater than 5%, the data will be discarded.

[0111] Filtering, including bandpass filtering and notch filtering. Bandpass filtering limits the frequency to 0.25Hz to 100Hz, and notch filtering is set according to the power spectrum density image to eliminate power frequency interference;

[0112] Perform independent component analysis on the data to remove electrooculogram and electromyography artifacts. Here, artifacts with a probability of more than 90% are removed to reduce the loss of key information;

[0113] According to the original sampling rate, choose whether to resample to ensure that the sampling rate is consistent with the resampled EEG;

[0114] Example 11, functional magnetic resonance imaging pretreatment

[0115] Manually check abnormal data, including image artifacts, signal anomalies, etc.

[0116] Use the MCFLIRT command in the FSL tool to perform head motion correction;

[0117] Use FSL's FLIRT command to normalize to the standard space, such as Figure 5 This is an anatomical slice diagram of a standard space MNI152;

[0118] Smooth the image using a Gaussian kernel;

[0119] Example 12: Analyzing frequency band signals extracted from electroencephalograms and magnetoencephalograms

[0120] Based on Example 1 and Example 2, the power spectral density images of the delta, theta, alpha, beta, and gamma frequency bands of the individual subject are obtained;

[0121] Combining the subject's gender, age and actual depression status, two structured information (gender, age, power spectrum density) and (depression / no depression) were constructed;

[0122] Repeat steps 1 and 2 for all objects in the training set to obtain two sets of structured data.

[0123] These two sets of data are input into the convolutional neural network and trained to obtain the collective decision result r1, which is the depression probability of all training sets;

[0124] Example 13: Analyzing Microstates Extracted from EEG

[0125] Based on Example 4, the microstate of the subject is obtained;

[0126] Combining the subjects' gender, age, and actual depression status, two structured data sets were constructed (gender, age, microstate category, microstate duration, microstate frequency, microstate time coverage), and (depressed / not depressed);

[0127] Repeat steps 1 and 2 for all objects in the training set to obtain two sets of structured data.

[0128] Input these two sets of data into the support vector machine and train to obtain the set decision result r2, which is the probability of depression for all training sets;

[0129] Example 14: Analyzing Event-Related Potentials and Event-Related Magnetic Field Data Extracted from EEG and MEG

[0130] Based on Example 5, event-related potential and event-related magnetic field data are obtained;

[0131] Combining the subjects' gender, age and actual depression status, two structured data (gender, age, event category, event-related potential, event-related magnetic field) and (depression / no depression) were constructed;

[0132] Repeat steps 1 and 2 for all objects in the training set to obtain two sets of structured data.

[0133] These two sets of data are input into the convolutional neural network and trained to obtain the set decision result r3, which is the depression probability of all training sets;

[0134] Example 15: Analysis of functional connectivity density extracted from functional magnetic resonance imaging

[0135] Based on Example 7, functional connectivity density data was obtained;

[0136] Combining the subjects' gender, age, and actual depression status, two structured data sets (gender, age, intra-network functional connectivity density, inter-network functional connectivity density) and (depressed / not depressed) were constructed.

[0137] Repeat steps 1 and 2 for all objects in the training set to obtain two sets of structured data.

[0138] Input these two sets of data into the support vector machine and train to obtain the set decision result r4, which is the depression probability of all training sets;

[0139] Example 16: Analysis of Gray Matter Volume Extracted from Functional Magnetic Resonance Imaging

[0140] Based on Example 8, gray matter volume data was obtained;

[0141] Combining the subjects' gender, age, and actual depression status, two structured data sets (gender, age, gray matter type, volume) and (depressed / not depressed) were constructed.

[0142] Repeat steps 1 and 2 for all objects in the training set to obtain two sets of structured data.

[0143] Input these two sets of data into the support vector machine and train to obtain the set decision result r5, which is the depression probability of all training sets;

[0144] Example 17, Decision Fusion

[0145] Based on Examples 12, 13, 14, 15, and 16, r1, r2, r3, r4, and r5 are obtained. If there are missing values in a set of data, they are marked with a specific value, such as -1;

[0146] According to the gender, age and actual depression status of each subject in the training set, two sets of structured data (gender, age, r1, r2, r3, r4, r5) and (depression / no depression) were constructed;

[0147] These two sets of data are fed into the Extreme Gradient Boosting Tree (XGBoost), which automatically handles missing values and trains a multimodal diagnostic model.

[0148] Overall steps for applying this method

[0149] Step 1. Collect experimental data, keeping the collection environment and equipment consistent as much as possible, and ensure that the patient is emotionally stable and has adequate sleep before collection. The process is the same as in Example 1;

[0150] Step 2. Preprocess the experimental data, as in Examples 9, 10, and 11;

[0151] Step 3. Extract features, the process is the same as in Examples 2, 3, 4, 5, 7, and 8;

[0152] Step 4. Analyze the features and obtain decisions such as r1, r2, r3, r4, and r5. The process is similar to Examples 12, 13, 14, 15, and 16.

[0153] Step 5. Fusion of decisions r1, r2, r3, r4, and r5, as in Example 17, to train a multimodal diagnostic model.

[0154] Step 6. Deploy the model to actual application scenarios and input real user data into the model.

[0155] Step 7. The user data is processed through the processes described in Steps 1, 2, 3, and 4 to obtain the required features.

[0156] Step 8. Input the features into the diagnostic model to obtain a probability of whether the user is depressed and output the feature information.

Claims

1. A multimodal feature-based auxiliary diagnosis system for depression, characterized by: The system comprises: Sampling unit: The same as the acquisition device based on the depression screening scale, electroencephalogram (EEG), magnetoencephalogram (MEG) and functional magnetic resonance imaging (fMRI) to construct a multimodal depression dataset; Feature extraction unit: used to extract multimodal brain features based on the depression dataset, including power spectral density (PSD) of delta, theta, alpha, beta, and gamma frequency bands obtained from electroencephalography, event-related potential (ERP), microstates, prefrontal gamma frequency band power spectral density and event-related magnetic field (ERF) obtained from magnetoencephalography, gray matter volume and resting-state functional connectivity density obtained from functional magnetic resonance imaging; Data preprocessing unit: used to preprocess the extracted multimodal brain features; Diagnostic model unit: used to build and train a multimodal depression diagnosis model based on multimodal brain features and combined with fusion strategies; Analysis and prediction unit: used to obtain analysis and prediction of depression based on the collected multimodal data and diagnostic model unit.

2. A multimodal feature-based depression auxiliary diagnosis system according to claim 1, characterized in that: The sampling unit comprises: Grouping unit: used to divide patients into patient group or healthy group according to Hamilton Depression Rating Scale; Experimental acquisition unit: includes resting state and task state. The resting state requires the subjects to maintain 10-15 minutes of closed-eye meditation with their heads still. The resting state experiment is conducted twice, recording the subjects' EEG, MEG data and functional magnetic resonance imaging data respectively; the task state is to conduct a set of emotion induction experiments and synchronously collect EEG and MEG signals.

3. The multimodal feature-based depression auxiliary diagnosis system according to claim 1, characterized in that: In the feature extraction unit, the power spectral density is extracted as follows: for each test signal, band-pass filtering is performed to obtain signals in four frequency bands of delta, theta, alpha, and beta under a single channel; the power spectral density under each frequency band is estimated using the Welch method to obtain the power spectral density of all frequency bands of all channels, and the power spectral densities of all channels are averaged to obtain four average power spectral densities.

4. The multimodal feature-based depression auxiliary diagnosis system according to claim 1, characterized in that: In the feature extraction unit, the power spectral density of the gamma frequency band of the frontal lobe obtained by magnetoencephalography is specifically as follows: the signal of the frontal lobe region is extracted based on the automatic anatomical labeling atlas AAL; the gamma signal in the region of interest (ROI) is extracted using a bandpass filtering method; the power spectral density of the regional channel is calculated using the Welch method; and the calculated power spectral density is averaged.

5. The depression auxiliary diagnosis system combined with multimodal features according to claim 1, characterized in that: In the feature extraction unit, the extraction of the microstate is specifically as follows: for each subject, the global field power GFP is calculated. The calculation formula of the global field power is: Among them, V i (t) is the voltage value of the i-th electrode at time t, is the average voltage of all electrodes; The K-means algorithm is used to cluster the topographic map of GFP peak points, identify a specified number of microstate categories, and back-fit to the original curve. The goal of the K-means algorithm is to minimize the sum of the squares of the distances from all data points to the center of their cluster. The objective function is: Among them, ||x i -μ j || 2 is the data point x i and cluster center μ j The square of the Euclidean distance; r ij is a binary indicator function that ensures that each data point belongs to only one cluster.

6. The multimodal feature-based depression auxiliary diagnosis system according to claim 1, characterized in that: In the feature extraction unit, the extraction of the resting-state functional connectivity density is specifically as follows: screening the region of interest, selecting the limbic system network, the frontoparietal control network, and the default mode network in the Yeo-7 network; calculating the intra-network connectivity density, that is, the average connectivity density of all voxel pairs in the network, first extracting the time series of the voxels in the network, calculating the Pearson correlation coefficient between all voxels in the network, setting the correlation threshold to r = 0.2, retaining connections above 0.2, obtaining a correlation coefficient matrix, and calculating the functional connectivity density. The calculation formula of the Pearson correlation coefficient is as follows: Among them, Xi, Yi are the two variables of the i-th observation; is the mean of variables X and Y; Then the correlation coefficient matrix is transformed by Fisher so that the result obeys the normal distribution. The formula of Fisher transformation is: Finally, the connection density between networks is calculated, and the correlation coefficient matrix between each pair of networks is calculated.

7. The multimodal feature-based depression auxiliary diagnosis system according to claim 1, characterized in that: The data preprocessing unit includes: EEG data preprocessing unit: Removes useless bad tracks; discards data if the number of bad tracks exceeds 5%; performs filtering, including bandpass filtering and notch filtering. Bandpass filtering limits the frequency to 0.25Hz to 40Hz, and notch filtering is set based on the power spectral density image to eliminate power frequency interference; re-references to the average reference; decides whether to resample based on the original sampling rate; performs independent component analysis (ICA) on the data to remove electrooculogram and electromyography artifacts; MEG signal preprocessing unit: Removes useless bad tracks; discards data if the number of bad tracks exceeds 5%. Filters, including bandpass and notch filtering. Bandpass filtering limits the frequency to 0.25Hz to 100Hz. Notch filtering is set based on the power spectral density image to eliminate power frequency interference. Independent component analysis is performed on the data to remove electrooculogram and electromyography artifacts. Resampling is selected based on the original sampling rate. Functional magnetic resonance imaging signal preprocessing unit: manually check abnormal data; use the MCFLIRT command in the FSL tool to correct head motion; use the FLIRT command in FSL to standardize to the standard space MNI; and smooth the image with a Gaussian kernel.

8. The multimodal feature-based depression auxiliary diagnosis system according to claim 1, characterized in that: The diagnostic model unit comprises: The unit analyzes frequency band signals extracted from the EEG and MEG: Based on the power spectral density images of the delta, theta, alpha, beta, and gamma frequency bands of the individual subjects, combined with the subjects' gender, age, and actual depression status, two structured information sets are constructed: the first set is gender, age, and power spectral density; the second set is depression / non-depression. These two sets of data are input into a convolutional neural network (CNN), and training is performed to obtain the collective decision result r1, which is the probability of depression for all training sets. Microstate analysis units extracted from magnetoencephalography: Based on the individual microstates of the subjects, combined with the subjects' gender, age, and actual depression status, two structured information sets were constructed. The first set contained gender, age, microstate category, microstate duration, microstate frequency, and microstate time coverage. The second set contained depression / non-depression data. These two sets of data were input into a support vector machine (SVM) and trained to obtain the collective decision result r2, which is the probability of depression for all training sets. The unit analyzes event-related potential and event-related magnetic field data extracted from the electroencephalogram (EEG) and magnetoencephalogram (MEG). Based on the obtained ERPM and ERFM data, combined with the subject's gender, age, and actual depression status, two structured data sets are constructed. The first set includes gender, age, event category, ERPM, and ERFM; the second set includes depression / non-depression. These two sets of data are input into a convolutional neural network, and training is performed to obtain the collective decision result r3, which is the probability of depression for all training sets. Functional connectivity density analysis unit extracted from functional magnetic resonance imaging: Based on functional connectivity density data, combined with the subject's gender, age, and actual depression status, two structured data sets were constructed. The first set included gender, age, intra-network functional connectivity density, and inter-network functional connectivity density; the second set included depression / non-depression. These two data sets were input into a support vector machine, and training was performed to obtain the collective decision result r4, which is the probability of depression for all training sets. Gray matter volume analysis unit extracted from functional magnetic resonance imaging: Based on gray matter volume data, combined with the subject's gender, age, and actual depression status, two structured data sets were constructed. The first set included gender, age, gray matter type, and volume; the second set included depression / non-depression. These two data sets were input into a support vector machine, and training was performed to obtain the collective decision result r5, which is the probability of depression for all training sets. Decision fusion unit: Based on the obtained r1, r2, r3, r4, and r5, two sets of structured data are constructed according to the gender, age, and actual depression status of each subject in the training set. The first set includes gender, age, r1, r2, r3, r4, and r5; the second set includes depression / non-depression. These two sets of data are input into the extreme gradient boosting tree XGBoost to train a multimodal diagnostic model.

Images