Depression auxiliary diagnosis system based on electroencephalogram and voice and data analysis method thereof

Abstract

The invention relates to an electroencephalogram and voice-based depression auxiliary diagnosis system and a data analysis method thereof, and the auxiliary diagnosis system comprises a data processing module, a detailed diagnosis mode module, a simple diagnosis mode module and a patient information management module. The data processing module comprises a data preprocessing unit and a fixed electroencephalogram feature extraction and visualization unit; the detailed diagnosis mode module is used for carrying out personalized feature extraction and mode recognition on electroencephalogram data by scientific researchers and checking the recognition accuracy rate, a visual graph and an evaluation index statistical table of a selected machine learning model, and the simple diagnosis mode module is used for rapid diagnosis. Comprising a single-mode electroencephalogram data processing unit, a single-mode voice data processing unit and an electroencephalogram voice fusion data processing unit, the simple diagnosis mode module generates a diagnosis result, and the result is in a healthy or depression character form. The result obtained by the auxiliary diagnosis system is objective, meticulous and reliable.

Description

Technical Field

[0001] The present invention relates to the fields of brain-computer interface technology and machine learning technology, and in particular to an electroencephalogram and speech-based depression auxiliary diagnosis system and a data analysis method thereof. Background Art

[0002] Depression is a complex neuropsychiatric disease with multiple factors intertwined. Patients will experience symptoms such as low mood, attention disorder, headache, muscle aches, and even suicidal behavior. Its pathogenesis involves multiple levels such as biology, psychology, social environment, and genetic genes, and is characterized by high incidence and high recurrence rate. According to statistics from the World Health Organization (WHO), the number of depression patients in the world has exceeded 350 million.

[0003] At present, depression cannot be examined by biochemical indicators through instruments and equipment, but is judged by clinicians' inquiries, observations and a series of auxiliary psychological tests, such as the 11th revision of the International Classification of Diseases (ICD-11), the Diagnostic and Statistical Manual of Mental Disorders (DSM), and psychological tests including the Hamilton Depression Scale (HAMD), the Beck Depression Inventory (BDI), the Self-rating Depression Scale (SDS) and the Patient Health Questionaire-9 (PHQ-9). The diagnostic process has the defects of high subjectivity, high uncertainty, and easy interference by the patient's psychological state, which makes it difficult to fully and accurately reflect the patient's true situation. Therefore, it is urgent to develop a more objective, detailed and reliable depression diagnosis and evaluation system to achieve early diagnosis and treatment of depression, alleviate the pain of patients, and reduce social burden.

[0004] Electroencephalography (EEG), as a non-invasive EEG recording method, has the advantage of high temporal resolution and can capture the rapid dynamic changes of brain neural activity in patients with depression. By analyzing the time domain, frequency domain, time-frequency domain and spatial domain characteristics of EEG signals of depression, abnormal changes in the brain of patients with depression can be revealed. Zeng Ling-Li's research shows that depression is related to dysfunction of neural networks distributed in the cerebral cortex. Functional brain network analysis is committed to understanding the communication and coordination patterns between different regions of the brain and is considered to be a powerful tool for early identification of potential mental and neurological diseases. Identification of depression based on brain network features is a current technical deficiency. Moreover, the diagnostic method based on a single EEG feature and model lacks comprehensiveness and comparison in practical application, and has a high degree of randomness and contingency. In addition, the existing diagnostic identification process lacks a visual display of the multi-dimensional features of the EEG time-frequency and spatial domains of patients with depression, and it is impossible to directly observe the feature differences. The diagnostic results of current depression research only display model evaluation indicators such as model classification accuracy, which is poorly readable and not conducive to patient information management and information viewing. In addition, the depression assessment process lacks personal files and paper reports that uniformly record features and diagnostic results.

[0005] In addition, the analysis of unimodal EEG data ignores other manifestations of the patient, such as the patient's voice information. Song Yonghao showed that there are significant differences in the amplitude-frequency characteristics of the speech of people with depression and non-depression. Solieman et al. demonstrated the feasibility of using speech signals to detect depression. This shows that unimodal speech data also has the potential to identify depression. The multimodal depression diagnosis method that integrates EEG and speech will take into account the patient's physiological and behavioral factors, further making up for the shortcomings of subjective diagnosis.

[0006] At the same time, the EEG data processing process of depression is very complicated, and machine learning technology is developing rapidly. Many researchers engaged in basic and clinical research on depression are not familiar with machine learning algorithm code writing and model building, and need to quickly adapt to and use these tools. Therefore, an online operating system that can perform multi-feature multi-model selection and parameter adjustment is crucial to the scientific research of depression. It allows researchers to standardize data analysis and model optimization without a programming background, lower technical barriers, improve research efficiency and repeatability of results, and promote scientific research development. The identification technology of fixed features, models, and model parameters cannot meet the scientific research needs of self-selected features and autonomous model parameter adjustment to observe depression characteristics and classification results, as well as the personalized diagnosis needs of patient diagnosis and treatment. Summary of the invention

[0007] The purpose of the present invention is to provide an auxiliary diagnosis system for depression based on EEG and speech in order to address the technical defects in the prior art.

[0008] Another object of the present invention is to provide a model training method for the auxiliary diagnosis system.

[0009] Another object of the present invention is to provide a data analysis method based on the auxiliary diagnosis system.

[0010] The technical solution adopted to achieve the purpose of the present invention is:

[0011] A depression auxiliary diagnosis system based on EEG and speech, comprising: a data processing module, a detailed diagnosis mode module, a simple diagnosis mode module and a patient information management module, wherein:

[0012] The data processing module includes a data preprocessing unit and a fixed EEG feature extraction and visualization unit; the data preprocessing unit is used to receive the voice data, EEG data and electrode files corresponding to the EEG data uploaded by the user, perform electrode positioning, re-reference of the whole brain average reference, filtering, ICA independent component analysis, interpolation and segmentation on the EEG data to obtain the segmented EEG data, and then remove the bad segments to obtain the preprocessed EEG data, and preprocess the voice data to obtain the spectrum features in the image format; the fixed EEG feature extraction and visualization unit is used to extract the time domain features, frequency domain features and spatial domain features of the EEG signal, and output the time domain waveform, EEG topographic map and brain network functional connection adjacency matrix diagram;

[0013] The detailed diagnosis mode module is used by scientific researchers to perform personalized feature extraction and pattern recognition on EEG data. The detailed diagnosis mode module includes an autonomous selection unit and a model evaluation unit; the autonomous selection unit includes EEG feature options, feature parameter options, machine learning model options, and model parameter options; in the model evaluation unit, the recognition accuracy, visualization graph, and evaluation index statistical table of the selected machine learning model are checked in combination with the data set formed by the EEG data of healthy people and depression patients that comes with the system or uploaded by users;

[0014] The simple diagnosis mode module is used for rapid diagnosis, including a single-modal EEG data processing unit, a single-modal speech data processing unit and an EEG speech fusion data processing unit; in the single-modal EEG data processing unit, the segmented EEG data is subjected to time-frequency and spatial domain feature extraction and input into a pre-trained traditional machine learning model to generate a diagnosis result. At the same time, the segmented EEG data is input into a pre-trained deep learning model to generate a diagnosis result in the form of text "healthy" or "depressed"; the single-modal speech data processing unit is used to input the spectrum graph features into a pre-trained deep residual network model ResNet34 to generate a diagnosis result; the EEG speech fusion data processing unit inputs the segmented EEG data and speech envelope of the same scale into a pre-trained Conformer model, and the Conformer model classifies the extracted features to generate a diagnosis result, and the diagnosis result is in the form of text "healthy" or "depressed";

[0015] The patient information management module includes an information input and storage unit and a diagnosis report output unit; the information input and storage unit is used to manage patient information, examination information and doctor information; the diagnosis report output unit generates a professional downloadable and printable diagnosis report based on the input information and the time domain waveform of the EEG signal, the EEG topography map, the brain network functional connection adjacency matrix diagram and the diagnosis results of the simple diagnosis mode module.

[0016] In the above technical solution, the working method of the data processing module includes the following steps:

[0017] Step S1 and step S2 are performed in sequence in the data preprocessing unit, step S1 is the EEG data preprocessing, and step S2 is the speech data preprocessing; step S3 is performed in the fixed EEG feature visualization unit, step S3 is to extract and visualize the fixed EEG features.

[0018] In the above technical solution, the specific process of step S1 includes: S101, uploading EEG data and the electrode file corresponding to the EEG data, performing electrode positioning on the EEG data, and determining the specific position of each lead; S102, using the average value of the whole head leads as a reference benchmark to re-reference the EEG data; S103, using a 0.5-80Hz bandpass filter to retain the key frequency band and a 50Hz notch filter to remove power frequency interference; S104, performing ICA independent component analysis to decompose the EEG signal into multiple independent components. S105, interpolate bad leads, mark the leads with energy greater than two times the standard deviation of the average power spectrum as bad leads and supplement them through surrounding interpolation, and replace the bad lead data with the mean of spatially adjacent leads; S106, divide the EEG data into 1s time windows to obtain the segmented EEG data; S107, use the absolute threshold method to eliminate the EEG bad segments exceeding ±100μV to obtain the preprocessed EEG data.

[0019] In the above technical solution, the specific process of step S2 includes: S201, uploading voice data, performing 200-3000 Hz bandpass filtering on the voice data, and removing background noise in the voice signal; S202, segmenting the voice segments based on the marking information in the voice data set, and removing the voice of the virtual interviewer; S203, splicing the voice segments of the subject; S204, extracting the spectrogram features of the voice signal every non-overlapping 7s, and saving the spectrogram features in picture format to obtain the spectrogram features in picture format.

[0020] In the above technical scheme, the specific process of step S3 includes: S301, based on the preprocessed EEG data obtained in S107, drawing a visualization diagram of the time domain characteristics of the subject's EEG signal, and obtaining time domain waveform diagrams of different brain regions, preferably, the brain regions include frontal region, parietal region, and occipital region; S302, based on the preprocessed EEG data obtained in S107, drawing a visualization diagram of the frequency domain characteristics of the subject's EEG signal, and obtaining EEG topographic maps of different frequency bands, preferably, the frequency bands include delta, theta, alpha, beta, and gamma; S303, based on the preprocessed EEG data obtained in S107, drawing a visualization diagram of the spatial domain characteristics of the subject's EEG signal, and obtaining adjacency matrix diagrams of brain network functional connections of different frequency bands.

[0021] In the above technical solution, the working method of the detailed diagnosis mode module includes the following steps:

[0022] Steps S4 to S7 are performed in sequence in the autonomous selection unit, wherein step S4 is to autonomously select the EEG features to be extracted; step S5 is to autonomously select feature parameters; step S6 is to autonomously select the machine learning model; step S7 is to autonomously select model parameters; step S8 is performed in the model evaluation unit, and step S8 is to view the model recognition accuracy, visualization graph and evaluation index statistical table.

[0023] In the above technical solution, in step S4, the EEG features include AR parameter features, wPLI features, DWT features and FBCSP features. In step S5, the feature parameters include wavelet basis functions and the number of feature extraction channels. In step S6, the machine learning model includes a traditional machine learning model and a deep learning model. The traditional machine learning model includes an SVM model, a KNN model, an RF model, a GBT model and an XGBoost model. The traditional machine learning model is classified based on the EEG features selected in step S4 and the feature parameters selected in step S5. The deep learning model includes a CNN model, an AE model and an EEGNET model. The deep model is classified based on the segmented EEG data obtained in step S1, and the classification result is "healthy" or "depressed". In step S7, when the traditional machine learning When the traditional machine learning model is an SVM model, the optional model parameters are kernel function selection, including linear, polynomial, RBF and Sigmoid. When the traditional machine learning model is a KNN model, the optional model parameter is the number of neighbors. When the traditional machine learning model is an RF model, the optional model parameters are the number and maximum depth of decision trees. When the traditional machine learning model is a GBT model, the optional model parameters are the learning rate and the number of base learners. When the traditional machine learning model is an XGBoost model, the optional model parameters are the learning rate and the number of base learners. When deep learning CNN, AE and EEGNET models are selected, the optional model parameters are the number of training times and the learning rate. In step S8, the visualization graph includes a confusion matrix graph and a ROC curve graph, and the evaluation indicators include macro average, micro average, weighted average, precision, recall and F1 score values.

[0024] In the above technical solution, the working method of the simple diagnosis mode module includes the following steps:

[0025] The single-modal EEG data processing unit performs step S9, which is specifically as follows: using the segmented EEG data obtained in step S106, calling the feature parameters and model parameters determined by pre-training, outputting the diagnosis results of the traditional machine learning models KNN, SVM, RF, GBT, XGBoost based on AR parameters, wPLI, and DWT features, and the diagnosis results of the deep learning models CNN, AE, and EEGNET, and the result is in the text form of "health" or "depression";

[0026] The single-modal voice data processing unit performs step S10, which is specifically as follows: using the spectrogram features in the picture format obtained in step S204, calling the ResNet model and the weight parameters determined by pre-training, and outputting the diagnosis result obtained by ResNet34 based on the transfer learning technology analysis, and the result is in the text form of "health" or "depression";

[0027] The EEG speech fusion data processing unit performs step S11, and step S11 is specifically as follows: S111, the clinical interview speech data of the person to be tested is downsampled and normalized; S112, the speech envelope is extracted using the Hilbert transform; S113, the speech envelope is segmented without overlap every 1 second; S114, the speech envelope is spliced ​​with the EEG data obtained after segmentation in step S106; S115, the EEG data and speech envelope of the same scale are input into the pre-trained Conformer model, and the Conformer model first extracts the low-order features of EEG and speech respectively through the convolution module, including using the convolution kernel to perform the convolution operation, and using the average pooling layer to reduce the data complexity operation; after passing through the convolution module, the EEG features and the speech features are spliced ​​in the channel dimension, and the fused sequence features are input into the attention module to capture the correlation between the features, and the extracted features are classified through the fully connected layer, and the classification form is the text form of "health" or "depression".

[0028] In the above technical solution, the working method of the information input and storage unit in the patient information management module includes steps S12-S15, wherein step S12 is for inputting the information of the person to be tested, and preferably, the patient information includes the name, gender, ID number, date of birth, contact information, remarks, last examination date, and whether the person suffers from other diseases; step S13 is for inputting the examination information, and preferably, the examination information includes the diagnosis result, examination method, examination findings, and diagnosis opinion; step S14 is for inputting the doctor information, and preferably, the doctor information includes the name of the reporting person and the name of the reviewer; step S15 is for saving the information input in steps S12-S14;

[0029] The working method of the diagnosis report output unit in the patient information management module includes steps S16 and S17, wherein step S16 is for downloading a PDF version of the diagnosis report; step S17 is for viewing a PDF version of the diagnosis report, wherein the diagnosis report includes the patient information of step S12, the examination information of step S13, the doctor information of step S14, and also includes the features and model names involved in the diagnosis process and the diagnosis results, as well as a specific description of the examination findings consisting of the original electroencephalogram, the electroencephalogram of delta, theta, alpha, beta, and gamma frequency bands, the brain network functional connection adjacency matrix diagram of delta, theta, alpha, beta, and gamma frequency bands, and the time domain waveform diagrams of the frontal area, the parietal area, and the occipital area.

[0030] Another aspect of the present invention also includes a method for training each model in the depression auxiliary diagnosis system, comprising the following steps:

[0031] Step a1, after obtaining the EEG data and speech data of healthy people and patients with depression, the EEG data are preprocessed through electrode positioning, whole-brain average reference, re-reference, filtering, ICA independent component analysis, interpolation bad leads, segmentation, and bad segment removal, wherein the EEG data after bad segment removal is used for fixed EEG feature extraction and visualization, and the EEG data before bad segment removal is used for classification and recognition of traditional machine learning models and deep learning models;

[0032] Step a2, extract the time-frequency and spatial domain features of the preprocessed EEG data, analyze the EEG waveforms of the frontal area, parietal area and occipital area in the time domain, and extract AR parameters; extract the power spectrum of the EEG data based on the AR parameter model in the frequency domain to obtain the signal spectrum of a specific frequency band, and draw the EEG topography of different frequency bands, that is, the plane figure of the spherical scalp with different colors representing the power values ​​in each frequency band of the EEG wave; use DWT in the time-frequency domain to decompose the signal into approximation coefficients (low-frequency components) and detail coefficients (high-frequency components) of different scales, and extract the wavelet coefficients in each layer of decomposition; extract the wPLI feature in the spatial domain to study the phase coupling between different electrodes in the EEG data, draw the average adjacency matrix and brain network diagram of the delta, theta, alpha, beta and gamma frequency bands, calculate and draw the parameter indicators of the brain network, including local efficiency, clustering coefficient, average node degree, global efficiency and assortative coefficient, to understand the connection mode and information transmission path between brain regions; extract the FBCSP feature by combining the frequency domain and the spatial domain;

[0033] Step a3, the AR parameters, DWT, FBCSP, and wPLI features extracted in step a2 are randomly divided into 80% training sets and 20% test sets, and the traditional machine learning models SVM, KNN, RF, GBT, and XGBoost models are selected for classification and recognition to find the optimal model parameters and feature parameters; the EEG data segmented in step a1 are divided into 80% training sets and 20% test sets for deep learning CNN, AE, and EEGNET models for pattern recognition; grid search and cross-validation techniques are used to tune parameters to find the optimal model parameter combination, and confusion matrix diagrams and ROC curve diagrams are output. The classification accuracy, precision, recall rate, F1 score, ROC curve, and AUC are used to evaluate the classification effects of traditional machine learning models and deep learning models.

[0034] Step a4: The original speech data of healthy people and patients with depression are preprocessed through filtering, segmentation, concatenation, and segmentation operations to extract speech spectrograms.

[0035] Step a5: Classify and detect the speech spectrogram extracted in step a4 based on the deep residual network ResNet34 and transfer learning technology, and migrate the ResNet34 model from the original 1000-class image classification field to the binary classification of depression and health to obtain a pre-trained ResNet34 model.

[0036] Step a6: downsample and normalize the original speech data of healthy people and patients with depression, use Hilbert transform to extract the speech envelope, segment the speech envelope, concatenate the EEG and speech data and input them into the Conformer model, train the Conformer model to obtain a pre-trained Conformer model, and realize the processing of multimodal EEG speech fusion data and depression recognition, wherein: the Conformer model includes convolution layer, BatchNorm layer, ELU layer, Avgpool layer, Conv2d layer, LayerNorm layer, Dropout layer, LayerNorm layer, fully connected layer, and Flatten layer.

[0037] Another aspect of the present invention also includes a data analysis method based on the depression auxiliary diagnosis system, comprising the following steps:

[0038] Step 1: Data processing module performs data preprocessing:

[0039] Obtaining EEG data and speech data of the person to be tested;

[0040] The EEG data is re-referenced, filtered, ICA independent component analysis, interpolated, and segmented to obtain segmented EEG data, and then the bad segments are removed to obtain pre-processed EEG data. The time-frequency and spatial domain features of the pre-processed EEG data are extracted, and the time domain waveform, EEG topography, and brain network functional connection adjacency matrix are drawn;

[0041] Preprocess the speech data to obtain the spectrogram features in image format;

[0042] Step 2: Detailed diagnosis mode module performs detailed diagnosis:

[0043] In the detailed diagnosis mode module, users can select EEG features, feature parameters, machine learning models, and model parameters on their own. Combined with the EEG data of healthy people and patients with depression that comes with the system or is uploaded by users, users can view the recognition accuracy, visualization graph, and evaluation index statistics of the machine learning model they selected.

[0044] Step 3: Simple diagnosis mode module performs simple diagnosis:

[0045] Single-modal EEG data processing: Use the segmented EEG data obtained in step 1, call the feature parameters and model parameters determined by pre-training, and output the diagnosis results of traditional machine learning models KNN, SVM, RF, GBT, XGBoost based on AR parameters, wPLI, and DWT features, and the diagnosis results of deep learning models CNN, AE, and EEGNET. The results are in the form of "healthy" or "depressed" text;

[0046] Single-modal speech data processing: Use the spectrogram features in the image format obtained in step 1, call the pre-trained ResNet model and weight parameters, and output the diagnosis results obtained by ResNet34 based on transfer learning technology analysis. The result is in the form of "healthy" or "depressed" text;

[0047] EEG and speech fusion data processing: The clinical interview speech data of the test person is downsampled and normalized; the Hilbert transform is used to extract the speech envelope; the speech envelope is segmented without overlap every 1 second; the speech envelope is spliced ​​with the segmented EEG data obtained in step 1; the EEG data and speech envelope of the same scale are input into the pre-trained Conformer model, which first extracts the low-order features of EEG and speech respectively through the convolution module, including the use of convolution kernels for convolution operations and the use of average pooling layers to reduce the complexity of data operations; after the convolution module, the EEG features and speech features are spliced ​​in the channel dimension, and the fused sequence features are input into the attention module to capture the correlation between the features, and the extracted features are classified through the fully connected layer in the form of "healthy" or "depressed" text;

[0048] Step 4: Enter information in the patient information management module to generate a PDF version of the diagnosis report:

[0049] Enter the patient information, examination information, and doctor information in the patient information management to generate a PDF version of the diagnosis report, which includes the features and model names involved in the diagnosis process and the diagnosis results, as well as a specific description of the examination findings consisting of the original EEG, EEG topography maps of delta, theta, alpha, beta, and gamma frequency bands, and adjacency matrix diagrams of brain network functional connections in delta, theta, alpha, beta, and gamma frequency bands, and time domain waveform diagrams of the frontal, parietal, and occipital regions.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] 1. The EEG and speech-based depression auxiliary diagnosis system provided by the present invention integrates a variety of traditional machine learning models and deep learning models and displays feature visualization graphs, which can solve the defects of strong subjectivity and high uncertainty in depression diagnosis, and provide a more objective, detailed and reliable depression diagnosis and evaluation tool. The present invention does not limit the use environment, and the EEG data is collected in a resting state, and the patient is not required to perform a certain task.

[0052] 2. The present invention comprehensively considers the physiological and behavioral factors of patients, and can identify depression by using single-modal EEG data or voice data, or by using multi-modal EEG and voice fusion data. The recognition accuracy of multi-modal EEG and voice fusion data is higher than that of single-modal EEG data and voice data, and the recognition accuracy of single-modal EEG data is higher than that of single-modal voice data.

[0053] 3. The present invention takes into account the needs of multiple parties and designs functional modules, including a data processing module, a detailed diagnosis mode module, a simple diagnosis mode module, and a patient information management module. The detailed diagnosis mode module is mainly designed for the needs of scientific researchers using machine learning models to analyze data. It can help researchers to standardize scientific research tasks such as data analysis and model optimization without a programming background, reduce technical barriers, improve research efficiency, and promote scientific research development; the simple diagnosis module can quickly output diagnostic results, and the diagnostic process only takes a few seconds, which will greatly shorten the time for medical consultation and waiting time, improve the doctor's diagnostic efficiency, provide convenience for patients with tight medical time, and optimize the medical experience.

[0054] 4. The present invention provides a PDF version of the diagnostic report for downloading and printing, which meets the patient's needs for carrying and retaining the report, makes up for the disadvantage that some elderly people lack the conditions for using the Internet, and has extremely important social value. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1This is the overall architecture diagram of the depression auxiliary diagnosis system based on EEG and speech.

[0056] Figure 2 Schematic diagram of preprocessing and partial feature visualization in the processing flow of single-modal EEG data.

[0057] Figure 3 Schematic diagram of pattern recognition and model visualization based on partial features in the processing flow of unimodal EEG data.

[0058] Figure 4 Schematic diagram of the processing flow of single-modal speech data.

[0059] Figure 5 Schematic diagram of the processing flow of multimodal EEG speech fusion data.

[0060] Figure 6 Schematic diagram of the Conformer model network structure.

[0061] Figure 7 Schematic diagram of the algorithm flow of the common space pattern (CSP). DETAILED DESCRIPTION

[0062] The present invention is further described in detail below in conjunction with specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0063] Example 1

[0064] like Figure 1-Figure 5 As shown, a depression auxiliary diagnosis system based on EEG and speech includes: a data processing module, a detailed diagnosis mode module, a simple diagnosis mode module and a patient information management module, wherein:

[0065] The data processing module includes a data preprocessing unit and a fixed EEG feature extraction and visualization unit; the data preprocessing unit is used to receive the voice data, EEG data and electrode files corresponding to the EEG data uploaded by the user, perform electrode positioning, re-reference of the whole brain average reference, filtering, ICA independent component analysis, interpolation and segmentation on the EEG data to obtain the segmented EEG data, and then remove the bad segments to obtain the preprocessed EEG data, and preprocess the voice data to obtain the spectrum features in the image format; the fixed EEG feature extraction and visualization unit is used to extract the time domain features, frequency domain features and spatial domain features of the EEG signal, and output the time domain waveform, EEG topographic map and brain network functional connection adjacency matrix diagram;

[0066] The specific process of the data processing module includes: S1, EEG data preprocessing; S2, speech data preprocessing and feature extraction; S3, fixed EEG feature visualization;

[0067] The specific process of step S1 is as follows: step S101, upload EEG data and the electrode file corresponding to the EEG data, perform electrode positioning on the EEG data, and determine the specific position of each lead; step S102, use the average value of the whole head leads as a reference benchmark to re-reference the EEG data; step S103, use a 0.5-80Hz bandpass filter to retain the key frequency band and a 50Hz notch filter to remove power frequency interference; step S104, perform ICA independent component analysis to decompose the EEG signal into multiple independent components, such as electrooculogram. S105, interpolate bad leads, mark the leads with energy greater than two times the standard deviation of the average power spectrum as bad leads and supplement them through surrounding interpolation, and replace the bad lead data with the mean of spatially adjacent leads; S106, divide the EEG data into 1s time windows to obtain the divided EEG data; S107, use the absolute threshold method to eliminate the EEG bad segments exceeding ±100μV to obtain the preprocessed EEG data.

[0068] The specific process of step S2 includes: S201, uploading voice data, performing 200-3000 Hz bandpass filtering on the voice data to remove background noise in the voice signal; S202, segmenting the voice segments based on the marking information in the voice data set, and removing the voice of the virtual interviewer; S203, splicing the voice segments of the subject; S204, extracting the spectrum features of the voice signal every 7 seconds without overlapping, and saving the spectrum features in image format.

[0069] The specific process of step S3 includes: 301, drawing a time domain waveform diagram of different brain regions (frontal region, parietal region, occipital region) of the time domain feature visualization diagram of the subject's EEG signal based on the EEG data after step S107; 302, drawing a frequency domain feature visualization diagram of the subject's EEG signal based on the EEG data after step S107, that is, the EEG topography diagram of delta (0.5-3Hz), theta (4-7Hz), alpha (8-12Hz), beta (13-30Hz), and gamma (31-80Hz) frequency bands obtained by power spectrum estimation based on the AR parameter model; 303, drawing a brain network functional connection adjacency matrix diagram of different frequency bands (delta, theta, alpha, beta, gamma) based on the spatial domain feature visualization diagram of the subject's EEG signal after step S107.

[0070] The detailed diagnosis mode module is used by scientific researchers to perform personalized feature extraction and pattern recognition on EEG data. The detailed diagnosis mode module includes an autonomous selection unit and a model evaluation unit; the autonomous selection unit includes EEG feature options to be extracted, feature parameter options, machine learning model options, and model parameter options; in the model evaluation unit, a data set (divided into 80% training set and 20% test set) formed by combining the EEG data of healthy people and depression patients that comes with the system or uploaded by users is used to view the recognition accuracy, visualization graph, and evaluation index statistics of the selected machine learning model;

[0071] The detailed diagnosis mode module uses the EEG data after step S106 is completed, specifically including: S4, autonomous selection of EEG features to be extracted; S5, autonomous selection of feature parameters; S6, autonomous selection of machine learning models; S7, autonomous selection of model parameters; S8, viewing the model recognition accuracy and visualization graph and evaluation index statistical table.

[0072] The specific contents of step S4 include: the selectable EEG features are divided into autoregressive (AR) parameters; the spatial domain weighted phase lag index (wPLI) features for exploring the topological structure and functional connection of the brain network; the time-frequency domain discrete wavelet transform (DWT) features obtained by decomposing the signal into a rough approximation of the original signal and detailed information; the spatial domain filter bank common spatial pattern (FBCSP) features obtained by using a bandpass filter to divide the frequency band and a common spatial pattern (CSP) algorithm.

[0073] The specific contents of step S5 include: the wavelet basis function of the discrete wavelet transform (DWT) feature can be selected independently, including db2, db4, db6 wavelet bases of the Daubechies wavelet basis (db) category, sym2, sym4, sym8 wavelet bases of the Symlets wavelet basis (sym) category, coif1, coif2, coif3 wavelet bases of the Coiflets wavelet basis (coif) category, and bior2.2, bior3.3, bior6.8 wavelet bases of the Biorthogonal wavelet basis (bior) category; the number of feature extraction channels of the filter bank common space pattern (FBCSP) feature can be adjusted, for example, 6 is selected.

[0074] The specific content of step S6 includes: the selectable machine learning classifier includes a traditional machine learning model and a deep learning model, wherein the traditional machine learning model is divided into a support vector machine (SVM) model, a k-nearest neighbor classification (KNN) model, a random forest (RF) model, a gradient boosting tree (GBT) model and an XGBoost (eXtreme GradientBoosting) model, and the traditional machine learning model is classified based on the features described in step S4 and the feature parameters described in step S5. The deep learning model is divided into a convolutional neural network (CNN) model, an auto-encoder (AE) model and an EEGNET model, and the deep learning model is identified based on the EEG data pre-processed in step S106.

[0075] The specific contents of step S7 include: the parameters of the traditional machine learning model and the deep learning model described in step S6 can be adjusted. For the support vector machine (SVM) model, a variety of kernel function options are provided (Table 2 shows the kernel function types), including linear, polynomial, RBF and Sigmoid; for the K nearest neighbor (KNN) model, the user can adjust the number of neighbors to optimize the classification effect; when the random forest (RF) model is selected, the user can set the number and maximum depth of decision trees; for the gradient boosted tree (GBT) model, the user can adjust the learning rate and the number of base learners; in addition, the learning rate and the number of base learners of the XGBoost model can also be set; the number of training times and learning rates of the deep learning CNN, AE and EEGNET models can be selected as needed.

[0076] The specific contents of step S8 include: after selecting features, feature parameters, models, and model parameters, the recognition accuracy of the model, the model visualization diagram, and the evaluation index statistical table are output, the visualization diagram includes a confusion matrix diagram and a ROC curve diagram, and the evaluation index statistical table includes macro average (macro avg), micro average (micro avg), weighted average (weighted avg), precision, recall rate, and F1 score values.

[0077] The simple diagnosis mode module includes a single-modal EEG data processing unit, a single-modal speech data processing unit and an EEG speech fusion data processing unit; in the single-modal EEG data processing unit, the segmented EEG data obtained in S106 is subjected to time-frequency and spatial domain feature extraction, and is input into a pre-trained traditional machine learning model to generate a diagnosis result. At the same time, the segmented EEG data obtained in S106 is input into a deep learning model to generate a diagnosis result in the form of "healthy" or "depressed" text; the single-modal speech data processing unit is used to input the spectrum graph features into a deep residual network model ResNet34 to generate a diagnosis result in the form of "healthy" or "depressed" text; the EEG speech fusion data processing unit inputs the EEG data and speech envelope of the same scale into a Conformer model, and the Conformer model classifies the extracted features to generate a diagnosis result in the form of "healthy" or "depressed" text;

[0078] The simple diagnosis mode module is used for rapid diagnosis, including: S9, processing unimodal EEG data; S10, processing unimodal speech data; S11, processing multimodal EEG speech fusion data. The specific content of step S9 includes: the input data is a unimodal EEG signal, the segmented EEG data obtained in step S106 is used, the feature parameters and model parameters determined by pre-training are directly called, and the diagnosis results of traditional machine learning models KNN, SVM, RF, GBT, XGBoost based on AR, wPLI, DWT features and the diagnosis results of deep learning models CNN, AE, EEGNET are output, and the result is in the form of "healthy" or "depressed" text.

[0079] The specific content of step S10 includes: when the input data is a speech signal, directly calling the pre-trained weight parameters of the model, outputting the deep residual network (Residual Networks, ResNet) model ResNet34 based on the transfer learning technology to analyze the patient's speech spectrogram after step S204 is completed and obtain the diagnosis result.

[0080] The specific contents of step S11 include: when the input data is an EEG signal and a speech signal, S111, the original speech data will be downsampled and normalized; S112, the speech envelope will be extracted using the Hilbert transform; S113, the speech envelope will be segmented without overlap every 1 second; S114, the speech envelope will be spliced ​​with the EEG data; S115, the EEG data and speech envelope processed in step S106 of the same scale are input into the Conformer network. The Conformer model first extracts the low-order features of EEG and speech respectively through the convolution module, including using the convolution kernel for convolution operation and using the average pooling layer to reduce the data complexity operation; after the convolution module, the EEG features and speech features are spliced ​​in the channel dimension, and the fused sequence features are input into the attention module to capture the correlation between the features, and the extracted features are classified through the fully connected layer.

[0081] The patient information management module includes an information input and storage unit and a diagnosis report output unit; the information input and storage unit is used to manage patient information, examination information and doctor information; the diagnosis report output unit generates a professional diagnosis report based on the diagnosis results of the detailed diagnosis mode module or the simple diagnosis mode module.

[0082] The patient information management module includes: S12, patient information input; S13, examination information input; S14, doctor information input; S15, information storage; S16, PDF version diagnosis report download; S17, PDF version diagnosis report viewing;

[0083] The patient information in step S12 includes the patient's name, gender, ID number, date of birth, contact information, notes, last examination date, and whether he suffers from other diseases. The examination information in step S13 includes the diagnosis result, examination method, examination findings, and diagnosis opinion. The doctor information in step S14 includes the name of the reporter and the name of the reviewer. In addition to the patient information described in step S12, the examination information described in step S13, and the doctor information described in step S14, the PDF version of the diagnostic report viewable in step S17 also includes the features and model names and diagnostic results involved in the diagnosis process, as well as the original EEG, EEG topography of delta, theta, alpha, beta, and gamma frequency bands, and functional connection adjacency matrix diagrams of delta, theta, alpha, beta, and gamma frequency bands, and time domain waveform diagrams of the frontal, parietal, and occipital regions. A specific description of the examination findings.

[0084] Embodiment 2 Before constructing the auxiliary diagnosis system, each model needs to be pre-trained and parameter tuned. The specific steps are as follows:

[0085] Step a, model parameter tuning phase:

[0086] Step a1, such as Figure 2 As shown in the figure, after obtaining the EEG data and speech data of healthy people (Healthy Control, HC) and patients with major depressive disorder (Major Depressive Disorder, MDD), the segmented EEG data are obtained through electrode positioning and whole-brain average reference, re-reference, filtering, ICA independent component analysis, interpolation and segmentation, and then the bad segments are removed to obtain the preprocessed EEG data, wherein the preprocessed EEG data is used for fixed EEG feature extraction and visualization, and the segmented EEG data is used for classification and recognition of traditional machine learning models and deep learning models;

[0087] Step a2, such as Figure 2 and Figure 3 As shown in the figure, the time-frequency and spatial domain features of the preprocessed EEG data are extracted, the EEG waveforms of the frontal area, parietal area and occipital area are analyzed in the time domain, and the AR parameters are extracted; the frequency domain extraction performs power spectrum analysis on the EEG data based on the AR parameter model to obtain the signal spectrum of a specific frequency band, and draws the EEG topography of different frequency bands, that is, the plane figure of the spherical scalp in which the power values ​​in each frequency band of the EEG are represented by different colors; the DWT is used in the time-frequency domain to decompose the signal into approximation coefficients (low-frequency components) and detail coefficients (high-frequency components) of different scales, and the wavelet coefficients in each layer of decomposition are extracted; the wPLI feature is extracted in the spatial domain to study the phase coupling between different electrodes in the EEG data, and the average adjacency matrix and brain network diagram of the delta, theta, alpha, beta and gamma frequency bands are drawn, and the parameter indicators of the brain network are calculated and drawn, including local efficiency, clustering coefficient, average node degree, global efficiency and assortative coefficient, to understand the connection pattern and information transmission path between brain regions; the FBCSP feature is extracted by combining the frequency domain and the spatial domain;

[0088] For details, see Zhang Bailin, Yang Chengzhi, Wu Hongchao. Performance analysis of Yule-Walker method and Burg method power spectrum estimation based on AR parameter model [J]. Computers and Digital Engineering, 2016, 44(05): 813-817. The process of performing power spectrum analysis on EEG data based on AR parameter model to extract EEG features includes: assuming that the system model that generates the random sequence x(n) is a linear difference equation, that is,

[0089]

[0090] w(n) represents a white noise sequence. Perform Z transform on the above equation, z = e jw , and assume that the power spectral density of the input white noise is The output power spectral density is:

[0091]

[0092] When the coefficient a j , b i and After all are determined, the power spectral density p of the random signal can be obtained x (w). When i>0, b i = 0, the difference equation of the system can be changed to

[0093]

[0094] Formula (3) is the autoregressive model, referred to as the AR parameter model. After performing the Z transform on it, the power spectrum of its output can be obtained as follows:

[0095]

[0096] When calculated and a k After that, the power spectrum density p of the random signal can be obtained. x (w).

[0097] Common spatial pattern (CSP) is a supervised learning algorithm that can reduce high-dimensional EEG signals to low dimensions by finding features in the correlation matrix of EEG signals and use linear projection transformation to find useful eigenvalues. This embodiment uses EEG signals in two cases, HC and MDD. The algorithm flow chart is as follows: Figure 7 As shown in the figure, the filter bank common spatial pattern (FBCSP) feature is realized by performing a spatial domain extraction method of the common spatial pattern (CSP) on each frequency band. It mainly includes four stages: frequency filtering, spatial filtering, feature selection and classification. In the first stage, EEG is band-pass filtered into j frequency bands X1, X2, ..., X using j band-pass filters of different frequency bands. j ∈R C*T ; In the second stage, the CSP algorithm is used to perform spatial filtering on each frequency band, so each pair of bandpass and spatial filters will produce CSP features within the frequency range of the bandpass filter; the third stage uses a feature selection algorithm to select distinctive CSP features from the filter bank; the fourth stage uses a classification algorithm to model and classify the selected CSP features.

[0098] The discrete wavelet transform (DWT) feature analyzes signals in different frequency bands by decomposing the signal into a rough approximation and detailed information. DWT is mathematically described as a set of inner products between a finite length sequence and a discrete wavelet basis. Each inner product results in a wavelet transform coefficient. Given a finite length discrete signal f(n), DWT can be expressed as:

[0099]

[0100] Where W is the DWT coefficient; ψ j,k (n) is the discrete wavelet basis:

[0101]

[0102] and k are the discrete versions of the scale parameter and the translation parameter, respectively.

[0103] Phase Lay Index (PLI) is an indicator used to measure the degree of phase synchronization between two signals. It is defined as the sum of the absolute values ​​of the signs of the imaginary parts of the complex cross-spectral density of two real-valued signals a(t) and b(t) over the entire frequency range. The weighted phase delay index (wPLI) introduces a weighting scheme based on PLI, which weights each phase difference according to the lead amplitude, thereby reducing the impact of phase differences near zero on the calculation results.

[0104] make is the imaginary component of the cross power of two time series a(t) and b(t):

[0105]

[0106] Here H[·] represents Hilbert transform (HT), is the conjugate of H[·], imag{·} represents the imaginary part, and the calculation formula of wPLI is as follows:

[0107]

[0108] Here <·> and |·| represent the mean and absolute value operators respectively, and sign is the sign function. 0≤WPLI≤1, (0): asynchronous, (1): synchronous (see reference: Research on action intention understanding and classification based on EEG signals, Xiong Xingliang).

[0109] In the construction of brain functional networks, the threshold has a great influence on the construction of adjacent networks and network parameters. This invention determines the threshold based on the mean K of all node degrees in the network, taking K>2lnN. Generally speaking, the range of K is between 3 and 12, where N is the network size. The minimum threshold T1 and the maximum threshold T2 of all samples are obtained. At this time, the value range of the threshold T is T1≤T≤T2. Set a suitable step size within the value range of T, construct the binary matrix respectively, and calculate various network parameters under different thresholds, including local efficiency, clustering coefficient, average node degree, global efficiency, and matching coefficient.

[0110] Step a3, the AR parameters, DWT, FBCSP, and wPLI features extracted in step a2 are randomly divided into 80% training sets and 20% test sets, and the traditional machine learning models SVM, KNN, RF, GBT, and XGBoost models are selected for classification and recognition to find the optimal model parameters and feature parameters; the EEG data divided in step a1 are divided into 80% training sets and 20% test sets for deep learning CNN, AE, and EEGNET models for pattern recognition; grid search and cross-validation techniques are used to tune parameters to find the optimal model parameter combination, and confusion matrix diagrams (Table 1 is the confusion matrix) and ROC curve diagrams are output. The classification accuracy, precision, recall rate, F1 score, ROC curve, and AUC are used to evaluate the classification effects of traditional machine learning models and deep learning models.

[0111] Table 1

[0112]

[0113] Among them, in data classification problems, the support vector machine (SVM) model can provide the largest separation margin for linearly separable data sets, but there will also be nonlinear problems, such as the XOR problem, so the data needs to be mapped to a higher dimensional space so that linear separation can be achieved. By using the kernel trick, there is no need to explicitly map the high-dimensional space because it returns a measure of similarity in such a space, namely k(x,y).

[0114] The kernel function used in this embodiment is shown in Table 2.

[0115] Table 2

[0116]

[0117] The classification accuracy of the machine learning model in this embodiment is calculated by assuming that each sample corresponds to a true label (0 or 1), the total number of training samples is recorded as N, and the number of correct labels output by the machine learning model for the input samples is recorded as N right , then it can be expressed as:

[0118]

[0119] Precision refers to the proportion of correctly predicted positives to all predicted positives:

[0120]

[0121] Recall refers to the proportion of correctly predicted positives to all actual positives:

[0122]

[0123] The value of the F1 score is the arithmetic mean divided by the geometric mean, and the larger the better. The value of the F1 score combines the precision and recall to form an indicator:

[0124]

[0125] The Receiver Operating Characteristic Curve (ROC) graph reflects sensitivity and specificity. Its horizontal axis represents the false positive rate (FPR), and the vertical axis represents the true positive rate (TPR). The closer the ROC curve is to the (0, 1) point, the better the classifier performance. The AUC (Area Under Curve) is defined as the area enclosed by the coordinate axis under the ROC curve. The closer the AUC is to 1.0, the higher the authenticity of the detection method.

[0126] Step a4: The original speech data of healthy people and patients with depression are preprocessed through filtering, segmentation, concatenation, and segmentation operations to extract speech spectrograms.

[0127] Step a5: Classify and detect the speech spectrogram extracted in step a4 based on the deep residual network ResNet34 and transfer learning technology to obtain a pre-trained ResNet34 model.

[0128] Among them, the residual neural network (ResNet) is a classic neural network. By introducing residual connections (ResidualConnection), the shallow input is directly added to the output through short-circuit connections (Shortcut Connections) to solve the gradient vanishing problem that occurs as the network depth increases. ResNet34 is a 34-layer deep residual network. In order to migrate the model from the original 1000-class image classification field to the binary classification of depression and health, the number of output nodes of the last fully connected layer of the model was changed from 1000 to 2. Transfer learning technology reuses the common features learned in shallow networks, so that a relatively ideal result can be quickly trained in a new field. This embodiment realizes transfer learning from the image classification field to the speech spectrogram field.

[0129] Step a6: Downsample and normalize the original speech data of healthy people and patients with depression, use Hilbert transform to extract the speech envelope, segment the speech envelope, concatenate the EEG and speech data and input them into the Conformer model, train the Conformer model, and obtain a pre-trained Conformer model to realize the processing of multimodal EEG speech fusion data and depression recognition. The main feature of Conformer is the combination of convolutional neural network and attention mechanism. The convolution layer can extract low-order short-time range features, and then the multi-head attention mechanism can capture the correlation between features in a long time range. Finally, the extracted features are classified through the fully connected layer. Among them: Figure 6 As shown, the Conformer model includes convolutional layer, BatchNorm layer, ELU layer, Avgpool layer, Conv2d layer, LayerNorm layer, Dropout layer, LayerNorm layer, fully connected layer, and Flatten layer;

[0130] Example 3

[0131] A data analysis method based on the auxiliary diagnosis system comprises the following steps:

[0132] Step 1: Data processing module performs data preprocessing:

[0133] Obtaining EEG data and speech data of the person to be tested;

[0134] The EEG data is re-referenced, filtered, ICA independent component analysis, interpolated, and segmented to obtain segmented EEG data, and then the bad segments are removed to obtain pre-processed EEG data. The time-frequency and spatial domain features of the pre-processed EEG data are extracted, and the time domain waveform, EEG topography, and brain network functional connection adjacency matrix are drawn;

[0135] Preprocess the speech data to obtain the spectrogram features in image format;

[0136] Step 2: Detailed diagnosis mode module performs detailed diagnosis:

[0137] Select EEG features, feature parameters, machine learning models, and model parameters in the detailed diagnosis mode module; combine the data set formed by the EEG data of healthy people and patients with depression that comes with the system or uploaded by users, and check the recognition accuracy, visualization graph, and evaluation index statistics of the self-selected machine learning model;

[0138] Step 3: Simple diagnosis mode module performs simple diagnosis:

[0139] Single-modal EEG data processing: Use the segmented EEG data obtained in step 1, call the feature parameters and model parameters determined by pre-training, and output the diagnosis results of traditional machine learning models KNN, SVM, RF, GBT, XGBoost based on AR parameters, wPLI, and DWT features, and the diagnosis results of deep learning models CNN, AE, and EEGNET. The results are in the form of "healthy" or "depressed" text;

[0140] Single-modal speech data processing: Use the spectrogram features in the image format obtained in step 1, call the pre-trained ResNet model and weight parameters, and output the diagnosis results obtained by ResNet34 based on transfer learning technology analysis. The result is in the form of "healthy" or "depressed" text;

[0141] EEG and speech fusion data processing: The clinical interview speech data of the test person is downsampled and normalized; the Hilbert transform is used to extract the speech envelope; the speech envelope is segmented without overlap every 1 second; the speech envelope is spliced ​​with the segmented EEG data obtained in step 1; the EEG data and speech envelope of the same scale are input into the pre-trained Conformer model, which first extracts the low-order features of EEG and speech respectively through the convolution module, including the use of convolution kernels for convolution operations and the use of average pooling layers to reduce the complexity of data operations; after the convolution module, the EEG features and speech features are spliced ​​in the channel dimension, and the fused sequence features are input into the attention module to capture the correlation between the features, and the extracted features are classified through the fully connected layer in the form of "healthy" or "depressed" text;

[0142] Step 4: Enter information in the patient information management module to generate a PDF version of the diagnosis report:

[0143] The patient information management module generates a diagnosis report (PDF version) based on the input patient information, examination information, doctor information and output diagnosis results, where the patient information includes the patient's name, gender, ID number, date of birth, contact information, notes, last examination date and whether he or she suffers from other diseases; the examination information includes the diagnosis results, examination methods, examination findings and diagnostic opinions; the doctor information includes the name of the reporter and the name of the reviewer. The diagnosis report includes the features and model names involved in the diagnosis process and the diagnosis results, as well as a specific description of the examination findings consisting of the original EEG, EEG topography of delta, theta, alpha, beta, and gamma frequency bands, and brain network functional connection adjacency matrix diagrams of delta, theta, alpha, beta, and gamma frequency bands, and time domain waveform diagrams of the frontal, parietal, and occipital regions.

[0144] The evaluation system of the above embodiment 1, the model training method of the embodiment 2 and the data analysis method of the embodiment 3 have the following characteristics:

[0145] 1. Preprocess and extract features of EEG data, extract multi-dimensional features of time-frequency-spatial domains of specific frequency bands and brain regions in a personalized manner, preprocess speech data, and fuse speech and EEG data; the processing process uses block processing to significantly reduce memory usage, avoid memory overflow, and ensure the efficiency and stability of the system; integrate real-time data visualization functions to intuitively view EEG signals, feature extraction results, and machine learning model prediction results.

[0146] 2. With the help of traditional machine learning and deep learning algorithms and transfer learning technology, an end-to-end architecture of multiple algorithms is designed, and the model parameters pre-trained before system design are called. The support vector machine (SVM) model, K nearest neighbor (KNN) model, random forest (RF) model, gradient boosted tree (GBT) model, XGBoost model, convolutional neural network (CNN) model, autoencoder (AE) model, EEGNet model, deep residual network (ResNet) model and Conformer model are integrated to achieve objective identification and diagnosis of depression. Grid search and cross-validation are used to tune model parameters. GPU acceleration, batch normalization and early stopping (EarlyStopping) technology are used to optimize the training process. The model saving and loading mechanism is set to support prediction at different time points and on different devices, thereby increasing system flexibility and practicality.

[0147] 3. The visualization system takes into account the needs of multiple parties and designs functional modules, including data processing module, detailed diagnosis mode module, simple diagnosis mode module and patient information management module, and automatically generates professional PDF version of diagnosis report for download and viewing; the interactive user-friendly interface design adopts the Streamlit framework, supports cloud computing and distributed processing, and cloud deployment uses cloud computing resources for efficient model training and data processing to ensure the high computing power and scalability of the system.

[0148] The above is only a preferred embodiment of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A depression auxiliary diagnosis system based on EEG and speech, characterized in that: It includes a data processing module, a detailed diagnosis mode module, a simple diagnosis mode module and a patient information management module, wherein: The data processing module includes a data preprocessing unit and a fixed EEG feature extraction and visualization unit; the data preprocessing unit is used to receive the voice data, EEG data and electrode files corresponding to the EEG data uploaded by the user, perform electrode positioning, re-reference of the whole brain average reference, filtering, ICA independent component analysis, interpolation and segmentation on the EEG data to obtain the segmented EEG data, and then remove the bad segments to obtain the preprocessed EEG data, and preprocess the voice data to obtain the spectrum features in the image format; the fixed EEG feature extraction and visualization unit is used to extract the time domain features, frequency domain features and spatial domain features of the EEG signal, and output the time domain waveform, EEG topographic map and brain network functional connection adjacency matrix diagram; The detailed diagnosis mode module is used by scientific researchers to perform personalized feature extraction and pattern recognition on EEG data. The detailed diagnosis mode module includes an autonomous selection unit and a model evaluation unit; the autonomous selection unit includes EEG feature options, feature parameter options, machine learning model options, and model parameter options; in the model evaluation unit, the recognition accuracy, visualization graph, and evaluation index statistical table of the selected machine learning model are checked in combination with the data set formed by the EEG data of healthy people and depression patients that comes with the system or uploaded by users; The simple diagnosis mode module is used for rapid diagnosis, including a single-modal EEG data processing unit, a single-modal speech data processing unit and an EEG speech fusion data processing unit; in the single-modal EEG data processing unit, the segmented EEG data is subjected to time-frequency and spatial domain feature extraction and input into a pre-trained traditional machine learning model to generate a diagnosis result. At the same time, the segmented EEG data is input into a pre-trained deep learning model to generate a diagnosis result in the form of "health" or "depression" text; the single-modal speech data processing unit is used to input the spectrum graph features into a pre-trained deep residual network model ResNet34 to generate a diagnosis result; the EEG speech fusion data processing unit inputs the segmented EEG data and speech envelope of the same scale into a pre-trained Conformer model, and the Conformer model classifies the extracted features to generate a diagnosis result, and the diagnosis result is in the form of "health" or "depression" text; The patient information management module includes an information input and storage unit and a diagnosis report output unit; the information input and storage unit is used to manage patient information, examination information and doctor information; the diagnosis report output unit generates a professional downloadable and printable diagnosis report based on the input information and the time domain waveform of the EEG signal, the EEG topography map, the brain network functional connection adjacency matrix diagram and the diagnosis results of the simple diagnosis mode module.

2. The depression auxiliary diagnosis system according to claim 1, characterized in that: The working method of the data processing module comprises the following steps: Step S1 and step S2 are performed in sequence in the data preprocessing unit, step S1 is the EEG data preprocessing, and step S2 is the speech data preprocessing; step S3 is performed in the fixed EEG feature visualization unit, step S3 is to extract and visualize the fixed EEG features.

3. The depression auxiliary diagnosis system according to claim 2, characterized in that: The specific process of step S1 includes: S101, uploading EEG data and the electrode file corresponding to the EEG data, positioning the electrodes of the EEG data, and determining the specific position of each lead; S102, re-referencing the EEG data using the average value of the whole head leads as a reference benchmark; S103, using a 0.5-80Hz bandpass filter to retain the key frequency band and a 50Hz notch filter to remove power frequency interference; S104, performing ICA independent component analysis to decompose the EEG signal into multiple independent components, such as eye Electrical artifacts, myoelectric artifacts, channel noise, etc., mark and manually remove bioelectric artifacts and noise; S105, interpolate bad leads, mark the leads with energy greater than two times the standard deviation of the average power spectrum as bad leads and supplement them through surrounding interpolation, and replace the bad lead data with the mean of spatially adjacent leads; S106, divide the EEG data into 1s time windows to obtain the segmented EEG data; S107, use the absolute threshold method to remove the EEG bad segments exceeding ±100μV to obtain the preprocessed EEG data.

4. The depression auxiliary diagnosis system according to claim 2, characterized in that: The specific process of step S2 includes: S201, uploading voice data, performing 200-3000 Hz bandpass filtering on the voice data, and removing background noise in the voice signal; S202, segmenting the voice segments based on the marking information in the voice data set, and removing the voice of the virtual interviewer; S203, splicing the voice segments of the subject; S204, extracting the spectrogram features of the voice signal every non-overlapping 7 seconds, and saving the spectrogram features in picture format to obtain the spectrogram features in picture format.

5. The depression auxiliary diagnosis system according to claim 2, characterized in that: The specific process of step S3 includes: S301, based on the preprocessed EEG data obtained in S107, drawing a visualization of the time domain characteristics of the subject's EEG signal, and obtaining time domain waveform diagrams of different brain regions, preferably, the brain regions include frontal region, parietal region, and occipital region; S302, based on the preprocessed EEG data obtained in S107, drawing a visualization of the frequency domain characteristics of the subject's EEG signal, and obtaining EEG topographic maps of different frequency bands, preferably, the frequency bands include delta, theta, alpha, beta, and gamma; S303, based on the preprocessed EEG data obtained in S107, drawing a visualization of the spatial domain characteristics of the subject's EEG signal, and obtaining adjacency matrix diagrams of brain network functional connections of different frequency bands.

6. The depression auxiliary diagnosis system according to claim 2, characterized in that: The working method of the detailed diagnosis mode module comprises the following steps: In the autonomous selection unit, steps S4 to S7 are performed in sequence, wherein step S4 is to autonomously select the EEG features to be extracted; step S5 is to autonomously select feature parameters; step S6 is to autonomously select a machine learning model; step S7 is to autonomously select model parameters; step S8 is performed in the model evaluation unit, and step S8 is to view the model recognition accuracy and visualization diagram and evaluation index statistical table; In step S4, the EEG features include AR parameter features, wPLI features, DWT features and FBCSP features. In step S5, the feature parameters include wavelet basis functions and the number of feature extraction channels. In step S6, the machine learning model includes a traditional machine learning model and a deep learning model; the traditional machine learning model includes an SVM model, a KNN model, an RF model, a GBT model and an XGBoost model. The traditional machine learning model is classified based on the EEG features selected in step S4 and the feature parameters selected in step S5. The deep learning model includes a CNN model, an AE model and an EEGNET model. The deep model is classified based on the segmented EEG data obtained in step S1, and the classification result is "healthy" or "depressed". In step S7, when the traditional machine learning model is S When the VM model is used, the optional model parameters are kernel function selection, including linear, polynomial, RBF and Sigmoid. When the traditional machine learning model is a KNN model, the optional model parameter is the number of neighbors. When the traditional machine learning model is an RF model, the optional model parameters are the number of decision trees and the maximum depth. When the traditional machine learning model is a GBT model, the optional model parameters are the learning rate and the number of base learners. When the traditional machine learning model is an XGBoost model, the optional model parameters are the learning rate and the number of base learners. When deep learning CNN, AE and EEGNET models are selected, the optional model parameters are the number of training times and the learning rate. In step S8, the visualization graph includes a confusion matrix graph and a ROC curve graph, and the evaluation indicators include the values ​​of macro average, micro average, weighted average, precision, recall and F1 score.

7. The depression auxiliary diagnosis system according to claim 1, characterized in that: The working method of the simple diagnosis mode module comprises the following steps: The single-modal EEG data processing unit performs step S9, which is specifically as follows: using the segmented EEG data obtained in step S106, calling the feature parameters and model parameters determined by pre-training, outputting the diagnosis results of the traditional machine learning models KNN, SVM, RF, GBT, XGBoost based on AR parameters, wPLI, and DWT features, and the diagnosis results of the deep learning models CNN, AE, and EEGNET, and the result is in the text form of "health" or "depression"; The single-mode speech data processing unit performs step S10, and step S10 is specifically as follows: using the spectrogram features in the picture format obtained in step S204, calling the ResNet model and the weight parameters determined by pre-training, and outputting the diagnosis result obtained by ResNet34 based on the transfer learning technology analysis, and the result is in the text form of "health" or "depression"; The EEG speech fusion data processing unit performs step S11, and step S11 is specifically as follows: S111, the clinical interview speech data of the person to be tested is downsampled and normalized; S112, the speech envelope is extracted using the Hilbert transform; S113, the speech envelope is segmented without overlap every 1 second; S114, the speech envelope is spliced ​​with the EEG data obtained after segmentation in step S106; S115, the EEG data and speech envelope of the same scale are input into the pre-trained Conformer model, and the Conformer model first extracts the low-order features of EEG and speech respectively through the convolution module, including using the convolution kernel for convolution operation, and using the average pooling layer to reduce the data complexity operation; after passing through the convolution module, the EEG features and speech features are spliced ​​in the channel dimension, and the fused sequence features are input into the attention module to capture the correlation between the features, and the extracted features are classified through the fully connected layer, and the classification form is the text form of "health" or "depression".

8. The depression auxiliary diagnosis system according to claim 2, characterized in that: The working method of the information input and storage unit in the patient information management module includes steps S12 to S15, wherein step S12 is for inputting the information of the person to be tested, and preferably, the patient information includes the name, gender, ID number, date of birth, contact information, remarks, last examination date, and whether the person suffers from other diseases; step S13 is for inputting the examination information, and preferably, the examination information includes the diagnosis result, examination method, examination findings, and diagnosis opinion; step S14 is for inputting the doctor information, and preferably, the doctor information includes the name of the reporting person and the name of the reviewing person; step S15 is for saving the information input in steps S12-S14; The working method of the diagnosis report output unit in the patient information management module includes steps S16 and S17, wherein step S16 is for downloading a PDF version of the diagnosis report; step S17 is for viewing a PDF version of the diagnosis report, wherein the diagnosis report includes the patient information of step S12, the examination information of step S13, the doctor information of step S14, and also includes the features and model names involved in the diagnosis process and the diagnosis results, as well as a specific description of the examination findings consisting of the original electroencephalogram, the electroencephalogram of delta, theta, alpha, beta, and gamma frequency bands, the brain network functional connection adjacency matrix diagram of delta, theta, alpha, beta, and gamma frequency bands, and the time domain waveform diagrams of the frontal area, the parietal area, and the occipital area.

9. The method for training a model in a depression auxiliary diagnosis system according to claim 1, characterized in that: The following steps are involved: Step a1, after obtaining the EEG data and speech data of healthy people and patients with depression, the EEG data are preprocessed through electrode positioning, whole-brain average reference, re-reference, filtering, ICA independent component analysis, interpolation bad leads, segmentation, and bad segment removal, wherein the EEG data after bad segment removal is used for fixed EEG feature extraction and visualization, and the EEG data before bad segment removal is used for classification and recognition of traditional machine learning models and deep learning models; Step a2, extract the time-frequency and spatial domain features of the preprocessed EEG data, analyze the EEG waveforms of the frontal area, parietal area and occipital area in the time domain, and extract AR parameters; extract the power spectrum of the EEG data based on the AR parameter model in the frequency domain to obtain the signal spectrum of a specific frequency band, and draw the EEG topography of different frequency bands, that is, the plane figure of the spherical scalp with different colors representing the power values ​​in each frequency band of the EEG wave; use DWT in the time-frequency domain to decompose the signal into approximation coefficients and detail coefficients of different scales, and extract the wavelet coefficients in each layer of decomposition; extract the wPLI feature in the spatial domain to study the phase coupling between different electrodes in the EEG data, draw the average adjacency matrix and brain network diagram of the delta, theta, alpha, beta and gamma frequency bands, calculate and draw the parameter indicators of the brain network, including local efficiency, clustering coefficient, average node degree, global efficiency and assortative coefficient, to understand the connection mode and information transmission path between brain regions; extract the FBCSP feature by combining the frequency domain and the spatial domain; Step a3, the AR parameters, DWT, FBCSP, and wPLI features extracted in step a2 are randomly divided into 80% training sets and 20% test sets, and the traditional machine learning models SVM, KNN, RF, GBT, and XGBoost models are selected for classification and recognition to find the optimal model parameters and feature parameters; the EEG data segmented in step a1 are divided into 80% training sets and 20% test sets for deep learning CNN, AE, and EEGNET models for pattern recognition; the parameters are tuned by grid search and cross-validation technology to find the optimal model parameter combination, and the confusion matrix and ROC curve are output. The classification accuracy, precision, recall rate, F1 score, ROC curve, and AUC are used to evaluate the classification effect of traditional machine learning models and deep learning models; Step a4: extracting speech spectrograms after preprocessing the original speech data of healthy people and patients with depression through filtering, segmentation, concatenation, and segmentation operations; Step a5: Classify and detect the speech spectrogram extracted in step a4 based on the deep residual network ResNet34 and transfer learning technology, and migrate the ResNet34 model from the original 1000-class image classification field to the binary classification of depression and health, and obtain a pre-trained ResNet34 model; Step a6: downsample and normalize the original speech data of healthy people and patients with depression, use Hilbert transform to extract the speech envelope, segment the speech envelope, concatenate the EEG and speech data and input them into the Conformer model, train the Conformer model to obtain a pre-trained Conformer model, and realize the processing of multimodal EEG speech fusion data and depression recognition, wherein: the Conformer model includes convolution layer, BatchNorm layer, ELU layer, Avgpool layer, Conv2d layer, LayerNorm layer, Dropout layer, LayerNorm layer, fully connected layer, and Flatten layer.

10. The data analysis method of the depression auxiliary diagnosis system according to claim 1, characterized in that: The following steps are included: Step 1: Data processing module performs data preprocessing: Obtaining EEG data and speech data of the person to be tested; The EEG data is re-referenced, filtered, ICA independent component analysis, interpolated, and segmented to obtain segmented EEG data, and then the bad segments are removed to obtain pre-processed EEG data. The time-frequency and spatial domain features of the pre-processed EEG data are extracted, and the time domain waveform, EEG topography, and brain network functional connection adjacency matrix are drawn; Preprocess the speech data to obtain the spectrogram features in image format; Step 2: Detailed diagnosis mode module performs detailed diagnosis: In the detailed diagnosis mode module, users can select EEG features, feature parameters, machine learning models, and model parameters on their own. Combined with the EEG data of healthy people and patients with depression that comes with the system or is uploaded by users, users can view the recognition accuracy, visualization graph, and evaluation index statistics of the machine learning model they selected. Step 3: Simple diagnosis mode module performs simple diagnosis: Single-modal EEG data processing: Use the segmented EEG data obtained in step 1, call the feature parameters and model parameters determined by pre-training, and output the diagnosis results of traditional machine learning models KNN, SVM, RF, GBT, XGBoost based on AR parameters, wPLI, and DWT features, and the diagnosis results of deep learning models CNN, AE, and EEGNET. The results are in the form of "healthy" or "depressed" text; Single-modal speech data processing: Use the spectrogram features in the image format obtained in step 1, call the pre-trained ResNet model and weight parameters, and output the diagnosis results obtained by ResNet34 based on transfer learning technology analysis. The result is in the form of "healthy" or "depressed" text; EEG and speech fusion data processing: The clinical interview speech data of the test subject is downsampled and normalized; the Hilbert transform is used to extract the speech envelope; the speech envelope is segmented without overlap every 1 second; the speech envelope is spliced ​​with the segmented EEG data obtained in step 1; the EEG data and speech envelope of the same scale are input into the pre-trained Conformer model, which first extracts the low-order features of EEG and speech respectively through the convolution module, including the use of convolution kernels for convolution operations and the use of average pooling layers to reduce the complexity of data operations; after the convolution module, the EEG features and speech features are spliced ​​in the channel dimension, and the fused sequence features are input into the attention module to capture the correlation between the features, and the extracted features are classified through the fully connected layer in the form of "healthy" or "depressed" text; Step 4: Enter information in the patient information management module to generate a PDF version of the diagnosis report: Enter the patient information, examination information, and doctor information in the patient information management to generate a PDF version of the diagnosis report, which includes the features and model names involved in the diagnosis process and the diagnosis results, as well as a specific description of the examination findings consisting of the original EEG, EEG topography of delta, theta, alpha, beta, and gamma frequency bands, and adjacency matrix diagrams of brain network functional connections in delta, theta, alpha, beta, and gamma frequency bands, and time domain waveform diagrams of the frontal, parietal, and occipital regions.

Images