Depression identification auxiliary decision-making system based on multi-modal fusion

Abstract

The invention discloses a depression identification auxiliary decision-making system based on multi-modal fusion. The depression identification auxiliary decision-making system comprises a data acquisition module; a data processing module; a feature extraction module; a depression identification model module; the model training module is used for training the corresponding depression recognition model module; the model integration module is used for fusing the plurality of groups of trained depression recognition model modules together and establishing a multi-mode depression recognition model for depression recognition; according to the method, features can be extracted from multiple information sources, data of different modes have complementarity in the aspect of capturing human physiological and psychological states, and the comprehensive data processing enables the system to more comprehensively capture the physiological and psychological states of a depression patient; depression recognition is carried out by extracting the multi-modal physiological data and constructing the multi-modal data fusion model, the defects of a single-modal data recognition method are overcome, and the depression recognition accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of depression identification, and in particular to a depression identification auxiliary decision-making system based on multimodal fusion. Background Art

[0002] Depression is a common mental disorder characterized by persistent frustration and loss of interest, which greatly affects the patient's health and even life safety. At present, the most common clinical diagnosis method for depression is for professional doctors to conduct interviews and assessments using depression scales (such as the Hamilton Depression Rating Scale (HAMD)). However, such a diagnostic method first requires that the doctor has received professional and good training. Secondly, the evaluation results obtained are also subjective and may lead to missed diagnoses or misdiagnoses. Therefore, in order to intervene in a timely manner and effectively prevent and treat depression symptoms, it is particularly important to adopt objective and reliable depression diagnosis techniques;

[0003] In recent years, there have been some depression recognition methods based on EEG, eye movement, brain imaging, speech, etc. that can reflect biological and psychological changes. These methods are of great significance in reducing subjective bias; however, such methods obviously have some limitations. On the one hand, due to the differences in individual physiological data, the method of diagnosing depression through single-modality physiological data often has poor generalization performance; at the same time, a single physiological indicator does not fully reflect the characteristics of depression, and it is impossible to construct an objective and effective model based on it, so the accuracy of depression recognition is low. Summary of the invention

[0004] The purpose of the present invention is to provide a depression identification decision-making auxiliary system based on multimodal fusion to solve the problem.

[0005] To achieve the above object, the present invention provides the following technical solution: a depression identification decision-making auxiliary system based on multimodal fusion, comprising:

[0006] A data acquisition module, the data acquisition module is used to obtain multiple sets of physiological data of the subject under the stimulation state;

[0007] A data processing module, which is used to pre-process the collected multiple sets of physiological data, including cleaning, sorting and labeling;

[0008] A feature extraction module, the feature extraction module is used to extract multiple groups of corresponding features that are helpful for depression identification from multiple groups of preprocessed physiological data;

[0009] A depression recognition model module is used to establish a corresponding depression recognition model module based on each set of data in the plurality of sets of physiological data;

[0010] A model training module, wherein the module training module is used to insert features that are helpful for depression identification into a corresponding depression identification model module, and the corresponding depression identification model module is trained. During the training process, the parameters of the corresponding depression identification model module are adjusted;

[0011] The model integration module is used to fuse multiple groups of trained depression recognition model modules together to establish a multimodal depression recognition model for depression recognition.

[0012] The multiple sets of physiological data include eye movement data, electrocardiogram data, electroencephalogram data, facial expression and posture data, and questionnaire data.

[0013] The data processing module cleans multiple sets of physiological data, specifically:

[0014] De-noising was performed on multiple sets of physiological data. For questionnaire data, useless punctuation and special characters were deleted. For eye movement data, ECG data, EEG data, facial expression and posture data, outliers caused by equipment errors were deleted.

[0015] The denoised or multiple sets of physiological data are converted into a standard format, the questionnaire data are encoded in a unified character code, and the eye movement data, electrocardiogram data, electroencephalogram data, facial expression and posture data are converted into the standard unit of sampling per minute.

[0016] The data processing module organizes multiple sets of physiological data, specifically:

[0017] Screen multiple groups of physiological data after denoising to find missing data, and then use the average value to fill in the missing data;

[0018] After multiple sets of physiological data are complete, data points are aligned in time series to ensure consistency of time nodes.

[0019] The data processing module labels multiple sets of physiological data, specifically:

[0020] In the sorted sets of physiological data, the depression values ​​that can represent depression are marked;

[0021] Then, we increase the diversity of the labeled depression values ​​by rotating, flipping, and cropping;

[0022] Then the increased annotated depression values ​​are divided into training set, validation set and test set;

[0023] Among them, the training set is input into the multimodal depression recognition model for model training, the validation set is input into the multimodal depression recognition model for depression result verification, and the test set is input into the multimodal depression recognition model for model testing.

[0024] The feature extraction module extracts multiple groups of corresponding features that are helpful for identifying depression, specifically:

[0025] For eye movement data, gaze duration features, scanning speed features, pupil size features and blink count features are extracted; the concentration of patients with depression is lower, and the data of gaze duration is lower than normal time, which is the gaze duration feature that needs to be extracted; the scanning speed of patients with depression is slower, and the data of scanning speed lower than that of normal people is the scanning speed feature that needs to be extracted; the pupil changes of patients with depression are not obvious, and the pupil size changes are lower than that of normal people, which is the pupil size feature that needs to be extracted; the blinking frequency of patients with depression decreases, and the data of blinking frequency lower than that of normal people is the blinking frequency feature that needs to be extracted.

[0026] The feature extraction module extracts multiple groups of corresponding features that are helpful for identifying depression, specifically:

[0027] For ECG data, the ECG variation amplitude feature is extracted. Depression patients have greater mood swings, and data with greater ECG variation amplitude are the ECG variation amplitude features that need to be extracted.

[0028] The feature extraction module extracts multiple groups of corresponding features that are helpful for identifying depression, specifically:

[0029] For EEG data, the EEG change amplitude characteristics are extracted. Depressed patients have greater mood swings, and data with larger EEG change amplitudes are the EEG change amplitude characteristics that need to be extracted.

[0030] The feature extraction module extracts multiple groups of corresponding features that are helpful for identifying depression, specifically:

[0031] For facial expression and posture data, the facial expression and posture change amplitude features are extracted. Depressed patients have less expression changes, and the data with smaller facial expression and posture change amplitudes are the facial expression and posture change amplitude features that need to be extracted.

[0032] The feature extraction module extracts multiple groups of corresponding features that are helpful for identifying depression, specifically:

[0033] For questionnaire data, the pathological information characteristics of depression in the questionnaire were extracted.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The multimodal fusion model proposed in the present invention not only has improved stability, but also has improved performance due to the introduction of multimodal information; it can extract features from multiple information sources, including eye movement data, electrocardiogram data, electroencephalogram data, facial expression and posture data, and questionnaire data. Data of different modalities are complementary in capturing human physiological and psychological states. This comprehensive data processing enables the system to more comprehensively capture the characteristics of the physiological and psychological states of patients with depression. By extracting multimodal physiological data and constructing a multimodal data fusion model for depression identification, the shortcomings of single-modal data identification methods are overcome, and the accuracy of depression identification is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the system flow structure of the present invention. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0038] Embodiment 1

[0039] See also Figure 1 The present invention provides a technical solution: a depression identification decision-making auxiliary system based on multimodal fusion, comprising:

[0040] A data acquisition module, wherein the data acquisition module is used to obtain multiple sets of physiological data of the subject under stimulation; the multiple sets of physiological data include eye movement data, electrocardiogram data, electroencephalogram data, facial expression and posture data, and questionnaire data;

[0041] A data processing module, which is used to pre-process the collected multiple sets of physiological data, including cleaning, sorting and labeling;

[0042] The data processing module cleans multiple sets of physiological data, specifically:

[0043] De-noising was performed on multiple sets of physiological data. For questionnaire data, useless punctuation and special characters were deleted. For eye movement data, ECG data, EEG data, facial expression and posture data, outliers caused by equipment errors were deleted.

[0044] The denoised or multiple sets of physiological data are converted into a standard format, the questionnaire data are encoded in a unified character code, and the eye movement data, electrocardiogram data, electroencephalogram data, facial expression and posture data are converted into the standard unit of sampling per minute.

[0045] Among them, the data processing module organizes multiple sets of physiological data, specifically:

[0046] Screen multiple groups of physiological data after denoising to find missing data, and then use the average value to fill in the missing data;

[0047] After multiple sets of physiological data are complete, data points are aligned in time series to ensure consistency of time nodes.

[0048] Among them, the data processing module labels multiple groups of physiological data, specifically:

[0049] In the sorted sets of physiological data, the depression values ​​that can represent depression are marked;

[0050] Then, we increase the diversity of the labeled depression values ​​by rotating, flipping, and cropping;

[0051] Then the increased annotated depression values ​​are divided into training set, validation set and test set;

[0052] Among them, the training set is input into the multimodal depression recognition model for model training, the validation set is input into the multimodal depression recognition model for depression result verification, and the test set is input into the multimodal depression recognition model for model testing.

[0053] A feature extraction module is used to extract multiple sets of corresponding features that are helpful for depression identification from multiple sets of preprocessed physiological data;

[0054] The feature extraction module extracts multiple sets of corresponding features that are helpful for identifying depression, specifically:

[0055] For eye movement data, the features of fixation duration, scanning speed, pupil size and blink count are extracted; the data of patients with depression whose concentration is lower and whose fixation duration is lower than that of normal people are the features of fixation duration that need to be extracted; the data of patients with depression whose scanning speed is slower and whose scanning speed is lower than that of normal people are the features of scanning speed that need to be extracted; the pupil changes of patients with depression are not obvious and whose pupil size changes are lower than that of normal people are the features of pupil size that need to be extracted; the data of patients with depression whose blink count is less and whose blink count is lower than that of normal people are the features of blink count that need to be extracted;

[0056] For ECG data, extract ECG variation amplitude features. Depression patients have greater mood swings, and data with greater ECG variation amplitudes are the ECG variation amplitude features that need to be extracted.

[0057] For EEG data, extract the EEG change amplitude feature. Depressed patients have large mood swings, and the data with large EEG change amplitude is the EEG change amplitude feature that needs to be extracted;

[0058] For facial expression and posture data, extract the facial expression and posture change amplitude features. Depressed patients have less expression changes, and the facial expression and posture change amplitude data with smaller facial expression and posture change amplitude features need to be extracted;

[0059] For questionnaire data, the pathological information characteristics of depression in the questionnaire were extracted.

[0060] A depression recognition model module is used to establish a corresponding depression recognition model module based on each set of data in the plurality of sets of physiological data;

[0061] A model training module, wherein the module training module is used to insert features that are helpful for depression identification into a corresponding depression identification model module, and the corresponding depression identification model module is trained. During the training process, the parameters of the corresponding depression identification model module are adjusted;

[0062] The model integration module is used to fuse multiple groups of trained depression recognition model modules together to establish a multimodal depression recognition model for depression recognition.

[0063] Embodiment 2

[0064] Based on Example 1, the specific implementation steps of this application are:

[0065] Step 1: First, multiple sets of physiological data of the subject under stimulation are obtained through a data acquisition module;

[0066] Step 2: Then, the collected multiple sets of physiological data are preprocessed by the data processing module, and the preprocessing includes cleaning, sorting and labeling;

[0067] Step 3, then extracting multiple groups of corresponding features that are helpful for depression identification from the preprocessed multiple groups of physiological data through a feature extraction module;

[0068] Step 4: Based on each set of data in the multiple sets of physiological data, a corresponding depression recognition model module is established;

[0069] Step 5: Then, the module training module inserts the features that are helpful for depression identification into the corresponding depression identification model module, and the corresponding depression identification model module is trained. During the training process, the parameters of the corresponding depression identification model module are adjusted;

[0070] Step 6: Then, multiple groups of trained depression recognition model modules are integrated together through a model integration module to establish a multimodal depression recognition model for depression recognition. Based on the multimodal depression recognition model, depression recognition is performed on the subjects.

[0071] In summary, the multimodal fusion model proposed in this application not only has improved stability, but also has improved performance due to the introduction of multimodal information; it can extract features from multiple information sources, including eye movement data, electrocardiogram data, electroencephalogram data, facial expression and posture data, and questionnaire data. Data of different modalities are complementary in capturing human physiological and psychological states. This comprehensive data processing enables the system to more comprehensively capture the characteristics of the physiological and psychological states of patients with depression. By extracting multimodal physiological data and constructing a multimodal data fusion model for depression identification, the shortcomings of single modality data identification methods are overcome, and the accuracy of depression identification is improved.

[0072] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.

Claims

1. A depression identification decision-making support system based on multimodal fusion, characterized in that: include: A data acquisition module, the data acquisition module is used to obtain multiple sets of physiological data of the subject under the stimulation state; A data processing module, which is used to pre-process the collected multiple sets of physiological data, including cleaning, sorting and labeling; A feature extraction module, the feature extraction module is used to extract multiple groups of corresponding features that are helpful for depression identification from multiple groups of preprocessed physiological data; A depression recognition model module is used to establish a corresponding depression recognition model module based on each set of data in the plurality of sets of physiological data; A model training module, wherein the module training module is used to insert features that are helpful for depression identification into a corresponding depression identification model module, and the corresponding depression identification model module is trained. During the training process, the parameters of the corresponding depression identification model module are adjusted; The model integration module is used to fuse multiple groups of trained depression recognition model modules together to establish a multimodal depression recognition model for depression recognition.

2. The depression identification decision support system based on multimodal fusion according to claim 1, characterized in that: The multiple sets of physiological data include eye movement data, electrocardiogram data, electroencephalogram data, facial expression and posture data, and questionnaire data.

3. The depression identification decision support system based on multimodal fusion according to claim 2, characterized in that: The data processing module cleans multiple sets of physiological data, specifically: De-noising was performed on multiple sets of physiological data. For questionnaire data, useless punctuation and special characters were deleted. For eye movement data, ECG data, EEG data, facial expression and posture data, outliers caused by equipment errors were deleted. The denoised or multiple sets of physiological data are converted into a standard format, the questionnaire data are encoded in a unified character code, and the eye movement data, electrocardiogram data, electroencephalogram data, facial expression and posture data are converted into the standard unit of sampling per minute.

4. The depression identification decision support system based on multimodal fusion according to claim 3 is characterized by: The data processing module organizes multiple sets of physiological data, specifically: Screen multiple groups of physiological data after denoising to find missing data, and then use the average value to fill in the missing data; After multiple sets of physiological data are complete, data points are aligned in time series to ensure consistency of time nodes.

5. The depression identification decision support system based on multimodal fusion according to claim 4, characterized in that: The data processing module labels multiple groups of physiological data, specifically: In the sorted sets of physiological data, the depression values ​​that can represent depression are marked; Then, we increase the diversity of the labeled depression values ​​by rotating, flipping, and cropping; Then the increased annotated depression values ​​are divided into training set, validation set and test set; Among them, the training set is input into the multimodal depression recognition model for model training, the validation set is input into the multimodal depression recognition model for depression result verification, and the test set is input into the multimodal depression recognition model for model testing.

6. The depression identification decision support system based on multimodal fusion according to claim 2, characterized in that: The feature extraction module extracts multiple groups of corresponding features that are helpful for identifying depression, specifically: For eye movement data, gaze duration features, scanning speed features, pupil size features and blink count features are extracted; the concentration of patients with depression is lower, and the data of gaze duration is lower than normal time, which is the gaze duration feature that needs to be extracted; the scanning speed of patients with depression is slower, and the data of scanning speed lower than that of normal people is the scanning speed feature that needs to be extracted; the pupil changes of patients with depression are not obvious, and the pupil size changes are lower than that of normal people, which is the pupil size feature that needs to be extracted; the blinking frequency of patients with depression decreases, and the data of blinking frequency lower than that of normal people is the blinking frequency feature that needs to be extracted.

7. The depression identification decision support system based on multimodal fusion according to claim 2, characterized in that: The feature extraction module extracts multiple groups of corresponding features that are helpful for identifying depression, specifically: For ECG data, the ECG variation amplitude feature is extracted. Depression patients have greater mood swings, and data with greater ECG variation amplitude are the ECG variation amplitude features that need to be extracted.

8. The depression identification decision support system based on multimodal fusion according to claim 2, characterized in that: The feature extraction module extracts multiple groups of corresponding features that are helpful for identifying depression, specifically: For EEG data, extract the EEG change amplitude feature. Depressed patients have greater mood swings, and data with larger EEG change amplitudes are the EEG change amplitude features that need to be extracted.

9. The depression identification decision support system based on multimodal fusion according to claim 2, characterized in that: The feature extraction module extracts multiple groups of corresponding features that are helpful for identifying depression, specifically: For facial expression and posture data, the facial expression and posture change amplitude features are extracted. Depressed patients have less expression changes, and the data with smaller facial expression and posture change amplitudes are the facial expression and posture change amplitude features that need to be extracted.

10. The depression identification decision support system based on multimodal fusion according to claim 2, characterized in that: The feature extraction module extracts multiple groups of corresponding features that are helpful for identifying depression, specifically: For questionnaire data, the pathological information characteristics of depression in the questionnaire were extracted.

Images