Depressive disorder recognition system based on multi-modal domain adaptation

Abstract

The invention discloses a depressive disorder recognition system based on multi-modal domain adaptation, and the system achieves the dynamic fusion and parameter optimization of signals through modeling the interaction relation of electroencephalogram, skin electricity and electrocardio signals into a deep learning framework. According to the method, complementary information of multi-modal signals is fully utilized, so that the model can extract more discriminative features from complex physiological data, and the accuracy and robustness of depressive disorder recognition are effectively improved; the cross-domain common features are successfully mined by drawing the data distribution of the source domain and the target domain twice; according to the cross-domain alignment strategy, the adaptive capacity of the model in different data domains is effectively improved, and the negative influence caused by domain offset is remarkably reduced. Generally speaking, the recognition method provided by the invention shows remarkable advantages in the aspects of improving the depressive disorder detection accuracy and enhancing the model generalization ability.

Description

Technical Field

[0001] The present invention belongs to the technical fields of intelligent medicine, psychophysiological computing, transfer learning, etc., and specifically relates to a depressive disorder recognition system based on multimodal domain adaptation. Background Art

[0002] Depressive disorder is a mental illness characterized by persistent low mood and loss of interest. With the increasing pressure of life and fierce competition in modern society, the incidence of depressive disorder has increased significantly and has become an important public health issue worldwide. According to statistics, more than 350 million people worldwide suffer from depressive disorder, which seriously affects the quality of life and work ability of individuals. Depressive disorder not only causes significant damage to emotional, cognitive and behavioral functions, but is also often accompanied by other mental or physical diseases, which imposes a huge burden on society and the economy. Although existing clinical methods can provide certain treatments for depressive disorders, due to the lack of effective objective diagnostic criteria and early identification methods, many patients' conditions have not been discovered and intervened in time, affecting the treatment effect. Therefore, the development of an efficient and objective method for identifying and evaluating depressive disorders is of great significance for improving the accuracy of disease identification, formulating personalized treatment plans, and reducing the social and economic burden brought by depressive disorders.

[0003] Depressive disorders not only affect the emotional state of individuals, but also have a profound impact on the emotional regulation mechanism. Emotional regulation is driven by a complex interaction between the central nervous system and the autonomic nervous system. When an individual is emotionally stimulated, the sensory system captures relevant information and triggers a series of physiological reactions. These reactions regulate physiological functions such as heart rate, blood pressure, and respiration through the sympathetic and parasympathetic nervous systems in the autonomic nervous system, and also trigger responses from the central nervous system, reflecting the individual's emotional state. Due to the impact of depressive disorders on the emotional regulation mechanism, it is of great significance to study the physiological signals of depressive disorders. EEG signals, skin electrodermal signals, and electrocardiogram signals are all important sources of physiological data for studying depressive disorders. EEG signals can directly capture the electrical activity of the central nervous system and provide detailed information about the dynamics of brain function and neural network activity. Skin electrodermal signals can reflect the activity of the autonomic nervous system and reveal the impact of emotional state on skin conductivity. Electrocardiogram signals provide key information about heart activity and indirectly reflect the regulatory effect of emotions on the autonomic nervous system. Although each physiological signal provides unique and valuable information, a single modality signal is often insufficient to fully capture the complex emotional state of an individual. In order to improve the accuracy and reliability of identifying depressive disorders, it is particularly important to comprehensively analyze multimodal data such as EEG signals, skin electrical signals and ECG signals.

[0004] Although the application of multimodal physiological signals in the identification of depressive disorders has made some progress, there are still many difficulties in how to effectively fuse data from different modalities and deal with the challenges brought by individual differences. EEG signals, skin electrical signals, and ECG signals each have unique physiological characteristics, and the relationships between them are complex and changeable. Multimodal domain adaptation technology provides new possibilities for solving these challenges. This technology aims to achieve effective integration between signals and improve the accuracy of depressive disorder identification by effectively fusing data from different modalities and aligning features in different domains. By learning and optimizing common features between different domains, the adaptability of the depressive disorder identification model can be improved. However, achieving this goal also faces challenges such as how to ensure the consistency of cross-modal information, reduce noise interference, and improve the generalization ability of the model on diverse data. In addition, existing studies often ignore the potential correlation and synergy between multimodal data, and fail to fully utilize the complementarity of these data to improve the effect of depressive disorder identification. Therefore, the present invention focuses on in-depth exploration of the interrelationships and synergies between data of different modalities, enhancing the accuracy and robustness of depressive disorder identification by optimizing multimodal domain adaptation strategies, and providing more comprehensive and scientific support for clinical diagnosis. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a depressive disorder recognition system based on multimodal domain adaptation, which collects multimodal data, including EEG signals, skin electrical conductance and ECG signals, and combines multimodal domain adaptation technology to improve the fusion ability between different data modalities and the feature alignment ability between different domains, thereby improving the accuracy and robustness of depressive disorder recognition. This method uses a deep transfer learning algorithm to achieve optimization in a multimodal feature space, enhance the recognition ability of depressive disorders, adapt to the physiological differences of different individuals, and effectively support real-time analysis and diagnosis.

[0006] A depression disorder recognition system based on multimodal domain adaptation, comprising a data acquisition module, a feature extraction module, a multimodal domain adaptation module, a neural network classification module, and a model training and depression detection module;

[0007] The data acquisition module is used to collect three modal signals of EEG, EG and ECG of the subjects in a resting state and an audio stimulation state;

[0008] The feature extraction module is used to extract the features of EEG, EGG and ECG signals;

[0009] The multimodal domain adaptation module is used to:

[0010] The three modal signal data to be used are divided into source domain data and target domain data;

[0011] For each source domain and target domain data, the EEG signal features, skin electrical signal features, and ECG signal features are used to form a tensor, which is then decomposed by CP, specifically expressed as:

[0012]

[0013] Among them, S represents the source domain feature tensor, T represents the target domain feature tensor; R is the rank of decomposition, which indicates the number of components used in the decomposition; Represents the scalar weight corresponding to the decomposition of source domain features, They represent the interactive features of EEG signals, skin electrical signals, and ECG signals at different ranks when the source domain features are decomposed, and they are all parameters that need to be learned; correspondingly, Represents the scalar weight corresponding to the decomposition of the target domain features, They respectively represent the interactive features of EEG signals, galvanic skin signals, and ECG signals at different ranks when the source and target domain features are decomposed, and are also parameters that need to be learned.

[0014] After tensor decomposition, the first alignment loss is calculated:

[0015]

[0016] Among them, φ() is the feature mapping function, which means mapping the original feature space into a higher-dimensional space; S i represents the i-th sample feature in the source domain feature S, T j represents the jth sample feature in the target domain feature T; n S and n T are the number of samples in the source domain and the target domain respectively;

[0017] The attention mechanism is used to extract relevant information between the source domain and the target domain. The specific steps are as follows:

[0018] Extract information related to the source domain S from the target domain T and use the attention mechanism:

[0019]

[0020] Construct a new target domain feature T′=T+A containing common information;

[0021] Among them, Q is the query vector, which is used to find relevant information; K is the key vector, which is used to match the query vector to find relevant information; V is the value vector, which represents the matched information, which is the final information of interest; d k is the dimension of the key;

[0022] For the input source domain features S and target domain features T, the corresponding query vector, key vector and value vector are obtained through linear transformation:

[0023] Q S =SW Q ,K S =SW K ,V S =SW V ;

[0024] Q T =TW Q ,K T =TW K ,V T =TW V ;

[0025] Among them, W Q , W K and W V is the weight matrix to be learned;

[0026] Extract information related to the target domain T from the source domain S and use the attention mechanism:

[0027]

[0028] Construct a new source domain feature S′=S+B containing common information;

[0029] Calculate the second alignment loss, expressed as:

[0030]

[0031] Among them, S′ i represents the i-th sample feature in the new source domain feature S′, T′ j represents the jth sample feature in the new target domain feature T′, n S′ and n T′ are the number of new source domain and target domain samples respectively;

[0032] The neural network classification module is used to classify the input feature data to obtain a classification result;

[0033] The model training and depression detection module is used to:

[0034] 1) Model training:

[0035] The source domain data is input into the neural network classification module, and the binary cross entropy loss is calculated based on the obtained classification results:

[0036]

[0037] Among them, y i is the true label of the feature data, is the predicted label;

[0038] Finally, the total loss function is calculated by combining dynamic fusion and cross-domain alignment to optimize the entire model:

[0039] L=λ1L mmd1 +λ2L mmd2 +L cls

[0040] Among them, λ1 and λ2 are hyperparameters used to balance the impact of different loss terms;

[0041] Based on the loss function L that combines dynamic fusion and cross-domain alignment, the neural network classification module and the parameters that need to be learned are trained, and the model weights are updated through the back-propagation algorithm to finally obtain an optimized depression detection system.

[0042] 2) Depression detection:

[0043] In practical applications, the multimodal signals collected from new subjects are input into the feature extraction module for preprocessing, and then input into the trained neural network classification module to generate classification results.

[0044] 2. A depressive disorder identification system based on multimodal domain adaptation as claimed in claim 1, characterized in that the model training and depression detection module also includes detection of the neural network classification module:

[0045] The classification performance of the neural network classification module is tested using source domain data and corresponding labels.

[0046] Furthermore, it also includes a data preprocessing module, including power frequency denoising, bandpass filtering, artifact removal and downsampling of EEG signals, low-pass filtering and smoothing of skin electrical signals, and power frequency denoising, bandpass filtering and smoothing of ECG signals to improve signal quality.

[0047] Preferably, the feature extraction module first performs manual feature extraction based on prior knowledge on the EEG signal, skin electrical signal and ECG signal, and then extracts deep features through a Transformer encoder.

[0048] Furthermore, the feature extraction module is also used for:

[0049] Three types of linear features and three types of nonlinear features are extracted from EEG signals; the linear features include mean value, power spectrum density, and center frequency; the nonlinear features include sample entropy, approximate entropy, and Lyapunov exponent.

[0050] Furthermore, the feature extraction module is also used for:

[0051] Heart rate variability, frequency ratio and fractal dimension features are extracted from ECG signals.

[0052] Furthermore, the feature extraction module is also used for:

[0053] The skin conductance level, peak number and variance characteristics are extracted from the skin electrical signal.

[0054] Preferably, the neural network classification module is composed of multiple fully connected layers, and each fully connected layer is connected to an activation function.

[0055] Furthermore, the neural network classification module also introduces batch normalization and Dropout technology to accelerate convergence and stabilize the training process.

[0056] The present invention has the following beneficial effects:

[0057] The present invention aims to solve the challenges of low recognition rate of single-modal physiological signals in depressive disorder detection and individual differences. Since different individuals have significant differences in physiological signal characteristics, these challenges are particularly prominent in the identification of depressive disorders. Therefore, a method for identifying depressive disorders based on multimodal domain adaptation is proposed. In the present invention, the domain specifically refers to the feature space of physiological signals of different individuals. By aligning the feature spaces between these different individuals, multimodal domain adaptation can overcome the influence of individual differences on the recognition effect, thereby improving the accuracy and robustness of depressive disorder identification (see Appendix 1). Figure 1 ). By integrating multimodal data such as EEG signals, skin electrical signals, and ECG signals, effective fusion between signals is achieved, and the complementary information of different modal signals is used to improve recognition accuracy. In addition, domain adaptation technology is introduced to solve the problem of data distribution deviation caused by individual differences, thereby enhancing the generalization ability and stability of the model. This method provides more efficient and accurate technical support for clinical applications.

[0058] The depressive disorder recognition method based on multimodal domain adaptation of the present invention realizes dynamic fusion and parameter optimization between signals by modeling the interactive relationship between EEG, skin electricity and ECG signals into a deep learning framework. The method makes full use of the complementary information of multimodal signals, so that the model can extract more discriminative features from complex physiological data, effectively improving the accuracy and robustness of depressive disorder recognition. In addition, the present invention successfully mines cross-domain common features by pulling the source domain and target domain data distribution closer twice. The first alignment uses the maximum mean difference to ensure the consistency of the source domain and the target domain in the feature space; secondly, the information features related to the target are further deeply mined and fused through the attention mechanism to construct a more accurate feature representation. This cross-domain alignment strategy effectively improves the adaptability of the model in different data domains and significantly reduces the negative impact of domain shift. In general, the recognition method provided by the present invention shows significant advantages in improving the accuracy of depressive disorder detection and enhancing the generalization ability of the model. It not only proves the effectiveness of multimodal fusion and domain adaptation, but also provides an innovative solution for the automated detection of depressive disorders in practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is a flow chart of the method for identifying depressive disorders based on multimodal domain adaptation of the present invention;

[0060] Figure 2 It is a schematic diagram of the position of the 3-conductance electrode, the position of the skin electricity and the position of the heart electricity;

[0061] Figure 3 Flowchart of the framework for identification of depressive disorders based on multimodal domain adaptation. DETAILED DESCRIPTION

[0062] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0063] The present invention proposes a depression disorder recognition system based on multimodal domain adaptation. First, the multimodal electrophysiological signals in resting state and audio stimulation state are preprocessed and corresponding features are extracted, wherein the extracted features are the commonly used artificial features and deep features of the corresponding modality; then, the dynamic interaction relationship between EEG signals, skin electrical signals, and ECG signals is modeled through CP tensor synthesis and decomposition; secondly, the model is trained by mining the common features across domains through the two maximum mean differences and attention mechanism; finally, the test set is used for depression detection to obtain the expected depression detection recognition rate. Figure 1 As shown, the system includes the following modules:

[0064] (1) Data collection module, collects the subject's number, name, gender, age and other information, and simultaneously collects the subject's EEG, EG and ECG signals in the resting state and audio stimulation state. The locations for collecting EEG, EG and ECG are as shown in the attached figure. Figure 2 .

[0065] In this embodiment, a self-developed three-channel EEG device (see attached Figure 2 Left) The EEG signal is collected by the software, and the skin electricity and ECG signals are collected by the secondary developed software. The two softwares record the timestamp at the same time to ensure that the data are aligned in time. All subjects are college students with normal hearing and intelligence and no other history of mental illness. No psychotropic drugs were taken before data collection, and all subjects were tested under the same laboratory conditions. The sampling rate of the EEG device is 250Hz, and the sampling rate of the skin electricity and ECG devices is 50Hz.

[0066] (2) Data preprocessing module, including power frequency denoising, bandpass filtering, artifact removal and downsampling of EEG signals, low-pass filtering and smoothing of skin electrical signals, and power frequency denoising, bandpass filtering and smoothing of ECG signals to improve signal quality.

[0067] In this embodiment, a notch filter is first used to remove 50Hz power line interference to eliminate power frequency noise. Next, the EEG signal, skin electrode signal and ECG signal are bandpass filtered, and the filter ranges are set to 0.5-50Hz, 0.5-2Hz and 0.5-40Hz, respectively, to retain the effective signal frequency band. Subsequently, artifact removal is performed to detect and remove abnormal data caused by motion artifacts or electrooculographic artifacts, thereby improving the quality of the signal. In order to reduce random fluctuations in the signal, the moving average technique is applied to smooth the signal and enhance the stability of the data. Finally, the EEG sampling rate is reduced from 250Hz to 100Hz by downsampling to reduce the amount of data and retain key information.

[0068] (3) Feature extraction module: extracts manual features based on prior knowledge from the preprocessed EEG signals, skin electrodermal signals, and ECG signals, and extracts deep features through the Transformer encoder. EEG signals: extract three types of linear features and three types of nonlinear features. Linear features include mean, power spectrum density, and center frequency; nonlinear features include sample entropy, approximate entropy, and Lyapunov exponent. ECG signals: extract features such as heart rate variability, frequency ratio, and fractal dimension. Skin electrodermal signals: extract features such as skin conductance level, number of peaks, and variance.

[0069] (4) The multimodal domain adaptation module first models the dynamic interaction between multimodal physiological signals through tensor synthesis and decomposition, and embeds it into the deep learning training process to optimize the model parameters. Then, it mines the common information of the source domain and the target domain, and uses the maximum mean difference method for cross-domain alignment, so as to train a model that performs well in the target domain, so that the source domain features can be effectively applied in the target domain.

[0070] (5) The neural network classification module consists of several fully connected layers, each of which is followed by an activation function to introduce nonlinearity and avoid the gradient vanishing problem. In addition, in order to enhance the generalization ability of the network and prevent overfitting, regularization techniques such as batch normalization and Dropout are introduced into the network to accelerate convergence and stabilize the training process.

[0071] (6) Model training and depression detection module (see Appendix Figure 3 ), use the trained model and neural network classification module to detect depression in the newly collected data to achieve efficient classification and recognition.

[0072] The data acquisition module comprises the following steps:

[0073] 1) Experimental design: First, collect multimodal physiological signals of the subjects in the closed-eye resting state. Then conduct an auditory stimulation experiment, using a variety of auditory stimuli with different emotional attributes (for example, 2 positive stimuli, 2 neutral stimuli, 2 negative stimuli), each auditory stimulation lasts for a set time. After each auditory stimulation, the subjects will have a short rest time. During the whole process, continuously collect EEG signals, skin electrical signals, and ECG signals of the subjects in the closed-eye state, and finally obtain complete data.

[0074] 2) EEG acquisition: EEG equipment that complies with the internationally widely used 10-20 system electrode position standard is used to collect EEG signals.

[0075] 3) Multimodal electrophysiological acquisition: Through multimodal acquisition equipment, the synchronous and stable acquisition of other physiological signals is achieved. The equipment has been optimized to ensure the accuracy and stability of signal acquisition.

[0076] The power frequency denoising in the data preprocessing stage uses a notch filter to remove 50Hz power line interference.

[0077] In the data preprocessing stage, the bandpass filtering ranges of the EEG signal, the galvanic skin signal and the electrocardiogram signal are respectively 0.5-50 Hz, 0.5-2 Hz and 0.5-40 Hz.

[0078] Artifact removal in the data preprocessing stage is to detect and remove data anomalies caused by motion artifacts or electrooculographic artifacts to improve signal quality.

[0079] The smoothing in the data preprocessing stage is to reduce the random fluctuation of the signal through the moving average technology to improve the stability of the data.

[0080] Downsampling in the data preprocessing stage is to reduce the sampling rate from 250 Hz to 100 Hz.

[0081] The feature extraction module: performs manual feature extraction based on prior knowledge on the preprocessed EEG signals, skin electrodermal signals and ECG signals, and extracts deep features through the Transformer encoder. For EEG signals, three types of linear features (average, power spectral density, center frequency) and three types of nonlinear features (sample entropy, approximate entropy, Lyapunov exponent) are extracted. For ECG signals, features such as heart rate variability, frequency ratio and fractal dimension are extracted. For skin electrodermal signals, features such as skin conductance level, number of peaks and variance are extracted. Subsequently, deep feature extraction is performed through the Transformer encoder, and the complex temporal patterns and global dependencies in the signal are automatically learned using the self-attention mechanism.

[0082] Power Spectral Density:

[0083]

[0084] Where x(t) is the time domain signal and f is the frequency.

[0085] Center Frequency:

[0086]

[0087] Among them, f1 is the lower frequency limit and f2 is the upper frequency limit.

[0088] Sample entropy:

[0089]

[0090] Among them, A i is the number of sequence pairs with matching length m+1, B i is the number of sequence pairs with matching length m, r is the tolerance, and N is the signal length.

[0091] Approximate entropy:

[0092] ApEn(m,r,N)=φ m (r)-φ m+1 (r)

[0093] in, is the number of all sequence pairs that are similar to the template sequence.

[0094] Lyapunov exponent:

[0095]

[0096] in, is a small disturbance at the i-th moment.

[0097] Heart Rate Variability:

[0098]

[0099] Among them, RR i is the RR interval between adjacent heartbeats, is the mean of the RR intervals.

[0100] Frequency ratio:

[0101]

[0102] Among them, HFB represents a high frequency band range, and LFB represents a low frequency band range.

[0103] Fractal dimension:

[0104]

[0105] Among them, ∈ is the scale parameter of the fractal structure and N is the length of the signal.

[0106] Skin conductance level:

[0107]

[0108] Among them, SC i is the skin conductance value at the ith time point.

[0109] The multimodal domain adaptation module performs the following method steps:

[0110] 1) Dynamic Fusion:

[0111] First, use the EEG signal features (feature dimension: I), skin electrical signal features (feature dimension: J), and ECG signal features (feature dimension: K) to form a tensor X∈R I×J×K The synthesized tensor X captures the joint effect of the three signals in each feature dimension. In order to extract meaningful interaction information from this synthesized tensor, CP decomposition is used. CP decomposition decomposes a high-dimensional tensor into a combination of multiple low-dimensional factor vectors to express the intrinsic structure of the tensor, which is specifically expressed as:

[0112]

[0113] Where R is the rank of the decomposition, which indicates the number of components used in the decomposition; λ ris a scalar weight indicating the importance of different components; a r 、b r and c r are the corresponding factor vectors, which come from the factor matrices of EEG signal, galvanic skin signal, and ECG signal features. represents the outer product, a r 、b r and c r Respectively represent the interactive features of EEG signals, skin electrical signals, and ECG signals at different ranks. r 、a r 、b r and c r are the parameters updated in embedded deep learning.

[0114] In this embodiment, the multimodal data of the subjects is divided into 10 parts using 10-fold cross validation, of which 9 folds are used as source domain data and the remaining 1 fold is used as target domain data. In the training phase, we can only use source domain data and labels, and target domain data. In the test phase, the remaining 1 fold of data, including target domain data and labels, is used.

[0115] The construction of source domain features S and target domain features T is as follows:

[0116]

[0117] in, Represents the scalar weight corresponding to the decomposition of source domain features, They represent the interactive features of EEG signals, skin electrical signals, and ECG signals at different ranks when the source domain features are decomposed, and they are all parameters that need to be learned; correspondingly, Represents the scalar weight corresponding to the decomposition of the target domain features, They respectively represent the interactive features of EEG signals, galvanic skin signals, and ECG signals at different ranks when the source and target domain features are decomposed, and are also parameters that need to be learned.

[0118] The above decomposition process is to extract low-dimensional representations of multimodal data and capture the complex interactions between signals; the synthesis is to recombine low-dimensional factors into new, more semantic and easy-to-align feature representations. X is multimodal data, the source domain is multimodal data, and the target domain is also multimodal data. Both of them need to go through these two steps. The source domain is the source domain features that have been decomposed and synthesized, and the target domain data is the same.

[0119] 2) Cross-domain alignment:

[0120] Both the source domain and the target domain contain EEG signals, skin electrical signals, and ECG signals. The present invention uses the maximum mean difference for alignment. After tensor decomposition, the source domain features are represented as S and the target domain features are represented as T. The first alignment loss can be expressed as:

[0121]

[0122] Among them, φ() is the feature mapping function, which means mapping the original feature space into a higher-dimensional space (reproducing kernel Hilbert space). i represents the i-th sample feature in the source domain feature S, T j represents the jth sample feature in the target domain feature T; n s and n T are the number of samples in the source domain and the target domain, respectively.

[0123] In practice, feature mapping is implemented by kernel functions, such as the Gaussian kernel (RBF kernel):

[0124]

[0125] Among them, S is the bandwidth hyperparameter of the kernel function, s i ∈S i ,s j ∈T j .

[0126] The attention mechanism is used to extract relevant information between the source domain and the target domain. The specific steps are as follows:

[0127] Extract information related to the source domain S from the target domain T and use the attention mechanism:

[0128]

[0129] Construct a new target domain feature T′=T+A containing common information. Q is the query vector, which is used to find relevant information; K is the key vector, which is used to match the query vector to find relevant information; V is the value vector, which represents the matched information, which is the final information of interest. In the source domain, the features of the target domain are used as queries (Q) to match the features of the source domain (K), and then extract information related to the target domain in the source domain, and vice versa. For the input source and target domain features (S, T), Q, K, and V are obtained through linear transformation.

[0130] Q S =SW Q ,K S =SW K ,V S =SW V

[0131] Among them, W Q , W K and W V is the weight matrix to be learned, d k is the dimension of the key, and so is the target domain, that is:

[0132] Q T =TW Q ,K T =TW K ,V T =TW V ;

[0133] Extract information related to the target domain T from the source domain S and use the attention mechanism:

[0134]

[0135] Construct a new source domain feature S′=S+B containing common information.

[0136] The second alignment loss is expressed as:

[0137]

[0138] Among them, S i ′ represents the i-th sample feature in the new source domain feature S′, T j ′ represents the jth sample feature in the new target domain feature T′, n S′ and n T′ is the number of new source domain and target domain samples.

[0139] The neural network classification module classifies the input new source domain features S′ and new target domain features T′ to obtain a classification result;

[0140] The model training and depression detection module is used to:

[0141] 1) Model training and testing:

[0142] The model is trained using source domain data, and the binary cross entropy loss is used for the classification task in the source domain:

[0143]

[0144] Among them, y i is the true label, is the predicted label.

[0145] Finally, the loss functions of dynamic fusion and cross-domain alignment are combined to optimize the entire model:

[0146] L=λ1L mmd1 +λ2L mmd2 +L cls

[0147] Among them, λ1 and λ2 are hyperparameters used to balance the impact of different loss terms.

[0148] Based on the loss function L that combines dynamic fusion and cross-domain alignment, the neural network classification module and the parameters that need to be learned are trained, and the model weights are updated through the back-propagation algorithm to finally obtain an optimized depression detection system.

[0149] At this stage, the 10-fold cross-validation method is used to train the model to improve the generalization ability of the model. In this method, the data set is divided into ten equal parts, one of which is selected as the test set each time, and the remaining nine are used as the training set, and the validation is repeated multiple times. In each validation, the current reserved data is input into the neural network classifier to obtain the prediction result of the model. The model output is compared with the true label, and the classification performance indicators, including accuracy, sensitivity, and specificity, are calculated to comprehensively evaluate the detection effect of the model. The whole process will be carried out on the data of multiple subjects.

[0150] 2) Depression detection:

[0151] In actual applications, the multimodal signals (electroencephalogram, skin electricity, electrocardiogram, etc.) collected from new subjects are input into the feature extraction module for preprocessing, and then input into the trained neural network model to generate prediction results. By analyzing the output of the model, efficient depression classification and identification can be performed to assist clinical diagnosis and treatment decisions.

[0152] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A depressive disorder recognition system based on multimodal domain adaptation, characterized in that: It includes data acquisition module, feature extraction module, multimodal domain adaptation module, neural network classification module, and model training and depression detection module; The data acquisition module is used to collect three modal signals of EEG, EG and ECG of the subjects in a resting state and an audio stimulation state; The feature extraction module is used to extract the features of EEG, EGG and ECG signals; The multimodal domain adaptation module is used to: The three modal signal data to be used are divided into source domain data and target domain data; For each source domain and target domain data, the EEG signal features, skin electrical signal features, and ECG signal features are used to form a tensor, which is then decomposed by CP, specifically expressed as: Among them, S represents the source domain feature tensor, T represents the target domain feature tensor; R is the rank of decomposition, which indicates the number of components used in the decomposition; Represents the scalar weight corresponding to the decomposition of source domain features, They represent the interactive features of EEG signals, skin electrical signals, and ECG signals at different ranks when the source domain features are decomposed, and they are all parameters that need to be learned; correspondingly, Represents the scalar weight corresponding to the decomposition of the target domain features, They respectively represent the interactive features of EEG signals, galvanic skin signals, and ECG signals at different ranks when the source and target domain features are decomposed, and are also parameters that need to be learned. After tensor decomposition, the first alignment loss is calculated: Among them, φ() is the feature mapping function, which means mapping the original feature space into a higher-dimensional space; S i represents the i-th sample feature in the source domain feature S, T j represents the jth sample feature in the target domain feature T; n S and n T are the number of samples in the source domain and the target domain respectively; The attention mechanism is used to extract relevant information between the source domain and the target domain. The specific steps are as follows: Extract information related to the source domain S from the target domain T and use the attention mechanism: Construct a new target domain feature T′=T+A containing common information; Among them, Q is the query vector, which is used to find relevant information; K is the key vector, which is used to match the query vector to find relevant information; V is the value vector, which represents the matched information, which is the final information of interest; d k is the dimension of the key; For the input source domain features S and target domain features T, the corresponding query vector, key vector and value vector are obtained through linear transformation: Q S =SW Q ,K S =SW K ,V S =SW V ; Q T =TW Q ,K T =TW K ,V T =TW V ; Among them, W Q , W K and W V is the weight matrix to be learned; Extract information related to the target domain T from the source domain S and use the attention mechanism: Construct a new source domain feature S′=S+B containing common information; Calculate the second alignment loss, expressed as: Among them, S i ′ represents the i-th sample feature in the new source domain feature S′, T j ′ represents the new target domain feature T ′ The jth sample feature in n S′ and n T′ are the number of new source domain and target domain samples respectively; The neural network classification module is used to classify the input feature data to obtain a classification result; The model training and depression detection module is used to: 1) Model training: The source domain data is input into the neural network classification module, and the binary cross entropy loss is calculated based on the obtained classification results: Among them, y i is the true label of the feature data, is the predicted label; Finally, the total loss function is calculated by combining dynamic fusion and cross-domain alignment to optimize the entire model: L=λ1L mmd1 +λ2L mmd2 +L cls Among them, λ1 and λ2 are hyperparameters used to balance the impact of different loss terms; Based on the loss function L that combines dynamic fusion and cross-domain alignment, the neural network classification module and the parameters that need to be learned are trained, and the model weights are updated through the back-propagation algorithm to finally obtain an optimized depression detection system. 2) Depression detection: In practical applications, the multimodal signals collected from new subjects are input into the feature extraction module for preprocessing, and then input into the trained neural network classification module to generate classification results.

2. A depressive disorder identification system based on multimodal domain adaptation as claimed in claim 1, characterized in that: The model training and depression detection module also includes the detection of the neural network classification module: The classification performance of the neural network classification module is tested using source domain data and corresponding labels.

3. A depressive disorder identification system based on multimodal domain adaptation as claimed in claim 1, characterized in that: It also includes a data preprocessing module, including power frequency denoising, bandpass filtering, artifact removal and downsampling of EEG signals, low-pass filtering and smoothing of skin electrical signals, and power frequency denoising, bandpass filtering and smoothing of ECG signals to improve signal quality.

4. The depressive disorder identification system based on multimodal domain adaptation according to claim 1, characterized in that: The feature extraction module first performs manual feature extraction based on prior knowledge on the EEG signal, the skin electrical signal and the ECG signal, and then extracts the deep features through the Transformer encoder.

5. A depressive disorder identification system based on multimodal domain adaptation as claimed in claim 4, characterized in that: The feature extraction module is also used for: Extract three types of linear features and three types of nonlinear features from EEG signals; Linear features include mean, power spectral density, and center frequency; nonlinear features include sample entropy, approximate entropy, and Lyapunov exponent.

6. A depressive disorder identification system based on multimodal domain adaptation as claimed in claim 5, characterized in that: The feature extraction module is also used for: Heart rate variability, frequency ratio and fractal dimension features are extracted from ECG signals.

7. A depressive disorder identification system based on multimodal domain adaptation as claimed in claim 6, characterized in that: The feature extraction module is also used for: The skin conductance level, peak number and variance characteristics are extracted from the skin electrical signal.

8. The depressive disorder identification system based on multimodal domain adaptation according to claim 1, characterized in that: The neural network classification module is composed of multiple fully connected layers, and each fully connected layer is connected to an activation function.

9. The depressive disorder identification system based on multimodal domain adaptation according to claim 1, characterized in that: The neural network classification module also introduces batch normalization and Dropout technology to accelerate convergence and stabilize the training process.

Images