A depressive disorder recognition system based on multimodal domain adaptation

Abstract

The present invention discloses a depressive disorder recognition system based on multimodal domain adaptation. By modeling the interactive relationship between three signals, namely, EEG, galvano-dermal, and electrocardiogram, into a deep learning framework, dynamic fusion and parameter optimization between signals are achieved. This method makes full use of the complementary information of multimodal signals, enabling the model to extract more discriminative features from complex physiological data, effectively improving the accuracy and robustness of depressive disorder recognition; by twice narrowing the source domain and target domain data distribution, cross-domain common features are successfully mined; this cross-domain alignment strategy effectively improves the adaptability of the model in different data domains and significantly reduces the negative impact of domain offset. Overall, the recognition method provided by the present invention has shown significant advantages in improving the accuracy of depressive disorder detection and enhancing the generalization ability of the model.

Description

Technical Field

[0001] The present invention belongs to the technical fields of intelligent medicine, psychophysiological computing, transfer learning, etc., and specifically relates to a depressive disorder identification system based on multimodal domain adaptation. Background Art

[0002] Depressive disorder is a psychiatric illness characterized by persistent low mood and loss of interest. With the increasing pressures and fierce competition of modern life, the incidence of depressive disorder has increased significantly, becoming a major public health issue worldwide. According to statistics, over 350 million people worldwide suffer from depressive disorder, which seriously affects their quality of life and work ability. Depressive disorder not only causes significant impairment in emotional, cognitive, and behavioral functioning but is also often accompanied by other mental or physical illnesses, placing a significant burden on society and the economy. Although existing clinical methods can provide some treatment for depressive disorder, the lack of effective objective diagnostic criteria and early identification methods means that many patients' conditions are not promptly detected and intervened, compromising treatment effectiveness. Therefore, developing an efficient and objective method for identifying and assessing depressive disorder is of great significance for improving the accuracy of disease identification, developing personalized treatment plans, and alleviating the social and economic burden of depressive disorder.

[0003] Depression not only affects an individual's emotional state but also profoundly impacts the mechanisms regulating emotion. Emotional regulation is driven by a complex interaction between the central nervous system and the autonomic nervous system. When an individual experiences emotional stimulation, the sensory system captures relevant information and triggers a series of physiological responses. These responses regulate physiological functions such as heart rate, blood pressure, and respiration through the sympathetic and parasympathetic nervous systems of the autonomic nervous system. They also trigger responses in the central nervous system, reflecting the individual's emotional state. Given the impact of depression on the mechanisms regulating emotion, studying the physiological signals of depression is crucial. EEG, electrodermal (galvanic skin) signals, and electrocardiogram (ECG) signals are all important sources of physiological data for studying depression. EEG signals directly capture electrical activity in the central nervous system, providing detailed information on brain function dynamics and neural network activity. Electrodermal (galvanic skin) signals reflect activity in the autonomic nervous system, revealing the impact of emotional state on skin conductivity. ECG signals provide key information on cardiac activity, indirectly reflecting the regulatory effects of emotion on the autonomic nervous system. While each physiological signal provides unique and valuable information, a single modality is often insufficient to fully capture the complex emotional state of an individual. In order to improve the accuracy and reliability of identifying depressive disorders, it is particularly important to comprehensively analyze multimodal data such as EEG signals, skin electrical signals and ECG signals.

[0004] While the application of multimodal physiological signals in depression identification has made some progress, many challenges remain in effectively fusing data from different modalities and addressing the challenges posed by individual variability. EEG, galvanic skin signals, and electrocardiogram (ECG) signals each possess unique physiological characteristics, and their interrelationships are complex and variable. Multimodal domain adaptation offers new possibilities for addressing these challenges. This technology aims to effectively integrate signals and improve the accuracy of depression identification by effectively fusing data from different modalities and aligning features across domains. By learning and optimizing common features across domains, the adaptability of depression identification models can be enhanced. However, achieving this goal also presents challenges such as ensuring cross-modal information consistency, reducing noise interference, and improving the model's generalization ability across diverse data. Furthermore, existing research often overlooks the potential correlations and synergies between multimodal data, failing to fully leverage their complementary nature to improve depression identification. Therefore, the present invention focuses on in-depth exploration of the interrelationships and synergies between different modal data, and by optimizing multimodal domain adaptation strategies, enhances the accuracy and robustness of depressive disorder identification and provides more comprehensive and scientific support for clinical diagnosis. Summary of the Invention

[0005] In light of this, the present invention aims to provide a depressive disorder identification system based on multimodal domain adaptation. By collecting multimodal data, including EEG signals, electrical skin conductance, and ECG signals, and combining them with multimodal domain adaptation technology, this system enhances the ability to fuse different data modalities and align features across domains, thereby improving the accuracy and robustness of depressive disorder identification. This method utilizes a deep transfer learning algorithm to achieve optimization within the multimodal feature space, enhancing the ability to identify depressive disorders, adapting to individual physiological differences, and effectively supporting real-time analysis and diagnosis.

[0006] A depression disorder identification system based on multimodal domain adaptation, including a data acquisition module, a feature extraction module, a multimodal domain adaptation module, a neural network classification module, and a model training and depression detection module;

[0007] The data acquisition module is used to collect three modal signals of EEG, EGG and ECG from the subject in a resting state and an audio stimulation state;

[0008] The feature extraction module is used to extract the features of EEG, EGG and ECG signals;

[0009] The multimodal domain adaptation module is used to:

[0010] The three modal signal data are divided into source domain data and target domain data;

[0011] For each source domain and target domain data, the EEG signal features, skin electrical signal features, and ECG signal features are used to form a tensor, which is then decomposed by CP, specifically expressed as:

[0012]

[0013] Where S represents the source domain feature tensor, T represents the target domain feature tensor; R is the rank of the decomposition, which indicates the number of components used in the decomposition; Represents the scalar weight corresponding to the decomposition of source domain features, They represent the interactive features of EEG signals, skin electrical signals, and ECG signals at different ranks when the source domain features are decomposed, and are all parameters that need to be learned; correspondingly, Represents the scalar weight corresponding to the decomposition of the target domain features, They represent the interactive features of EEG signals, skin electrical signals, and ECG signals at different ranks when the source and target domain features are decomposed, and are also parameters that need to be learned.

[0014] After tensor decomposition, the first alignment loss is calculated:

[0015]

[0016] Among them, φ() is the feature mapping function, which means mapping the original feature space into a higher-dimensional space; S i Represents the i-th sample feature in the source domain feature S, T j Represents the jth sample feature in the target domain feature T; n S and n T are the number of samples in the source domain and target domain respectively;

[0017] The attention mechanism is used to extract relevant information between the source domain and the target domain. The specific steps are as follows:

[0018] Extract information related to the source domain S from the target domain T and use the attention mechanism:

[0019]

[0020] Construct a new target domain feature T′=T+A containing common information;

[0021] Among them, Q is the query vector, which is used to find relevant information; K is the key vector, which is used to match the query vector to find relevant information; V is the value vector, which represents the matched information, which is the information of final concern; d k is the dimension of the key;

[0022] For the input source domain features S and target domain features T, the corresponding query vector, key vector and value vector are obtained through linear transformation:

[0023] Q S =SW Q ,K S =SW K ,V S =SW V ;

[0024] Q T =TW Q ,K T =TW K ,V T =TW V ;

[0025] Among them, W Q 、W K and W V is the weight matrix to be learned;

[0026] Extract information related to the target domain T from the source domain S and use the attention mechanism:

[0027]

[0028] Construct a new source domain feature S′=S+B containing common information;

[0029] Calculate the second alignment loss, expressed as:

[0030]

[0031] Among them, S′ i Represents the i-th sample feature in the new source domain feature S′, T′ j Represents the jth sample feature in the new target domain feature T′, n S′ and n T′ are the number of new source domain and target domain samples respectively;

[0032] The neural network classification module is used to classify the input feature data to obtain a classification result;

[0033] The model training and depression detection module is used to:

[0034] 1) Model training:

[0035] The source domain data is input into the neural network classification module, and the binary cross entropy loss is calculated based on the obtained classification results:

[0036]

[0037] Among them, y i is the true label of the feature data, is the predicted label;

[0038] Finally, the total loss function is calculated by combining dynamic fusion and cross-domain alignment to optimize the entire model:

[0039] L=λ1L mmd1 +λ2L mmd2 +L cls

[0040] Among them, λ1 and λ2 are hyperparameters used to balance the impact of different loss terms;

[0041] Based on the loss function L that combines dynamic fusion and cross-domain alignment, the neural network classification module and the parameters that need to be learned are trained, and the model weights are updated through the backpropagation algorithm, ultimately obtaining an optimized depression detection system.

[0042] 2) Depression detection:

[0043] In practical applications, the multimodal signals collected from new subjects are input into the feature extraction module for preprocessing, and then input into the trained neural network classification module to generate classification results.

[0044] 2. The depressive disorder identification system based on multimodal domain adaptation according to claim 1, wherein the model training and depression detection module further includes testing of the neural network classification module:

[0045] The classification performance of the neural network classification module is tested using source domain data and corresponding labels.

[0046] Furthermore, it also includes a data preprocessing module, including power frequency denoising, bandpass filtering, artifact removal and downsampling of EEG signals, low-pass filtering and smoothing of skin electrical signals, and power frequency denoising, bandpass filtering and smoothing of ECG signals to improve signal quality.

[0047] Preferably, the feature extraction module first performs manual feature extraction based on prior knowledge on the EEG signal, skin electrical signal and ECG signal, and then extracts deep features through the Transformer encoder.

[0048] Furthermore, the feature extraction module is also used to:

[0049] Three types of linear features and three types of nonlinear features are extracted from EEG signals; the linear features include mean value, power spectrum density, and center frequency; the nonlinear features include sample entropy, approximate entropy, and Lyapunov exponent.

[0050] Furthermore, the feature extraction module is also used to:

[0051] Heart rate variability, frequency ratio and fractal dimension features are extracted from ECG signals.

[0052] Furthermore, the feature extraction module is also used to:

[0053] The skin conductance level, peak number and variance characteristics of the skin electrical signal are extracted.

[0054] Preferably, the neural network classification module is composed of multiple fully connected layers, and each fully connected layer is connected to an activation function.

[0055] Furthermore, the neural network classification module also introduces batch normalization and Dropout technology to accelerate convergence and stabilize the training process.

[0056] The present invention has the following beneficial effects:

[0057] The present invention aims to solve the challenges of low recognition rate of single modality physiological signals in the detection of depressive disorders and the challenges caused by individual differences. Since different individuals have significant differences in physiological signal characteristics, these challenges are particularly prominent in the identification of depressive disorders. Therefore, a method for identifying depressive disorders based on multimodal domain adaptation is proposed. In the present invention, the domain specifically refers to the physiological signal feature space of different individuals. By aligning the feature spaces between these different individuals, multimodal domain adaptation can overcome the influence of individual differences on the recognition effect, thereby improving the accuracy and robustness of depressive disorder identification (see Appendix). Figure 1 By integrating multimodal data such as EEG, galvanic skin signals, and electrocardiogram (ECG) signals, effective signal fusion is achieved, leveraging the complementary information of different modal signals to improve recognition accuracy. Furthermore, domain adaptation technology is introduced to address data distribution shifts caused by individual differences, thereby enhancing the model's generalization and stability. This approach provides more efficient and accurate technical support for clinical applications.

[0058] The present invention's method for identifying depressive disorders based on multimodal domain adaptation models the interactive relationship between EEG, electrodermal, and electrocardiographic signals within a deep learning framework, enabling dynamic signal fusion and parameter optimization. This method fully leverages the complementary information of multimodal signals, enabling the model to extract more discriminative features from complex physiological data, effectively improving the accuracy and robustness of depressive disorder identification. Furthermore, the present invention successfully mines cross-domain common features by twice aligning the data distributions of the source and target domains. The first alignment uses the maximum mean difference to ensure consistency between the source and target domains in the feature space. Second, an attention mechanism is used to further deeply mine and fuse target-related information features to construct a more accurate feature representation. This cross-domain alignment strategy effectively enhances the model's adaptability across different data domains and significantly reduces the negative impact of domain shift. Overall, the identification method provided by the present invention demonstrates significant advantages in improving the accuracy of depressive disorder detection and enhancing model generalization. It not only demonstrates the effectiveness of multimodal fusion and domain adaptation but also provides an innovative solution for the automated detection of depressive disorders in practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is a flow chart of the depressive disorder identification method based on multimodal domain adaptation of the present invention;

[0060] Figure 2 Schematic diagram of the three-conductor electrode position, skin electrode position and heart electrode position;

[0061] Figure 3 Flowchart of the framework for identification of depressive disorders based on multimodal domain adaptation. DETAILED DESCRIPTION

[0062] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0063] The present invention proposes a depression disorder recognition system based on multimodal domain adaptation. First, the multimodal electrophysiological signals in the resting state and audio stimulation state are preprocessed and the corresponding features are extracted. The extracted features are the commonly used artificial features and deep features of the corresponding modalities. Then, the dynamic interaction relationship between EEG signals, skin electrical signals, and ECG signals is modeled through CP tensor synthesis and decomposition. Secondly, the model is trained by mining cross-domain common features through the two maximum mean differences and attention mechanism. Finally, the test set is used for depression detection to obtain the expected depression detection recognition rate. Figure 1 As shown, the system includes the following modules:

[0064] (1) Data acquisition module, collects the subject's number, name, gender and age information, and simultaneously collects the subject's EEG, EG and ECG signals in the resting state and audio stimulation state. The EEG, EG and ECG collection locations are as shown in the attached figure. Figure 2 .

[0065] In this embodiment, a self-developed three-channel EEG device (see attached Figure 2 (Left) EEG signals were collected using software, while galvano-skin and ECG signals were collected using redeveloped software. Both software programs simultaneously recorded timestamps to ensure temporal alignment of the data. All subjects were college students with normal hearing and intelligence and no history of other mental illnesses. No psychiatric medications were taken before data collection, and all subjects were tested under identical laboratory conditions. The sampling rate for the EEG device was 250Hz, and the sampling rate for the galvano-skin and ECG devices was 50Hz.

[0066] (2) Data preprocessing module, including power frequency denoising, bandpass filtering, artifact removal and downsampling of EEG signals, low-pass filtering and smoothing of skin electrical signals, and power frequency denoising, bandpass filtering and smoothing of ECG signals to improve signal quality.

[0067] In this embodiment, a notch filter is first used to remove 50Hz power line interference to eliminate power frequency noise. Next, the EEG signal, skin electrode signal and ECG signal are band-pass filtered, and the filter ranges are set to 0.5-50Hz, 0.5-2Hz and 0.5-40Hz, respectively, to retain the effective signal frequency band. Subsequently, artifact removal is performed to detect and remove abnormal data caused by motion artifacts or electrooculogram artifacts, thereby improving the quality of the signal. In order to reduce random fluctuations in the signal, a moving average technique is applied to smooth the signal and enhance the stability of the data. Finally, the EEG sampling rate is reduced from 250Hz to 100Hz by downsampling, reducing the amount of data and retaining key information.

[0068] (3) Feature extraction module: This module extracts manual features based on prior knowledge from the preprocessed EEG signals, skin electrodermal signals, and ECG signals, and extracts deep features through the Transformer encoder. EEG signals: Three types of linear features and three types of nonlinear features are extracted. Linear features include mean, power spectrum density, and center frequency; nonlinear features include sample entropy, approximate entropy, and Lyapunov exponent. ECG signals: Features such as heart rate variability, frequency ratio, and fractal dimension are extracted. Skin electrodermal signals: Features such as skin conductance level, number of peaks, and variance are extracted.

[0069] (4) The multimodal domain adaptation module first models the dynamic interaction between multimodal physiological signals through tensor synthesis and decomposition, and embeds it into the deep learning training process to optimize model parameters. Then, it mines the common information of the source and target domains and uses the maximum mean difference method for cross-domain alignment to train a model that performs well in the target domain, so that the source domain features can be effectively applied in the target domain.

[0070] (5) The neural network classification module consists of several fully connected layers, each followed by an activation function to introduce nonlinearity and avoid the vanishing gradient problem. In addition, to enhance the generalization ability of the network and prevent overfitting, regularization techniques such as batch normalization and dropout are introduced into the network to accelerate convergence and stabilize the training process.

[0071] (6) Model training and depression detection module (see Appendix Figure 3 ), use the trained model and neural network classification module to perform depression detection on the newly collected data to achieve efficient classification and recognition.

[0072] The data acquisition module includes the following steps:

[0073] 1) Experimental Design: First, multimodal physiological signals were collected from subjects in a resting state with their eyes closed. Subsequently, an auditory stimulation experiment was conducted, using a variety of auditory stimuli with different emotional attributes (e.g., two positive stimuli, two neutral stimuli, and two negative stimuli). Each auditory stimulation lasted for a set duration. After each auditory stimulation, the subjects had a short rest period. Throughout this process, EEG signals, skin conduction signals, and ECG signals were continuously collected while the subjects were in the closed-eye state, ultimately obtaining complete data.

[0074] 2) EEG acquisition: EEG signals were collected using EEG equipment that complies with the internationally widely used 10-20 system electrode position standard.

[0075] 3) Multimodal electrophysiological acquisition: Multimodal acquisition equipment enables the simultaneous and stable acquisition of other physiological signals. The equipment is optimized to ensure the accuracy and stability of signal acquisition.

[0076] In the data preprocessing stage, the power frequency denoising is performed by using a notch filter to remove 50 Hz power line interference.

[0077] In the data preprocessing stage, the bandpass filtering ranges of the EEG signal, the galvanic skin signal and the electrocardiogram signal are respectively 0.5-50 Hz, 0.5-2 Hz and 0.5-40 Hz.

[0078] Artifact removal in the data preprocessing stage is to detect and remove data anomalies caused by motion artifacts or electrooculogram artifacts to improve signal quality.

[0079] The smoothing in the data preprocessing stage is to reduce the random fluctuation of the signal by using the moving average technology to improve the stability of the data.

[0080] Downsampling in the data preprocessing stage is to reduce the sampling rate from 250 Hz to 100 Hz.

[0081] The feature extraction module: performs manual feature extraction based on prior knowledge on the preprocessed EEG signals, skin electrode signals and ECG signals, and extracts deep features through the Transformer encoder. For EEG signals, three types of linear features (average value, power spectral density, center frequency) and three types of nonlinear features (sample entropy, approximate entropy, Lyapunov exponent) are extracted. For ECG signals, features such as heart rate variability, frequency ratio and fractal dimension are extracted. For skin electrode signals, features such as skin conductance level, number of peaks and variance are extracted. Subsequently, deep feature extraction is performed through the Transformer encoder, and the self-attention mechanism is used to automatically learn the complex temporal patterns and global dependencies in the signal.

[0082] Power spectral density:

[0083]

[0084] Where x(t) is the time domain signal and f is the frequency.

[0085] Center Frequency:

[0086]

[0087] Among them, f1 is the lower frequency limit and f2 is the upper frequency limit.

[0088] Sample entropy:

[0089]

[0090] Among them, A i is the number of sequence pairs with matching length m+1, B i is the number of sequence pairs with matching length m, r is the tolerance, and N is the signal length.

[0091] Approximate entropy:

[0092] ApEn(m,r,N)=φ m (r)-φ m+1 (r)

[0093] in, is the number of all sequence pairs that are similar to the template sequence.

[0094] Lyapunov exponent:

[0095]

[0096] in, is the small perturbation at the i-th moment.

[0097] Heart rate variability:

[0098]

[0099] Among them, RR i is the RR interval between adjacent heartbeats, is the mean of the RR intervals.

[0100] Frequency ratio:

[0101]

[0102] Here, HFB represents a high frequency band, and LFB represents a low frequency band.

[0103] Fractal dimension:

[0104]

[0105] Where ∈ is the scale parameter of the fractal structure and N is the length of the signal.

[0106] Skin conductance level:

[0107]

[0108] Among them, SC i is the skin conductance value at the i-th time point.

[0109] The multimodal domain adaptation module performs the following method steps:

[0110] 1) Dynamic Fusion:

[0111] First, use the EEG signal features (feature dimension: I), skin electrical signal features (feature dimension: J), and ECG signal features (feature dimension: K) to form a tensor X∈R I×J×K The synthesized tensor X captures the joint effect of the three signals in each feature dimension. In order to extract meaningful interaction information from this synthesized tensor, CP decomposition is used. CP decomposition decomposes a high-dimensional tensor into a combination of multiple low-dimensional factor vectors to express the intrinsic structure of the tensor, specifically expressed as:

[0112]

[0113] Where R is the rank of the decomposition, which indicates the number of components used in the decomposition; λ ris a scalar weight indicating the importance of different components; a r 、b r and c r are the corresponding factor vectors, which come from the factor matrices of EEG signal, skin electrical signal, and ECG signal features respectively. represents the outer product, a r 、b r and c r Respectively represent the interactive features of EEG signals, skin electrical signals, and ECG signals at different ranks. r 、a r 、b r and c r are parameters that are updated in embedded deep learning.

[0114] In this example, the multimodal data of the subjects is divided into 10 parts using 10-fold cross-validation. 9 of these folds are used as source domain data, and the remaining 1 fold is used as target domain data. During the training phase, only the source domain data and labels, along with the target domain data, are used. During the testing phase, the remaining 1 fold, including the target domain data and labels, is used.

[0115] The construction of source domain features S and target domain features T is as follows:

[0116]

[0117] in, Represents the scalar weight corresponding to the decomposition of source domain features, They represent the interactive features of EEG signals, skin electrical signals, and ECG signals at different ranks when the source domain features are decomposed, and are all parameters that need to be learned; correspondingly, Represents the scalar weight corresponding to the decomposition of the target domain features, They represent the interactive features of EEG signals, skin electrical signals, and ECG signals at different ranks when the source and target domain features are decomposed, and are also parameters that need to be learned.

[0118] The decomposition process described above is intended to extract low-dimensional representations of multimodal data, capturing the complex interactions between signals. Synthesis, on the other hand, is to recombine these low-dimensional factors into new, more semantically relevant, and easily aligned feature representations. If X represents multimodal data, and both the source and target domains are multimodal, both require these two steps. The source domain is the source domain features that have undergone decomposition and synthesis, and the target domain data is the same.

[0119] 2) Cross-domain alignment:

[0120] Both the source and target domains contain EEG signals, skin electrical signals, and ECG signals. This paper uses maximum mean difference for alignment. After tensor decomposition, the source domain features are represented as S and the target domain features are represented as T. The first alignment loss can be expressed as:

[0121]

[0122] Among them, φ() is the feature mapping function, which means mapping the original feature space into a higher-dimensional space (reproducing kernel Hilbert space). i Represents the i-th sample feature in the source domain feature S, T j Represents the jth sample feature in the target domain feature T; n s and n T are the number of samples in the source domain and target domain respectively.

[0123] In practice, feature mapping is implemented by kernel functions, such as Gaussian kernel (RBF kernel):

[0124]

[0125] Among them, S is the bandwidth hyperparameter of the kernel function, s i ∈S i ,s j ∈T j .

[0126] The attention mechanism is used to extract relevant information between the source domain and the target domain. The specific steps are as follows:

[0127] Extract information related to the source domain S from the target domain T and use the attention mechanism:

[0128]

[0129] Construct a new target domain feature T′=T+A containing common information. Q is the query vector, used to find relevant information; K is the key vector, used to match the query vector to find relevant information; V is the value vector, representing the matched information, which is the information of ultimate interest. In the source domain, the target domain feature is used as the query (Q) to match the source domain feature (K), thereby extracting information in the source domain related to the target domain, and vice versa. For the input source and target domain features (S, T), Q, K, and V are obtained through linear transformation.

[0130] Q S =SW Q ,K S =SW K ,V S =SW V

[0131] Among them, W Q 、W K and W V is the weight matrix to be learned, d k is the dimension of the key, and so is the target domain, i.e.:

[0132] Q T =TW Q ,K T =TW K ,V T =TW V ;

[0133] Extract information related to the target domain T from the source domain S and use the attention mechanism:

[0134]

[0135] Construct a new source domain feature S′=S+B containing common information.

[0136] The second alignment loss is expressed as:

[0137]

[0138] Among them, S i ′ represents the i-th sample feature in the new source domain feature S′, T j ′ represents the jth sample feature in the new target domain feature T′, n S′ and n T′ is the number of new source and target domain samples.

[0139] The neural network classification module classifies the input new source domain features S′ and new target domain features T′ to obtain a classification result;

[0140] The model training and depression detection module is used to:

[0141] 1) Model training and testing:

[0142] The model is trained using source domain data, and binary cross entropy loss is used for the source domain classification task:

[0143]

[0144] Among them, y i is the true label, is the predicted label.

[0145] Finally, the loss functions of dynamic fusion and cross-domain alignment are combined to optimize the entire model:

[0146] L=λ1L mmd1 +λ2L mmd2 +L cls

[0147] Among them, λ1 and λ2 are hyperparameters used to balance the impact of different loss terms.

[0148] Based on the loss function L that combines dynamic fusion and cross-domain alignment, the neural network classification module and the parameters that need to be learned are trained, and the model weights are updated through the backpropagation algorithm, ultimately obtaining an optimized depression detection system.

[0149] During this phase, the model is trained using a ten-fold cross-validation method to improve its generalization capabilities. This method involves splitting the dataset into ten equal parts, selecting one part each time as the test set and the remaining nine parts as the training set, and repeating this process multiple times. During each validation phase, the remaining part of the data is fed into the neural network classifier to generate the model's predictions. The model output is then compared with the true labels, and classification performance metrics such as accuracy, sensitivity, and specificity are calculated to comprehensively evaluate the model's detection performance. This entire process is performed on data from multiple subjects.

[0150] 2) Depression detection:

[0151] In practical applications, multimodal signals (e.g., EEG, electrodermal conduction, and electrocardiogram) collected from new subjects are fed into a feature extraction module for preprocessing and then fed into a trained neural network model to generate predictions. By analyzing the model's output, efficient depression classification and identification can be performed to assist in clinical diagnosis and treatment decisions.

[0152] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A depressive disorder recognition system based on multimodal domain adaptation, characterized by: It includes data acquisition module, feature extraction module, multimodal domain adaptation module, neural network classification module, and model training and depression detection module; The data acquisition module is used to collect three modal signals of EEG, EGG and ECG from the subject in a resting state and an audio stimulation state; The feature extraction module is used to extract the features of EEG, EGG and ECG signals; The multimodal domain adaptation module is used to: The three modal signal data of the subjects are divided into source domain data and target domain data; For each source domain and target domain data, the EEG signal features, skin electrical signal features, and ECG signal features are used to form a tensor, which is then decomposed by CP, specifically expressed as: Where S represents the source domain feature tensor, T represents the target domain feature tensor; R is the rank of the decomposition, which indicates the number of components used in the decomposition; Represents the scalar weight corresponding to the decomposition of source domain features, They represent the interactive features of EEG signals, skin electrical signals, and ECG signals at different ranks when the source domain features are decomposed, and are all parameters that need to be learned; correspondingly, Represents the scalar weight corresponding to the decomposition of the target domain features, They represent the interactive features of EEG signals, skin electrical signals, and ECG signals at different ranks when the target domain features are decomposed, and are also parameters that need to be learned; After tensor decomposition, the first alignment loss is calculated: Among them, φ() is the feature mapping function, which means mapping the original feature space into a higher-dimensional space; S i Represents the i-th sample feature in the source domain feature S, T j Represents the jth sample feature in the target domain feature T; n S and n T are the number of samples in the source domain and target domain respectively; The attention mechanism is used to extract relevant information between the source domain and the target domain. The specific steps are as follows: Extract information related to the source domain S from the target domain T and use the attention mechanism: Construct a new target domain feature T′=T+A containing common information; Among them, Q is the query vector, which is used to find relevant information; K is the key vector, which is used to match the query vector to find relevant information; V is the value vector, which represents the matched information, which is the information of final concern; d k is the dimension of the key; For the input source domain features S and target domain features T, the corresponding query vector, key vector and value vector are obtained through linear transformation: Q S =SW Q ,K S =SW K ,V S =SW V ; Q T =TW Q ,K T =TW K ,V T =TW V ; Among them, W Q 、W K and W V is the weight matrix to be learned; Extract information related to the target domain T from the source domain S and use the attention mechanism: Construct a new source domain feature S′=S+B containing common information; Calculate the second alignment loss, expressed as: Among them, S i ′ represents the i-th sample feature in the new source domain feature S′, T j ′ represents the new target domain feature T ′ The jth sample feature in n S′ and n T′ are the number of new source domain and target domain samples respectively; The neural network classification module is used to classify the input feature data to obtain a classification result; The model training and depression detection module is used to: 1) Model training: The source domain data is input into the neural network classification module, and the binary cross entropy loss is calculated based on the obtained classification results: Among them, y i is the true label of the feature data, is the predicted label; Finally, the total loss function is calculated by combining dynamic fusion and cross-domain alignment to optimize the entire model: L=λ1L mmd1 +λ2L mmd2 +L cls Among them, λ1 and λ2 are hyperparameters used to balance the impact of different loss terms; Based on the loss function L that combines dynamic fusion and cross-domain alignment, the neural network classification module and the parameters that need to be learned are trained, and the model weights are updated through the backpropagation algorithm, ultimately obtaining an optimized depression detection system. 2) Depression detection: In practical applications, the multimodal signals collected from new subjects are input into the feature extraction module for preprocessing, and then input into the trained neural network classification module to generate classification results.

2. The depressive disorder identification system based on multimodal domain adaptation according to claim 1, characterized in that: The model training and depression detection module also includes testing of the neural network classification module: The classification performance of the neural network classification module is tested using source domain data and corresponding labels.

3. The depressive disorder identification system based on multimodal domain adaptation according to claim 1, characterized in that: It also includes a data preprocessing module, which includes power frequency denoising, bandpass filtering, artifact removal and downsampling of EEG signals, low-pass filtering and smoothing of skin electrical signals, and power frequency denoising, bandpass filtering and smoothing of ECG signals to improve signal quality.

4. The depressive disorder identification system based on multimodal domain adaptation according to claim 1, characterized in that: The feature extraction module first performs manual feature extraction based on prior knowledge on the EEG signal, skin electrical signal and ECG signal, and then extracts deep features through the Transformer encoder.

5. The depressive disorder identification system based on multimodal domain adaptation according to claim 4, characterized in that: The feature extraction module is also used to: Extract three types of linear features and three types of nonlinear features from EEG signals; Linear features include mean, power spectral density, and center frequency; nonlinear features include sample entropy, approximate entropy, and Lyapunov exponent.

6. The depressive disorder identification system based on multimodal domain adaptation according to claim 5, characterized in that: The feature extraction module is also used to: Heart rate variability, frequency ratio and fractal dimension features are extracted from ECG signals.

7. The depressive disorder identification system based on multimodal domain adaptation according to claim 6, characterized in that: The feature extraction module is also used to: The skin conductance level, peak number and variance characteristics of the skin electrical signal are extracted.

8. The depressive disorder identification system based on multimodal domain adaptation according to claim 1, characterized in that: The neural network classification module is composed of multiple layers of fully connected layers, and each fully connected layer is connected to an activation function.

9. The depressive disorder identification system based on multimodal domain adaptation according to claim 1, characterized in that: The neural network classification module also introduces batch normalization and Dropout techniques to accelerate convergence and stabilize the training process.

Images