Depression recognition system based on electroencephalogram-voice dual-mode decision fusion

Abstract

The application discloses a depression recognition method based on electroencephalogram-voice dual-mode decision fusion, and belongs to the technical field of digital signal processing.The application aims to adopt a D-S evidence theory decision fusion strategy combining a capsule graph neural network and a Transform network, so as to improve the depression recognition system based on electroencephalogram-voice dual-mode decision fusion in terms of objectivity and accuracy of depression population recognition and classification of a detection device.The steps of the application are as follows: collecting electroencephalogram and voice signals, data preprocessing, electroencephalogram and voice feature extraction, construction and training of the capsule graph neural network and the Transform network, and depression classification and recognition of a test set.The application overcomes the problems of great individual difference influence and low depression classification recognition degree caused by only relying on a single mode, effectively mines and fuses depression factors contained in electroencephalogram and voice information, and improves the depression classification recognition accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of digital signal processing. BACKGROUND

[0002] Depression is a common mental disorder, which is manifested as persistent depression and loss of interest, and greatly affects the life and health of patients even life safety. At present, the most common clinical diagnosis method of depression is that professional doctors assess through depression scale (such as Hamilton Depression Scale HAMD). However, such diagnosis method requires that the doctors have received professional and good training first, and the evaluation results obtained also have a certain subjectivity, which may lead to missed diagnosis or misdiagnosis. Therefore, in order to intervene and effectively prevent and treat depression symptoms in time, it is particularly important to adopt an objective and reliable depression diagnosis technology.

[0003] In recent years, many researchers have devoted to developing objective depression auxiliary detection technology. The brain electrical signals and speech signals of individuals are considered to be reliable basis for evaluating depression. Studies have shown that the brain electrical activity of some depression patients in different brain cortex regions and different rhythms has changed, and the brain network has abnormalities in spatial structure and hemispheric symmetry. In addition, scholars have studied the potential depression factors in speech according to the comprehensive performance of depression patients, such as reduced expression of will, slow speech speed when communicating, monotonous tone and single content. Therefore, the existing method often analyzes the brain electrical or speech information, and constructs a machine learning or deep learning model to realize the classification and recognition of depression. However, since the neural mechanism of depression is still unclear, and the performance of brain electrical activity and speech changes of different patients is not uniform, the depression recognition result obtained by using a single mode may not be accurate due to the influence of individual differences. Therefore, how to use the existing reliable multi-modal information that can be collected to realize more accurate and higher generalization performance of the device for depression recognition is a problem that needs to be considered at present. SUMMARY

[0004] The purpose of the application is to use the D-S evidence theory decision fusion strategy combined with capsule graph neural network and Transformer network, aiming to improve the objectivity and accuracy of the depression recognition system based on brain electrical-speech dual modal decision fusion of the detection device for recognizing and classifying depression population.

[0005] Construct and train the capsule graph neural network and the Transformer network:

[0006] S1, for the brain electrical feature, a capsule graph neural network is used for processing, which includes: graph structure input, graph convolution layer, initial capsule generation layer, attention mechanism, graph capsule generation layer, classification capsule generation layer, and brain electrical mass function Output:

[0007] a, graph structure input: power spectrum entropy features is constructed as a graph structure , wherein is a node set composed of N electroencephalogram leads, is an adjacency matrix, is a node feature; the correlation coefficient between the features of each node is obtained as a Pearson correlation matrix:

[0008] (1)

[0009] wherein, represents the correlation coefficient of the th node feature in the th node;

[0010] comparing the size of and the threshold value , the correlation matrix is converted into an adjacency matrix composed of elements 0 and 1 , that is

[0011] (2)

[0012] b, graph convolution layer: according to the graph structure input, a layer graph convolution layer is used to learn higher quality node embedding, and the formula of each layer graph convolution layer is

[0013] (3)

[0014] wherein, represents the node feature of the th node in the th layer graph convolution layer, and , ; represents the feature transformation weight from the th node in the th layer graph convolution layer to the th node in the th layer, which constitutes a matrix ;

[0015] represents an activation function, which is specifically function to enhance the nonlinear expression ability of the features, and the formula is

[0016] (4);

[0017] c, initial capsule generation layer: recombining and stacking the output of each layer of graph convolution layer to generate primary capsules; the output of each layer of graph convolution layer recombinant into a three-dimensional matrix, wherein denotes the number of nodes, denotes the number of capsules established for each node, denotes the dimension of each capsule, and ; each three-dimensional matrix is stacked along dimension to obtain a primary capsule , wherein ; the primary capsule can be regarded as being composed of groups of node capsules, each group of node capsules being , wherein denotes the th node, th capsule of the th node;

[0018] d, attention mechanism: recalibration is performed on each group of node capsules, and the process is as follows:

[0019] (5)

[0020] wherein, is obtained by tiling , that is, , denotes a fully connected layer, the number of input neurons of which is , and the number of output neurons of which is , the attention coefficient of each capsule being obtained from the normalized value of the output of the fully connected layer , and the group of node capsules is recalibrated using the attention coefficient to obtain ;

[0021] e, graph capsule generation layer: the primary capsule after attention calibration is subjected to a dynamic routing process (cycling times) to generate a graph capsule , wherein denotes the number of graph capsules, denotes the dimension of each graph capsule;

[0022] f, classification capsule generation layer: the graph capsule is subjected to a dynamic routing process (cycling times) to generate a classification capsule ; wherein denotes the number of classification capsules, denotes the dimension of each classification capsule;

[0023] g, electroencephalogram mass function Output: Calculate the modulus of each capsule in the classification capsule, which corresponds to the recognition probability of each category; the model judges the probability of the input feature being each category and obtains and , and further obtains the basic probability distribution mass function of electroencephalogram , that is

[0024] (6)

[0025] S2, for speech features, a Transformer network is used, which includes: Fbank feature input, lightweight multi-head self-attention module, multi-layer perception feedforward module, classification layer, speech mass function Output:

[0026] a, Fbank feature input: Fbank feature is input into the Transformer network, ;

[0027] b, lightweight multi-head self-attention module: input feature is first subjected to batch normalization BN operation to obtain , the number of attention heads is set to , and is divided into groups: query matrix , key matrix , and value matrix , that is, the matrix size of each group is transformed into , and the dimension of and is reduced by using average pooling operation, so the dimension of each group and is , where s is the step, and and are used to represent, respectively, the and of each group are subjected to product scaling, flexible maximum value calculation, and dropout strategy to obtain the attention coefficient matrix of each head, and the coefficient matrix is multiplied by to obtain the output after each attention head:

[0028] (7)

[0029] where ; then, the attention outputs of all heads are spliced, represented by the symbol ||, and fed into a linear layer; finally, a residual operation is performed:

[0030] (8);

[0031] c. Multi-layer perceptron feedforward module: the input of the module is ; the module is composed of two two-dimensional convolution layers, and the input end is first subjected to batch normalization BN operation, and activation function is used between the two two-dimensional convolution layers:

[0032] (9)

[0033] (10)

[0034] (11)

[0035] (12)

[0036] wherein, represents the bias of the first convolution layer, represents the learnable weight matrix of the first convolution layer, and respectively represent the dimension size of , and represents the number of output channels, represents the bias of the second convolution layer, represents the learnable weight matrix of the second convolution layer, and respectively represent the dimension size of , and represents the number of output channels; finally, the output is subjected to residual operation, that is,

[0037] (13)

[0038] Continue to repeat processes b and c, taking as the input of process b, until times of calculation are completed;

[0039] d. Classification layer: composed of one fully connected layer and softmax layer, the outputs of the classifier are respectively represented as and ;

[0040] e. Speech mass function Output: the classifier outputs and constitute the speech mass function , specifically

[0041] (14)

[0042] S3, D-S evidence theory decision layer fusion:

[0043] Set the classification universe as , according to steps S1 and S2, the basic probability distribution mass functions of two kinds of evidence of electroencephalogram features and speech features are obtained and , the probability distribution of the universe is calculated , wherein the probability distribution of each state is:

[0044] (15)

[0045] (16)

[0046] wherein, is the conflict coefficient between evidences, ; finally, compare and , the hypothesis with the maximum probability is determined to be true, and the state corresponding to this maximum probability is set as the final decision result;

[0047] S4, overall loss function of the model:

[0048] In the training process of the model, three kinds of loss functions are used to update the parameters in the model, i.e. marginal loss , cross-entropy loss , and classification loss ;

[0049] Marginal loss based on capsule graph neural network:

[0050] (17)

[0051] wherein, when the prediction result is , otherwise , denotes the module length of the th classification capsule, denotes the marginal coefficient;

[0052] Cross-entropy loss based on Transformer network:

[0053] (18)

[0054] wherein, denotes the classification label, when the prediction result is true , otherwise ; denotes the probability of being predicted as true;

[0055] Classification loss of D-S decision fusion:

[0056] (19)

[0057] wherein, is the predicted class, is the real class;

[0058] The training target of the model is to minimize the loss function, that is,

[0059] (20)

[0060] wherein, and are proportional coefficients, is the optimal parameter set of the model after training update.

[0061] The application overcomes the problem of large individual difference and low depression classification recognition caused by relying on a single mode only, effectively fuses the brain electrical and speech dual modal information, adopts the D-S evidence theory decision fusion algorithm with better fault tolerance and anti-interference, reduces the influence of individual difference on depression recognition, and realizes a more accurate depression recognition method. In addition, the application adopts a capsule graph neural network, considers the spatial information often ignored in the brain electrical signal, learns more optimal graph embedding in the form of capsules (vectors), designs a Transformer network, pays attention to the time global context semantic relationship of the speech in a more lightweight architecture, and the effective combination of the two models enriches the existing analysis algorithm about the brain electrical signal and the speech signal. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 is the overall flowchart of the application;

[0063] Figure 2 is the hierarchical graph of the capsule graph neural network of the application;

[0064] Figure 3 is the hierarchical graph of the Transformer network of the application;

[0065] Figure 4 is the lightweight multi-head self-attention module in the Transformer network of the application;

[0066] Figure 5 is the multi-layer perceptron feedforward module in the Transformer network of the application. DETAILED DESCRIPTION

[0067] The application will be described in detail below in combination with the drawings:

[0068] Step 1: Collecting EEG and speech signals under specific experimental tasks;

[0069] The subjects (depression patients or normal subjects) participate in two experimental tasks and collect signals: 1) collect EEG signals under the closed-eye resting task; 2) record speech signals under the expert interview, material reading and description task.

[0070] The subjects are in a quiet, light and temperature suitable experimental environment, and start the experiment in a comfortable sitting position. The process of collecting EEG signals is: after the subject is in a stable relaxed state, use the EEG collection cap to collect the subject's 5-minute closed-eye resting EEG signals; the process of collecting speech signals is: use a microphone to record the subject's speech signals, and record the content including expert interview replies, short article reading and picture description speech.

[0071] Step 2: Data preprocessing;

[0072] For EEG signals: EEG signals are sampled at 1000Hz and downsampled to 250Hz. A digital bandpass filter of 1-80Hz is selected, and a 50Hz notch filter is used to remove power frequency interference. ICA method is used to remove eye artifacts, and the part disturbed by invalid electrodes or other unrelated signals is manually removed.

[0073] For speech signals: the speech signals are sampled at 44.1kHz, and pre-emphasis (pre-emphasis coefficient is 0.98) is used to compensate for the loss of high frequencies in the speech signals. The speech signals are divided into 25ms frames with a frame shift of 10ms. Hamming window is used for each frame of signal to reduce spectral leakage. End-point detection is used to remove silent segments in the speech signal.

[0074] Data preprocessing is to take down-sampling, band-pass filtering, 50Hz power frequency interference notch filtering, ICA artifact removal operation on the original EEG signal; and take sampling, pre-emphasis, frame, windowing, endpoint detection operation on the original speech signal.

[0075] Step 3: Feature extraction of EEG and speech signals;

[0076] The feature extraction of EEG and speech is to extract the power spectrum entropy features of each 1-second time slice on six EEG rhythms from the preprocessed EEG signal to construct the EEG feature set; and extract the Fbank features of each 9-second time slice from the preprocessed speech signal to construct the speech feature set.

[0077] For EEG signal: extract the power spectrum entropy feature of each 1-second time slice EEG signal in delta (1-3Hz), theta (4-8Hz), alpha (8-13Hz), beta (13-30Hz), low gamma (30-50Hz), high gamma (50-80Hz) rhythm where N represents the number of EEG acquisition leads, represents the number of EEG rhythms.

[0078] For speech signal: extract the Fbank feature of each 9-second time slice speech signal where M represents the number of mel filters, and T represents the number of frame slices.

[0079] First, according to the frequency band range of each rhythm, a band-pass filter is used for filtering, and the EEG signal is subjected to fast Fourier transform with a time window of 1 second, and the calculation formula is:

[0080] (21)

[0081] wherein, represents the filtered EEG signal of each second, which is composed of E points, is the EEG frequency domain signal after fast Fourier transform.

[0082] Next, the power spectrum can be calculated

[0083] (22).

[0084] Finally, the power spectrum entropy feature of each 1-second signal in each rhythm frequency band is calculated, and the calculation formula is

[0085] (23).

[0086] Therefore, the EEG feature is the power spectrum entropy feature of each 1-second time slice where N represents the number of EEG acquisition leads, represents the number of EEG rhythms.

[0087] For speech signal: extract the Fbank feature of each 9-second time slice speech signal First, the fast Fourier transform is performed on each frame signal in the time slice, that is,

[0088] (24)

[0089] wherein, represents the preprocessed speech signal of each frame, which is composed of S points, is the speech frequency domain signal after fast Fourier transform.

[0090] Next, the linear spectrum is mapped into the mel-frequency scale, which is a perceptual scale of hearing, and converted to the cepstrum. The formulas for converting between frequency and mel-frequency are

[0091] (25)

[0092] where m denotes the mel-frequency of the speech signal, in Mels, denotes the frequency of the speech signal, in Hz.

[0093] Finally, M mel-scale triangular filters (a mel-filterbank) are selected to act on the frequency-domain energy of each frame of speech signal and logarithmically operated, and the obtained logarithmic spectrum is the Fbank feature, whose calculation formula is

[0094] (26)

[0095] where denotes the m-th transfer function of the mel-filterbank, whose expression is

[0096] (27).

[0097] Thus, the speech feature is the Fbank feature of each 9-second time slice where M denotes the number of mel-filters, and T denotes the number of frame of time slice.

[0098] Step 4: Building and training the capsule graph neural network and the Transformer network: First, for the EEG feature set and the speech feature set obtained in step 3, the two feature sets are respectively divided into 10 equal parts, and one part is taken as the test set in turn, and the remaining 9 parts are taken as the training set for ten-fold cross-validation. Second, a hybrid model combining the capsule graph neural network and the Transformer network is constructed, and the training set is used as the input of the hybrid model. Among them, the training set of EEG features is used as the input of the capsule graph neural network, and the training set of speech features is used as the input of the Transformer network. At the beginning of training, the required hyperparameters in the model are set, and the initial values of the weights and bias parameters in the capsule graph neural network and the Transformer are randomly configured. After each iteration of training, the capsule graph neural network outputs the depression recognition result for EEG features and forms the EEG mass function M1, and the Transformer network outputs the depression recognition result for speech features and forms the speech mass function M2, and the final depression classification decision is made according to M1 and M2 using the D-S decision fusion algorithm. The loss function of the hybrid model includes: marginal loss, cross-entropy loss, and classification loss. In the training process of the model, the Adam optimizer is used to optimize the model, and the weights and bias parameters in the hybrid model are continuously updated until the overall loss function converges, and the ideal depression recognition result can be output.

[0099] The model training strategy adopts ten-fold cross-validation, that is, the EEG power spectrum entropy features and Fbank features are divided into 10 equal parts, and one part of the EEG features and one part of the speech features are selected to form the test set in turn, and the remaining 9 parts form the training set. During the ten training processes of the model, all hyperparameters remain the same.

[0100] Step 4.1: For EEG features, the capsule graph neural network is used for processing, which includes: graph structure input, graph convolution layer, initial capsule generation layer, attention mechanism, graph capsule generation layer, classification capsule generation layer, and EEG mass function M1 output.

[0101] a、Graph structure input: power spectrum entropy features Constructed as a graph structure , wherein is a node set composed of N EEG leads, is an adjacency matrix, is a node feature. First, the correlation coefficient values between nodes are calculated to obtain the Pearson correlation matrix, and the calculation formula of each element is

[0102] (1)

[0103] wherein, represents the first element of the and the Node characteristics The correlation coefficient.

[0104] Compare and threshold The size of the correlation matrix is ​​used to transform it into an adjacency matrix consisting of elements 0 and 1. ,Right now

[0105] (2).

[0106] b. Graph Convolutional Layer: Based on the graph structure input, it employs... Graph convolutional layers learn higher-quality node embeddings; the formula for each graph convolutional layer is...

[0107] (3)

[0108] in, Indicates the first Layer Graph Convolutional Layer The node characteristics of each node, and have , ; Indicates the first The first layer of the graph convolutional layer The node to the first The first in the layer The feature transformation weights of each node constitute a matrix. .

[0109] This represents the activation function, specifically... A function is used to enhance the nonlinear expressive power of features, and its formula is:

[0110] (4).

[0111] c. Initial capsule generation layer: Recombination and stacking of the output of each graph convolutional layer Generate primary capsules; specifically, the output of each graph convolutional layer. Reorganized into A three-dimensional matrix, where Indicates the number of nodes. This represents the number of capsules created at each node. This represents the dimension of each capsule, and ; along By stacking the three-dimensional matrices, we obtain the primary capsule. ,in Primary capsules It can be seen as being caused by It consists of groups of node capsules, each group of node capsules is wherein represents the i-th capsule of the j-th node, .

[0112] d, attention mechanism: used to recalibrate each group of node capsules. Its calculation process is as follows:

[0113] (5)

[0114] wherein, is obtained by tiling, i.e. , represents a fully connected layer, the number of input neurons is , the number of output neurons is , and the attention coefficient of each capsule is obtained by normalizing the output of the fully connected layer , and further, the attention coefficient is used to recalibrate each group of node capsules to obtain .

[0115] e, graph capsule generation layer: the primary capsule after attention calibration is generated into graph capsules through a dynamic routing process (cycling times), wherein represents the number of graph capsules, represents the dimension of each graph capsule.

[0116] f, classification capsule generation layer: the graph capsule is generated into classification capsules through a dynamic routing process (cycling times); wherein represents the number of classification capsules, represents the dimension of each classification capsule.

[0117] g, electroencephalogram mass function Output: calculate the modulus (2-norm) of each capsule in the classification capsule, which corresponds to the recognition probability of different categories; the classification task is two categories (K=2): normal subjects (label 0) and depressed patients (label 1). The model discriminates the probability of each category according to the input features, and obtains and , and further obtains the electroencephalogram basic probability distribution mass function , i.e.

[0118] (6).

[0119] ​​Step 4.2: For speech features, a Transformer network is adopted, which includes: Fbank feature input, lightweight multi-head self-attention module, multi-layer perceptron feedforward module, classification layer, speech mass function Output.

[0120] a. Fbank feature input: Fbank features are taken as the input of the Transformer network, .

[0121] b. Lightweight multi-head self-attention module: input features First, batch normalization (BN) operation is performed to obtain The number of attention heads is set to . Divide into group matrices, and each group is represented as (query matrix), (key matrix), (value matrix), and , i.e., the vector size of each group is transformed to . In order to save computing resources, average pooling (step size s) operation is used to reduce the dimensions of and , so the dimensions of each group and are reduced to , denoted as and respectively. Next, each group's and are subjected to product scaling, flexible maximum value calculation, and dropout strategy to obtain the attention coefficient matrix of each head. Multiply this coefficient matrix with to obtain the output after each attention head, and the calculation formula is represented as:

[0122] (7)

[0123] where .

[0124] Then, concatenate all the attention outputs of the heads, represented by the symbol ||, and feed them into a linear layer; finally, perform a residual operation:

[0125] (8).

[0126] c. Multi-layer perceptron feedforward module: the input of the module is The module is composed of two two-dimensional convolution layers, and the input end is first subjected to batch normalization (BN) operation, and the activation function is used between the two two-dimensional convolution layers The calculation formula of the process is:

[0127] (9)

[0128] (10)

[0129] (11)

[0130] (12)

[0131] wherein, represents the bias of the first convolution layer, represents the learnable weight matrix of the first convolution layer, and respectively represent the dimension size, represents the output channel number, represents the bias of the second convolution layer, represents the learnable weight matrix of the second convolution layer, and respectively represent the dimension size, represents the output channel number.

[0132] Finally, the output is subjected to residual operation, that is,

[0133] (13).

[0134] Next, the processes b and c are continuously repeated, taking as the input of process b, until times of calculation are completed.

[0135] d, classification layer: composed of one fully connected layer and softmax layer, the classification task is two categories: normal subjects (label 0) and depression patients (label 1), and the outputs of the classifier are respectively represented as and .

[0136] e, speech mass function Output: the classifier outputs and constitute the speech mass function , specifically

[0137] (14).

[0138] Step 4.3: D-S evidential theory decision layer fusion:

[0139] Let the classification universe be , and the basic probability assignment mass functions of the two evidences of electroencephalogram features and speech features are and respectively according to step 4.1 and step 4.2. The probability assignment of the universe is calculated as , and the calculation formula of each state probability assignment is

[0140] (15)

[0141] (16)

[0142] where is the conflict coefficient between evidences, ; finally, compare and , determine the true hypothesis with the maximum probability, and set the state corresponding to this maximum probability as the final decision result.

[0143] Step 4.4: Overall loss function of the model:

[0144] In the training process of the model, three loss functions are used to update the parameters in the model, i.e., marginal loss , cross-entropy loss , and classification loss .

[0145] Marginal loss based on capsule graph neural network:

[0146] (17)

[0147] where when the prediction result is , otherwise , denotes the module length (2-norm) of the th classification capsule, denotes the marginal coefficient.

[0148] Cross-entropy loss based on Transformer network:

[0149] (18)

[0150] where denotes the classification label, and when the prediction result is true, otherwise ; denotes the probability of prediction as true.

[0151] Classification loss of D-S decision fusion:

[0152] (19)

[0153] wherein, is the predicted class, is the real class.

[0154] The training target of the model is to minimize the loss function, that is,

[0155] (20)

[0156] wherein, and are proportional coefficients, is the optimal parameter set of the model after training update.

[0157] The test set of the electroencephalogram and the speech feature is respectively input into the trained capsule graph neural network and the mixed model of the Transformer, to obtain the electroencephalogram mass function M1 output by the capsule graph neural network and the speech mass function M2 output by the Transformer network. Then, according to M1 and M2, the D-S evidence theory decision fusion algorithm is adopted to obtain the final depression classification result considering the electroencephalogram and the speech information comprehensively.

[0158] The following is the actual verification of the present application:

[0159] Reference Figure 1 , Figure 1 is the overall flowchart of the present application. First, the electroencephalogram signal of the subject after completing task 1 (closed-eye resting state) and the speech signal after completing task 2 (expert interview, material reading and description) are collected; second, the collected electroencephalogram signal and speech signal are respectively preprocessed to obtain purer electroencephalogram and speech signals; then, the power spectrum entropy feature of the preprocessed electroencephalogram signal is extracted, and the Fbank feature of the preprocessed speech signal is extracted; then, the electroencephalogram feature training set is taken as the input of the capsule graph neural network, and the speech feature training set is taken as the input of the Transformer network, the capsule graph neural network outputs the depression recognition probability based on the electroencephalogram feature, and constitutes the electroencephalogram mass function M1, the Transformer network outputs the depression recognition probability based on the speech feature, and constitutes the speech mass function M2, the D-S evidence theory is used to fuse M1 and M2, and in the training process, the overall loss function of the model is: marginal loss , cross-entropy loss and classification loss The model's loss function is reduced through training until convergence. Finally, the test sets of EEG and speech features are input into the trained model. The DS evidence theory is used to fuse the outputs of the EEG mass function M1 and the speech mass function M2 to obtain the final depression recognition probability. The predicted probabilities of "normal" and "depressed" states are compared, and the hypothesis corresponding to the higher probability value is judged as true, thus obtaining the final depression recognition result.

[0160] See Figure 2 , Figure 2 This is a layered diagram of the capsule graph neural network of the present invention. The capsule graph neural network uses EEG power spectral entropy features as input and includes the following structure: graph structure input, graph convolutional layer, primary capsule generation layer, attention mechanism, graph capsule generation layer, classification capsule generation layer, and EEG mass function M1.

[0161] The layering details for each layer are as follows:

[0162] First, the EEG signal is sliced ​​for 1 second, resulting in a signal matrix of size [size missing]. Where 128 represents the acquisition channels and 250 represents the number of sampling points. Power spectral entropy features of each 1-second slice on the EEG rhythm are extracted. The size of the obtained feature matrix is The 6 represents six rhythm frequency bands: delta (1-3Hz), theta (4-8Hz), alpha (8-13Hz), beta (13-30Hz), low gamma (30-50Hz), and high gamma (50-80Hz).

[0163] a. Graph structure input layer: The extracted power spectral entropy features are Transform it into a graph structure , where the node set for EEG leads, adjacency matrix The size is .

[0164] b. Graph Convolutional Layer: Consists of 3 layers of graph convolutional networks, with the input being... After graph convolution, the output of each layer is obtained as follows: ,in .

[0165] c. Primary capsule generation layer: For each layer output of the graph convolutional layer Reorganize to obtain a three-dimensional matrix ,in Along The second dimension will encompass all Stacking them together yields primary capsules. , primary capsules can be seen as being composed of 128 groups of node capsules, each group of node capsules being , where represents the th node, th capsule of the th node, , .

[0166] d. Attention mechanism: each group of is flattened to get , and is calculated using a fully connected layer with 54 input neurons and 9 output neurons. The attention coefficient of each capsule is obtained from the normalized value of the output of the fully connected layer, and this attention coefficient is used to recalibrate the groups of node capsules to obtain the attention-calibrated primary capsules .

[0167] e. Graph capsule generation layer: the attention-calibrated primary capsules ( ) are subjected to a dynamic routing process (cycled twice) to generate graph capsules , . Where 16 represents the number of graph capsules, and 6 represents the dimension of each graph capsule.

[0168] f. Classification capsule generation layer: the graph capsules are subjected to a dynamic routing process (cycled twice) to generate classification capsules , . Where 2 represents the number of classification capsules, and 6 represents the dimension of each classification capsule.

[0169] g. EEG mass function : the classification task is two classes (K = 2), normal subjects (label 0), and patients with depression (label 1). The modulus of each capsule in the classification capsule is calculated using the 2-norm to obtain and , as the recognition probability of different classes, therefore, the EEG mass function can be expressed as

[0170]

[0171] The hyperparameter settings adopted in the capsule graph neural network are summarized in Table (1):

[0172] Table 1

[0173] .

[0174] Referring toFigure 3 , Figure 3 Layered diagram of the Transformer network of the present application. The Transformer network takes speech Fbank features as input, which includes the following parts: Fbank feature input, lightweight multi-head self-attention module, multi-layer perceptron feedforward module, classification layer, speech mass function .

[0175] The details of each layer are as follows:

[0176] a. Fbank feature input layer: For the preprocessed speech signal, a 9-second time slice is intercepted and Fbank features are extracted. The sampling frequency of the speech signal is 44.1 kHz, the frame length is 25 ms, the frame displacement is 10 ms, and the number of triangular filters on the Mel scale is 26, so the size of the extracted Fbank features is .

[0177] b. Lightweight multi-head self-attention module: refer to Figure 4 , Figure 4 Lightweight multi-head self-attention module in the Transformer network of the present application. The input feature is first subjected to batch normalization operation to obtain . The number of attention heads is set to 8. The input feature is defined as 8 groups , each group of which is transformed into , . In order to save computing resources, average pooling (step size 4) operation is used to reduce the dimensions of and , so the dimensions of each group and are reduced to , denoted by and respectively. Next, each group of and is subjected to product scaling (changed to ), flexible maximum value calculation (changed to ), dropout strategy (changed to ) to obtain the attention coefficient matrix of each head, and the coefficient matrix is multiplied by (changed to ) to obtain the output after each attention head. Finally, all attention heads are spliced and transposed (changed to ), and the final output (changed to ) is obtained after passing through a linear layer.

[0178] c. Multi-layer perceptron feedforward module: refer to Figure 5 , Figure 5 The multi-layer perceptron feedforward module in the Transformer network of the present application is subjected to batch normalization operation, two-layer convolution kernel two-dimensional convolution with Mish function as the activation function. Finally, the output of the convolution layer is subjected to residual calculation to obtain the final output ( ).

[0179] Next, the processes b and c are continued to be repeated, taking as the input of process b until the calculation is completed after a total of 3 times.

[0180] d. Classification layer: composed of one fully connected layer and a softmax layer. The classification task is two categories: normal subjects (label 0) and depression patients (label 1). The outputs of the classifier are respectively and .

[0181] e. Speech mass function Output: classifier output and as the speech mass function , specifically

[0182]

[0183] The summary of the hyperparameter settings adopted in the Transformer is shown in Table (2):

[0184] Table 2

[0185] .

[0186] Model training process: the training set of the electroencephalogram features and the speech features is taken as the input of the overall model to optimize the parameters of the capsule graph neural network and the Transformer network in the model. The optimization strategy of the model uses Adam algorithm, and the loss function adopted includes marginal loss, cross-entropy loss and classification loss. The training strategy of the model uses ten-fold cross-validation method.

[0187] The training optimization strategy adopted by the overall model is shown in Table (3):

[0188] Table 3

[0189] .

[0190] Test process of the model: After the model is optimized, the parameters in the model are frozen, and the test set of EEG and speech features is used as the input of the model. The capsule graph neural network and the Transformer network output the recognition probability of normal subjects and depression patients, respectively. The D-S evidence theory strategy is used to fuse the decision of EEG and speech information to output the final depression recognition probability.

Claims

1. A depression recognition system based on electroencephalogram-voice dual-modal decision fusion, characterized in that: Capsule graph neural network and Transformer network are constructed and trained: S1, for the electroencephalogram feature, a capsule graph neural network is adopted for processing, which includes a graph structure input, a graph convolution layer, an initial capsule generation layer, an attention mechanism, a graph capsule generation layer, a classification capsule generation layer, and an electroencephalogram mass function Output: a、graph structure input: power spectrum entropy features configured as a graph structure wherein is a node set consisting of N electroencephalogram leads, is an adjacency matrix, is a node feature; the feature correlation coefficient between each node obtains a Pearson correlation matrix: （1） wherein, represents in the and the correlation coefficient of the node features of the Comparing and a threshold value The correlation matrix is converted into an adjacency matrix consisting of elements 0 and 1 by comparing i.e. （2）； b. Graph convolutional layer: According to the graph structure input, adopt Each layer of graph convolutional layer learns higher quality node embedding, and the formula of each layer of graph convolutional layer is （3） in, Indicates the first Layer Graph Convolutional Layer The node characteristics of each node, and have , ; Indicates the first The first layer of the graph convolutional layer The node to the first The first in the layer The feature transformation weights of each node constitute a matrix. ; This represents the activation function, specifically... A function is used to enhance the nonlinear expressive power of features, and its formula is: （4）； c. Initial capsule generation layer: Recombination and stacking of the output of each graph convolutional layer Generate primary capsules; convert the output of each graph convolutional layer. Reorganized into a three-dimensional matrix ,in Indicates the number of nodes. This represents the number of capsules created at each node. This represents the dimension of each capsule, and ; along By stacking the three-dimensional matrices, we obtain the primary capsule. ,in Primary capsules It can be seen as being caused by It consists of groups of node capsules, each group of node capsules is ,in Indicates the first The node of the first One capsule, ; d, Attention mechanism: recalibrate each group of node capsules, the process is as follows: （5） wherein, is obtained from tiled, i.e. , denotes a fully connected layer with input neurons and output neurons, the attention coefficient for each capsule is obtained from the normalized values of the output of the fully connected layer, using this attention coefficient to recalibrate the respective group of node capsules results in ; e. Graph capsule generation layer: primary capsules whose attention is calibrated through a dynamic routing procedure (loop Graph capsule generation layer: secondary capsules where represents the number of graph capsules, represents the dimensionality of each graph capsule; f. Classification capsule generation layer: generate the graph capsules Through the dynamic routing process (cycle Second) to generate classification capsules ; Wherein N represents the number of classification capsules, D represents the dimension of each classification capsule; g, electroencephalogram mass function Output: Calculate the modulus length of each capsule in the classification capsule, which corresponds to the identification probability of each category; the model judges the probability of each category according to the input features, and obtains and , and further obtains the electroencephalogram basic probability distribution mass function , that is （6） S2, for speech features, a Transformer network is used, which includes: Fbank feature input, lightweight multi-head self-attention module, multi-layer perceptron feedforward module, classification layer, speech mass function Output: a、Fbank feature input: Fbank features as input to the Transformer network, ; b. Lightweight multi-head self-attention module: input features First, the batch normalization BN operation is performed to obtain The number of attention heads is set to Divide into groups: query matrix , key matrix , value matrix , that is, the matrix size of each group is transformed to , the dimension reduction operation is performed on the dimensions of and , therefore, the dimensions of each group and are , wherein s is the step length, and and are used to represent, respectively, the product scaling, flexible maximum value calculation, and dropout strategy are performed on each group and to obtain the attention coefficient matrix of each head, and the product of this coefficient matrix and is obtained to obtain the output after each attention head. （7） wherein ; then, all heads' attention outputs are concatenated, denoted by ||, and fed into a linear layer; finally, a residual operation is performed: （8）； c. Multi-layer perceptron feedforward module: the input of the module is ; the module is composed of two two-dimensional convolution layers, and the input end is first subjected to batch normalization BN operation, and activation function is used between the two two-dimensional convolution layers （9） （10） （11） （12） in, This indicates the bias of the first convolutional layer. This represents the learnable weight matrix of the first convolutional layer. and They represent The size of the dimension, Indicates the number of output channels. This indicates the bias of the second convolutional layer. This represents the learnable weight matrix of the second convolutional layer. and They represent The size of the dimension, Indicates the number of output channels; finally, the output... Perform residual operations, i.e. （13） The process b and c are continued to As input to process b, until a total of The calculation is completed after 2 d. Classification layer: consists of a fully connected layer and a softmax layer. The output of the classifier is denoted as and ; e. Speech mass function Output: Classifier output And Speech mass function In particular （14）； S3, D-S evidence theory decision layer fusion: The classification universe is set as , the basic probability assignment mass functions of the two kinds of evidence of the brain electrical characteristics and the speech characteristics are obtained according to the step S1 and the step S2 respectively and , the probability assignment of the universe is calculated , wherein the probability assignment of each state is: （15） （16） wherein, is a conflict coefficient between evidences, ; finally, comparing and , the hypothesis with the maximum probability is determined as true, and the state corresponding to this maximum probability is set as the final decision result; S4, The overall loss function of the model: In the training process of the model, three loss functions are used to update the parameters in the model, i.e., marginal loss , cross-entropy loss , and classification loss . Capsule graph neural network based marginal loss: （17） wherein, when the prediction result is a class , otherwise , denotes the module length of the th capsule of the classification, denotes the marginal coefficient; Cross entropy loss based on Transformer network: （18） wherein, represents a classification label, when the prediction result is true , otherwise ; represents a probability that the prediction is true; Classification loss of D-S decision fusion: （19） wherein, is the predicted class, is the true class; The training goal of the model is to minimize the loss function, that is （20） wherein, and is a proportionality coefficient, is the optimal set of parameters of the model after training update.

Images