A method and system for identifying depression based on multimodal data

Through the fusion of multimodal data and neural network model, combined with wav2vec and transfer learning model, the problem of low accuracy in recognition of depression is solved, and efficient depression recognition and early warning is achieved, which is suitable for early intervention in adolescents.

CN118656732BInactive Publication Date: 2024-11-22SOUTH CENTRAL UNIVERSITY FOR NATIONALITIES
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411140974.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing depression identification methods have low accuracy, making it difficult to effectively identify depression characteristics in different modal data, and the cost of biomarker analysis is high and the data collection equipment is high, which limits the accurate diagnosis of depression.

Method used

Using a multimodal data-based depression recognition method, by obtaining speech data and physiological data, using pre-trained wav2vec model and transfer learning model to extract low-dimensional speech features, and fuse it with physiological data to build a neural network model for depression recognition , combining wavelet transform and soft threshold denoising processing, introduce attention mechanism and bidirectional GRU model to optimize feature extraction and classification.

Benefits of technology

It improves the accuracy of depression recognition, reduces noise interference, shortens calculation costs, enhances the model's attention to contextual characteristics, and achieves high-quality depression recognition, which is suitable for timely early warning and early intervention in adolescents.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118656732B_ABST
    Figure CN118656732B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and system for depression identification based on multimodal data, belonging to the technical field of depression identification, obtaining voice data, physiological data and labels of whether or not a plurality of experimental users suffer from depression, and preprocessing the voice data and physiological data respectively; extracting low-dimensional voice features of the preprocessed voice data through a pre-trained wav2vec model and a transfer learning model; fusing the low-dimensional voice features with the preprocessed physiological data into multimodal data, making a data set based on the multimodal data and the labels of whether or not suffering from depression, and training a depression identification model based on a neural network; and identifying depression in a user to be tested through the trained depression identification model. The present invention constructs a depression identification model based on a neural network, which can realize the identification of depression in combination with the voice data and physiological data of the user to be tested, and improve the accuracy of depression identification.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of depression identification, and specifically relates to a method and system for depression identification based on multimodal data. Background Art

[0002] Depression is a common mental health disease. The pathogenesis of depression is still unclear, and there are many challenges in judging depression. The current diagnosis method is more based on self-reports and clinical observations of patients. In addition, there are obvious individual differences in depression. Different subjects have unique reactions and experiences to depression, which also increases the difficulty of depression detection.

[0003] In recent years, research on depression detection technology has seen rapid development, especially with the development of artificial intelligence technology. More and more researchers are using artificial intelligence technology to identify and study depression. Some researchers explore combining data from different sources to more comprehensively describe the characteristics of depression. For example, functional magnetic resonance imaging and electroencephalograms are combined to analyze brain activity patterns, while voice analysis technology is used to capture changes in intonation and rhythm, and language usage habits on social media are combined to discover depressive tendencies. However, since the amount of data required for this method is too large and it relies on high-precision data acquisition equipment, it is not conducive to accurate computer calculations and equipment deployment.

[0004] Since individual differences in depression are more obvious, some researchers have begun to use deep learning technology to analyze large amounts of clinical data and patient reports to identify the unique manifestations of depression. However, during the network training process, a high recognition accuracy rate is often not obtained, and the global optimal solution of the model cannot be found. Therefore, the model cannot show satisfactory performance. Moreover, this method has large differences in the recognition of different modal data, so it is difficult to break through the recognition accuracy rate.

[0005] In addition to the above-mentioned AI-based depression detection methods, researchers are also working hard to find biomarkers related to depression, such as specific protein levels in the blood, to develop objective depression diagnostic tools. However, biomarkers are often difficult to obtain, and the cost of analyzing biomarkers is high, so the technology of using biomarkers to identify depression is also immature.

[0006] Therefore, a new depression identification method is needed that can integrate multimodal data and improve the accuracy of depression identification. Summary of the invention

[0007] In view of this, the present invention proposes a method and system for identifying depression based on multimodal data, which is used to solve the problem that the existing depression identification is not accurate enough.

[0008] In a first aspect, the present invention discloses a method for identifying depression based on multimodal data, the method comprising:

[0009] Acquire voice data, physiological data, and labels of whether or not a plurality of experimental users suffer from depression, and pre-process the voice data and physiological data respectively;

[0010] Extract low-dimensional speech features of preprocessed speech data through the pre-trained wav2vec model and transfer learning model;

[0011] The low-dimensional speech features are fused with the preprocessed physiological data into multimodal data, and a data set is created based on the multimodal data and the label of whether the person suffers from depression;

[0012] Constructing a depression recognition model based on a neural network, and training the depression recognition model using the data set;

[0013] The multimodal data of the user to be tested is extracted, and depression is identified in the user to be tested through the trained depression recognition model.

[0014] On the basis of the above technical solution, preferably, the physiological data includes EEG data and ECG data;

[0015] The preprocessing of the voice data and the physiological data respectively specifically includes:

[0016] Performing wavelet transform and soft threshold denoising processing on the EEG data and the ECG data respectively;

[0017] The data enhancement is performed on the speech data and the EEG data and ECG data after denoising respectively by sliding window truncation.

[0018] On the basis of the above technical solution, preferably, the low-dimensional speech features of the preprocessed speech data extracted by the pre-trained wav2vec model and the transfer learning model specifically include:

[0019] Obtaining a preset number of speech training samples; the labels of the speech training samples are low-dimensional speech features corresponding to the speech sample data;

[0020] Extracting high-dimensional speech features of the speech training sample through a pre-trained wav2vec model;

[0021] Training a transfer learning model using the high-dimensional speech features as input and the labels of the speech training samples as output;

[0022] Extracting high-dimensional speech features of the preprocessed speech data through a pre-trained wav2vec model;

[0023] The high-dimensional speech features of the preprocessed speech data are input into the trained transfer learning model to obtain corresponding low-dimensional speech features.

[0024] On the basis of the above technical solution, preferably, the transfer learning model includes a feature compression module, a forward GRU module, a backward GRU module, a feature splicing module and a feature encoding module;

[0025] The input ends of the forward GRU module and the backward GRU module are respectively connected to the output end of the feature compression module, and the output ends of the forward GRU module and the backward GRU module are respectively connected to the input end of the feature splicing module; the output end of the feature splicing module is connected to the input end of the feature encoding module;

[0026] The feature compression module is used to extract and compress the input high-dimensional speech features and output compressed speech features;

[0027] The forward GRU module is used to extract the hidden state of the compressed speech feature at the current time step and the previous time step, and output the forward hidden feature;

[0028] The backward GRU module is used to extract the hidden state of the compressed speech feature at the current time step and the next time step, and output the backward hidden feature;

[0029] The feature concatenation module is used to concatenate the forward hidden feature and the backward hidden feature to output a speech feature containing context information of the speech data;

[0030] The feature encoding module is used to encode the output of the feature splicing module and output the low-dimensional speech features corresponding to the preprocessed speech data; the data size of the low-dimensional speech features is the same as the size of the preprocessed physiological data.

[0031] On the basis of the above technical solution, preferably, the depression recognition model includes a feature compression module, a position encoding module, an attention module, a bidirectional GRU module and a classification layer connected in sequence;

[0032] The feature compression module is used to extract and compress the input multimodal data and output multimodal fusion features;

[0033] The position encoding module comprises an embedding layer, a position encoding layer and a splicing layer, the input ends of the embedding layer and the position encoding layer are respectively connected to the output end of the feature compression module, and the output ends of the embedding layer and the position encoding layer are respectively connected to the input end of the splicing layer;

[0034] The embedding layer is used to embed multimodal fusion features, the position encoding layer is used to encode the position of multimodal data, and the splicing layer is used to splice the position information with the embedded multimodal fusion features to obtain embedded data;

[0035] The attention module is used to calculate the attention weights of different positions of the embedded data, and calculate the context feature vector through the attention weights;

[0036] The bidirectional GRU module is used to fuse the feature information of the forward time step and the backward time step based on the context feature vector, and perform feature mapping through a fully connected layer;

[0037] The classification layer is used to perform binary classification on the feature mapping results through the softmax function, and output the probability of whether the patient suffers from depression.

[0038] On the basis of the above technical solution, preferably, the feature compression module includes a first convolution subunit, a second convolution subunit, a third convolution subunit, a jump connection subunit and a feature splicing subunit;

[0039] The first convolution subunit, the second convolution subunit, and the third convolution subunit each include a convolution layer, a batch normalization layer, and a ReLU activation layer connected in sequence;

[0040] The first convolution subunit, the second convolution subunit, and the third convolution subunit are connected in sequence to extract and compress the input multi-channel data in sequence to obtain single-channel data;

[0041] The output end of the third convolution subunit is connected to the input end of the feature splicing subunit;

[0042] The input end of the jump connection subunit is connected to the input end of the first convolution subunit, and the output end of the jump connection subunit is connected to the input end of the feature splicing subunit, so as to jump-connect the input data of the feature compression module to the feature splicing subunit, and compress the input multi-channel data through the convolution layer to obtain single-channel data;

[0043] The feature splicing subunit is used to splice the single-channel data output by the third convolution subunit with the single-channel data output by the jump connection subunit.

[0044] On the basis of the above technical solution, preferably, the step of calculating the attention weights of different positions of the embedded data and calculating the context feature vector by the attention weights specifically includes:

[0045] Calculate the attention weight for the embedded data, the calculation formula is as follows:

[0046]

[0047] in, It's location i About Location j The attention weight, i , j =1,2,.., n , n is the sequence length of multimodal fusion features, The location i , j The attention matrix obtained after different linear transformations of the embedded data;

[0048] The context vector is calculated by attention weight, the formula is as follows:

[0049]

[0050] in, It's location j The transformation matrix obtained by multiplying the embedded data of by the attention matrix is It's location i The context vector of .

[0051] In a second aspect, the present invention discloses a system for identifying depression based on multimodal data, the system comprising:

[0052] Data collection module, questionnaire module, cloud server data analysis, display module, early warning notification module

[0053] Data collection module: used to collect voice data and physiological data of multiple experimental users, and obtain labels of whether each experimental user suffers from depression; upload the collected data to the cloud server;

[0054] Data analysis module: deployed on the cloud server, used to pre-process the voice data and physiological data collected by the data collection module respectively; extract low-dimensional voice features of the pre-processed voice data through the pre-trained wav2vec model and the transfer learning model; fuse the low-dimensional voice features with the pre-processed physiological data into multimodal data, and create a data set based on the multimodal data and the label of whether the patient suffers from depression; construct a depression recognition model based on a neural network, and train the depression recognition model through the data set;

[0055] Identification and detection module: deployed on the cloud server, used to extract multimodal data of the user to be tested, identify depression in the user to be tested through the trained depression recognition model, and output the recognition results to the display module;

[0056] Display module: used to display the depression recognition results of the tested user;

[0057] Early warning notification module: used to send early warning notifications to relevant personnel when it is identified that the user to be tested suffers from depression.

[0058] In a third aspect of the present invention, an electronic device is disclosed, comprising: at least one processor, at least one memory, a communication interface and a bus;

[0059] Wherein, the processor, memory, and communication interface communicate with each other via the bus;

[0060] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the method according to the first aspect of the present invention.

[0061] According to a fourth aspect of the present invention, a computer-readable storage medium is disclosed, wherein the computer-readable storage medium stores computer instructions, and the computer instructions enable a computer to implement the method described in the first aspect of the present invention.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] 1) The present invention extracts high-quality speech features from speech data through a pre-trained wav2vec model and a transfer learning model, integrates the high-quality speech features with multiple physiological data into multimodal data, and inputs them into a neural network-based depression recognition model for training. Through the depression recognition model, the speech data and physiological data of the user to be tested can be combined to realize the recognition of depression, thereby greatly improving the accuracy of depression recognition.

[0064] 2) In the preprocessing of physiological data, the present invention introduces a soft threshold denoising method based on wavelet packet transform, and filters the physiological data of different frequency bands after wavelet transform decomposition, which can improve the quality of EEG data and ECG data, make them contain more information related to depression, and reduce noise interference.

[0065] 3) The present invention extracts high-dimensional speech features of the preprocessed speech data through a pre-trained wav2vec model, and then further compresses the data through a feature compression module of a transfer learning model, so that the width of the speech data matches the physiological parameters to shorten the gap between data of different modalities, and extracts contextual feature information related to depression through a forward GRU module, a backward GRU module, and a feature splicing module, so that the transfer learning model can accurately learn features related to depression from the large-scale high-dimensional speech features obtained by the wav2vec model; the speech feature extraction method based on the wav2vec model and the transfer learning model of the present invention can reduce the complexity and computational cost of the model while retaining key information, and ultimately obtain low-dimensional speech features that are most conducive to depression identification.

[0066] 4) The present invention introduces a feature extraction module and an attention mechanism into the bidirectional GRU model to form a lightweight depression detection model. The feature extraction module avoids problems such as gradient vanishing or gradient exploding in the model. The attention mechanism calculates the attention weight of the embedded data, and calculates the context vector through the attention weight. Finally, the forward and backward context information is fused through the bidirectional GRU network, so that the model itself pays more attention to the information related to depression in the context, outputs high-quality multimodal features, optimizes the problem that the traditional network model cannot pay attention to the connection between contextual features, and further improves the accuracy of depression recognition.

[0067] 5) The depression identification system of the present invention has the function of timely warning and early intervention of adolescent depression, is suitable for scenes such as schools and hospitals, and has great use value. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0069] Figure 1 This is a flow chart of a method for identifying depression based on multimodal data according to the present invention;

[0070] Figure 2 A schematic diagram of the structure of the transfer learning model constructed for the present invention;

[0071] Figure 3 It is a structural schematic diagram of the feature compression module of the present invention;

[0072] Figure 4 It is a structural schematic diagram of the depression recognition model of the present invention;

[0073] Figure 5 This is a schematic diagram of the system structure for identifying depression based on multimodal data according to the present invention. DETAILED DESCRIPTION

[0074] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] See also Figure 1The present invention discloses a method for identifying depression based on multimodal data, the method comprising:

[0076] S1. Acquire speech data, physiological data, and labels of whether or not a plurality of experimental users suffer from depression, and pre-process the speech data and physiological data respectively.

[0077] We interviewed multiple experimental users through questionnaires, collected voice data and physiological data of multiple experimental users using voice, EEG and ECG collection equipment, and recorded labels of whether each experimental user suffered from depression. Among them, the physiological data included EEG data and ECG data. By integrating EEG data, ECG data, voice data and other different modal data, it is helpful to improve the accuracy of depression identification.

[0078] The present invention takes dual-channel EEG data and single-channel ECG data as examples to preprocess the speech data and physiological data respectively, and the preprocessing specifically includes:

[0079] S11: Preliminary denoising processing.

[0080] For the collected EEG data and ECG data, the power frequency interference is removed by using a 50HZ notch filter, and simple denoising is performed, with a sampling rate of 250HZ.

[0081] The collected voice data is subjected to simple denoising processing, and the sampling rate is 16000HZ.

[0082] S12, performing wavelet transform and soft threshold denoising processing on the EEG data and the ECG data respectively.

[0083] In an embodiment of the present invention, the sym8-based wavelet function is used to perform 6-layer wavelet decomposition on the EEG data and the ECG data, and a dynamic threshold method is used to dynamically adjust the soft threshold of denoising, and finally the baseline drift of the ECG data and the EEG data is removed.

[0084] Specifically, different modal noises have different corresponding noise thresholds during the denoising process. To facilitate processing, the present invention uses a dynamic threshold method to calculate a universal threshold T as a soft threshold for denoising. First, the standard deviation σ of the ECG and EEG data of a single sample is calculated, and then the sample length mean L of different modalities is calculated. t After logarithmic transformation, take the second root, that is, , the product of the two is obtained to obtain the global universal threshold of different modes The universal threshold method can take into account the impact of data volume, use data length as one of the dynamic threshold factors, and eliminate the impact of dynamic threshold setting caused by different lengths of data in different modalities.

[0085] S13, performing data enhancement on the speech data and the EEG data and ECG data after denoising respectively by means of sliding window truncation.

[0086] Due to insufficient data volume, the present invention also uses a 3s sliding window and 1s step-length truncation method to enhance the EEG data and ECG data after denoising. For speech data, the same sliding window truncation method is used for data enhancement.

[0087] S2. Extract low-dimensional speech features of the preprocessed speech data through the pre-trained wav2vec model and the transfer learning model.

[0088] Step S2 specifically includes the following sub-steps:

[0089] S21. Obtain a preset number of speech training samples; the labels of the speech training samples are low-dimensional speech features corresponding to the speech sample data.

[0090] S22. Extracting high-dimensional speech features of the speech training sample through a pre-trained wav2vec model.

[0091] In an embodiment of the present invention, the wav2vec model adopts the wav2vec 2.0 pre-trained model. Wav2vec is a speech recognition model that converts an audio signal into a high-dimensional speech representation through an autoencoder, and can better capture the speech features in the audio signal.

[0092] S23. Construct a transfer learning model, and train the transfer learning model by taking the high-dimensional speech features as input and taking the labels of the speech training samples as output.

[0093] like Figure 2 The figure shows a schematic diagram of the structure of the transfer learning model constructed by the present invention, wherein the transfer learning model includes a feature compression module, a forward GRU module, a backward GRU module, a feature splicing module and a feature encoding module;

[0094] Among them, the input ends of the forward GRU module and the backward GRU module are respectively connected to the output end of the feature compression module, and the output ends of the forward GRU module and the backward GRU module are respectively connected to the input end of the feature splicing module; the output end of the feature splicing module is connected to the input end of the feature encoding module.

[0095] The feature compression module is used to extract and compress the input high-dimensional speech features and output compressed speech features.

[0096] like Figure 3: It is a schematic diagram of the structure of the feature compression module of the present invention, and the feature compression module includes a first convolution subunit, a second convolution subunit, a third convolution subunit, a skip connection subunit and a feature splicing subunit;

[0097] The first convolution subunit, the second convolution subunit, the third convolution subunit and the skip connection subunit have the same structure, and all include a convolution layer, a batch normalization layer and a ReLU activation layer connected in sequence;

[0098] The first convolution subunit, the second convolution subunit, and the third convolution subunit are connected in sequence to extract and compress the input multi-channel data in sequence to obtain single-channel data;

[0099] The output end of the third convolution subunit is connected to the input end of the feature splicing subunit;

[0100] The input end of the jump connection subunit is connected to the input end of the first convolution subunit, and the output end of the jump connection subunit is connected to the input end of the feature splicing subunit, so as to jump-connect the input data of the feature compression module to the feature splicing subunit, and compress the input multi-channel data through the convolution layer to obtain single-channel data;

[0101] The feature splicing subunit is used to splice the single-channel data output by the third convolution subunit with the single-channel data output by the jump connection subunit to obtain one-dimensional speech feature data.

[0102] Specifically, according to the voice data format, the time step needs to be used as the channel input feature compression module. Therefore, the input channels of the first and second convolution subunits and the jump connection subunit are all 1024. The third convolution subunit will convert the input multi-channel data into single-channel data. The convolution kernel size should be set to the same 1x3, and then after the maximum pooling, normalization, L1 regularization and ReLU activation operations, the structure of the jump connection subunit is the same. The three convolution subunits and a layer of jump connection subunits are directly spliced ​​and output, so that the spliced ​​data can contain some original data information.

[0103] The forward GRU module is used to extract the hidden states of the compressed speech features at the current time step and the previous time step, and output the forward hidden features.

[0104] The backward GRU module is used to extract the hidden state of the compressed speech feature at the current time step and the next time step, and output the backward hidden feature. The feature concatenation module is used to concatenate the forward hidden feature and the backward hidden feature, and output the speech feature containing the context information of the speech data.

[0105] In an embodiment of the present invention, the forward GRU module, the backward GRU module, and the feature splicing module constitute a Bi-GRU network, and the Bi-GRU network can combine the two information transmission modes of the forward GRU and the backward GRU. Among them, the forward GRU module can realize the extraction of the hidden state of the current time step and the previous time step, output the forward hidden feature, and can take into account the influence of the previous element on the current element in the sequence. The backward GRU module is opposite to the forward GRU module, and can capture the subsequent information, realize the extraction of the hidden state of the current time step and the next time step, and output the backward hidden feature; and the feature splicing module splices the hidden state generated by the forward module and the backward GRU module at the same time step to obtain the speech feature containing the context information of the speech data. The features extracted by the migration model of the present invention will contain the context information of the entire sequence, which is helpful to extract richer and more meaningful feature representations, thereby improving the model's representation ability for sequence data.

[0106] The feature encoding module is used to encode the output of the feature concatenation module and output the low-dimensional speech features corresponding to the preprocessed speech data; the data size of the low-dimensional speech features is the same as the size of the preprocessed physiological data. In the feature encoding module, the number of fully connected layers is designed according to the size of its input and output, and a ReLU activation function is added after each encoding layer to enable it to fit nonlinear relationships.

[0107] Specifically, in the embodiment of the present invention, the number of neurons in the Bi-GRU network is set to 128, and the speech feature obtained by the forward GRU module is :{ x 1 , x 2 ,…, x 128}, the speech features obtained by the backward GRU module are :{ x 1 , x 2 ,…, x 128}, and finally concatenate the outputs of the last hidden layers of the two to obtain X :{ x 1 , x 2 ,…, x 256}. Since the slice length of ECG data and EEG data is 750, it is necessary to XThe input feature coding layer performs speech feature encoding, which is completed by the fully connected layer and the activation layer so that the output can match the input size of the multimodal model. The input scale of the fully connected layer is 256 and the output scale is set to 750.

[0108] The present invention trains a transfer learning model through high-dimensional speech features and sample labels extracted by a pre-trained wav2vec model, so that the transfer learning model can compress the feature output of the upstream model and obtain the low-dimensional speech features that are most conducive to depression identification, while shortening the gap between different modal data to meet the input size of the multimodal model.

[0109] S24. Extracting high-dimensional speech features of the preprocessed speech data through a pre-trained wav2vec model.

[0110] S25, inputting the high-dimensional speech features of the pre-processed speech data into the trained transfer learning model to obtain corresponding low-dimensional speech features. The speech feature extraction method based on the wav2vec model and the transfer learning model of the present invention can reduce the complexity and computational cost of the model while retaining key information, and finally obtain the low-dimensional speech features that are most conducive to depression identification.

[0111] S3. Fusing the low-dimensional speech features with the preprocessed physiological data into multimodal data, and creating a data set based on the multimodal data and the label of whether the individual suffers from depression.

[0112] The low-dimensional speech features extracted by the transfer learning model are spliced ​​with the ECG data and EEG data into multimodal data, and a data set is created based on the multimodal data and the corresponding labels of whether the patient suffers from depression.

[0113] S4. Construct a depression recognition model based on a neural network, and train the depression recognition model using the data set.

[0114] The present invention constructs a depression recognition model based on a neural network, such as Figure 4 The figure shows a schematic diagram of the structure of a depression recognition model based on a neural network of the present invention. The depression recognition model comprises a feature compression module, a position encoding module, an attention module, a bidirectional GRU module and a classification layer which are connected in sequence.

[0115] The feature compression module is used to extract and compress the input multimodal data and output multimodal fusion features;

[0116] The feature compression module has the same main structure as the feature compression module of the transfer learning model, with only the number of channels being different. In the depression recognition model of the present invention, the number of channels of the input data of the feature compression module should be adjusted to 4, of which two are EEG data channels, one is an ECG data channel, and one is a voice data channel.

[0117] The position encoding module includes an embedding layer, a position encoding layer and a splicing layer, wherein the input ends of the embedding layer and the position encoding layer are respectively connected to the output end of the feature compression module, and the output ends of the embedding layer and the position encoding layer are respectively connected to the input end of the splicing layer; wherein the embedding layer is used to embed multimodal fusion features, the position encoding layer is used to encode the position of multimodal data, and the splicing layer is used to splice the position information with the embedded multimodal fusion features to obtain embedded data; in an embodiment of the present invention, the embedding dimension is set to 512, that is, each feature is embedded as a 512-length data.

[0118] The attention module is used to calculate the attention weights of different positions above and below the full text, and normalize them through the softmax function to weighted sum all the attention weights at the same time step, so as to obtain the feature information of different time steps and contexts.

[0119] Specifically, in the embodiment of the present invention, the number of attention heads is set to 8, and the feedforward inner layer dimension is set to 2048. The attention weight is calculated for the embedded data, and the calculation formula is as follows:

[0120]

[0121] in, It's location i About Location j The attention weight, i , j =1,2,.., n , n is the sequence length of multimodal fusion features, The location i , j The attention matrix obtained after different linear transformations of the embedded data;

[0122] The context vector is calculated by attention weight, the formula is as follows:

[0123]

[0124] in, It's location j The transformation matrix obtained by multiplying the embedded data of by the attention matrix is It's location i The context vector here also needs to be probabilistically normalized using the softmax function to obtain normalized features at different positions.

[0125] The bidirectional GRU module is used to fuse the feature information of the forward time step and the backward time step based on the context feature vector, and perform feature mapping through a fully connected layer;

[0126] In the bidirectional GRU module, the forward GRU unit includes a reset gate, an update gate, a candidate hidden state gate, and a forward hidden state. The backward GRU unit is similar to the forward GRU unit, except that the forward hidden unit is replaced by a backward hidden unit so that it considers the impact of the information after the time step on the current data. The context vector obtained by the attention module is input into the bidirectional GRU network, and the feature information of the forward time step and the backward time step is further extracted, and the feature information of the forward time step and the backward time step is fused. The fused features are then encoded, that is, the output features are mapped to data of length 2 through a fully connected layer.

[0127] The classification layer is used to perform binary classification on the feature mapping results through the softmax function, and output the probability of whether the patient suffers from depression.

[0128] After feature mapping, the classification layer is connected to classify the multimodal data. The model is trained using the binary classification label of whether or not the patient suffers from depression. The loss function selects the binary cross entropy loss function. The specific formula is as follows:

[0129]

[0130] In the above formula, is the loss function, p is the actual label (0 or 1), q is the probability that the model predicts the positive class. Through this formula, the error between the true label and the predicted label can be obtained, and the model can be optimized based on the loss function.

[0131] In the embodiment of the present invention, the optimizer is set to the Adam optimizer. This optimization function combines the momentum method to calculate the exponential moving average (EMA), thereby realizing dynamic learning rate adjustment and reaching the upper limit of the model performance. The formula is as follows:

[0132]

[0133] Where α is the initial state learning rate, is the weighted first-order moment of the gradient, is the second-order moment, is the weight at the current moment, is the weight at the next moment, is a constant, This is to avoid the problem that the learning rate iteration is 0 when the initial state or distribution is 0.

[0134] In the embodiment of the present invention, the initial learning rate of the optimizer is set to 0.1 to optimize the model. This optimization function can reduce the momentum in the flat area or near the optimal solution, and reduce the learning rate. However, since the learning rate in the flat area is too small, the model cannot be optimized to the global optimal solution. Therefore, the present invention introduces a cosine annealing restart mechanism to adjust the learning rate, and periodically restarts the learning rate to make it jump out of the local optimal area and reach the global optimal solution as much as possible. The specific adjustment formula is as follows:

[0135]

[0136] In the above formula Represents the current moment t The learning rate, represents the minimum learning rate at the end of the cycle, is the maximum learning rate at the beginning of the cycle, usually Several times of represents the current number of iterations (ie, the number of epochs), is the length of a single cycle, and the learning rate is Reduce to The number of iterations required. It can be seen from the above formula that the optimizer can periodically restart the learning rate when encountering a local optimum, that is, re-adjust the learning rate to the maximum value, thereby jumping out of the local optimal solution and then finding the global optimal solution. In the present invention, the maximum learning rate is set to 0.001, the minimum learning rate is 0, and the restart period is 60.

[0137] In the above-mentioned improved lightweight depression recognition model, the feature compression module and the bidirectional GRU network enable the model itself to pay more attention to depression-related information in the context, output high-quality multimodal features, and add L1 regularization to the network to reduce the risk of model overfitting. At the same time, the cosine annealing restart strategy is used to enable the optimizer to calculate the global optimal solution of the model, thereby further improving the accuracy.

[0138] S5. Extract multimodal data of the user to be tested, and use the trained depression recognition model to identify depression in the user to be tested.

[0139] Acquire the voice data and physiological data of the user to be tested, preprocess the physiological data using the same preprocessing method as step S1, then extract low-dimensional voice features using the same feature extraction method as step S2, and then fuse the preprocessed physiological data and low-dimensional voice features of the user to be tested into multimodal data.

[0140] Based on the above method embodiments, the present invention also proposes a system for identifying depression based on multimodal data, such as Figure 5 As shown, the system comprises:

[0141] Data collection module, questionnaire module, cloud server data analysis, display module, early warning notification module

[0142] Data collection module: used to collect voice data and physiological data of multiple experimental users, and obtain labels of whether each experimental user suffers from depression; upload the collected data to the cloud server;

[0143] Data analysis module: deployed on the cloud server, used to pre-process the voice data and physiological data collected by the data collection module respectively; extract low-dimensional voice features of the pre-processed voice data through the pre-trained wav2vec model and the transfer learning model; fuse the low-dimensional voice features with the pre-processed physiological data into multimodal data, and create a data set based on the multimodal data and the label of whether the patient suffers from depression; construct a depression recognition model based on a neural network, and train the depression recognition model through the data set;

[0144] Identification and detection module: deployed on the cloud server, used to extract multimodal data of the user to be tested, identify depression in the user to be tested through the trained depression recognition model, and output the recognition results to the display module;

[0145] Display module: used to display the depression recognition results of the tested user;

[0146] Early warning notification module: used to send early warning notifications to relevant personnel when it is identified that the user to be tested suffers from depression.

[0147] The system for identifying depression based on multimodal data proposed in the present invention is suitable for scenarios such as schools and hospitals. The early warning notification module mainly notifies the counselors and school psychologists of the test results according to the college or department to be tested, which facilitates the counselors to provide timely psychological counseling to the tested persons, and is also conducive to the psychologists to formulate treatment plans suitable for the tested persons, etc. Timely early warning and early intervention of adolescent depression can be achieved.

[0148] Specifically, when the user to be tested uses this system to detect depression, the user needs to measure the ECG data, EEG data and voice data of no less than 3s and transmit them to the server. After the data is uploaded, it is first automatically preprocessed and low-dimensional voice feature extracted to obtain multimodal data. Then the depression recognition model classifies the multimodal data of the user to be tested and displays the test results in the display module. If the user to be tested has a tendency to depression, the test results will be sent to the counselor of the department to which he belongs based on the basic information of the user to be tested to help him provide psychological counseling for the student.

[0149] The above system embodiments are implemented based on the method embodiments. Please refer to the method embodiments for a brief description of the system embodiments.

[0150] The present invention also discloses an electronic device, comprising: at least one processor, at least one memory, a communication interface and a bus; wherein the processor, memory, and communication interface communicate with each other through the bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the aforementioned method of the present invention.

[0151] The present invention also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, wherein the computer instructions enable the computer to implement all or part of the steps of the method described in the embodiment of the present invention. The storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk, and other media that can store program codes.

[0152] The system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be distributed on multiple network units. A person skilled in the art may select some or all of the modules according to actual needs to achieve the purpose of the solution of this embodiment without creative work.

[0153] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for identifying depression based on multimodal data, characterized in that: The method comprises: Acquire voice data, physiological data, and labels of whether or not a plurality of experimental users suffer from depression, and pre-process the voice data and physiological data respectively; Extract low-dimensional speech features of preprocessed speech data through the pre-trained wav2vec model and transfer learning model; The low-dimensional speech features are fused with the preprocessed physiological data into multimodal data, and a data set is created based on the multimodal data and the label of whether the person suffers from depression; Constructing a depression recognition model based on a neural network, and training the depression recognition model using the data set; Extract multimodal data of the user to be tested, and use the trained depression recognition model to identify depression in the user to be tested; The method of extracting low-dimensional speech features of preprocessed speech data by using the pre-trained wav2vec model and the transfer learning model specifically includes: Obtaining a preset number of speech training samples; the labels of the speech training samples are low-dimensional speech features corresponding to the speech sample data; Extracting high-dimensional speech features of the speech training sample through a pre-trained wav2vec model; Training a transfer learning model using the high-dimensional speech features as input and the labels of the speech training samples as output; Extracting high-dimensional speech features of the preprocessed speech data through a pre-trained wav2vec model; Inputting the high-dimensional speech features of the preprocessed speech data into the trained transfer learning model to obtain corresponding low-dimensional speech features; The transfer learning model includes a feature compression module, a forward GRU module, a backward GRU module, a feature splicing module and a feature encoding module; The input ends of the forward GRU module and the backward GRU module are respectively connected to the output end of the feature compression module, and the output ends of the forward GRU module and the backward GRU module are respectively connected to the input end of the feature splicing module; the output end of the feature splicing module is connected to the input end of the feature encoding module; The feature compression module is used to extract and compress the input high-dimensional speech features and output compressed speech features; The forward GRU module is used to extract the hidden state of the compressed speech feature at the current time step and the previous time step, and output the forward hidden feature; The backward GRU module is used to extract the hidden state of the compressed speech feature at the current time step and the next time step, and output the backward hidden feature; The feature concatenation module is used to concatenate the forward hidden feature and the backward hidden feature to output a speech feature containing context information of the speech data; The feature encoding module is used to encode the output of the feature splicing module and output the low-dimensional speech features corresponding to the preprocessed speech data; the data size of the low-dimensional speech features is the same as the size of the preprocessed physiological data.

2. The method for identifying depression based on multimodal data according to claim 1, characterized in that: The physiological data includes electroencephalogram data and electrocardiogram data; The preprocessing of the voice data and the physiological data respectively specifically includes: Performing wavelet transform and soft threshold denoising processing on the EEG data and the ECG data respectively; The data enhancement is performed on the speech data and the EEG data and ECG data after denoising respectively by sliding window truncation.

3. The method for identifying depression based on multimodal data according to claim 1, characterized in that: The depression recognition model includes a feature compression module, a position encoding module, an attention module, a bidirectional GRU module and a classification layer connected in sequence; The feature compression module is used to extract and compress the input multimodal data and output multimodal fusion features; The position encoding module comprises an embedding layer, a position encoding layer and a splicing layer, the input ends of the embedding layer and the position encoding layer are respectively connected to the output end of the feature compression module, and the output ends of the embedding layer and the position encoding layer are respectively connected to the input end of the splicing layer; The embedding layer is used to embed multimodal fusion features, the position encoding layer is used to encode the position of multimodal data, and the splicing layer is used to splice the position information with the embedded multimodal fusion features to obtain embedded data; The attention module is used to calculate the attention weights of different positions of the embedded data, and calculate the context feature vector through the attention weights; The bidirectional GRU module is used to fuse the feature information of the forward time step and the backward time step based on the context feature vector, and perform feature mapping through a fully connected layer; The classification layer is used to perform binary classification on the feature mapping results through the softmax function, and output the probability of whether the patient suffers from depression.

4. The method for identifying depression based on multimodal data according to claim 3, characterized in that: The feature compression module includes a first convolution subunit, a second convolution subunit, a third convolution subunit, a jump connection subunit and a feature splicing subunit; The first convolution subunit, the second convolution subunit, and the third convolution subunit each include a convolution layer, a batch normalization layer, and a ReLU activation layer connected in sequence; The first convolution subunit, the second convolution subunit, and the third convolution subunit are connected in sequence to sequentially extract and compress the input multi-channel data to obtain single-channel data; The output end of the third convolution subunit is connected to the input end of the feature splicing subunit; The input end of the jump connection subunit is connected to the input end of the first convolution subunit, and the output end of the jump connection subunit is connected to the input end of the feature splicing subunit, so as to jump-connect the input data of the feature compression module to the feature splicing subunit, and compress the input multi-channel data through the convolution layer to obtain single-channel data; The feature splicing subunit is used to splice the single-channel data output by the third convolution subunit with the single-channel data output by the jump connection subunit.

5. The method for identifying depression based on multimodal data according to claim 3, characterized in that: The step of calculating the attention weights of different positions of the embedded data and calculating the context feature vector by using the attention weights specifically includes: Calculate the attention weight for the embedded data, the calculation formula is as follows: ; in, It's location i About Location j The attention weight, i , j =1,2,.., n , n is the sequence length of multimodal fusion features, The location i , j The attention matrix obtained after different linear transformations of the embedded data; The context vector is calculated by attention weights, and the formula is as follows: ; in, It's location j The transformation matrix obtained by multiplying the embedded data of by the attention matrix is It's location i The context vector of .

6. A system for identifying depression based on multimodal data using the method according to any one of claims 1 to 5, characterized in that: The system comprises: Data collection module: used to collect voice data and physiological data of multiple experimental users, and obtain labels of whether each experimental user suffers from depression; upload the collected data to the cloud server; Data analysis module: deployed on the cloud server, used to pre-process the voice data and physiological data collected by the data collection module respectively; extract low-dimensional voice features of the pre-processed voice data through the pre-trained wav2vec model and the transfer learning model; fuse the low-dimensional voice features with the pre-processed physiological data into multimodal data, and create a data set based on the multimodal data and the label of whether the patient suffers from depression; construct a depression recognition model based on a neural network, and train the depression recognition model through the data set; The method of extracting low-dimensional speech features of preprocessed speech data by using the pre-trained wav2vec model and the transfer learning model specifically includes: Obtaining a preset number of speech training samples; the labels of the speech training samples are low-dimensional speech features corresponding to the speech sample data; Extracting high-dimensional speech features of the speech training sample through a pre-trained wav2vec model; Training a transfer learning model using the high-dimensional speech features as input and the labels of the speech training samples as output; Extracting high-dimensional speech features of the preprocessed speech data through a pre-trained wav2vec model; Inputting the high-dimensional speech features of the preprocessed speech data into the trained transfer learning model to obtain corresponding low-dimensional speech features; Identification and detection module: deployed on the cloud server, used to extract multimodal data of the user to be tested, identify depression in the user to be tested through the trained depression recognition model, and output the recognition results to the display module; Display module: used to display the depression recognition results of the tested user; Early warning notification module: used to send early warning notifications to relevant personnel when it is identified that the user to be tested suffers from depression.

7. An electronic device, characterized in that: include: at least one processor, at least one memory, a communication interface, and a bus; Wherein, the processor, memory, and communication interface communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions enable a computer to implement the method according to any one of claims 1 to 5.