A multimodal depression recognition system based on multi-level feature fusion

Abstract

The present invention discloses a multimodal depression recognition system based on multi-level feature fusion, which relates to the technical field of multimodal fusion. The system comprises: an acquisition unit: used for acquiring multimodal depression data, preprocessing the multimodal depression data, and obtaining multimodal data; a first feature unit: used for performing feature extraction on the multimodal data to obtain low-level depression features of a single modality; performing feature extraction on the low-level depression features to obtain high-level depression features of a single modality; a fusion unit: used for time dimension, space dimension and the high-level depression features to obtain time-space fusion depression features; a relationship graph unit: used for the time-space fusion depression features and the high-level depression features to obtain a modal fusion relationship graph E; a second feature unit: used for processing the balance relationship of the modalities of the modal fusion relationship graph E to obtain multimodal depression features; a prediction unit: used for inputting the multimodal depression features into a depression prediction module to obtain a prediction result, which can solve the problem that the existing multimodal depression recognition model lacks attention to the high emotion perception features within each modality and the processing of the value differences within the modality itself, resulting in poor accuracy and generalization ability of model prediction.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal fusion, and in particular to a multimodal depression recognition system based on multi-level feature fusion. Background Art

[0002] At present, depression recognition is mainly based on multimodal recognition (using multiple information sources in text, voice, and vision). This method collects multiple physiological feature data such as the respondent's voice, movement, and electroencephalogram, and inputs the feature data into the depression recognition network model for fusion and training. At the same time, the model parameters are updated based on the output of the network model. Finally, a multimodal depression recognition model is obtained to predict whether the respondent is depressed.

[0003] However, in this approach, features with high emotion perception are not taken into account within the modality, and features within the modality are not completely associated with depression. Also, in the multimodal fusion process, only the fusion and balance between modalities are focused on, and there is a lack of processing of value differences within the modalities themselves, which results in poor prediction accuracy and generalization ability of the multimodal depression recognition model. Summary of the invention

[0004] In order to solve the problem that the existing multimodal depression recognition model lacks attention to the high emotion perception features within each modality and lacks processing of the value differences within the modality itself, resulting in poor model prediction accuracy and generalization ability, the present invention provides a multimodal depression recognition system based on multi-level feature fusion, the system comprising:

[0005] An acquisition unit: used for acquiring multimodal depression data based on an acquisition device, and preprocessing the multimodal depression data based on a processor to obtain multimodal data;

[0006] A first feature unit is used for performing feature extraction on the multimodal data to obtain a single-modal low-level depression feature; performing feature extraction on the low-level depression feature to obtain a single-modal high-level depression feature;

[0007] Fusion unit: used for the time dimension, space dimension and the advanced depression feature to obtain the spatiotemporal fusion depression feature;

[0008] Relationship graph unit: used for the spatiotemporal fusion depression feature and the advanced depression feature to obtain a modality fusion relationship graph E;

[0009] The second feature unit is used to process the balance relationship of the modes of the modal fusion relationship graph E to obtain a multi-modal depression feature;

[0010] Prediction unit: used for inputting the multimodal depression features into the depression prediction model, obtaining the prediction results, and transmitting the prediction results to the display device.

[0011] Modality definition: Modality representation is the process of extracting semantic information from one or more modal data in machine learning. It can be divided into two forms: unimodal representation and multimodal representation.

[0012] Single-modal representation: linear or nonlinear mapping is performed on a single modal information to obtain a high-order semantic feature representation of the modal information. The single modality can be a sentence, image, video, sound, etc.

[0013] Multimodal representation: Based on unimodal representation, the results of unimodal representation are constrained to achieve shared semantic representation between modalities or semantic consistency between modalities. Multimodal representation processes the information of each modality in mixed data so that modalities with similar semantics have similar representation results. Unimodal representation is the basis of multimodal representation, and multimodal representation should contain information of each modality in mixed data.

[0014] Mel-frequency cepstral coefficient (MFCC) is an important speech feature. Its unique cepstral-based feature extraction method is more in line with the human hearing principle. It is based on the linear transformation of the logarithmic energy spectrum of the nonlinear Mel scale of the sound frequency.

[0015] The Geneva Minimalist Acoustic Parameter Set (eGeMAPS), an extension of GeMAPS, is a set of acoustic features rich in emotional information. It consists of 25 low-dimensional features, including frequency, energy, duration, and spectrogram features in audio.

[0016] MFCC features and eGeMAPS features are selected as input features of the audio modality to more comprehensively learn the potential depressive features in the audio.

[0017] Wav2Vec2 is an open-source pre-trained model for automatic speech recognition (ASR) and is used to process speech tasks. It is a model that has been trained on the Librispeech dataset. The model has learned acoustic features from the audio dataset in advance and has the knowledge to extract acoustic features. Wav2Vec2 feature representation: The features are obtained by extracting features from the output of the last hidden layer of the Wav2Vec2 pre-trained model.

[0018] The DepRoBERTa model is a model trained on the RoBERTa model using a depression dataset. Compared with the traditional RoBERTa model, it can better capture the emotional characteristics of depression. The RoBERTa model is a pre-trained language model based on the Transformer architecture, which is a variant of the BERT model.

[0019] The first video feature AUs, the second video feature Gaze, and the third video feature Pose refer to facial expression features, gaze direction features, and head posture features, respectively.

[0020] Tensors are multidimensional arrays, and node tensors represent nodes of a graph using multidimensional arrays.

[0021] The principle of the invention is as follows: selecting multimodal features with high emotional representation and high representation ability for depressive emotions, and paying more attention to features with high emotional value, can improve the recognition effect of depression; using a feedforward neural network for audio modalities and video modalities containing multiple internal features to allow the features to be fused in the time and space dimensions respectively, which can associate the information between the modalities and reduce the value difference between the features; constructing a modal fusion relationship graph according to the fusion method between modalities, and using a graph attention mechanism to balance this relationship, which can achieve a balance between modalities, thereby improving the accuracy and generalization ability of model prediction.

[0022] Regarding the selection of multimodal features, MFCC features with subject information in the interviewee’s audio, eGeMAPS features with high emotion representation, and Wav2Vec2 features with prior knowledge were selected; expression features, gaze direction features, and head posture features in the interviewee’s video were selected; for text features, the DepRoBERTa model, a pre-trained model with prior knowledge, was used to obtain features with high representation ability for depressive emotions; paying more attention to features with high emotional value can improve the recognition effect of depression.

[0023] Furthermore, the acquisition unit specifically includes:

[0024] Data unit: used for acquiring the multimodal depression data based on the acquisition device;

[0025] A division unit: configured to divide the multimodal depression data based on a file format to obtain first data, wherein the first data includes audio data, video data, and text data;

[0026] Audio unit: used for extracting features from the audio data to obtain audio features, where the audio features include a first audio feature MFCC, a second audio feature eGeMAPS, and a third audio feature Wav2Vec2;

[0027] Video unit: used for extracting features from the video data to obtain video features, where the video features include a first video feature AUs, a second video feature Gaze, and a third video feature Pose;

[0028] Text unit: used for performing word segmentation and encoding on the text data to obtain second data, and performing feature extraction on the second data based on the pre-trained model DepRoBERTa to obtain text features;

[0029] Slicing unit: used for slicing the audio features, the video features and the text features to obtain the multimodal data.

[0030] Furthermore, the first characteristic unit specifically includes:

[0031] The first advanced unit is used to unify the dimensions of the audio feature and the video feature, obtain the first feature and the second feature, respectively enhance the weights of the first feature and the second feature, and obtain the advanced audio depression feature F a and advanced video depression features F v ;

[0032] The second advanced unit is used to capture the association between the contexts of the text features, obtain the associated text features, and weight the associated text features to obtain the advanced text depression feature F t ;

[0033] The third advanced unit: for advanced audio depression features based on the F a 、The advanced video depression feature F v and the advanced text depression feature F t The high-level depressive features are obtained.

[0034] Furthermore, the fusion unit specifically includes:

[0035] The first fusion unit is used to combine the high-level audio depression feature F a and the advanced video depression feature F v Splice them separately to obtain the audio fusion feature F1 a And video fusion feature F1 v ; The audio fusion feature F1 a And the video fusion feature F1 v Transpose them respectively to obtain audio fusion feature tensor A and video fusion feature tensor B;

[0036] The second fusion unit is used to fuse the audio fusion feature tensor A and the video fusion feature tensor B based on the time dimension and the space dimension to obtain the spatiotemporal audio fusion depression feature F2 a Fusion of depression features with spatiotemporal video F2 v ; Based on the spatiotemporal audio fusion depression feature F2 a And the spatiotemporal video fusion depression feature F2 v Obtaining the spatiotemporal fusion depression characteristics;

[0037] Get the audio fusion feature F1 a And the video fusion feature F1v The first calculation method is:

[0038]

[0039] Among them, F1 a and F1 v Represent audio fusion features and video fusion features respectively, Indicates splicing, and Respectively represent the first audio feature MFCC, the second audio feature eGeMAPS and the third audio feature Wav2Vec2, and They respectively represent the first video feature AUs, the second video feature Gaze and the third video feature Pose.

[0040] The audio fusion feature F1a and the video fusion feature F1v are transposed respectively so that each row has only one type of feature and each column contains all category features in the same time period; for the audio modality and video modality containing multiple internal features, a feedforward neural network is used to fuse the features in the time and space dimensions respectively, which can associate the information within the modalities and reduce the value difference between features.

[0041] Further, the spatiotemporal audio fusion depression feature F2 is obtained a The specific methods include:

[0042] Based on the time dimension, the audio fusion feature tensor A is input into the feedforward neural network, and after being processed by the activation function, it is input into the feedforward neural network again to obtain the first fusion feature A1; the first fusion feature A1 is transposed to obtain the first feature tensor A2; based on the space dimension, the first feature tensor A2 is input into the feedforward neural network, and after being processed by the activation function, it is input into the feedforward neural network again to obtain the spatiotemporal audio fusion depression feature F2 a ;

[0043] Obtain the spatiotemporal audio fusion depression feature F2 a The second calculation method is:

[0044] A1=MLP(ηMLP((F1 a ) T ));

[0045] F2 a =MLP(ηMLP((A1) T ));

[0046] Obtain the spatiotemporal video fusion depression feature F2 v The specific steps include:

[0047] Based on the time dimension, the video fusion feature tensor B is input into the feedforward neural network, and after being processed by the activation function, it is input into the feedforward neural network again to obtain the second fusion feature B1; the second fusion feature B1 is transposed to obtain the second feature tensor B2; based on the space dimension, the second feature tensor B2 is input into the feedforward neural network, and after being processed by the activation function, it is input into the feedforward neural network again to obtain the spatiotemporal video fusion depression feature F2 v ;

[0048] Obtain the spatiotemporal video fusion depression feature F2 v The third calculation method is:

[0049] B1=MLP(ηMLP((F1 v ) T ));

[0050] F2 v =MLP(ηMLP((B1) T ));

[0051] Among them, A1 and B1 represent the first fusion feature A1 and the second fusion feature B1, F1 a and F1 v Respectively represent audio fusion features and video fusion features, F2 a and F2 v They respectively represent the spatiotemporal audio fusion depression features and the spatiotemporal video fusion depression features, MLP() represents a feedforward neural network, η represents an activation function, and T represents transpose.

[0052] The audio fusion feature F1 a And video fusion feature F1 v It is input into a feedforward neural network, then processed by the GELU activation function and input into a feedforward neural network again, so that the features can be fully integrated along the time dimension, and then transposed to obtain a tensor in which each row contains all category features at the same time, and then the same processing method is used to make the features fully integrated along the spatial dimension.

[0053] Furthermore, the relationship diagram unit specifically includes:

[0054] Single modal unit: used to fuse the spatiotemporal audio with the depression feature F2 a 、The spatiotemporal video fusion depression feature F2 v and the advanced text depression feature F t They are defined as a node tensor respectively, and the auditory node tensor, visual node tensor and text node tensor are obtained, which are recorded as N a =F2 a 、N v=F2 v 、N t =F t , the first node set V1 is defined as: V1 = {N a , N v , N t}, obtaining a single-modal node layer based on the first node set V1;

[0055] Bimodal unit: used for obtaining any node tensor of the first node set V1 based on the single-modal node layer, obtaining the first node tensor, combining the first node tensor with the non-first node tensor by similarity, and obtaining the second node set V2, wherein the second node set V2 is defined as: V2 = {N av , N at , N va , N vt , N ta , N tv}; Obtain any node tensor of the second node set V2 to obtain a second node tensor, establish an edge between the second node tensor and all the first node tensors constituting the second node tensor, obtain a first matrix E1, and obtain a bimodal node layer based on the first matrix E1;

[0056] Among them, N av represents the node tensor obtained by combining the auditory node tensor and the visual node tensor by similarity; N at N represents the node tensor obtained by combining the auditory node tensor and the text node tensor by similarity; va represents the node tensor obtained by combining the visual node tensor and the auditory node tensor by similarity; N vt Represents the node tensor obtained by combining the visual node tensor and the text node tensor by similarity; N ta N represents the node tensor obtained by combining the text node tensor and the auditory node tensor by similarity; tv Represents the node tensor obtained by combining the text node tensor and the visual node tensor by similarity;

[0057] Multimodal unit: used to concatenate all the second node tensors in pairs to obtain an aggregate feature matrix E2, wherein the aggregate feature matrix E2 is defined as: E2 = {M av , M at , M va , M vt , M ta , M tv}; Aggregate the nodes of the unimodal node layer, the bimodal node layer and the aggregate feature matrix E2 to obtain a third node set V3, wherein the third node set V3 is defined as: V3 = {fav , f at , f va , f vt , f ta , f tv}; Establish an edge between any two node tensors of the third node set V3, obtain any node tensor of the third node set V3, obtain the third node tensor, establish an edge between the third node tensor and the parent node tensor of the third node tensor, obtain the second matrix E3, and obtain a multimodal node layer based on the second matrix E3;

[0058] Among them, M av Represents the second node tensor N av The node obtained by concatenating the tensor of each second node; M at Represents the second node tensor N at The node obtained by concatenating the tensor of each second node; M va Represents the second node tensor N va The node obtained by concatenating the tensor of each second node; M vt Represents the second node tensor N vt The node obtained by concatenating the tensor of each second node; M ta Represents the second node tensor N ta The node obtained by concatenating the tensor of each second node; M tv Represents the second node tensor N tv The node obtained by concatenating with each second node tensor;

[0059] Among them, f av Represents node M av The nodes obtained after aggregation, f at Represents node M at The nodes obtained after aggregation, f va Represents node M va The nodes obtained after aggregation, f vt Represents node M vt The nodes obtained after aggregation, f ta Represents node M ta The nodes obtained after aggregation, f tv Represents node M tv The nodes obtained after aggregation;

[0060] Fusion relationship diagram unit: used to obtain the modal fusion relationship diagram E based on the first matrix E1 and the second matrix E3, and the modal fusion relationship diagram E is defined as: E={E1, E3}.

[0061] The existing methods of constructing relationship graphs simply associate the nodes of each fused modality according to similarity, but this will result in the high-value information being ignored in some data with low high-value occupancy rate, and the existing methods simply splice various modalities, which will cause some data with relatively large noise to affect the overall situation. In addition, if a modality is lost in actual application, the feature dimension will change, and the model will not be able to operate. However, when constructing the graph structure, this system fully considers the value of each modality, and fuses each modality layer by layer from the inside out. First, the internal features of each modality are fused in time and space, and then the modal fusion relationship graph is constructed layer by layer between the modalities for fusion. This can reduce the impact of noise data and the loss of valuable data. In addition, the modeling is adopted layer by layer, so even if a modality is lost, it will not affect the normal operation of the model.

[0062] Furthermore, the second characteristic unit specifically includes:

[0063] Node unit: used to obtain a node set V based on the first node set V1, the second node set V2 and the third node set V3, wherein the node set V is defined as: V = {V1, V2, V3};

[0064] Weight unit: used to calculate the attention weight coefficients of any two adjacent nodes of the node set V and the modality fusion relationship graph E; normalize the attention weight coefficients to obtain the weight coefficients;

[0065] Depression feature unit: used to perform weighted summation on the node set V and the nodes of the modal fusion relationship graph E based on the weight coefficient to obtain the multimodal depression feature.

[0066] Using the graph attention mechanism to balance the relationship between the modalities in the modal fusion relationship graph can improve the accuracy and generalization ability of the model prediction.

[0067] Furthermore, a fourth calculation method for obtaining the advanced audio depression feature and the advanced video depression feature is:

[0068]

[0069]

[0070]

[0071] Get the high-level text depression feature F t The fifth calculation method is:

[0072] h t =BiLSTM(X t );

[0073]

[0074] in, and Represent the first feature and the second feature respectively, MLP() represents a feedforward neural network, and Represent audio features and video features respectively, and They represent high-level audio depression features and high-level video depression features, respectively. Softmax() represents the normalized exponential function. BiLSTM() represents the bidirectional long short-term memory network. t represents the associated text feature, X t represents the text feature, F t represents high-level text depression features, T represents transposition, and W q , W k and W v Both represent learnable weight matrix parameters, D k Represents the scaling point coefficient, i corresponds to the first audio feature MFCC, the second audio feature eGeMAPS, the third audio feature Wav2Vec2 of the audio feature and the first video feature AUs, the second video feature Gaze, and the third video feature Pose of the video feature.

[0075] Furthermore, a sixth calculation method for obtaining the second node set V2 is:

[0076]

[0077]

[0078] in, Indicates relevant parameters, and Both represent the node tensor of the first node set V1, Represents a node tensor The dimension, T represents transpose, N xy Represents the node tensor of the second node set V2, and n represents the current node tensor The length of j represents the node tensor The j-th sequence of , x and y represent variable parameters, a, v and t represent elements in the set;

[0079] The seventh calculation method for obtaining the aggregate feature matrix E2 is:

[0080]

[0081] M xy ={cross_att(Nxy ,N),x,y∈{a,v,t} and x≠y,N∈V2};

[0082] Among them, cross_att() represents the correlation function, softmax() represents the normalized exponential function, d represents the dimension of the node tensor N1, N1, N2 and N all represent the node tensors of the second node set, V2 represents the second node set, T represents transpose, M xy Represents the node tensor of the aggregate feature matrix E2, x and y represent variable parameters, a, v and t represent elements in the set;

[0083] An eighth calculation method for obtaining the third node set V3 is:

[0084]

[0085]

[0086]

[0087] fusion(z)=z⊙e(z),z∈E2;

[0088] Among them, MaxPool(z) b,c 、AvgPool(z) b,c , e(z) and fusion(z) all represent correlation functions, z represents the input feature, b and c represent the position of the input feature in the output feature map, e and g represent the relative position of the input feature in the pooling window, represents the pooling window size, η represents the activation function; MLP() represents a feedforward neural network, represents concatenation, ⊙ represents aggregation, maxwindow represents the maximum pooling window, poolwindow represents the average pooling window, and E2 represents the aggregated feature matrix.

[0089] Furthermore, a ninth calculation method for obtaining the multimodal depression feature is:

[0090] S(N q , N p )=LeakyReLU(a[WN q ||WN p ]);

[0091]

[0092] N h =Σ k∈μ α q，p WN k ;

[0093] Among them, S(Nq ,N p ) represents the correlation function, α q,p Represents relevant parameters, N q represents node q, N p represents the neighbor node p of node q, LeakyReLU() represents the LeakyReLU function, || represents concatenation, a represents the attention parameter, W represents the learnable weight matrix, μ represents the set of neighbor nodes of node q, k represents the kth sequence in μ, and N k represents the kth feature sequence of node q, N h Represents multimodal depressive traits.

[0094] One or more technical solutions provided by the present invention have at least the following technical effects or advantages:

[0095] 1. Regarding the selection of multimodal features, we selected MFCC features with subject information in the interviewee’s audio, eGeMAPS features with high emotional representation, and Wav2Vec2 features with prior knowledge; we selected facial expression features, gaze direction features, and head posture features in the interviewee’s video; for text features, we used the DepRoBERTa model, a pre-trained model with prior knowledge, to obtain features with high representation capabilities for depressive emotions; we paid more attention to features with high emotional value, which can improve the recognition effect of depression.

[0096] 2. For audio and video modalities containing multiple internal features, a feedforward neural network is used to fuse the features in the time and space dimensions respectively, which can associate the information within the modalities and reduce the value difference between the features.

[0097] 3. When constructing the graph structure, the value of each modality is fully considered, and each modality is fused layer by layer from the inside out. The internal features of the audio modality and the video modality are first fused in time and space, and then the modal fusion relationship graph is constructed layer by layer between the modalities for fusion. This can reduce the impact of noise data and the loss of valuable data. By modeling each layer separately, even if one modality is lost, it will not affect the normal operation of the model.

[0098] 4. Construct a modal fusion relationship graph based on the layer-by-layer fusion method between modalities, and use the graph attention mechanism to balance the relationship between modalities to achieve a balance between modalities, thereby improving the accuracy and generalization ability of model prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of the present invention, and do not constitute a limitation on the embodiments of the present invention;

[0100] Figure 1It is a flowchart of a multimodal depression recognition system based on multi-level feature fusion in the present invention;

[0101] Figure 2 It is a structural schematic diagram of a multimodal depression recognition system based on multi-level feature fusion in the present invention;

[0102] Figure 3 It is a schematic diagram of the modal multi-scale aggregation structure of a multimodal depression recognition system based on multi-level feature fusion in the present invention. DETAILED DESCRIPTION

[0103] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0104] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those within the scope of this description. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0105] Embodiment 1

[0106] refer to Figure 1-Figure 3 This embodiment provides a multimodal depression recognition system based on multi-level feature fusion, the system comprising:

[0107] Acquisition unit: used for acquiring multimodal depression data based on an acquisition device, and preprocessing the multimodal depression data based on a processor to obtain multimodal data; in this embodiment, the acquisition device may include a camera, a video recorder, a recorder, a computer, an openSMILE toolkit, an OpenFace toolkit, and a speech-to-text toolkit, etc.; the processor may be a controller, a chip or a computer, and these devices may contain computer programs, computational model programs or related computational functions, etc.; the acquisition device is connected to the processor data, and the processor may be built into the acquisition device or may not be installed on the acquisition device, and the acquisition device transmits the acquired data to the processor for processing.

[0108] Wherein, the acquisition unit specifically includes:

[0109] Data unit: used for acquiring the multimodal depression data based on the acquisition device;

[0110] A division unit: used for dividing the multimodal depression data based on the file format to obtain first data, wherein the first data includes audio data, video data and text data; for example, dividing the data ending with "wav" or "mp4" into audio data and video data, and using Google Cloud speech recognition service to convert speech into text data; at the same time, dividing the multimodal depression data into a training set, a validation set and a test set in a data ratio of 6:2:2, for training the prior knowledge model Wav2Vec2;

[0111] Audio unit: used to extract features from the audio data to obtain audio features, wherein the audio features include a first audio feature MFCC, a second audio feature eGeMAPS, and a third audio feature Wav2Vec2; such as using the openSMILE toolkit to extract MFCC features and eGeMAPS features of the audio, and using the pre-trained model Wav2Vec2 with prior knowledge to extract audio features containing prior knowledge;

[0112] Video unit: used for extracting features from the video data to obtain video features, wherein the video features include a first video feature AUs, a second video feature Gaze, and a third video feature Pose; for example, using the OpenFace toolkit to extract facial information of a person in the video;

[0113] Text unit: used to segment and encode the text data to obtain second data, and extract features from the second data based on the pre-trained model DepRoBERTa to obtain text features; such as segmenting the text data, converting the text data into indexes in the vocabulary of the pre-trained model DepRoBERTa, and padding or truncating the text to ensure that the length of the input sequence is the same; then, using the pre-trained model DepRoBERTa model with prior knowledge to extract features with high representation ability for depressive emotions;

[0114] Slicing unit: used to slice the audio features, the video features and the text features to obtain the multimodal data. For example, if the multimodal depression data is sliced ​​at intervals of 6 seconds, the last slice of less than 6 seconds will be discarded, which can improve the computing efficiency.

[0115] A first feature unit is used for performing feature extraction on the multimodal data to obtain a single-modal low-level depression feature; performing feature extraction on the low-level depression feature to obtain a single-modal high-level depression feature;

[0116] Wherein, the first characteristic unit specifically includes:

[0117] The first advanced unit is used to unify the dimensions of the audio feature and the video feature, obtain the first feature and the second feature, respectively enhance the weights of the first feature and the second feature, and obtain the advanced audio depression feature F a and advanced video depression features F v ; For example, a layer of feedforward neural network is used to unify the dimensions of the two features, and different attention weights are assigned to different features through the self-attention mechanism to obtain the advanced audio depression feature F a and advanced video depression features F v ;

[0118] The second advanced unit is used to capture the association between the contexts of the text features, obtain the associated text features, and weight the associated text features to obtain the advanced text depression feature F t ; For example, two bidirectional long short-term memory networks are used to aggregate the contextual information of text features and their important information is weighted through the self-attention mechanism to obtain the advanced text depression feature F t ;

[0119] The third advanced unit: for advanced audio depression features based on the F a 、The advanced video depression feature F v and the advanced text depression feature F t The high-level depressive features are obtained.

[0120] Fusion unit: used for the time dimension, space dimension and the advanced depression feature to obtain the spatiotemporal fusion depression feature;

[0121] Wherein, the fusion unit specifically includes:

[0122] The first fusion unit is used to combine the high-level audio depression feature F a and the advanced video depression feature F v Splice them separately to obtain the audio fusion feature F1 a And video fusion feature F1 v ; The audio fusion feature F1 a And the video fusion feature F1 v Transpose them respectively to obtain audio fusion feature tensor A and video fusion feature tensor B;

[0123] The second fusion unit is used to fuse the audio fusion feature tensor A and the video fusion feature tensor B based on the time dimension and the space dimension to obtain the spatiotemporal audio fusion depression feature F2 a Fusion of depression features with spatiotemporal video F2 v ; Based on the spatiotemporal audio fusion depression feature F2 aAnd the spatiotemporal video fusion depression feature F2 v Obtaining the spatiotemporal fusion depression characteristics;

[0124] Among them, the spatiotemporal audio fusion depression feature F2 is obtained a The specific methods include:

[0125] Based on the time dimension, the audio fusion feature tensor A is input into the feedforward neural network, and after being processed by the activation function, it is input into the feedforward neural network again to obtain the first fusion feature A1; the first fusion feature A1 is transposed to obtain the first feature tensor A2; based on the space dimension, the first feature tensor A2 is input into the feedforward neural network, and after being processed by the activation function, it is input into the feedforward neural network again to obtain the spatiotemporal audio fusion depression feature F2 a ;

[0126] Obtain the spatiotemporal video fusion depression feature F2 v The specific steps include:

[0127] Based on the time dimension, the video fusion feature tensor B is input into the feedforward neural network, and after being processed by the activation function, it is input into the feedforward neural network again to obtain the second fusion feature B1; the second fusion feature B1 is transposed to obtain the second feature tensor B2; based on the space dimension, the second feature tensor B2 is input into the feedforward neural network, and after being processed by the activation function, it is input into the feedforward neural network again to obtain the spatiotemporal video fusion depression feature F2 v .

[0128] Relationship graph unit: used for the spatiotemporal fusion depression feature and the advanced depression feature to obtain a modality fusion relationship graph E;

[0129] Wherein, the relationship diagram unit specifically includes:

[0130] Single modal unit: used to fuse the spatiotemporal audio with the depression feature F2 a 、The spatiotemporal video fusion depression feature F2 v and the advanced text depression feature F t They are defined as a node tensor respectively, and the auditory node tensor, visual node tensor and text node tensor are obtained, which are recorded as N a =F2 a 、N v =F2 v 、N t =F t , the first node set V1 is defined as: V1 = {N a , N v , N t}, obtaining a single-modal node layer based on the first node set V1;

[0131] Bimodal unit: used for obtaining any node tensor of the first node set V1 based on the single-modal node layer, obtaining the first node tensor, combining the first node tensor with the non-first node tensor by similarity, and obtaining the second node set V2, wherein the second node set V2 is defined as: V2 = {N av , N at , N va , N vt , N ta , N tv}; Get any node tensor of the second node set V2, obtain the second node tensor, establish an edge between the second node tensor and all the first node tensors constituting the second node tensor, obtain the first matrix E1, and obtain the bimodal node layer based on the first matrix E1; the first matrix E1 can be expressed as:

[0132]

[0133] Among them, N av represents the node tensor obtained by combining the auditory node tensor and the visual node tensor by similarity; N at N represents the node tensor obtained by combining the auditory node tensor and the text node tensor by similarity; va represents the node tensor obtained by combining the visual node tensor and the auditory node tensor by similarity; N vt Represents the node tensor obtained by combining the visual node tensor and the text node tensor by similarity; N ta N represents the node tensor obtained by combining the text node tensor and the auditory node tensor by similarity; tv Represents the node tensor obtained by combining the text node tensor and the visual node tensor by similarity;

[0134] Multimodal unit: used to concatenate all the second node tensors in pairs to obtain an aggregate feature matrix E2, wherein the aggregate feature matrix E2 is defined as: E2 = {M av , M at , M va , M vt , M ta , M tv}; Aggregate the nodes of the unimodal node layer, the bimodal node layer and the aggregate feature matrix E2 to obtain a third node set V3, wherein the third node set V3 is defined as: V3 = {f av , f at , f va , f vt , f ta , ftv}; Establish an edge between any two node tensors of the third node set V3, obtain any node tensor of the third node set V3, obtain the third node tensor, establish an edge between the third node tensor and the parent node tensor of the third node tensor, obtain the second matrix E3, and obtain a multimodal node layer based on the second matrix E3;

[0135] The second matrix E3 can be expressed as:

[0136]

[0137] Among them, M av Represents the second node tensor N av The node obtained by concatenating the tensor of each second node; M at Represents the second node tensor N at The node obtained by concatenating the tensor of each second node; M va Represents the second node tensor N va The node obtained by concatenating the tensor of each second node; M vt Represents the second node tensor N vt The node obtained by concatenating the tensor of each second node; M ta Represents the second node tensor N ta The node obtained by concatenating the tensor of each second node; M tv Represents the second node tensor N tv The node obtained by concatenating with each second node tensor;

[0138] Among them, f av Represents node M av The nodes obtained after aggregation, f at Represents node M at The nodes obtained after aggregation, f va Represents node M va The nodes obtained after aggregation, f vt Represents node M vt The nodes obtained after aggregation, f ta Represents node M ta The nodes obtained after aggregation, f tv Represents node M tv The nodes obtained after aggregation;

[0139] Fusion relationship diagram unit: used to obtain the modal fusion relationship diagram E based on the first matrix E1 and the second matrix E3, and the modal fusion relationship diagram E is defined as: E={E1, E3}.

[0140] The second feature unit is used to process the balance relationship of the modes of the modal fusion relationship graph E to obtain a multi-modal depression feature;

[0141] Wherein, the second characteristic unit specifically includes:

[0142] Node unit: used to obtain a node set V based on the first node set V1, the second node set V2 and the third node set V3, wherein the node set V is defined as: V = {V1, V2, V3};

[0143] Weight unit: used to calculate the attention weight coefficients of any two adjacent nodes of the node set V and the modality fusion relationship graph E; normalize the attention weight coefficients to obtain the weight coefficients;

[0144] Depression feature unit: used to perform weighted summation on the node set V and the nodes of the modal fusion relationship graph E based on the weight coefficient to obtain the multimodal depression feature.

[0145] Prediction unit: used to input the multimodal depression feature into the depression prediction model, obtain the prediction result, and transmit the prediction result to the display device. For example, after the multimodal depression feature is input into a fully connected layer, the Softmax activation function is used to predict whether the patient suffers from depression. In this embodiment, the display device can be a display screen, a computer, etc.

[0146] Embodiment 2

[0147] On the basis of the first embodiment, in this embodiment, the audio fusion feature F1 is obtained. a And the video fusion feature F1 v The first calculation method is:

[0148]

[0149] Among them, F1 a and F1 v Represent audio fusion features and video fusion features respectively, Indicates splicing, and Respectively represent the first audio feature MFCC, the second audio feature eGeMAPS and the third audio feature Wav2Vec2, and They respectively represent the first video feature AUs, the second video feature Gaze and the third video feature Pose.

[0150] Obtain the spatiotemporal audio fusion depression feature F2 a The second calculation method is:

[0151] A1=MLP(ηMLP((F1 a ) T ));

[0152] F2 a =MLP(ηMLP((A1) T ));

[0153] Obtain the spatiotemporal video fusion depression feature F2 v The third calculation method is:

[0154] B1=MLP(ηMLP((F1 v ) T ));

[0155] F2 v =MLP(ηMLP((B1) T ));

[0156] Among them, A1 and B1 represent the first fusion feature A1 and the second fusion feature B1, F1 a and F1 v Respectively represent audio fusion features and video fusion features, F2 a and F2 v They respectively represent the spatiotemporal audio fusion depression features and the spatiotemporal video fusion depression features, MLP() represents a feedforward neural network, η represents an activation function, and T represents transpose.

[0157] A fourth calculation method for obtaining the advanced audio depression feature and the advanced video depression feature is:

[0158]

[0159]

[0160]

[0161] Get the high-level text depression feature F t The fifth calculation method is:

[0162] h t =BiLSTM(X t );

[0163]

[0164] in, and Represent the first feature and the second feature respectively, MLP() represents a feedforward neural network, and Represent audio features and video features respectively, and They represent high-level audio depression features and high-level video depression features, respectively. Softmax() represents the normalized exponential function. BiLSTM() represents the bidirectional long short-term memory network. t represents the associated text feature, X t represents the text feature, F t represents high-level text depression features, T represents transposition, and W q , W k and W v Both represent learnable weight matrix parameters, D k Represents the scaling point coefficient, i corresponds to the first audio feature MFCC, the second audio feature eGeMAPS, the third audio feature Wav2Vec2 of the audio feature and the first video feature AUs, the second video feature Gaze, and the third video feature Pose of the video feature.

[0165] A sixth calculation method for obtaining the second node set V2 is:

[0166]

[0167]

[0168] in, Indicates relevant parameters, and Both represent the node tensor of the first node set V1, Represents a node tensor The dimension, T represents transpose, N xy Represents the node tensor of the second node set V2, and n represents the current node tensor The length of j represents the node tensor The j-th sequence of , x and y represent variable parameters, a, v and t represent elements in the set;

[0169] The seventh calculation method for obtaining the aggregate feature matrix E2 is:

[0170]

[0171] M xy ={cross_att(N xy , N), x, y∈{a, v, t} and x≠y, N∈V2};

[0172] Among them, cross_att() represents the correlation function, softmax() represents the normalized exponential function, d represents the dimension of the node tensor N1, N1, N2 and N all represent the node tensors of the second node set, V2 represents the second node set, T represents transpose, M xyRepresents the node tensor of the aggregate feature matrix E2, x and y represent variable parameters, a, v and t represent elements in the set;

[0173] An eighth calculation method for obtaining the third node set V3 is:

[0174]

[0175]

[0176]

[0177] fusion(z)=z⊙e(z), z∈E2;

[0178] Among them, MaxPool(z) b，c 、AvgPool(z) b，c , e(z) and fusion(z) all represent correlation functions, z represents the input feature, b and c represent the position of the input feature in the output feature map, e and g represent the relative position of the input feature in the pooling window, represents the pooling window size, η represents the activation function; MLP() represents a feedforward neural network, represents concatenation, ⊙ represents aggregation, maxwindow represents the maximum pooling window, poolwindow represents the average pooling window, and E2 represents the aggregated feature matrix.

[0179] The ninth calculation method for obtaining the multimodal depression feature is:

[0180] S(N q , N p )=LeakyReLU(a[WN q ||WN p ]);

[0181]

[0182] N h =∑ k∈μ α q，p WN k ;

[0183] Among them, S(N q ,N p ) represents the correlation function, α q,p Represents relevant parameters, N q represents node q, N prepresents the neighbor node p of node q, LeakyReLU() represents the LeakyReLU function, || represents concatenation, a represents the attention parameter, W represents the learnable weight matrix, μ represents the set of neighbor nodes of node q, k represents the kth sequence in μ, and N k represents the kth feature sequence of node q, N h Represents multimodal depressive traits.

[0184] Embodiment 3

[0185] On the basis of the above embodiments, in this embodiment, the acquisition device and the input device are connected to the memory, the memory is connected to the voice processing chip and the CPU, the CPU is connected to the voice processing chip and the display device, and the CPU has built-in depression prediction model, pre-trained model DepRoBERTa and depression recognition model Wav2Vec2. In this implementation, the acquisition device can be a camera to obtain video data, the input device can be a microphone to obtain audio data, all data are transmitted to the memory, audio data and video data are transmitted to the CPU, and audio data is also transmitted to the voice processing chip, which is converted from speech to text to obtain text data, and then transmitted to the CPU. The CPU processes all data to obtain prediction results, which are transmitted to the display device, and the display device can be a display screen.

[0186] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0187] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multimodal depression recognition system based on multi-level feature fusion, characterized in that: The system comprises: An acquisition unit: used for acquiring multimodal depression data based on an acquisition device, and preprocessing the multimodal depression data based on a processor to obtain multimodal data; A first feature unit is used for performing feature extraction on the multimodal data to obtain a single-modal low-level depression feature; performing feature extraction on the low-level depression feature to obtain a single-modal high-level depression feature; Fusion unit: used for the time dimension, space dimension and the advanced depression feature to obtain the spatiotemporal fusion depression feature; Relationship graph unit: used for the spatiotemporal fusion depression feature and the advanced depression feature to obtain a modality fusion relationship graph E; The second feature unit is used to process the balance relationship of the modes of the modal fusion relationship graph E to obtain a multi-modal depression feature; Prediction unit: used for inputting the multimodal depression feature into a depression prediction model, obtaining a prediction result, and transmitting the prediction result to a display device; The acquisition unit specifically includes: Data unit: used for acquiring the multimodal depression data based on the acquisition device; A dividing unit: configured to divide the multimodal depression data based on a file format to obtain first data, wherein the first data includes audio data, video data and text data; Audio unit: used for extracting features from the audio data to obtain audio features, where the audio features include a first audio feature MFCC, a second audio feature eGeMAPS, and a third audio feature Wav2Vec2; Video unit: used for extracting features from the video data to obtain video features, where the video features include a first video feature AUs, a second video feature Gaze, and a third video feature Pose; Text unit: used for performing word segmentation and encoding on the text data to obtain second data, and performing feature extraction on the second data based on the pre-trained model DepRoBERTa to obtain text features; Slicing unit: used for slicing the audio feature, the video feature and the text feature to obtain the multimodal data; The first characteristic unit specifically includes: The first advanced unit is used to unify the dimensions of the audio feature and the video feature, obtain the first feature and the second feature, respectively enhance the weights of the first feature and the second feature, and obtain the advanced audio depression feature F a and advanced video depression features F v ; The second advanced unit is used to capture the association between the contexts of the text features, obtain the associated text features, and weight the associated text features to obtain the advanced text depression feature F t ; The third advanced unit: for advanced audio depression features based on the F a 、The advanced video depression feature F v and the advanced text depression feature F t obtaining the advanced depressive features; The fusion unit specifically includes: The first fusion unit is used to combine the high-level audio depression feature F a and the advanced video depression feature F v Splice them separately to obtain the audio fusion feature F1 a And video fusion feature F1 v ; The audio fusion feature F1 a And the video fusion feature F1 v Transpose them respectively to obtain audio fusion feature tensor A and video fusion feature tensor B; The second fusion unit is used to fuse the audio fusion feature tensor A and the video fusion feature tensor B based on the time dimension and the space dimension to obtain the spatiotemporal audio fusion depression feature F2 a Fusion of depression features with spatiotemporal video F2 v ; Based on the spatiotemporal audio fusion depression feature F2 a And the spatiotemporal video fusion depression feature F2 v Obtaining the spatiotemporal fusion depression characteristics; Get the audio fusion feature F1 a And the video fusion feature F1 v The first calculation method is: Among them, F1 a and F1 v Represent audio fusion features and video fusion features respectively, Indicates splicing, and Respectively represent the first audio feature MFCC, the second audio feature eGeMAPS and the third audio feature Wav2Vec2, and They represent the first video feature AUs, the second video feature Gaze, and the third video feature Pose respectively; Obtain the spatiotemporal audio fusion depression feature F2 a The specific methods include: Based on the time dimension, the audio fusion feature tensor A is input into the feedforward neural network, and after being processed by the activation function, it is input into the feedforward neural network again to obtain the first fusion feature A1; the first fusion feature A1 is transposed to obtain the first feature tensor A2; based on the space dimension, the first feature tensor A2 is input into the feedforward neural network, and after being processed by the activation function, it is input into the feedforward neural network again to obtain the spatiotemporal audio fusion depression feature F2 a ; Obtain the spatiotemporal audio fusion depression feature F2 a The second calculation method is: A1=MLP(ηMLP((F1 a ) T )); F2 a =MLP(ηMLP((A1) T )); Obtain the spatiotemporal video fusion depression feature F2 v The specific steps include: Based on the time dimension, the video fusion feature tensor B is input into the feedforward neural network, and after being processed by the activation function, it is input into the feedforward neural network again to obtain the second fusion feature B1; the second fusion feature B1 is transposed to obtain the second feature tensor B2; based on the space dimension, the second feature tensor B2 is input into the feedforward neural network, and after being processed by the activation function, it is input into the feedforward neural network again to obtain the spatiotemporal video fusion depression feature F2 v ; Obtain the spatiotemporal video fusion depression feature F2 v The third calculation method is: B1=MLP(ηMLP((F1 v ) T )); F2 v =MLP(ηMLP((B1) T )); Among them, A1 and B1 represent the first fusion feature A1 and the second fusion feature B1, F1 a and F1 v Respectively represent audio fusion features and video fusion features, F2 a and F2 v They represent the spatiotemporal audio fusion depression feature and the spatiotemporal video fusion depression feature, MLP() represents a feedforward neural network, η represents an activation function, and T represents transposition; The relationship diagram unit specifically includes: Single modal unit: used to fuse the spatiotemporal audio with the depression feature F2 a 、The spatiotemporal video fusion depression feature F2 v and the advanced text depression feature F t They are defined as a node tensor respectively, and the auditory node tensor, visual node tensor and text node tensor are obtained, which are recorded as N a =F2 a 、N v =F2 v 、N t =F t , the first node set V1 is defined as: V1 = {N a , N v , N t }, obtaining a single-modal node layer based on the first node set V1; Bimodal unit: used for obtaining any node tensor of the first node set V1 based on the single-modal node layer, obtaining the first node tensor, combining the first node tensor with the non-first node tensor by similarity, and obtaining the second node set V2, wherein the second node set V2 is defined as: V2 = {N av , N at , N va , N vt , N ta , N tv }; Obtain any node tensor of the second node set V2 to obtain a second node tensor, establish an edge between the second node tensor and all the first node tensors constituting the second node tensor, obtain a first matrix E1, and obtain a bimodal node layer based on the first matrix E1; Among them, N av represents the node tensor obtained by combining the auditory node tensor and the visual node tensor by similarity; N at N represents the node tensor obtained by combining the auditory node tensor and the text node tensor by similarity; va represents the node tensor obtained by combining the visual node tensor and the auditory node tensor by similarity; N vt Represents the node tensor obtained by combining the visual node tensor and the text node tensor by similarity; N ta N represents the node tensor obtained by combining the text node tensor and the auditory node tensor by similarity; tv Represents the node tensor obtained by combining the text node tensor and the visual node tensor by similarity; Multimodal unit: used to concatenate all the second node tensors in pairs to obtain an aggregate feature matrix E2, wherein the aggregate feature matrix E2 is defined as: E2 = {M av , M at , M va , M vt , M ta , M tv }; Aggregate the nodes of the unimodal node layer, the bimodal node layer and the aggregate feature matrix E2 to obtain a third node set V3, wherein the third node set V3 is defined as: V3 = {f av , f at , f va , f vt , f ta , f tv }; Establish an edge between any two node tensors of the third node set V3, obtain any node tensor of the third node set V3, obtain the third node tensor, establish an edge between the third node tensor and the parent node tensor of the third node tensor, obtain the second matrix E3, and obtain a multimodal node layer based on the second matrix E3; Among them, M av Represents the second node tensor N av The node obtained by concatenating the tensor of each second node; M at Represents the second node tensor N at The node obtained by concatenating the tensor of each second node; M va Represents the second node tensor N va The node obtained by concatenating the tensor of each second node; M vt Represents the second node tensor N vt The node obtained by concatenating the tensor of each second node; M ta Represents the second node tensor N ta The node obtained by concatenating the tensor of each second node; M tv Represents the second node tensor N tv The node obtained by concatenating with each second node tensor; Among them, f av Represents node M av The nodes obtained after aggregation, f at Represents node M at The nodes obtained after aggregation, f va Represents node M va The nodes obtained after aggregation, f vt Represents node M vt The nodes obtained after aggregation, f ta Represents node M ta The nodes obtained after aggregation, f tv Represents node M tv The nodes obtained after aggregation; Fusion relationship graph unit: used for obtaining the modality fusion relationship graph E based on the first matrix E1 and the second matrix E3, wherein the modality fusion relationship graph E is defined as: E={E1, E3}; The second characteristic unit specifically includes: Node unit: used to obtain a node set V based on the first node set V1, the second node set V2 and the third node set V3, wherein the node set V is defined as: V = {V1, V2, V3}; Weight unit: used to calculate the attention weight coefficients of any two adjacent nodes of the node set V and the modality fusion relationship graph E; normalize the attention weight coefficients to obtain the weight coefficients; Depression feature unit: used to perform weighted summation on the node set V and the nodes of the modal fusion relationship graph E based on the weight coefficient to obtain the multimodal depression feature.

2. According to claim 1, a multimodal depression recognition system based on multi-level feature fusion is characterized in that: A fourth calculation method for obtaining the advanced audio depression feature and the advanced video depression feature is: Get the high-level text depression feature F t The fifth calculation method is: h t =BiLSTM(X t ); in, and Represent the first feature and the second feature respectively, MLP() represents a feedforward neural network, and Represent audio features and video features respectively, and They represent advanced audio depression features and advanced video depression features, respectively. Softmax() represents the normalized exponential function. BiLSTM() represents the bidirectional long short-term memory network. t represents the associated text feature, X t represents the text feature, F t represents high-level text depression features, T represents transposition, and W q , W k and W v Both represent learnable weight matrix parameters, D k Represents the scaling point coefficient, i corresponds to the first audio feature MFCC, the second audio feature eGeMAPS, the third audio feature Wav2Vec2 of the audio feature and the first video feature AUs, the second video feature Gaze, and the third video feature Pose of the video feature.

3. The multimodal depression recognition system based on multi-level feature fusion according to claim 1 is characterized in that: A sixth calculation method for obtaining the second node set V2 is: x,y∈{a,v,t} and x≠y; x,y∈{a,v,t} and x≠y; in, Indicates relevant parameters, and Both represent the node tensor of the first node set V1, Represents a node tensor The dimension, T represents transpose, N xy Represents the node tensor of the second node set V2, and n represents the current node tensor The length of j represents the node tensor The j-th sequence of , x and y represent variable parameters, a, v and t represent elements in the set; The seventh calculation method for obtaining the aggregate feature matrix E2 is: N1,N2∈V2 and N1≠N2; M xy = {cross_att(N xy , N), where x, y ∈ {a, v, t} and x ≠ y, and N ∈ V2}; Among them, cross_att() represents the correlation function, softmax() represents the normalized exponential function, d represents the dimension of the node tensor N1, N1, N2 and N all represent the node tensors of the second node set, V2 represents the second node set, T represents transpose, M xy Represents the node tensor of the aggregate feature matrix E2, x and y represent variable parameters, a, v and t represent elements in the set; An eighth calculation method for obtaining the third node set V3 is: e(z)=η(MaxPool(MLP(z)))⊕η(AvgPool(MLP(z))); fusion(z)=z⊙e(z),z∈E2; Among them, MaxPool(z) b,c 、AvgPool(z) b,c , e(z) and fusion(z) all represent correlation functions, z represents the input feature, b and c represent the position of the input feature in the output feature map, e and g represent the relative position of the input feature in the pooling window, represents the pooling window size, η represents the activation function; MLP() represents a feedforward neural network, ⊕ represents concatenation, ⊙ represents aggregation, maxwindow represents the maximum pooling window, poolwindow represents the average pooling window, and E2 represents the aggregated feature matrix.

4. The multimodal depression recognition system based on multi-level feature fusion according to claim 1 is characterized in that: The ninth calculation method for obtaining the multimodal depression feature is: S(N q ,N p )=LeakyReLU(a[WN q ||WN p ]); N h =∑ k∈μ a q,p WN k ; Among them, S(N q ,N p ) represents the correlation function, α q,p Represents relevant parameters, N q represents node q, N p represents the neighbor node p of node q, LeakyReLU() represents the LeakyReLU function, || represents concatenation, a represents the attention parameter, W represents the learnable weight matrix, μ represents the set of neighbor nodes of node q, k represents the kth sequence in μ, and N k represents the kth feature sequence of node q, N h Represents multimodal depressive traits.

Images