Multi-modal emotion recognition method and system based on audio and video

Abstract

The invention provides a multi-mode emotion recognition method and system based on audios and videos, and relates to the technical field of emotion recognition. Comprising the following steps: inputting preprocessed face video data and audio data into a DepITCM network model to obtain an aggregation feature of the face video data and an aggregation feature of the audio data; extracting features of the aggregation features of the face video data and the aggregation features of the audio data through a time attention module, a channel attention module and a space attention module in sequence to obtain final features of the face video data and final features of the audio data; and fusing the final features of the face video data and the final features of the audio data to obtain fused features, inputting the fused features into a multi-task learning module, and finally classifying whether the person with the emotion to be recognized contains depression emotion or not through a binary classification task module. The method can improve the detection precision of whether the person with the emotion to be recognized contains the depression emotion or not.

Description

Technical Field

[0001] The present invention belongs to the technical field of emotion recognition, and in particular to a multimodal emotion recognition method and system based on audio and video. Background Art

[0002] Emotion recognition technology has great scientific significance and economic value, and has broad application prospects. At the same time, its application scenarios are not limited to the field of human-computer interaction, but can play an important role in many other fields, such as dialogue generation, social media analysis, and intelligent systems. Since emotions have a certain degree of ambiguity, how to accurately identify human emotions has become a problem that needs to be solved.

[0003] In existing emotion recognition systems, a single method is often used for analysis, or multiple methods are used but the results of various methods can only be simply combined. For example, in the Chinese authorized invention patent with announcement number CN107609572B, a multimodal emotion recognition method and system based on neural network and transfer learning is proposed. The deep neural network is trained based on large-scale data and the audio feature extractor and video feature extractor are obtained through transfer learning, and then the audio features and video features of the multimodal emotion data are extracted, so as to identify the probability of each speech emotion category and the probability of each video emotion category, and the final emotion category is judged by the probability value.

[0004] The Chinese authorized invention patent with publication number CN 117556084 B discloses a multi-modal video emotion analysis system, wherein the audio and video acquisition module is used to collect the user's facial video information and voice information, the expression recognition module is used to analyze and process the facial video information, the voice analysis module is used to analyze and process the voice information, and the emotion comprehensive analysis module obtains the user's emotion information based on the video analysis results and the voice analysis results. By analyzing the two modes of video information and audio information and fusing the two analysis results, more accurate emotion results can be obtained.

[0005] The inventors found that the models based on visual and audio data proposed in the above patents still have certain limitations. For example, it is difficult to effectively extract and select useful multimodal information and features from audio and video data, and few studies can simultaneously focus on information in the three dimensions of time, channel and space in depression detection. This leads to a certain room for improvement in the accuracy of emotion recognition results. Summary of the invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a multimodal emotion recognition method and system based on audio and video, which adopts a phased feature extraction strategy, first global and then local, and can capture global features while paying more attention to the information fusion of time, channel and space, so as to more comprehensively understand and characterize the characteristics of depressive emotions and depressive feelings, and improve the detection accuracy of whether the person to be identified has depressive emotions.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0008] The first aspect of the present invention provides a multimodal emotion recognition method based on audio and video.

[0009] The multimodal emotion recognition method based on audio and video includes the following steps:

[0010] Obtain facial video data and audio data of the person whose emotion is to be identified and perform preprocessing;

[0011] The preprocessed facial video data and audio data are input into the DepITCM network model, and global features are extracted and feature aggregated to obtain aggregate features of facial video data and aggregate features of audio data.

[0012] The aggregated features of the facial video data and the aggregated features of the audio data are extracted through the temporal attention module, the channel attention module, and the spatial attention module respectively, to obtain the final features of the facial video data and the final features of the audio data;

[0013] The final features of the facial video data and the final features of the audio data are fused to obtain the fused features, which are input into the multi-task learning module. The regression task module is used to learn the features of depression of the person to be identified, and the learned features of depression of the person to be identified are shared with the binary classification task module. Finally, the binary classification task module is used to classify whether the person to be identified has depression.

[0014] The second aspect of the present invention provides a multimodal emotion recognition system based on audio and video.

[0015] Multimodal emotion recognition system based on audio and video, including:

[0016] The data acquisition module is configured to: acquire facial video data and audio data of the person to be emotion-recognized and perform pre-processing;

[0017] The global feature extraction module is configured to: input the preprocessed facial video data and audio data into the DepITCM network model, extract global features respectively and perform feature aggregation respectively to obtain aggregated features of the facial video data and aggregated features of the audio data;

[0018] The multi-attention feature extraction module is configured to: extract features from the aggregated features of the facial video data and the aggregated features of the audio data through the time attention module, the channel attention module, and the spatial attention module in sequence, to obtain final features of the facial video data and final features of the audio data;

[0019] The classification module is configured to: fuse the final features of the facial video data and the final features of the audio data to obtain fused features, input the fused features into the multi-task learning module, use the regression task module to learn the features of depression of the person to be identified, and share the learned features of depression of the person to be identified with the binary classification task module, and finally classify whether the person to be identified has depression through the binary classification task module.

[0020] The third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the multimodal emotion recognition method based on audio and video as described in the first aspect of the present invention.

[0021] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the multimodal emotion recognition method based on audio and video as described in the first aspect of the present invention are implemented.

[0022] One or more of the above technical solutions have the following beneficial effects:

[0023] The present invention provides a multimodal emotion recognition method and system based on audio and video. When classifying whether a person to be identified has depressive emotions, a phased feature extraction strategy is adopted, first global and then local, which can capture global features while paying more attention to the information fusion of time, channel and space. This method can more comprehensively understand and characterize the characteristics of depressive emotions and depressive emotions, and improve the detection accuracy of whether a person to be identified has depressive emotions.

[0024] After using Inception Dilated Conv to obtain the aggregated features of facial video data and the aggregated features of audio data, the present invention extracts features again through the time attention module, the channel attention module and the space attention module in sequence, breaking the limitations of the existing technology, and can effectively extract and select useful multimodal information and features from audio and video data, and can simultaneously pay attention to information in the three dimensions of time, channel and space in depressive emotion detection.

[0025] The addition of the multi-task learning module of the present invention enables the model to fully consider the shared feature extraction layer, which can improve the richness of feature representation. Therefore, this method can further utilize the complementary information between different tasks on the basis of multimodal depression emotion recognition, improve the accuracy and generalization ability of the model, and provide a convenient and reliable screening method for early identification of depression emotion and auxiliary clinical diagnosis and treatment.

[0026] The present invention has good scalability and can be further combined with text to detect depression.

[0027] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0029] Figure 1 This is the structural diagram of DepITCM of the first embodiment.

[0030] Figure 2 This is the structure diagram of ITCM Encoder of the first embodiment. DETAILED DESCRIPTION

[0031] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0032] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0033] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0034] Embodiment 1

[0035] Depression is a common mental illness. Depression often manifests itself in symptoms such as lack of energy, low mood, loss of interest and pleasure, and severe depression may also manifest symptoms of self-harm and suicide. Therefore, early detection and treatment of depression is crucial. Depressed people often have certain depressive emotions.

[0036] In order to help clinicians work more effectively, the first step in treating depression is to assess whether a person has depression. At present, the clinical diagnosis of depression mainly relies on non-laboratory methods, which are carried out through structured or semi-structured interviews. During the interview, clinicians communicate with patients to assess their mental state based on standardized depression scales (such as HAMD). However, this method is affected by the doctor's experience and the patient's cooperation, and has a certain degree of subjectivity.

[0037] In addition to language differences, patients with depression also have certain differences in behavioral characteristics. Patients with depression usually show painful expressions, low voices, drooping corners of the mouth, slow movements, etc. The Facial Action Coding System (FACS) is usually used to systematically describe facial expressions, but these changes are often subtle and it is difficult for doctors to capture these changes. Therefore, the emergence of deep learning technology helps to overcome these challenges.

[0038] A series of medical studies have shown that some behavioral characteristics of patients with depression (such as facial activity units (AUs) and voice intonation) are significantly different from those of normal people. With the development of computer technology, vision and audio are gradually being used in the clinical detection of mental illness. However, there are still certain limitations in the deep models based on visual and audio data proposed so far. For example, it is difficult to effectively extract and select useful multimodal information and features from audio and video data, and few studies can simultaneously focus on information in the three dimensions of time, channel and space in depression detection. Solving these problems is crucial to building a depression emotion detection model. Based on this problem, we propose a multimodal emotion recognition method based on audio and video, which can be applied to the detection of depression emotions as an auxiliary means for early detection and treatment of depression.

[0039] The multimodal emotion recognition method based on audio and video disclosed in this embodiment is as follows: Figure 1 As shown, the following steps may be included:

[0040] Obtain facial video data and audio data of the person whose emotion is to be identified and perform preprocessing;

[0041] The preprocessed facial video data and audio data are input into the DepITCM network model, and global features are extracted and feature aggregated to obtain aggregate features of facial video data and aggregate features of audio data.

[0042] The aggregated features of the facial video data and the aggregated features of the audio data are extracted through the temporal attention module, the channel attention module, and the spatial attention module respectively, to obtain the final features of the facial video data and the final features of the audio data;

[0043] The final features of the facial video data and the final features of the audio data are fused to obtain the fused features, which are input into the multi-task learning module. The regression task module is used to learn the features of depression of the person to be identified, and the learned features of depression of the person to be identified are shared with the binary classification task module. Finally, the binary classification task module is used to classify whether the person to be identified has depression.

[0044] This embodiment adopts a phased feature extraction strategy when designing a multimodal emotion recognition model based on audio and video, first global and then local, which can capture global features while paying more attention to the information fusion of time, channel and space. The addition of the multi-task learning module makes the model pay more attention to the complementary information between different tasks, improving the accuracy and generalization ability of the model.

[0045] This embodiment includes the following steps:

[0046] (1) Preprocess the data set;

[0047] (2) Feature selection;

[0048] (3) Use the audio-video based multimodal emotion recognition model DepITCM to detect the emotions of the people to be identified.

[0049] In step (1), the public datasets AVEC 2017 and AVEC 2019 are used to train the audio and video based multimodal emotion recognition model DepITCM.

[0050] For the AVEC 2017 dataset, during the audio preprocessing process, COVAREP features were first extracted from the raw data for each subject and processed according to the start and end time of the conversation and the conversation labels of the participants. Then, the COVAREP features were standardized to ensure that different features had similar scales and ranges. Finally, non-speech frames and rows with response durations less than 1 second were deleted. To ensure data consistency, they were padded with 0 to have a uniform duration.

[0051] In the visual preprocessing process, the data is first subsampled, sampling every 0.3 seconds. Then, the frames that have not been successfully detected are deleted, and the number of frames of each subject's data is calculated, and the maximum and minimum number of frames are recorded to ensure consistency when processing and analyzing the data.

[0052] For the preprocessing method of AVEC 2019, this embodiment uses the method provided by AVEC 2019DDS.

[0053] Among them, in step (2), considering that there may be noise or long pauses in the data, the features in such samples change very little, and their contribution to the classification and regression tasks is also small. They do not have enough changes between different samples to distinguish different values ​​of the target variable. Therefore, we measure the importance of the feature by calculating the variance of the feature and filter out the features with low variance.

[0054] The calculation formula of characteristic variance is:

[0055]

[0056] Var(X) represents the variance of feature X. i is the eigenvalue of the i-th sample, μ is the mean of feature X, and n is the total number of samples.

[0057] Wherein, in step (3), if Figure 1 and Figure 2 As shown in Figure 1, the multimodal emotion recognition model DepITCM based on audio and video is composed of ITCM Encoder and Multi-Task Learning module. ITCM Encoder is composed of InceptionDilated Conv and TCPCA modules; TCPCA module is composed of Temporal Attention Module (TAM) and Channel Priority Convolution Attention Mechanism (CPCA). The structure of DepITCM is as follows: Figure 1 As shown, the ITCM Encoder architecture is as follows Figure 2 shown.

[0058] The task of Inception Dilated Conv is to extract the global features of the data to comprehensively consider the multi-scale long-range information of visual and audio modal data. The convolutional layer (Conv) in the Inception structure introduces an expansion factor to supplement the global information and increase the diversity of the model. The information is aggregated through the Maxpool layer after Inception Dilated Conv.

[0059] Taking the visual modality as an example, Inception Dilated Conv uses parallel 1×1 convolution, 3×3 convolution, 5×5 convolution, and 3×3 maximum pooling layers for calculation. The calculation method is as follows:

[0060] IDC(x)=Contact(f 1×1 (x),f 3×3 (x),f 5×5 (x),f 1×1 (Maxpool(x)))

[0061] Among them, x represents the input of Inception Dilated Conv, Contact refers to the concatenation of features by dimension, and f 1×1 , f 3×3 , f 5×5 They represent 1×1 convolution, 3×3 convolution, and 5×5 convolution respectively.

[0062] The final feature feature is calculated by stacking Inception Dilated Conv and Maxpool. The calculation method is as follows:

[0063] Feature(x)=Maxpool(IDR(x)).

[0064] The TCPCA module works to solve the problem of information loss caused by simple concatenation, so that the model can better focus on local features, especially focusing on the information fusion of time, channel and space dimensions. Among them, the temporal attention module (TAM) and the channel-priority convolutional attention mechanism (CPCA) construct a mapping method between global information and local important information, through which local features are extracted and used to enhance the original features. At the same time, these attention mechanisms can be directly integrated into the backbone network without the need for a separate network.

[0065] The specific calculation method is as follows:

[0066] When a feature sequence F is given in , the TCPCA module calculates the temporal attention module TAM(F in ), channel attention module CAM (F in ) and the spatial attention module SAM(F in ), the whole calculation process is as follows:

[0067]

[0068] Among them, F ′ 、F ″″ 、F ″″′ They represent the computational outputs of the input features along the temporal attention, channel attention, and spatial attention, respectively. Represents the multiplication of corresponding elements.

[0069] The role of the temporal attention module (TAM) is to highlight the dependencies in the temporal dimension, thereby enhancing the extraction of temporal information. The calculation method is as follows:

[0070] TAM(F in )=σ(MLP(AvgPool(F in ))+MLP(MaxPool(F in )))

[0071] F in Represents the input of TAM, σ represents the sigmoid activation function, MLP represents a fully connected neural network, AvgPool and MaxPool represent average pooling and maximum pooling.

[0072] The channel-priority convolutional attention mechanism (CPCA) is mainly composed of two key components: the channel attention module (CAM) and the spatial attention module (SAM). The processing flow of the CPCA module is as follows:

[0073] CAM(TAM(F in ))

[0074] =σ(MLP(AvgPool(TAM(F in )))+MLP(MaxPool(TAM(F in ))))

[0075] TAM(F in ) represents the input of CAM, σ represents the sigmoid activation function, MLP represents the fully connected neural network, AvgPool and MaxPool represent average pooling and maximum pooling. The calculation method of SAM is as follows:

[0076]

[0077] CAM(TAM(F in )) represents the input of SAM, DwConv represents depth-wise convolution, which performs convolution operation on the input features, and each branch performs independent convolution operation; Branch i ,i∈{0,1,2,3} represents the i-th branch, which is used to extract spatial features of different scales; Conv 1×1 is the convolution operation.

[0078] The role of the multi-task learning module is to use the complementary information between different tasks to improve the generalization ability of the model, so that the model can enhance the feature representation by sharing the feature extraction layer. To this end, we added an auxiliary task (regression task) to enhance the main task of depression classification. In order to achieve the purpose of multi-task learning, after feature extraction, features from different modalities are concatenated and fused into a unified feature representation. This ensures that the model can utilize the complementary information of visual and audio data. The fused features are sent to the FC block to perform the PHQ-8 regression and binary classification tasks respectively. The FC block includes the ReLU activation function, the Dropout layer, and the linear layer.

[0079] The multi-task loss function in the training phase can be expressed as:

[0080] Loss = a*L re +b*L cl

[0081] Among them, L re and L cl are the loss functions for PHQ-8 regression and binary classification, respectively. a and b are designed to exploit the coefficients between the two tasks and can be set as hyperparameters. Specifically, we use the commonly used cross entropy loss function as the loss function for the binary classification task, which can be expressed as:

[0082]

[0083] Where N is the number of samples, y i is the true label of the i-th sample, is the predicted probability of the ith sample.

[0084] The loss function of PHQ-8 regression can be expressed as:

[0085]

[0086] Where M is the number of samples, x j and are the true and predicted PHQ-8 scores of the jth sample, respectively.

[0087] The regression task helps the model learn the characteristics of depression in more detail by predicting the PHQ-8 score, providing additional contextual information, and these details help improve the accuracy of the classification task. By optimizing the regression and classification tasks simultaneously, the model is able to share the learned features and representations, which promotes the learning of the classification task because it can use the rich information provided by the regression task to improve the classification performance.

[0088] After the DepITCM network model training is completed:

[0089] (1) Obtain facial video data and audio data of the person whose emotion is to be identified and perform preprocessing:

[0090] Sub-sampling the facial video data, obtaining multiple frames of images and deleting frames that have not been successfully detected, and identifying facial features in the multiple frames of images;

[0091] Use the COVAREP tool to extract COVAREP features from audio data, standardize the COVAREP features, and filter the frame data;

[0092] Calculate the variance of facial features and COVAREP features, measure the importance of features, and filter out features with variances below a preset threshold.

[0093] (2) Input the preprocessed facial video data and audio data into the DepITCM network model, extract global features and perform feature aggregation respectively, and obtain the aggregated features of facial video data and the aggregated features of audio data:

[0094] Extract global features from the preprocessed facial video data and audio data through the Inception Dilated Conv in the DepITCM network model, where the Inception Dilated Conv includes multiple convolutional layers and maximum pooling layers in parallel;

[0095] The extracted global features are aggregated separately through the Maxpool layer after Inception Dilated Conv to obtain the aggregated features of facial video data and the aggregated features of audio data respectively.

[0096] (3) The aggregated features of the facial video data and the aggregated features of the audio data are extracted through the temporal attention module, the channel attention module, and the spatial attention module respectively to obtain the final features of the facial video data and the final features of the audio data. The specific calculation process is as follows:

[0097] TAM(F in )=σ(MLP(AvgPool(F in ))+MLP(MaxPool(F in )));

[0098] CAM(TAM(F in ))=

[0099] σ(MLP(AvgPool(TAM(F in )))+MLP(MaxPool(TAM(F in ))));

[0100]

[0101] Among them, TAM represents the temporal attention module; CAM represents the channel attention module; SAM represents the spatial attention module; F in represents the input of TAM, TAM(F in ) represents the input of CAM; CAM(TAM(F in )) represents the input of SAM; σ represents the sigmoid activation function, MLP represents a fully connected neural network, AvgPool and MaxPool represent average pooling and maximum pooling; DwConv represents deep convolution, Branch i ,i∈{0,1,2,3} represents the i-th branch.

[0102] (4) The final features of the facial video data and the final features of the audio data are fused to obtain fused features, which are then input into the multi-task learning module. The regression task module is used to predict the PHQ-8 score of the person to be identified to quantify his or her depressive emotional characteristics. The learned PHQ-8 score is shared with the binary classification task module. The binary classification task module uses the predicted value of the regression model and the fused features as input, and judges whether the person to be identified has depressive emotions based on this information, thereby improving the accuracy of classification.

[0103] The regression task module adopts PHQ-8 regression, predicts PHQ-8 scores by integrating features, and learns the characteristics of depression of the person to be identified;

[0104] The binary classification task module promotes the learning of the classification task by sharing the depressive emotion features of the person to be identified in the regression task module.

[0105] It can be understood that using PHQ-8 regression to learn the characteristics of depression emotions of people to be identified and predicting the PHQ-8 scores of people to be identified is an existing technology and will not be elaborated here.

[0106] It is also an existing technology to classify whether the person to be identified has depressive emotions through a binary classification task module, which will not be described in detail.

[0107] This embodiment has the following advantages:

[0108] (1) When designing a depression recognition model based on audio and video, a phased feature extraction strategy is adopted, first global and then local, which can capture global features while paying more attention to the information fusion of time, channel and space. This method can more comprehensively understand and characterize the emotional characteristics of patients with depression and improve the accuracy of depression detection.

[0109] (2) The addition of the multi-task learning module enables the model to fully consider that the shared feature extraction layer can improve the richness of feature representation. Therefore, based on multimodal depression recognition, this method can further utilize the complementary information between different tasks to improve the accuracy and generalization ability of the model, providing a convenient and reliable screening method for early identification of depression and auxiliary clinical diagnosis and treatment.

[0110] (3) This embodiment has good scalability and can be further combined with text to detect depression.

[0111] Embodiment 2

[0112] This embodiment discloses a multimodal emotion recognition system based on audio and video.

[0113] Multimodal emotion recognition system based on audio and video, including:

[0114] The data acquisition module is configured to: acquire facial video data and audio data of the person to be emotion-recognized and perform pre-processing;

[0115] The global feature extraction module is configured to: input the preprocessed facial video data and audio data into the DepITCM network model, extract global features respectively and perform feature aggregation respectively to obtain aggregated features of the facial video data and aggregated features of the audio data;

[0116] The multi-attention feature extraction module is configured to: extract features from the aggregated features of the facial video data and the aggregated features of the audio data through the time attention module, the channel attention module, and the spatial attention module in sequence, to obtain final features of the facial video data and final features of the audio data;

[0117] The classification module is configured to: fuse the final features of the facial video data and the final features of the audio data to obtain fused features, input the fused features into the multi-task learning module, use the regression task module to learn the features of depression of the person to be identified, and share the learned features of depression of the person to be identified with the binary classification task module, and finally classify whether the person to be identified has depression through the binary classification task module.

[0118] Embodiment 3

[0119] The purpose of this embodiment is to provide a computer-readable storage medium.

[0120] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the multimodal emotion recognition method based on audio and video as described in Example 1 of the present disclosure.

[0121] Embodiment 4

[0122] The purpose of this embodiment is to provide an electronic device.

[0123] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the multimodal emotion recognition method based on audio and video as described in Example 1 of the present disclosure are implemented.

[0124] The steps involved in the apparatuses of the above embodiments 2, 3 and 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0125] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0126] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A multimodal emotion recognition method based on audio and video, characterized in that: The following steps are involved: Obtain facial video data and audio data of the person whose emotion is to be identified and perform preprocessing; The preprocessed facial video data and audio data are input into the DepITCM network model, and global features are extracted and feature aggregated to obtain aggregate features of facial video data and aggregate features of audio data. The aggregated features of the facial video data and the aggregated features of the audio data are extracted through the temporal attention module, the channel attention module, and the spatial attention module respectively, to obtain the final features of the facial video data and the final features of the audio data; The final features of the facial video data and the final features of the audio data are fused to obtain the fused features, which are input into the multi-task learning module. The regression task module is used to learn the features of depression of the person to be identified, and the learned features of depression of the person to be identified are shared with the binary classification task module. Finally, the binary classification task module is used to classify whether the person to be identified has depression.

2. The multimodal emotion recognition method based on audio and video as claimed in claim 1, characterized in that: Preprocess the facial video data and audio data of the person whose emotion is to be recognized, including: Sub-sampling the facial video data, obtaining multiple frames of images and deleting frames that have not been successfully detected, and identifying facial features in the multiple frames of images; Use the COVAREP tool to extract COVAREP features from audio data, standardize the COVAREP features, and filter the frame data.

3. The multimodal emotion recognition method based on audio and video as claimed in claim 2, characterized in that: The pre-processing process further includes: Calculate the variance of facial features and COVAREP features, measure the importance of features, and filter out features with variances below a preset threshold.

4. The multimodal emotion recognition method based on audio and video as claimed in claim 1, characterized in that: The aggregate features of facial video data and audio data are obtained by: Extract global features from the preprocessed facial video data and audio data through the Inception Dilated Conv in the DepITCM network model, where the Inception Dilated Conv includes multiple convolutional layers and maximum pooling layers in parallel; The extracted global features are aggregated separately through the Maxpool layer after Inception Dilated Conv to obtain the aggregated features of facial video data and the aggregated features of audio data respectively.

5. The multimodal emotion recognition method based on audio and video as claimed in claim 1, characterized in that: The aggregate features of facial video data and the aggregate features of audio data are extracted through the time attention module, channel attention module and spatial attention module respectively to obtain the final features of facial video data and audio data. The specific calculation process is as follows: TAM(F in )=σ(MLP(AvgPool(F in ))+MLP(MaxPool(F in ))); GLASS(FULL(F in ))= σ(MLP(AvgPool(TAM(F in )))+MLP(MaxPool(TAM(F in )))); Among them, TAM represents the temporal attention module; CAM represents the channel attention module; SAM represents the spatial attention module; F in represents the input of TAM, TAM(F in ) represents the input of CAM; CAM(TAM(F in )) represents the input of SAM; σ represents the sigmoid activation function, MLP represents a fully connected neural network, AvgPool and MaxPool represent average pooling and maximum pooling; DwConv represents deep convolution, Branch i ,i∈{0,1,2,3} represents the i-th branch.

6. The multimodal emotion recognition method based on audio and video according to claim 1, characterized in that: The regression task module adopts PHQ-8 regression, predicts PHQ-8 scores by integrating features, and learns the characteristics of depression of the person to be identified; The binary classification task module promotes the learning of the classification task by sharing the depressive emotion features of the person to be identified in the regression task module.

7. The multimodal emotion recognition method based on audio and video as claimed in claim 6, characterized in that: The loss function Loss of the multi-task learning module is: Loss=a*L re +b*L cl ; Among them, L re and L cl are the loss functions of the PHQ-8 regression task module and the binary classification task module respectively; a and b are hyperparameters; The cross entropy loss function is used as the loss function for the binary classification task: Where N is the number of samples, y i is the true label of the i-th sample, is the predicted probability of the i-th sample; The loss function for the PHQ-8 regression task is: Where M is the number of samples, x j and are the true and predicted PHQ-8 scores of the jth sample, respectively.

8. A multimodal emotion recognition system based on audio and video, characterized in that: include: The data acquisition module is configured to: acquire facial video data and audio data of the person to be emotion-recognized and perform pre-processing; The global feature extraction module is configured to: input the preprocessed facial video data and audio data into the DepITCM network model, extract global features respectively and perform feature aggregation respectively to obtain aggregated features of the facial video data and aggregated features of the audio data; The multi-attention feature extraction module is configured to: extract features from the aggregated features of the facial video data and the aggregated features of the audio data through the time attention module, the channel attention module, and the spatial attention module in sequence, to obtain final features of the facial video data and final features of the audio data; The classification module is configured to: fuse the final features of the facial video data and the final features of the audio data to obtain fused features, input the fused features into the multi-task learning module, use the regression task module to learn the features of depression of the person to be identified, and share the learned features of depression of the person to be identified with the binary classification task module, and finally classify whether the person to be identified has depression through the binary classification task module.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the multimodal emotion recognition method based on audio and video as described in any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the multimodal emotion recognition method based on audio and video as described in any one of claims 1-7 are implemented.

Images