Deep learning-based multi-modal deep-scale sentiment feature fusion sentiment recognition method

Abstract

The application relates to a deep learning-based multi-modal deep-scale emotion feature fusion emotion recognition method. A section of voice in a wav format and a transcription text are input, and a ME module is used to obtain general text features, speaker features and general voice features of the section of voice through BERT, ECAPA-TDNN and huBERT pre-training models; a DST module is used to condense and extract emotion-related features in a multi-level and deep-scale feature range; an MT module is used to calculate the cross-correlation information of multi-modal information, and speaker features are introduced into speech-level feature representation to obtain speaker-related emotion features; an MFC module is used to perform attention weighting on multi-modal cross-correlation information, speaker-related features and deep-scale emotion features, and an emotion recognition result matrix is output; finally, the emotion category with the maximum probability in the emotion recognition matrix is output, emotion recognition of a section of voice and a transcription text is completed, and the emotion recognition efficiency of a multi-modal emotion recognition task in human-computer interaction is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of multi-modal emotion recognition, and particularly relates to a multi-modal deep-scale emotion feature fusion based emotion recognition method. BACKGROUND

[0002] Multimodal Emotion Recognition (MER) refers to processing multimedia resources and detecting the emotional state thereof, and is a very active research topic in affective computing, which has been widely studied in recent years. With the rapid development of Artificial Intelligent (AI), it has become increasingly popular to study how to improve the experience of automatic human-computer application in the field of Human-Computer Interaction (HCI). If the current emotional state of people can be accurately grasped and the corresponding reaction can be obtained in the interaction process between the artificial intelligence product and the people, the experience of the user to the artificial intelligence product can be improved to a certain extent. Therefore, emotion recognition plays a crucial role in human-computer interaction, and can effectively interact with machines. Considering its important significance in product recommendation, public opinion monitoring, human-computer dialogue and the like, multi-modal emotion recognition is undoubtedly a valuable research field.

[0003] In the past decades, more efforts have been paid to investigate valuable and effective approaches to improve the performance of multi-modal emotion recognition. In the early stage, researchers mainly applied various probabilistic statistical models (e.g., Hidden Markov Model and Gaussian Mixture Model) to classify the affective states of utterances. In their approaches, the derived features of natural pitch and energy contours of speech signals were used to organize the global statistical framework of utterances by Gaussian Mixture Model to accomplish the classification of affective states. In recent years, due to the significant feature extraction and data fitting performance of Deep Neural Network (DNN), various DNN structures have been applied to multi-modal emotion recognition. Generally, these approaches include two stages: 1) a pre-processing system computes low-level descriptors from multimedia resources and extracts robust universal multi-modal embeddings. 2) a multi-modal fusion mechanism aggregates multi-modal emotion-related features into utterance-level representations. Compared with traditional approaches, DNNs are more feasible to extract emotion features from rich data and more effective to distinguish emotions. However, some limitations have not been solved, including: (1) data scarcity. There is a common data scarcity problem in multi-modal corpora. The strong fitting capability of DNNs is vulnerable to noise information, and the lack of emotion-labeled data, especially the unbalanced distribution of data, increases the influence of noise pollution period. (2) difficult-to-distinguish emotion features. Speech and text are composed of multiple small components. A feasible approach to utilize deep and fine-grained features is the key to compatibly model the affective states of utterances. (3) fragile multi-modal inter-related information. In applications, the lengths of speech and text embeddings are usually inconsistent, and there is a lack of fusion of multi-modal features at the segment level, which limits the extraction of inter-information in deep dimensions. SUMMARY

[0004] The present application provides a multi-modal deep dimension emotion feature fusion based on deep learning emotion recognition method, which is used to improve the emotion recognition efficiency of multi-modal emotion recognition task in human-computer interaction.

[0005] The present application provides an electronic device.

[0006] The present application provides a non-transitory computer readable storage medium.

[0007] The present application is implemented by the following technical solutions:

[0008] A multi-modal deep dimension emotion feature fusion based on deep learning emotion recognition method, the implementation method comprises the following steps:

[0009] Step 1: input a segment of speech in wav format and transcription text, and use ME module to obtain its universal text features, speaker features and universal speech features through BERT, ECAPA-TDNN and huBERT pre-training models respectively;

[0010] Step 2: Use the DST module to condense and extract emotion-related features from the text features and speech features obtained in Step 1 in a multi-level, deep-scale feature range, obtaining multi-modal deep-scale emotion features;

[0011] Step 3: Use the MT module to calculate the mutual correlation information of the multi-modal information obtained in Step 1, and introduce speaker features in the utterance-level feature representation to obtain speaker-related multi-modal mutual correlation emotion features;

[0012] Step 4: Use the MFC module to weight the multi-modal deep-scale emotion features in Step 2 and the speaker feature-related multi-modal mutual correlation emotion features in Step 3, calculate and output the emotion recognition result matrix;

[0013] Step 5: Use the SoftMax function to calculate the emotion recognition probability matrix of the result matrix obtained in Step 4, output the maximum probability confidence emotion category, and realize emotion recognition of multi-modal information.

[0014] A kind of emotion recognition method based on deep learning multi-modal deep-scale emotion feature fusion, and the step 1 specifically includes the following steps:

[0015] Step 1.1: collect the speech waveform information of people through a recording device, and obtain a speech file in wav format;

[0016] Step 1.2: use an ASR automatic speech recognition system to complete text transcription of the speech file in Step 1.1, and obtain a speech text transcription file;

[0017] Step 1.3: use huBERT and ECAPA-TDNN pre-training models to extract general speech features and speaker features from the speech file in Step 1.1;

[0018] Step 1.4: use a BERT pre-training model to extract general text features from the text transcription file in Step 1.2.

[0019] Step 1.5: concatenate the general speech features and speaker features in Step 1.3 and the general text features in Step 1.4 to generate a multi-modal information dataset.

[0020] A kind of emotion recognition method based on deep learning multi-modal deep-scale emotion feature fusion, and the step 2 specifically includes the following steps:

[0021] Step 2.1: based on the multi-modal information dataset generated in Step 1, scale the speech and text features with shared weights to obtain a deep-scale feature dataset;

[0022] Step 2.2: Feature extraction using a Transformer encoder structure on the deep-scale feature dataset of Step 2.1, respectively constructing and capturing fine-grained and deep-scale sentiment-related features of text and speech information;

[0023] Step 2.3: Repeat the operations described in Step 2.1 and Step 2.2, concatenate the sentiment-related features of Step 2.2 to obtain a multi-level, deep-scale sentiment feature set;

[0024] Step 2.4: Using statistical methods, calculate the maximum pooling and average pooling features of the sentiment feature set of Step 2.3, and concatenate to obtain a multi-modal deep-scale sentiment feature.

[0025] A deep learning-based multi-modal deep-scale sentiment feature fusion emotion recognition method, said Step 3 specifically includes the following steps:

[0026] Step 3.1: Based on the multi-modal information dataset generated in Step 1, use the multi-head self-attention mechanism to calculate the multi-modal mutual information of speech and text;

[0027] Step 3.2: Use the shared weight Transformer structure to extract sentiment features from speech and text data;

[0028] Step 3.3: Using statistical methods, use maximum pooling and average pooling methods to reduce dimensionality and concatenate speaker features, mutual information of Step 3.1, and sentiment features of Step 3.2 to obtain speaker-related multi-modal mutual information sentiment features.

[0029] A deep learning-based multi-modal deep-scale sentiment feature fusion emotion recognition method, said Step 4 includes the following steps,

[0030] Step 4.1: Concatenate the multi-modal deep-scale sentiment feature matrix of Step 2 and the multi-modal mutual information sentiment feature matrix of Step 3 to obtain an emotion recognition matrix;

[0031] Step 4.2: Use the attention mechanism to calculate the attention weighting score based on the emotion recognition matrix of Step 4.1;

[0032] Step 4.3: Dot product calculation of the attention weighting score and the emotion recognition matrix of Step 4.1 to obtain an emotion recognition result matrix.

[0033] A deep learning-based multi-modal deep-scale sentiment feature fusion emotion recognition method, said Step 5 includes the following steps:

[0034] Step 5.1: Use the SoftMax function to scale the emotion recognition result matrix generated in Step 4 to obtain an emotion recognition probability matrix.

[0035] Step 5.2: Based on the emotion recognition probability matrix of step 5.1, output the emotion category with the maximum emotion probability as the emotion recognition result.

[0036] The emotion recognition method based on deep learning multi-modal deep-scale emotion feature fusion, the weight sharing feature scaling of step 2.1 is specifically,

[0037] The data dimension and data length of the speech and text information of step 1 are scaled and aligned by using one-dimensional convolution.

[0038] The speech and text are simultaneously weighted in time by using a double-layer shared weight gated recurrent neural network to generate attention-aligned speech and text data information.

[0039] The emotion recognition method based on deep learning multi-modal deep-scale emotion feature fusion, the step 3.1 calculates the multi-modal mutual information, and specifically,

[0040] The multi-head self-attention mechanism is used, wherein the speech information is used as the query, the text information is used as the key and the value, and the multi-head attention is calculated as the mutual information of the speech to the text.

[0041] The multi-head self-attention mechanism is used, wherein the text information is used as the query, the speech information is used as the key and the value, and the multi-head attention is calculated as the mutual information of the text to the speech.

[0042] An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the implementation method of any of the above.

[0043] A non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the implementation method of any of the above.

[0044] The beneficial effects of the present application are:

[0045] The present application can effectively organize multi-modal emotion data information, and realize emotion recognition by using deep-scale, multi-level multi-modal emotion features.

[0046] The present application uses a pre-trained model to generate robust and universal multi-modal embedding representation, and improves the expansibility and portability of the emotion recognition method.

[0047] The present application uses speaker characteristics to enhance the emotion correlation of multi-modal mutual information features, and improves the performance of multi-modal emotion recognition. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 It is the schematic diagram of the architecture of the present application.

[0049] Figure 2 is a schematic diagram of the DST module of the present application.

[0050] Figure 3 is a schematic diagram of the MT module of the present application. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0052] Currently, multi-modal emotion recognition is mainly limited by the small amount of labeled data, and it is difficult to overcome the overfitting effect caused by noise during training. At the same time, the emotion features extracted by the current mainstream emotion recognition model are still not discriminative enough, and the confusion between neutral and other emotions is very serious, mainly due to the difficulty of the emotion recognition task and the insufficient utilization of paralinguistic information for auxiliary decision-making. In addition, the multi-modal information fusion method is not perfect enough, and the interaction of voice and text information of different data lengths is obviously limited. Based on the existence of the above problems, the present application mainly solves the problems of data scarcity, feature sparsity, and insufficient fusion level in emotion recognition. The specific method mainly includes the following stages:

[0053] Multi-modal emotion feature extraction stage.

[0054] In order to extract general and robust feature representations, enhance the performance of the model fitting data, and reduce the noise pollution problem caused by insufficient labeled data. The present application uses a general and robust pre-trained embedding to avoid data scarcity problems. In addition, the deep-scale feature extraction module DST uses attention alignment to extract emotion-related time periods from multimedia resources, and then captures fine-grained features through the Transformer structure. Next, the mutual correlation information extraction module MT is specially used to extract multi-modal mutual correlation information related to the speaker.

[0055] A multi-modal deep-scale emotion feature fusion emotion recognition method based on deep learning, in the multi-modal emotion feature extraction stage, mainly divided into the following several parts:

[0056] Step 1: input a segment of voice in wav format and transcribed text, use the ME module to get its general text features, speaker features and general voice features through BERT, ECAPA-TDNN and huBERT pre-training models respectively;

[0057] Step 2: Use the DST module to condense and extract emotion-related features from the text features and speech features obtained in Step 1 in a multi-level, deep-scale feature range, obtaining multi-modal deep-scale emotion features;

[0058] Step 3: Use the MT module to calculate mutual correlation information of the multi-modal information obtained in Step 1, and introduce speaker features in the utterance-level feature representation to obtain speaker-related multi-modal mutual correlation emotion features;

[0059] Further, Step 1 specifically includes the following steps:

[0060] Step 1.1: Collect human speech waveform information through a recording device to obtain a wav format speech file;

[0061] Step 1.2: Use the ASR automatic speech recognition system to complete text transcription of the speech file in Step 1.1 to obtain a speech text transcription file;

[0062] Step 1.3: Use huBERT and ECAPA-TDNN pre-training models to extract general speech features and speaker features from the speech file in Step 1.1;

[0063] Step 1.4: Use the BERT pre-training model to extract general text features from the text transcription file in Step 1.2.

[0064] Step 1.5: Concatenate the general speech features and speaker features in Step 1.3 and the general text features in Step 1.4 to generate a multi-modal information dataset.

[0065] Further, Step 2 specifically includes the following steps:

[0066] Step 2.1: Based on the multi-modal information dataset generated in Step 1, perform feature scaling on the speech and text features with shared weights to obtain a deep-scale feature dataset;

[0067] Step 2.2: Use the Transformer encoder structure to extract features from the deep-scale feature dataset in Step 2.1 to construct and capture fine-grained and deep-scale emotion-related features of text and speech information respectively;

[0068] Step 2.3: Repeat the operations described in Steps 2.1 and 2.2, concatenate the emotion-related features in Step 2.2 to obtain a multi-level, deep-scale emotion feature set;

[0069] Step 2.4: Use statistical methods to calculate maximum pooling and average pooling features from the emotion feature set in Step 2.3, and concatenate to obtain multi-modal deep-scale emotion features.

[0070] Further, the step 3 specifically comprises the following steps:

[0071] Step 3.1: Based on the multi-modal information dataset generated in step 1, use multi-head self-attention mechanism to calculate the multi-modal inter-correlation information of speech and text;

[0072] Step 3.2: Use the shared weight Transformer structure to extract the sentiment features of speech and text data;

[0073] Step 3.3: Use statistical methods to reduce dimensionality and concatenate speaker features, step 3.1 inter-correlation information, and step 3.2 sentiment features using max-pooling and average-pooling methods to obtain speaker-related multi-modal inter-correlation information.

[0074] Multi-modal emotion recognition stage

[0075] A multi-modal deep-scale emotion feature fusion-based emotion recognition method based on deep learning, in the multi-modal emotion recognition stage, mainly includes the following parts:

[0076] Step 4: Use the MFC module to perform attention weighting on the multi-modal deep-scale emotion features in step 2 and the multi-modal inter-correlation information in step 3, calculate and output the emotion recognition result matrix;

[0077] Step 5: Use the SoftMax function to calculate the emotion recognition probability matrix of the result matrix obtained in step 4, output the maximum probability confidence emotion category, and realize multi-modal information emotion recognition.

[0078] Further, the step 4 comprises the following steps:

[0079] Step 4.1: Concatenate the multi-modal deep-scale emotion feature matrix in step 2 and the multi-modal inter-correlation information feature matrix in step 3 to obtain the emotion recognition matrix;

[0080] Step 4.2: Use the attention mechanism to calculate the attention weighting score based on the emotion recognition matrix in step 4.1;

[0081] Step 4.3: Perform dot product calculation on the attention weighting score and the emotion recognition matrix in step 4.1 to obtain the emotion recognition result matrix.

[0082] Further, the step 5 comprises the following steps:

[0083] Step 5.1: Use the SoftMax function to scale the emotion recognition result matrix generated in step 4 to obtain the emotion recognition probability matrix.

[0084] Step 5.2: Based on the emotion recognition probability matrix of step 5.1, output the emotion category with the maximum emotion probability as the emotion recognition result.

[0085] An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the method for implementing as claimed in any one of the above.

[0086] A non-transitory computer-readable storage medium having stored thereon a computer program, wherein the computer program is executed by a processor to implement the method for implementing as claimed in any one of the above.

[0087] In the following detailed description, reference will be made to the accompanying drawings, which form a part of this description. As shown in the drawings, Figure 1 The emotion recognition method based on deep learning multi-modal deep-scale emotion feature fusion mainly includes two stages: a multi-modal emotion feature extraction stage and a multi-modal emotion recognition stage. The multi-modal emotion feature extraction stage mainly includes multi-modal deep-scale emotion feature extraction and multi-modal inter-correlation emotion feature extraction. The purpose is to more effectively extract emotion-related features in a multi-level, deep-scale feature range, while effectively aligning multi-modal information, using speaker-related inter-correlation information to enhance emotion recognition performance. The multi-modal emotion recognition stage is based on generated emotion features to highlight emotion-relatedness and complete the emotion recognition task of a specific speech.

[0088] The multi-modal emotion feature extraction stage mainly includes three steps: a data preprocessing stage, a multi-modal deep-scale emotion feature extraction stage, and a multi-modal inter-correlation emotion feature extraction stage.

[0089] The multi-modal emotion feature extraction stage.

[0090] Step 1: Data preprocessing stage, input a segment of speech in wav format and transcription text, use ME module to get its general text feature T', speaker feature S and general speech feature A' through BERT, ECAPA-TDNN, huBERT pre-training model respectively;

[0091] The step 1 specifically comprises the following steps:

[0092] Step 1.1: Collect the speech waveform information of a person through a recording device, with a sampling frequency of 16000HZ, each sentence is saved separately, and a speech waveform file A stored in wav format is obtained.

[0093] Step 1.2: Use the ASR automatic speech recognition system to read in the continuous wav format file A, complete the automatic speech recognition of the speech file A of step 1.1, and obtain a continuous speech text transcription file T.

[0094] Step 1.3: General speech feature extraction and speaker feature extraction are performed on the speech file in step 1.1 using the huBERT and ECAPA-TDNN pre-training models. The pre-training model of the speech feature is trained using a mask language model. By masking 15% of a segment of speech, the model can restore the masked part of the speech based on the features, thereby extracting general and robust speech feature representations. The continuous speech file is input into the huBERT pre-training model to obtain speech feature A', and the continuous speech file is input into the ECAPA-TDNN model to obtain speaker feature S.

[0095] Step 1.4: General text feature extraction is performed on the text transcription file T in step 1.2 using the BERT pre-training model. The pre-training target is similar to step 1.3. 15% of the words are randomly replaced with other words or masks, and the model can restore the masked word information based on the trained features, thereby obtaining the general features of the text T'.

[0096] Step 1.5: The general speech features A' and speaker features S in step 1.3 and the general text features T' in step 1.4 are spliced to generate a multi-modal information dataset n represents the number of files in the corpus.

[0097] Step 2: Multi-modal deep-scale emotion feature extraction stage, as shown in Figure 2 The DST module is used to condense and extract emotion-related features from the text features T' and speech features A' obtained in step 1 in a multi-level, deep-scale feature range, obtaining multi-modal deep-scale emotion features D and H.

[0098] The step 2 specifically includes the following steps:

[0099] Step 2.1: Based on the multi-modal information dataset The speech and text features are scaled with shared weights, and the feature dimension scaling is completed by CNN1D. The formula is as shown in (1):

[0100] T' = CNN1D(T'). (1)

[0101] In particular, the convolution kernel size used in CNN1D is 1x1, and the step is 1. Through one-dimensional convolution algorithm, the feature dimension size and sequence length of A' and T' can be scaled to be consistent. In addition, in order to be able to fuse the information of different modalities with each other, the information of different modalities needs to be aligned, otherwise the model will incorrectly learn the relationship between the unequal information and ignore the connection of the related emotional words. The weight scaling method of the application mainly uses a double-layer gated recurrent neural network GRU with shared weights to realize the attention weighted multi-modal information alignment method, and the formula is as shown in formula (2):

[0102]

[0103]

[0104]

[0105]

[0106] The output generated by GRU at each time step is determined by the previous information of voice and text, and simultaneously acts on the information of the current time step input. Through the sharing weight method, the voice and text at each time step are jointly weighted based on the past information to align the emotional relevance, so that the feature area with the same emotional significance can be well reflected in the multi-modal information. Thus, we obtain the deep scale feature dataset

[0107] Step 2.2: Aligning the deep scale feature dataset of step 2.1 The Transformer encoder structure is used for feature extraction, and the fine-grained and deep-scale emotional related features of text and voice information are constructed and captured respectively, and the formula is as shown in formula (3):

[0108] H=L(G T +σ(W T (L(G T +Attn(G T )))+b)). (3)

[0109] Wherein Attn represents the abbreviation of self-attention mechanism, W and b are learnable parameters, and the formula of self-attention mechanism is as shown in formula (4):

[0110] Q=W Q X,

[0111] K=W K X,

[0112] V=W V X,

[0113]

[0114] Wherein Q, K, V represent the input query, key and value, X is the input voice feature A' and text feature T'.

[0115] Step 2.3: repeat the operation described in steps 2.1 and 2.2, splice the emotion-related features in step 2.2 to obtain a multi-level, deep-scale emotion feature set N represents the number of repetitions;

[0116] Step 2.4: using statistical methods, the emotion feature set in step 2.3 Calculate the dimensionality reduction features of MaxPool and AvgPool, and splice to obtain a multi-modal deep-scale emotion feature B.

[0117] Step 3: multi-modal interrelated emotional feature extraction stage, as shown in Figure 3 The MT module is used to calculate the interrelated information of the multi-modal data set obtained in step 1, and introduce speaker features S into the utterance-level feature representation to obtain speaker-related multi-modal interrelated emotional features U.

[0118] The step 3 specifically includes the following steps:

[0119] Step 3.1: based on the multi-modal information data set generated in step 1, use multi-head self-attention mechanism Attn, formula as shown in (5):

[0120]

[0121]

[0122] Calculate the multi-modal interrelated information C AT , C TA of the voice and text, which respectively represent the part associated with the text information in the feature set based on the voice information, and the part associated with the voice information in the feature set based on the text information. The present application effectively constructs the interinformation relationship between multi-modal by using the MT module, and further highlights the emotional features by using such dependence and correlation, formula as shown in (6):

[0123] C AT = L(T+σ(W T (L(T+Attn(T,A)))+b)). (6)

[0124] Step 3.2: Extract the sentiment feature C of the speech and text data using the Transformer structure with shared weights. For non-aligned multi-modal information, we not only need to calculate their mutual correlation feature set, but also need to focus on the construction features between the two modalities. Unlike the DST module, the MT module of the present application introduces a Transformer structure with shared weights, which can extract the intersection of multi-modal features in different modalities and multi-modal feature sets through powerful self-attention mechanism calculation. This feature intersection represents the feature range of the interaction between modalities, which can further reduce the influence of noise and help improve the performance of sentiment recognition. The formula is shown in equation (7):

[0125] C = Transformer (Concat (A, T)). (7)

[0126] Step 3.3: Use statistical methods to reduce the dimensionality using average pooling AvgPool and concatenate the speaker feature, the mutual correlation information C AT , TA and the sentiment feature C of step 3.2 to obtain the speaker-related multi-modal mutual information state feature U. The formula is shown in equation (8):

[0127] U = Concat (AvgPool (S, C, AT C TA , C)) (8)

[0128] Multi-modal sentiment recognition stage.

[0129] Step 4: Use the MFC module to perform attention weighting on the speech feature A' of step 1, the text feature T', the multi-modal deep-scale sentiment feature B of step 2, and the speaker feature-related multi-modal mutual correlation state feature U of step 3, and calculate and output the sentiment recognition result matrix P.

[0130] The step 4 includes the following steps:

[0131] Step 4.1: Concatenate the speech feature A' of step 1, the text feature T', the multi-modal deep-scale sentiment feature matrix B of step 2, and the multi-modal mutual information state feature matrix U of step 3 to obtain the sentiment recognition matrix O. The formula is shown in equation (9):

[0132] O = Concat (B, U, A', T') (9)

[0133] Step 4.2: Use the attention mechanism to calculate the attention weighting score a based on the sentiment recognition matrix O of step 4.1. The formula is shown in equation (10):

[0134] a = sigmoid(W T O) (10)

[0135] Step 4.3: Dot product calculation is performed on the attention weight score a and the emotion recognition matrix O obtained in step 4.1 to obtain an emotion recognition result matrix P, as shown in equation (11):

[0136] P = a T O (11)

[0137] Step 5: The SoftMax function is used to calculate the emotion recognition probability matrix P' of the result matrix P obtained in step 4, and the maximum probability confidence emotion class emo is output to achieve emotion recognition of multi-modal information.

[0138] The step 5 includes the following steps:

[0139] Step 5.1: The SoftMax function is used to perform dimension scaling on the emotion recognition result matrix P generated in step 4 to obtain the emotion recognition probability matrix P', as shown in equation (12):

[0140] P' = σ(W T P + b) (12)

[0141] Step 5.2: Based on the emotion recognition probability matrix P' obtained in step 5.1, the emotion class emo with the maximum emotion probability is output as the emotion recognition result, as shown in equation (13):

[0142] emo = MAX(P') (13).

Claims

1. A deep learning-based multi-modal deep-scale emotion feature fusion-based emotion recognition method, characterized in that, The implementation method comprises the following steps: Step 1: input a section of voice in wav format and transcription text, and use the ME module to obtain its general text features, speaker features and general voice features through BERT, ECAPA-TDNN and huBERT pre-training models respectively; Step 2: use the DST module to condense and extract the emotion-related features of the text features and voice features obtained in step 1 in the multi-level and deep-scale feature range, and obtain multi-modal deep-scale emotion features; Step 3: use the MT module to calculate the mutual correlation information of the multi-modal information obtained in step 1, and introduce speaker features in the utterance-level feature representation to obtain speaker-related multi-modal mutual correlation emotion features; Step 4: use the MFC module to weight the multi-modal deep-scale emotion features in step 2 and the multi-modal mutual correlation emotion features related to the speaker features in step 3, calculate and output the emotion recognition result matrix; Step 5: use the SoftMax function to calculate the emotion recognition probability matrix of the result matrix obtained in step 4, output the maximum probability confidence emotion category, and realize emotion recognition of multi-modal information.

2. The method according to claim 1, wherein, The step 1 specifically comprises the following steps: Step 1.1: collect the voice waveform information of a person through a recording device to obtain a voice file in wav format; Step 1.2: use an ASR automatic speech recognition system to complete the text transcription of the voice file in step 1.1 to obtain a voice text transcription file; Step 1.3: use huBERT and ECAPA-TDNN pre-training models to extract general voice features and speaker features from the voice file in step 1.1; Step 1.4: use the BERT pre-training model to extract general text features from the text transcription file in step 1.2; Step 1.5: splice the general voice features and speaker features in step 1.3 and the general text features in step 1.4 to generate a multi-modal information dataset. 3.The method according to claim 1, characterized in that, The step 2 specifically comprises the following steps: Step 2.1: based on the multi-modal information dataset generated in step 1, perform feature scaling on the voice and text features with shared weights to obtain a deep-scale feature dataset; Step 2.2: use the Transformer encoder structure to extract features from the deep-scale feature dataset in step 2.1 to construct and capture fine-grained and deep-scale emotion-related features of text and voice information respectively; Step 2.3: repeat the operations in steps 2.1 and 2.2, splice the emotion-related features in step 2.2 to obtain a multi-level and deep-scale emotion feature set; Step 2.4: use statistical methods to calculate the maximum pooling and average pooling features of the emotion feature set in step 2.3, and splice to obtain multi-modal deep-scale emotion features.

4. The method according to claim 3, wherein, The shared weight feature scaling in step 2.1 is specifically, Use one-dimensional convolution to scale and align the data dimensions and data lengths of the voice and text information in step 1; Use a double-layer shared weight gated recurrent neural network to simultaneously weight the voice and text in time sequence to generate attention-aligned voice and text data information.

5. The method of claim 1, wherein the method is based on deep learning and multi-modal deep-scale emotion feature fusion. The step 3 specifically comprises the following steps: Step 3.1: Based on the multi-modal information dataset generated in step 1, the multi-modal mutual correlation information of speech and text is calculated using a multi-head self-attention mechanism; Step 3.2: The shared weight Transformer structure is used to extract the emotional features of the speech and text data; Step 3.3: Using statistical methods, the speaker features, step 3.1 mutual correlation information, and step 3.2 emotional features are reduced and spliced using maximum pooling and average pooling methods to obtain speaker-related multi-modal mutual information emotional features.

6. The method according to claim 5, wherein the method is a deep learning based multi-modal deep feature fusion emotion recognition method. The step 3.1 calculates the multi-modal mutual information, which is, Using a multi-head self-attention mechanism, taking speech information as the query and text information as the key and value, and calculating multi-head attention as the mutual information of speech to text; Using a multi-head self-attention mechanism, taking text information as the query and speech information as the key and value, and calculating multi-head attention as the mutual information of text to speech.

7. The method of claim 1, wherein the method is based on deep learning and multi-modal deep-scale emotion feature fusion. The step 4 is specifically, Step 4.1: Splice the multi-modal deep-scale emotional feature matrix of step 2 and the multi-modal mutual information emotional feature matrix of step 3 to obtain an emotional recognition matrix; Step 4.2: Using attention mechanism, calculate the attention weighted score based on the emotional recognition matrix of step 4.1; Step 4.3: Dot product calculation of attention weighted score and step 4.1 emotional recognition matrix to obtain emotional recognition result matrix. 8.The method according to claim 1, wherein, The step 5 includes the following steps: Step 5.1: Use the SoftMax function to scale the emotional recognition result matrix generated in step 4 to obtain an emotional recognition probability matrix; Step 5.2: Based on the emotional recognition probability matrix of step 5.1, output the maximum emotional probability emotional category as the emotional recognition result.

9. An electronic device, comprising: The computer program is stored in the memory and can be run on the processor, and when the processor executes the computer program, the implementation method of any one of claims 1-8 is realized.

10. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the implementation method of any one of claims 1-8. The computer program is executed by the processor to realize the implementation method of any one of claims 1-8.

Images