Multi-level fusion multimodal emotion recognition method and system based on Wav2vec2.0 and BERT

Abstract

The present invention provides a method and system for multi-level fusion multimodal emotion recognition based on Wav2vec 2.0 and BERT, including: extracting speech frame-level feature vectors using a large-scale pre-trained speech model Wav2vec 2.0; extracting speech word-level, phoneme-level, and syllable-level feature vectors from speech frame-level feature vectors using word-level, phoneme-level, and syllable-level forced alignment respectively; extracting text feature vectors using large-scale pre-trained BERT for text; fusing the early fusion decision and late fusion decision of the speech frame-level feature vector, the speech word-level feature vector, the speech phoneme-level feature vector, the speech syllable-level feature vector, and the text feature vector; and fusing the early and late combined fusion decision of the early fusion decision and the late fusion decision. The present invention uses a large-scale pre-trained model to extract feature vectors of speech and text respectively, and proposes a multi-level fusion strategy to achieve more accurate multimodal emotion recognition.

Description

Technical Field

[0001] The present invention relates to the field of multimodal processing of text and speech, and in particular, to a method and system for multi-level fusion multimodal emotion recognition based on Wav2vec2.0 and BERT. Background Art

[0002] Humans have a wide range of emotions, and the expression of emotions is common in life. Humans also have a strong ability to discern these emotions in order to generate appropriate responses. Similarly, machines with the ability to discern emotions can make them closer to humans, so there are many application scenarios. Therefore, emotion recognition is a very important task.

[0003] The present invention considers the multimodal emotion recognition task of text and speech. Both modalities can be used alone to recognize emotions, but the complementary use of the two can improve the recognition effect. This is because when there is only text but no speech, the model cannot distinguish sarcasm and other situations. If it is combined with speech information, it will help to judge; if there is only speech but no text, it is difficult to accurately obtain the wording, and the speaker's words will directly or indirectly reflect his emotions. For the two modalities, how to extract their features separately and effectively integrate them is the focus of multimodal emotion recognition.

[0004] Patent document CN112489635A, application number 202011397667.1 discloses a multimodal emotion recognition method based on an enhanced attention mechanism. FBank acoustic features are extracted for speech information, and feature vectors are extracted for text information through the large-scale pre-trained model BERT. In the fusion stage, a self-attention model is used for the two modalities respectively, and then the alignment matrix is ​​obtained by dot multiplication, which is then calibrated with the original features of the modality to obtain more interactive information. Finally, the two modalities are passed through the self-attention model respectively. The above patent considers a powerful pre-training model to extract features for text, but only uses simple frequency domain features for speech, which will result in strong text modal performance and weak speech performance, which is not conducive to fusion; when fusion, only the early fusion step similar to the joint attention of this patent is considered, but the later fusion step is not considered, and these two steps are fused at different stages, which can obtain information at different levels; at the same time, it also does not consider the use of phoneme-level, word-level, and syllable-level speech features to assist recognition, and these features are all related to rhythm, and rhythm can reflect the characteristics of a sentence, such as emotional state, because it contains information about the ups and downs of the speech signal.

[0005] Patent document CN113506562A (application number: 202110812076.4) discloses an end-to-end speech synthesis method and system based on the fusion of acoustic features and text emotion features, and constructs an end-to-end speech synthesis model. The rhythmic latent features of the text can be obtained according to the embedding vector of the phoneme sequence, and the phonemes and Mel spectrums are aligned. The acoustic features at the phoneme level and the sentence level are obtained respectively through the acoustic encoder, and the latent features are added to the acoustic features. At the same time, the text emotion features are obtained from the text; the acoustic features and the text emotion features are fused again, and input into the variance adapter of FastSpeech2 for feature enhancement and expansion; finally, the output of the variance adapter is input into the Mel decoder of FastSpeech2 to realize parallel decoding to obtain the spectrum diagram of the synthesized speech; the spectrum diagram features are mapped into a sound waveform using a vocoder to obtain the synthesized speech. Summary of the invention

[0006] In view of the defects in the prior art, the purpose of the present invention is to provide a method and system for multi-level fusion multimodal emotion recognition based on Wav2vec2.0 and BERT.

[0007] According to the present invention, a multi-level fusion multi-modal emotion recognition method based on Wav2vec2.0 and BERT is provided, comprising:

[0008] Step S1: extracting the frame-level speech feature vector of the input speech using the Wav2vec 2.0 model;

[0009] Step S2: Using the input text and the input speech to perform word-level mandatory alignment on the frame-level speech feature vector to obtain a word-level speech feature vector;

[0010] Step S3: using the input text and the input speech to perform phoneme-level forced alignment on the frame-level speech feature vector to obtain a phoneme-level speech feature vector;

[0011] Step S4: using the input text and the input speech to perform syllable-level forced alignment on the frame-level speech feature vector to obtain a syllable-level speech feature vector;

[0012] Step S5: extracting a text feature vector of the input text using the BERT model;

[0013] Step S6: Use the speech feature vector, the word-level speech feature vector, the phoneme-level speech feature vector, the syllable-level speech feature vector and the text feature vector to respectively concatenate the feature vectors through the common attention model, and fuse the concatenated feature vectors through the linear model to obtain the probability distribution of each emotion category fused in the early stage;

[0014] Step S7: using the speech feature vector, the word-level speech feature vector, the phoneme-level speech feature vector, the syllable-level speech feature vector and the text feature vector to obtain updated feature vectors through linear models respectively, averaging the updated feature vectors, and then fusing them through the linear model to obtain the probability distribution of each emotion category after fusion;

[0015] Step S8: averaging the probability distribution of each emotion category fused in the early stage and the probability distribution of each emotion category fused in the late stage to obtain the probability distribution of the combined fusion of the early stage and the late stage;

[0016] The input text is the text content corresponding to the input voice.

[0017] Preferably, the step S1 adopts:

[0018] Fixed the parameters of the Wav2vec 2.0 model, input the speech into the Wav2vec 2.0 model, extract the hidden state of each layer of the preset number of Transformer encoding layers in the Wav2vec 2.0 model, and calculate the weighted average of different layers. The formula is as follows:

[0019]

[0020] in, represents the nth speech sample of the input; W represents the Wav2vec 2.0 model; F represents the frame-level feature;

[0021]

[0022] in, represents the weight of the lth layer of the frame-level speech feature vector, Indicates U F,n The f-th frame of the l-th layer of ; LN represents the layer normalization operation.

[0023] Preferably, the step S2 adopts: using the input text and the input audio to obtain word-level mandatory alignment annotations, using the word-level mandatory alignment annotations to segment the frame-level speech feature vectors, obtaining word-level speech feature vectors, and calculating weighted averages for different layers, the formula is as follows:

[0024]

[0025] in, represents the kth part of the lth layer word-level speech feature vector of the nth sample, s represents the starting frame, e represents the ending frame, Indicates U F,n The fth frame of the lth layer;

[0026]

[0027] in, represents the weight of the lth layer of the word-level speech feature vector; LN represents the layer normalization operation.

[0028] Preferably, the step S3 adopts: using the input text and the input audio to obtain phoneme-level mandatory alignment annotation, using the phoneme-level mandatory alignment annotation to segment the frame-level speech feature vector, obtaining the phoneme-level speech feature vector, and calculating the weighted average of different layers, the formula is as follows:

[0029]

[0030] in, represents the kth part of the lth layer phoneme-level speech feature vector of the nth sample; s represents the starting frame, and e represents the ending frame; Indicates U F,n The fth frame of the lth layer;

[0031]

[0032] in, represents the weight of the lth layer of the phoneme-level speech feature vector; LN represents the layer normalization operation.

[0033] Preferably, the step S4 adopts: using the input text and the input audio to obtain syllable-level forced alignment annotation, using the syllable-level forced alignment annotation to segment the frame-level speech feature vector, to obtain the syllable-level speech feature vector, and calculating the weighted average of different layers, the formula is as follows:

[0034]

[0035] in, represents the kth part of the lth layer syllable-level speech feature vector of the nth sample; s represents the start frame; e represents the end frame; Indicates U F,n The fth frame of the lth layer;

[0036]

[0037] in, Represents the weight of the lth layer of the word syllable speech feature vector; LN represents the layer normalization operation.

[0038] Preferably, step S5 adopts: fixing the BERT model parameters, inputting the text into the BERT model, extracting the hidden state of each layer of a preset number of Transformer encoding layers in the BERT model, and calculating the weighted average of different layers, and the formula is as follows:

[0039]

[0040] in, represents the nth text sample of the input; B represents the BERT model; T represents the text feature;

[0041]

[0042] in, Represents the weight of the lth layer of the text feature vector; Indicates U T,n The kth part of the lth layer of ; LN represents the layer normalization operation.

[0043] Preferably, the step S6 adopts:

[0044] Step S6.1: The phoneme-level speech feature vector, word-level speech feature vector, syllable-level speech feature vector, frame-level speech feature vector and text feature vector are respectively subjected to four common attention models and global average to obtain the phoneme-level speech feature vector, word-level speech feature vector, syllable-level speech feature vector and frame-level speech feature vector that integrate text information;

[0045] Step S6.2: concatenate the phoneme-level speech feature vector, word-level speech feature vector, syllable-level speech feature vector and frame-level speech feature vector of the fused text information, fuse them through a linear model and output a predicted probability distribution;

[0046]

[0047] in, Represents the splicing operation; c n Represents the feature vector containing text and four types of speech information; c P,n The phoneme-level speech feature vector representing the nth sample fused with text information, c W,n represents the word-level speech feature vector of the nth sample fused with text information, c S,n represents the syllable-level speech feature vector representing the nth sample fused with text information, c F,n represents the frame-level speech feature vector representing the nth sample fused with text information;

[0048] After concatenation, feed it into the linear model:

[0049] g n =LN(Relu(W7c n +b7)+c n )

[0050]

[0051] Among them, g n represents the feature vector containing text and four types of speech information after linear change, W7, W8 represent weight parameters; b7, b8 represent bias parameters; Represents the probability distribution of sentiment prediction obtained in the previous fusion step.

[0052] Preferably, the step S7 adopts:

[0053] Step S7.1: respectively applying the phoneme-level speech feature vector, the word-level speech feature vector, the syllable-level speech feature vector, the frame-level speech feature vector and the text feature vector to the linear model to obtain updated phoneme-level speech feature vector, word-level speech feature vector, syllable-level speech feature vector, frame-level speech feature vector and text feature vector;

[0054]

[0055]

[0056] Among them, P, W, S, F, T represent phoneme level, word level, syllable level, frame level, and text respectively; Relu represents the Relu nonlinear function, and W and b are parameters; represents the kth part of the eigenvector after a linear change of the nth sample mode i, represents the kth part of the eigenvector after the quadratic linear change of the nth sample mode i;

[0057] Step S7.2: using global averaging to fuse the features of different parts of the phoneme-level speech feature vector, word-level speech feature vector, syllable-level speech feature vector, frame-level speech feature vector, and text feature vector to obtain a phoneme-level speech feature vector, word-level speech feature vector, syllable-level speech feature vector, frame-level speech feature vector, and text feature vector representing all parts;

[0058]

[0059] Among them, l i,n is the sentence length, P, W, S, F, and T represent phoneme level, word level, syllable level, frame level, and text, respectively;

[0060] Step S7.3: The phoneme-level speech feature vector, word-level speech feature vector, syllable-level speech feature vector, frame-level speech feature vector, and text feature vector are fused through a linear model and a predicted probability distribution is output;

[0061]

[0062] Where W and b are parameters, P, W, S, F, T represent phoneme level, word level, syllable level, frame level, and text respectively; softmax represents the softmax function; represents the probability distribution of sentiment prediction obtained in the later fusion step, is the weight parameter, Represents the bias parameter.

[0063] Preferably, the step S8 adopts:

[0064] The front and rear combined fusion steps are to take the average of the output of the early fusion step and the output of the late fusion step, and the formula is as follows:

[0065]

[0066] According to the present invention, a multi-level fusion multi-modal emotion recognition system based on Wav2vec2.0 and BERT is provided, comprising:

[0067] Module M1: Extract the frame-level speech feature vector of the input speech using the Wav2vec 2.0 model;

[0068] Module M2: Using the input text and the input speech to perform word-level forced alignment on the frame-level speech feature vector to obtain a word-level speech feature vector;

[0069] Module M3: Using the input text and the input speech to perform phoneme-level forced alignment on the frame-level speech feature vector to obtain a phoneme-level speech feature vector;

[0070] Module M4: Using the input text and the input speech to perform syllable-level forced alignment on the frame-level speech feature vector to obtain a syllable-level speech feature vector;

[0071] Module M5: Use the BERT model to extract the text feature vector of the input text;

[0072] Module M6: Use speech feature vectors, word-level speech feature vectors, phoneme-level speech feature vectors, and syllable-level speech feature vectors to concatenate with text feature vectors through a common attention model to obtain feature vectors, and fuse the concatenated feature vectors through a linear model to obtain the probability distribution of each emotion category fused in the early stage;

[0073] Module M7: Use the speech feature vector, word-level speech feature vector, phoneme-level speech feature vector, syllable-level speech feature vector and text feature vector to obtain updated feature vectors through linear models respectively, average the updated feature vectors, and then fuse them through linear models to obtain the probability distribution of each emotion category in the later stage;

[0074] Module M8: average the probability distribution of each emotion category fused in the early stage and the probability distribution of each emotion category fused in the late stage to obtain the probability distribution of the combined fusion of the early stage and the late stage;

[0075] The input text is the text content corresponding to the input voice.

[0076] Compared with the prior art, the present invention has the following beneficial effects:

[0077] 1. This paper explores the use of large-scale pre-trained models to extract features of both speech and text, and considers both speech and text as the main modalities of multimodal emotion recognition, rather than just considering speech as an auxiliary modality of text. The acquisition of powerful features by both speech and text is the prerequisite for effective fusion;

[0078] 2. The present invention proposes to use not only conventional frame-level speech features, but also the information after forced alignment to obtain syllable-level, phoneme-level, and word-level speech feature vectors to assist in the task of emotion recognition. These three speech feature vectors are related to prosody, and prosody can convey the characteristics of a sentence, such as emotional state, because it contains information about the intonation of the speech signal, and is therefore suitable as auxiliary information for emotion recognition;

[0079] 3. For the above-mentioned multimodal features, the present invention proposes a front-end and late-stage combined fusion method based on different levels. This method not only uses the front-end fusion method based on common attention, but also uses the late-stage fusion method. These two methods fuse the two modalities from different levels, so the angles of the information obtained are different. The present invention combines the information to obtain a more complete fusion result, thereby improving the effect of multimodal emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0081] Figure 1 A flow chart of a method in an embodiment of the present invention;

[0082] Figure 2 A schematic diagram of a system in an embodiment of the present invention;

[0083] Figure 3 This is a structural diagram of the joint attention model in an embodiment of the present invention. DETAILED DESCRIPTION

[0084] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0085] Example 1

[0086] According to a multi-level fusion multi-modal emotion recognition method based on Wav2vec2.0 and BERT provided by the present invention, Figure 1 As shown, including:

[0087] Frame-level speech feature vector extraction step: For the input speech, use the large-scale speech pre-training model Wav2vec2.0 to extract the frame-level speech feature vector;

[0088] The word-level speech feature vector extraction step: performing word-level forced alignment on the frame-level speech feature vector to obtain a word-level speech feature vector;

[0089] The phoneme-level speech feature vector extraction step: performing phoneme-level forced alignment on the frame-level speech feature vector to obtain a phoneme-level speech feature vector;

[0090] The syllable-level speech feature vector extraction step: performing syllable-level forced alignment on the syllable-level speech feature vector to obtain a syllable-level speech feature vector;

[0091] Text feature vector extraction step: For the input text, use the large-scale text pre-training model BERT to extract the text feature vector;

[0092] Preliminary fusion step: use the speech frame-level feature vector, speech word-level feature vector, speech phoneme-level feature vector, and speech syllable-level feature vector respectively with the text feature vector through a common attention model, and then pass the concatenated feature vectors through a linear model to obtain the probability distribution of each emotion category in the preliminary fusion.

[0093] Late fusion step: Use the speech frame-level feature vector, speech word-level feature vector, speech phoneme-level feature vector, speech syllable-level feature vector, and text feature vector to pass through linear models respectively, and average the obtained feature vectors to obtain the probability distribution of each emotion category for late fusion.

[0094] The step of combining and fusing the front and rear ends is as follows: averaging the probability distribution of the front and rear ends fusion and obtaining the probability distribution of the front and rear ends fusion.

[0095] Specifically, the frame-level speech feature vector extraction step includes: fixing the Wav2vec 2.0 model parameters, inputting the speech into the model, extracting the hidden state of each layer of the 12-layer Transformer coding layer in the Wav2vec 2.0 model, and calculating the weighted average of different layers. The formula is as follows:

[0096]

[0097] in, represents the nth speech sample of the input, W represents the Wav2vec 2.0 model, and F represents the frame-level features.

[0098]

[0099] in, represents the weight of the lth layer of the frame-level speech feature vector. This weight is learnable. It's U F ,n The f-th frame of the l-th layer.

[0100] Specifically, the word-level speech feature vector extraction step includes: using the input text and the input audio to obtain the word-level mandatory alignment annotation, using the word-level mandatory alignment annotation to segment the frame-level speech feature vector, obtaining the word-level speech feature vector, and calculating the weighted average of different layers, the formula is as follows:

[0101]

[0102] in is the kth part of the lth layer word-level speech feature vector of the nth sample, s is the start frame, e is the end frame, It's U F,n The f-th frame of the l-th layer.

[0103]

[0104] in Represents the weight of the lth layer of the word-level speech feature vector, which is learnable. LN represents the layer normalization operation.

[0105] Specifically, the phoneme-level speech feature vector extraction step includes: using the input text and the input audio to obtain the phoneme-level forced alignment annotation, using the phoneme-level forced alignment annotation to segment the frame-level speech feature vector, obtaining the phoneme-level speech feature vector, and calculating the weighted average of different layers. The formula is as follows:

[0106]

[0107] in is the kth part of the lth layer phoneme-level speech feature vector of the nth sample, s is the start frame, e is the end frame, It's U F,n The f-th frame of the l-th layer.

[0108]

[0109] in Represents the weight of the lth layer of the phoneme-level speech feature vector, which is learnable. LN represents the layer normalization operation.

[0110] Specifically, the syllable-level speech feature vector extraction step includes: using the input text and the input audio to obtain syllable-level forced alignment annotations, using the syllable-level forced alignment annotations to segment the frame-level speech feature vectors, obtaining syllable-level speech feature vectors, and calculating weighted averages for different layers. The formula is as follows:

[0111]

[0112] in is the kth part of the lth layer syllable-level speech feature vector of the nth sample, s is the start frame, e is the end frame, It's U F,n The f-th frame of the l-th layer.

[0113]

[0114] in Represents the weight of the lth layer of the word syllable speech feature vector, which is learnable. LN represents the layer normalization operation.

[0115] Specifically, the text feature vector extraction step includes: fixing the BERT model parameters, inputting the text into the model, extracting the hidden state of each layer of the 12-layer Transformer encoding layer in the BERT model, and calculating the weighted average of different layers. The formula is as follows:

[0116]

[0117] in, represents the nth text sample of the input, B represents the BERT model, and T represents the text feature.

[0118]

[0119] in Represents the weight of the lth layer of the text feature vector. This weight is learnable. It's U T,n The kth part of the lth layer of , LN represents the layer normalization operation.

[0120] Specifically, the late fusion step includes: the late fusion step respectively obtains updated phoneme-level speech feature vectors, word-level speech feature vectors, syllable-level speech feature vectors, frame-level speech feature vectors, and text feature vectors through a linear model, and the formula is as follows:

[0121]

[0122]

[0123] Where P, W, S, F, T represent phoneme level, word level, syllable level, frame level, and text respectively. Relu represents the Relu nonlinear function, and W and b are both parameters and can be learned.

[0124] Then, the phoneme-level speech feature vector, word-level speech feature vector, syllable-level speech feature vector, frame-level speech feature vector, and text feature vector are respectively fused using global average to obtain the phoneme-level speech feature vector, word-level speech feature vector, syllable-level speech feature vector, frame-level speech feature vector, and text feature vector representing all parts. The formula is as follows:

[0125]

[0126] Among them l i,n is the sentence length, and P, W, S, F, and T represent phoneme level, word level, syllable level, frame level, and text, respectively.

[0127] Then, the phoneme-level speech feature vector, word-level speech feature vector, syllable-level speech feature vector, frame-level speech feature vector, and text feature vector are fused through a linear model and a predicted probability distribution is output. The formula is as follows:

[0128]

[0129] Where W and b are parameters and are learnable. P, W, S, F, and T represent phoneme level, word level, syllable level, frame level, and text, respectively. Softmax represents the softmax function.

[0130] Specifically, the preliminary fusion step includes: respectively passing the phoneme-level speech feature vector, word-level speech feature vector, syllable-level speech feature vector, frame-level speech feature vector and text feature vector through four common attention models and global averaging, and obtaining the phoneme-level speech feature vector, word-level speech feature vector, syllable-level speech feature vector and frame-level speech feature vector of fused text information, and the formula is as follows:

[0131] Each joint attention model contains three identical joint attention layers connected one after the other. The joint attention model is as follows Figure 3 As shown. The input of the joint attention layer is a speech feature vector and a text feature vector. The speech feature vector and the text feature vector are respectively sent to two multi-head attention models as the Q value of the multi-head attention model. Without loss of generality, the single-head attention model is introduced here:

[0132]

[0133]

[0134]

[0135] in i and j represent that they come from different modalities. Then the output of the single-head attention model can be obtained:

[0136]

[0137] Where d represents the dimension of the feature. The multi-head attention model performs this process multiple times in order to learn more information from different representation subspaces. The outputs of the two multi-head attention models pass through the layer normalization layer and the fully connected layer twice respectively. The joint attention layer is repeated three times to obtain the output of the joint attention model, which is then globally averaged to obtain c i,n , where i∈{P,W,S,F}, P, W, S, F, T represent phoneme level, word level, syllable level, and frame level respectively. The formula for global average is the same as formula 1.

[0138] Then, the phoneme-level speech feature vector, word-level speech feature vector, syllable-level speech feature vector, and frame-level speech feature vector are concatenated, fused through a linear model, and the predicted probability distribution is outputted. The formula is as follows:

[0139]

[0140] in Represents the concatenation operation. After concatenation, it is fed into the linear model:

[0141] g n =LN(Relu(W7c n +b7)+c n )

[0142]

[0143] Specifically, the front and rear combined fusion steps are to take the average value of the output of the early fusion step and the output of the late fusion step, and the formula is as follows:

[0144]

[0145] According to a second aspect of the present invention, a system for multi-level fusion multimodal emotion recognition based on Wav2vec 2.0 and BERT is provided, such as Figure 2 As shown, including:

[0146] Frame-level speech feature vector extraction module: For the input speech, the large-scale speech pre-training model Wav2vec2.0 is used to extract the frame-level speech feature vector;

[0147] A word-level speech feature vector extraction module: performs word-level forced alignment on the frame-level speech feature vector to obtain a word-level speech feature vector;

[0148] A phoneme-level speech feature vector extraction module: performs phoneme-level forced alignment on the frame-level speech feature vector to obtain a phoneme-level speech feature vector;

[0149] A syllable-level speech feature vector extraction module: performs syllable-level forced alignment on the syllable-level speech feature vector to obtain a syllable-level speech feature vector;

[0150] Text feature vector extraction module: For the input text, use the large-scale text pre-training model BERT to extract text feature vectors;

[0151] Early fusion module: Use the speech frame-level feature vector, speech word-level feature vector, speech phoneme-level feature vector, and speech syllable-level feature vector with the text feature vector respectively through a common attention model, and then pass the concatenated feature vectors through a linear model to obtain the probability distribution of each emotion category in the early fusion.

[0152] Late fusion module: Use the speech frame-level feature vector, speech word-level feature vector, speech phoneme-level feature vector, speech syllable-level feature vector, and text feature vector respectively through linear models, and average the obtained feature vectors to obtain the probability distribution of each emotion category for late fusion.

[0153] Front-end and rear-end combined fusion module: average the front-end fusion probability distribution and the rear-end fusion probability distribution to obtain the front-end and rear-end combined fusion probability distribution.

[0154] The present invention uses cross entropy as the training loss function, Adam optimizer as the optimizer, the early fusion step uses a learning rate of 5e-5, the late fusion step uses a learning rate of 1e-3, the batch size is 32, and early stopping is used in training. Dropout layers are added to all linear layers except the last linear layer to prevent overfitting with a probability of 0.2. The large-scale pre-training models use wav2vec 2.0-base and BERT-base un-cased models, and their feature dimensions are both 768 dimensions.

[0155] In summary, the present invention provides a method for multi-level fusion multimodal emotion recognition based on Wav2vec2.0 and BERT. The present invention explores the use of large-scale pre-trained models to extract features of speech and text, and regards both speech and text as the main modalities of multimodal emotion recognition, rather than only regarding speech as the auxiliary modality of text. Both speech and text obtain powerful features, which is the premise for effective fusion; the present invention proposes not only to use conventional frame-level speech features, but also to use the information after forced alignment to obtain syllable-level, phoneme-level, and word-level speech feature vectors to assist the emotion recognition task. These three speech feature vectors are related to rhythm, and rhythm can convey the characteristics of sentences, such as emotional state, because it contains the information of the ups and downs of speech signals, so it is suitable as auxiliary information for emotion recognition; the present invention proposes a combined fusion method based on different levels of front and back ends for the above-mentioned multimodal features. This method not only uses the front fusion method based on common attention, but also uses the back fusion method. These two methods fuse two modalities from different levels, so the angles of the information obtained are different. The present invention combines its information to obtain a more complete fusion result, which improves the effect of multimodal emotion recognition.

[0156] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0157] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A multi-level fusion multimodal emotion recognition method based on Wav2vec2.0 and BERT, characterized in that: include: Step S1: extracting the frame-level speech feature vector of the input speech using the Wav2vec 2.0 model; Step S2: Using the input text and the input speech to perform word-level mandatory alignment on the frame-level speech feature vector to obtain a word-level speech feature vector; Step S3: using the input text and the input speech to perform phoneme-level forced alignment on the frame-level speech feature vector to obtain a phoneme-level speech feature vector; Step S4: using the input text and the input speech to perform syllable-level forced alignment on the frame-level speech feature vector to obtain a syllable-level speech feature vector; Step S5: extracting a text feature vector of the input text using the BERT model; Step S6: Use the frame-level speech feature vector, the word-level speech feature vector, the phoneme-level speech feature vector, and the syllable-level speech feature vector to concatenate with the text feature vector through a common attention model to obtain a feature vector, and fuse the concatenated feature vectors through a linear model to obtain the probability distribution of each emotion category fused in the early stage; Step S7: using the frame-level speech feature vector, the word-level speech feature vector, the phoneme-level speech feature vector, the syllable-level speech feature vector and the text feature vector to obtain updated feature vectors through linear models respectively, averaging the updated feature vectors, and then fusing them through the linear model to obtain the probability distribution of each emotion category after fusion; Step S8: averaging the probability distribution of each emotion category fused in the early stage and the probability distribution of each emotion category fused in the late stage to obtain the probability distribution of the combined fusion of the early stage and the late stage; The input text is the text content corresponding to the input voice; The step S6 adopts: Step S6.1: The phoneme-level speech feature vector, word-level speech feature vector, syllable-level speech feature vector, frame-level speech feature vector and text feature vector are respectively subjected to four common attention models and global average to obtain the phoneme-level speech feature vector, word-level speech feature vector, syllable-level speech feature vector and frame-level speech feature vector that integrate text information; Step S6.2: concatenate the phoneme-level speech feature vector, word-level speech feature vector, syllable-level speech feature vector and frame-level speech feature vector of the fused text information, fuse them through a linear model and output a predicted probability distribution; c n =c P,n ⊕c W,n ⊕c S,n ⊕c F,n Among them, ⊕ represents the splicing operation; c n Represents the feature vector containing text and four types of speech information; c P,n The phoneme-level speech feature vector representing the nth sample fused with text information, c W,n represents the word-level speech feature vector of the nth sample fused with text information, c S,n represents the syllable-level speech feature vector representing the nth sample fused with text information, c F,n represents the frame-level speech feature vector representing the nth sample fused with text information; After concatenation, feed it into the linear model: g n =LN(Relu(W7c n +b7)+c n ) Among them, g n represents the feature vector containing text and four types of speech information after linear change, W7, W8 represent weight parameters; b7, b8 represent bias parameters; Represents the probability distribution of sentiment prediction obtained in the previous fusion step.

2. The method for multi-level fusion multimodal emotion recognition based on Wav2vec2.0 and BERT according to claim 1, characterized in that: The step S1 adopts: Fixed the parameters of the Wav2vec 2.0 model, input the speech into the Wav2vec 2.0 model, extract the hidden state of each layer of the preset number of Transformer encoding layers in the Wav2vec 2.0 model, and calculate the weighted average of different layers. The formula is as follows: in, represents the nth speech sample of the input; W represents the Wav2vec 2.0 model; F represents the frame-level feature; in, represents the weight of the lth layer of the frame-level speech feature vector, Indicates U F,n The f-th frame of the l-th layer of ; LN represents the layer normalization operation.

3. The method for multi-level fusion multimodal emotion recognition based on Wav2vec2.0 and BERT according to claim 1, characterized in that: The step S2 adopts: using the input text and the input audio to obtain word-level mandatory alignment annotations, using the word-level mandatory alignment annotations to segment the frame-level speech feature vectors, obtaining word-level speech feature vectors, and calculating weighted averages for different layers, the formula is as follows: in, represents the kth part of the lth layer word-level speech feature vector of the nth sample, s represents the starting frame, e represents the ending frame, Indicates U F,n The fth frame of the lth layer; in, represents the weight of the lth layer of the word-level speech feature vector; LN represents the layer normalization operation.

4. The method for multi-level fusion multimodal emotion recognition based on Wav2vec2.0 and BERT according to claim 1, characterized in that: The step S3 adopts: using the input text and the input audio to obtain the phoneme-level mandatory alignment annotation, using the phoneme-level mandatory alignment annotation to segment the frame-level speech feature vector, to obtain the phoneme-level speech feature vector, and to calculate the weighted average of different layers, the formula is as follows: in, represents the kth part of the lth layer phoneme-level speech feature vector of the nth sample; s represents the starting frame, and e represents the ending frame; Indicates U F,n The fth frame of the lth layer; in, represents the weight of the lth layer of the phoneme-level speech feature vector; LN represents the layer normalization operation.

5. The method for multi-level fusion multimodal emotion recognition based on Wav2vec2.0 and BERT according to claim 1, characterized in that: The step S4 adopts: using the input text and the input audio to obtain syllable-level mandatory alignment annotation, using the syllable-level mandatory alignment annotation to segment the frame-level speech feature vector, obtaining the syllable-level speech feature vector, and calculating the weighted average of different layers, the formula is as follows: in, represents the kth part of the lth layer syllable-level speech feature vector of the nth sample; s represents the start frame; e represents the end frame; Indicates U F,n The fth frame of the lth layer; in, Represents the weight of the lth layer of the word syllable speech feature vector; LN represents the layer normalization operation.

6. The method for multi-level fusion multimodal emotion recognition based on Wav2vec2.0 and BERT according to claim 1, characterized in that: The step S5 adopts: fixing the BERT model parameters, inputting the text into the BERT model, extracting the hidden state of each layer of a preset number of Transformer encoding layers in the BERT model, and calculating the weighted average of different layers. The formula is as follows: in, represents the nth text sample of the input; B represents the BERT model; T represents the text feature; in, Represents the weight of the lth layer of the text feature vector; Indicates U T,n The kth part of the lth layer of ; LN represents the layer normalization operation.

7. The method for multi-level fusion multimodal emotion recognition based on Wav2vec2.0 and BERT according to claim 1, characterized in that: The step S7 adopts: Step S7.1: respectively applying the phoneme-level speech feature vector, the word-level speech feature vector, the syllable-level speech feature vector, the frame-level speech feature vector and the text feature vector to the linear model to obtain updated phoneme-level speech feature vector, word-level speech feature vector, syllable-level speech feature vector, frame-level speech feature vector and text feature vector; Among them, P, W, S, F, T represent phoneme level, word level, syllable level, frame level, and text respectively; Relu represents the Relu nonlinear function, and W and b are parameters; represents the kth part of the eigenvector after a linear change of the nth sample mode i, represents the kth part of the eigenvector after the quadratic linear change of the nth sample mode i; Step S7.2: using global averaging to fuse the features of different parts of the phoneme-level speech feature vector, word-level speech feature vector, syllable-level speech feature vector, frame-level speech feature vector, and text feature vector to obtain a phoneme-level speech feature vector, word-level speech feature vector, syllable-level speech feature vector, frame-level speech feature vector, and text feature vector representing all parts; Among them, l i,n is the sentence length, P, W, S, F, and T represent phoneme level, word level, syllable level, frame level, and text, respectively; Step S7.3: The phoneme-level speech feature vector, word-level speech feature vector, syllable-level speech feature vector, frame-level speech feature vector, and text feature vector are fused through a linear model and a predicted probability distribution is output; Where W and b are parameters, P, W, S, F, T represent phoneme level, word level, syllable level, frame level, and text respectively; softmax represents the softmax function; represents the probability distribution of sentiment prediction obtained in the later fusion step, is the weight parameter, Represents the bias parameter.

8. The method for multi-level fusion multi-modal emotion recognition based on Wav2vec2.0 and BERT according to claim 1, characterized in that: The step S8 adopts: The front and rear combined fusion steps are to take the average of the output of the early fusion step and the output of the late fusion step, and the formula is as follows:

9. A multi-level fusion multimodal emotion recognition system based on Wav2vec2.0 and BERT, characterized in that: include: Module M1: Extract the frame-level speech feature vector of the input speech using the Wav2vec 2.0 model; Module M2: Using the input text and the input speech to perform word-level forced alignment on the frame-level speech feature vector to obtain a word-level speech feature vector; Module M3: Using the input text and the input speech to perform phoneme-level forced alignment on the frame-level speech feature vector to obtain a phoneme-level speech feature vector; Module M4: Using the input text and the input speech to perform syllable-level forced alignment on the frame-level speech feature vector to obtain a syllable-level speech feature vector; Module M5: Use the BERT model to extract the text feature vector of the input text; Module M6: Use the frame-level speech feature vector, word-level speech feature vector, phoneme-level speech feature vector, and syllable-level speech feature vector to concatenate with the text feature vector through a common attention model to obtain feature vectors, and fuse the concatenated feature vectors through a linear model to obtain the probability distribution of each emotion category fused in the early stage; Module M7: Use the frame-level speech feature vector, word-level speech feature vector, phoneme-level speech feature vector, syllable-level speech feature vector and text feature vector to obtain updated feature vectors through linear models respectively, average the updated feature vectors, and then fuse them through linear models to obtain the probability distribution of each emotion category in the later stage; Module M8: average the probability distribution of each emotion category fused in the early stage and the probability distribution of each emotion category fused in the late stage to obtain the probability distribution of the combined fusion of the early stage and the late stage; The input text is the text content corresponding to the input voice; The module M6 adopts: Module M6.1: The phoneme-level speech feature vector, word-level speech feature vector, syllable-level speech feature vector, frame-level speech feature vector and text feature vector are respectively passed through four common attention models and globally averaged to obtain the phoneme-level speech feature vector, word-level speech feature vector, syllable-level speech feature vector and frame-level speech feature vector that integrate text information; Module M6.2: Concatenate the phoneme-level speech feature vector, word-level speech feature vector, syllable-level speech feature vector and frame-level speech feature vector of the fused text information, fuse them through a linear model and output the predicted probability distribution; c n =c P,n ⊕c W,n ⊕c S,n ⊕c F,n Among them, ⊕ represents the splicing operation; c n Represents the feature vector containing text and four types of speech information; c P,n The phoneme-level speech feature vector representing the nth sample fused with text information, c W,n represents the word-level speech feature vector of the nth sample fused with text information, c S,n represents the syllable-level speech feature vector representing the nth sample fused with text information, c F,n represents the frame-level speech feature vector representing the nth sample fused with text information; After concatenation, feed it into the linear model: g n =LN(Relu(W7c n +b7)+c n ) Among them, g n represents the feature vector containing text and four types of speech information after linear change, W7, W8 represent weight parameters; b7, b8 represent bias parameters; Represents the probability distribution of sentiment prediction obtained in the previous fusion step.

Images