A wide temporal emotion recognition method for audio and text based on transfer learning

Transfer learning with VGGish and BERTbase enhances emotion recognition in audio and text modalities by leveraging pre-trained models to address dataset size limitations, achieving improved accuracy through combined feature extraction and classification.

CN117633587BActive Publication Date: 2024-12-06BEIJING INST OF TECH ZHUHAI CAMPUS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311573437.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2023-11-20
Filing Date
2023-11-23
Publication Date
2024-12-06
Estimated Expiration
2043-11-23

AI Technical Summary

Technical Problem

Existing datasets for emotion recognition in audio and text modalities are small in size, making it challenging to effectively train deep learning models, and there is a lack of comparison of methods in natural environments.

Method used

Employ transfer learning using pre-trained models VGGish for audio and BERTbase for text to extract emotional features, combining factual and situational descriptions to enhance emotion recognition in a wide time range.

Benefits of technology

Improves emotion recognition accuracy by leveraging pre-trained models to handle small datasets, providing more accurate emotional state inference through combined audio and text features.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117633587B_ABST
    Figure CN117633587B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for emotion recognition in a wide time range of audio and text based on transfer learning. The method extracts emotion features of the two modalities of audio and text by using pre-trained models VGGish and BERTbase, and analyzes the emotion of a target person from different angles. The method extracts emotion features by using factual description and textual description, and makes a more accurate inference on the current emotional state of the target person within a wide time range. Experiments on the audio and text description data of the SVCEmotion dataset show that both the VGGish and BERTbase pre-trained models can achieve good results on the dataset used in this article, and the parameters learned by the models during the pre-training process can effectively improve their performance on the target tasks. Comparative experiments show that the situational description introduced in the SVCEmotion dataset for emotion recognition tasks in a wide time range can provide clues for emotion recognition, and can greatly improve the emotion recognition effect when combined with the factual description.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the technical field of facial capture, and in particular to a method for emotion recognition in a wide time range of audio and text based on transfer learning. Background Art

[0002] In the process of building a multimodal emotion recognition system, the model's ability to represent the emotional features of each single modality is the basis for the realization of emotion recognition. Both audio and text can provide effective information for emotion recognition, so it is necessary to study emotion recognition methods for the audio and text modalities in the dataset. Deep learning models need to be trained on a large number of data samples to achieve a more ideal feature representation capability. However, due to the high time cost of collecting and annotating the SVCEmotion dataset, the current dataset is small in size and is not enough as training data to build an effective deep learning model. Transfer learning can use the knowledge learned by the pre-trained model in the source domain to assist it in achieving better recognition results on the tasks in the target domain.

[0003] The text content studied in this application is different from the text that records the content of character conversations in conventional data sets. It presents the events that occurred in the video and the characters' past experiences in the form of text, which is divided into two parts: fact description and situation description. The situation description is a situational context description of the character's past experiences related to the current state, so that the model can mine feature representations associated with emotions from text information within a wide time range. Deep learning models need to be trained on a large number of data samples to achieve a more ideal feature representation capability. However, due to the high time cost of collecting and annotating the SVCEmotion data set, the current data set is small in size and is not enough as training data to build an effective deep learning model. Transfer learning can use the knowledge learned by the pre-trained model in the source domain to assist it in achieving better recognition effects on tasks in the target domain.

[0004] At present, speech emotion recognition (SER) has become an indispensable function in the process of human-computer interaction. The speech emotion recognition system extracts and classifies speech signals from preprocessed speech signals. In the research on speech emotion recognition based on deep learning, the combination of convolutional neural network (CNN) and long short-term memory (LSTM) can achieve better results in speech emotion recognition tasks. LSTM has advantages due to its inherent characteristics in learning contextual information of features, while CNN is used because it overcomes the scalability problem of conventional neural networks. The model proposed in reference

[59] uses parallel CNN convolutional layers to control different temporal resolutions in feature extraction blocks. After connecting the features of each part, the convolution kernel maximum pooling operation is used to obtain better feature representation, which is then sent to LSTM to learn the temporal feature relationship. Finally, the classification representation is obtained through the fully connected layer. This model captures short-term and long-term interactions, thereby improving the performance of the speech emotion recognition system. Reference

[60] proposes a basic sequence segment selection based on radial basis function network (RBFN). The selected sequence is converted into a spectrogram using short-time Fourier transform and passed to the CNN model to extract features. The features of the CNN are then normalized and fed into a deep bidirectional long short-term memory (BiLSTM) network to learn temporal features and thus recognize emotions. The above methods were all experimented on the IEMOCAP dataset, which was collected from artificial performances under laboratory conditions. There is no comparison of the results based on datasets in natural environments. Aftab et al.

[61] proposed an efficient and lightweight fully convolutional neural network that extracts various feature maps through three parallel paths with convolution kernels of different sizes, which helps the deep convolution block extract high-level features while ensuring sufficient separability. The obtained features are concatenated and passed through multiple modules consisting of convolutional layers, batch normalization, ReLu layers, and average pooling layers. The extracted features are used to classify the emotions of the input speech segment. This method was experimented on the IEMOCAP and EMO-DB datasets and achieved the best recognition results at the time.

[0005] Text classification has always been a major research direction in the field of natural language processing. Madasu et al.

[62] proposed a sequential convolutional attention recursive network (SCARN). Since conventional convolutional networks lose some position information when processing text information and are unable to capture continuous information, it can robustly create transformations that can handle any task-specific noise. Therefore, the paper proposed using a recursive network structure in the model. The overall network architecture contains two modules: a convolutional recursive subnetwork and a recursive attention subnetwork. The first module extracts high-level features through a convolutional network, and the output of the convolution is trained sequentially with an LSTM network. In the second module, the input vector is trained using LSTM. In order to better focus on the most relevant words, the attention mechanism is applied to the output of the LSTM at each time step. The outputs of the two modules are cascaded and fed into the classifier to obtain the classification results. Compared with CNN and LSTM of the same scale, this method can achieve better recognition results in sentiment analysis and classification problems. Lai et al. proposed a syntax-based graph convolutional network (GCN) model to enhance the understanding of different grammatical structures of Chinese Weibo content. The model mainly consists of three parts: First, a Bi-LSTM network is used to extract preliminary lexical features of a given text. Then, the preliminary word features and the dependency parse tree built for each microblog content are fed into a single-layer graph convolutional network (GCN) to exploit the sentiment characteristics of the microblogs. Finally, a pooling or fully connected layer is used to obtain the probability distribution. In addition, a percentile-based ensemble method is proposed to improve the accuracy of the model. Experimental results show that the model can effectively utilize the information of dependency parsing to improve the performance of sentiment detection. Summary of the invention

[0006] In order to solve the above problems, this application provides a wide-time range emotion recognition method for audio and text based on transfer learning, uses the pre-trained models VGGish and BERTbase to extract emotional features of both audio and text modalities, analyzes the emotions of the target person from different angles, and improves the emotion recognition effect.

[0007] In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0008] A method for emotion recognition in a wide time range of audio and text based on transfer learning is provided, the method comprising the following steps:

[0009] Process the input signal of the VGGish pre-trained model; call the FFmpeg tool to extract the audio from the video, and then resample it to 16kHz mono audio; use a Hann window with a window length of 25ms to intercept the audio clip, implement short-time Fourier transform on the audio clip with a frame shift of 10ms to obtain a spectrogram, use a Mel-scale filter bank, calculate log(mel-spectrum+0.01), and convert the spectrum into a Mel spectrogram. Each frame has a duration of 10ms and contains 64 Mel frequency bands. The Mel spectrograms of each frame are combined without overlapping, and each group is 0.96s long, and each group of Mel spectrograms with an input size of 96×64 is obtained;

[0010] Establish a VGGish pre-training model; the VGGish pre-training model is pre-trained on a large-scale audio event dataset AudioSet dataset, wherein the AudioSet dataset consists of more than 2 million YouTube audio clips with a duration of 10 seconds, and the AudioSet dataset annotates the ontology of 623 audio event classes;

[0011] The converted Mel spectrogram is used as the input of the VGGish pre-training model. Each set of input is extracted to obtain a 128-dimensional feature representation. During the training process, a set of feature representations of the dimension of the audio sample is randomly selected each time and sent to two fully connected layers to obtain a 28-dimensional classification representation H. v ;

[0012] During the verification process, the mean of all feature representations of each sample is taken as the feature representation at the video level and sent to the classifier to obtain the recognition result.

[0013] Furthermore, the BERTbase model was used as the feature extraction network for text sentiment classification. The text information contained two parts: factual description and situational description of each video. In order to verify the auxiliary role of situational information in a wide time range for emotion recognition, the two text descriptions of each sample were concatenated as the third text description.

[0014] Furthermore, the BERTbase model network has a total of 12 encoding layers, the hidden layer size is 768, the number of multi-head self-attention modules in each layer is 12, and the maximum sequence length is set to 128.

[0015] Furthermore, by using the query and key corresponding to the input vector to calculate the dot product, we get the correlation between each word vector and other word vectors. After Softmax normalization, we get the attention distribution corresponding to the word vector at each position. We multiply the attention value of the corresponding position by the value and then sum them up. The vector representation of each word vector incorporates the semantic information of other positions in the input sequence. The calculation process of self-attention is as follows:

[0016]

[0017] where d k The purpose is to make the elements of the obtained attention map conform to the standard Gaussian distribution with a mean of 0 and a variance of 1.

[0018] Furthermore, for each input sequence, the output dimension after feature extraction is 128×768, and the output features corresponding to the classification label [CLS] are mapped to the output space through the fully connected layer to obtain the classification result of each text.

[0019] Furthermore, feature extraction and sentiment classification are performed on the fact description, situation description and the combination of the two, and the classification results are expressed as P t , P c and P tc .

[0020] Furthermore, the BCEWithLogitsLoss loss function is selected, and the expression of the BCEWithLogitsLoss loss function for multi-label classification with a total number of sentiment categories C is as follows:

[0021]

[0022] l n,c =-w n,c [p c y n,c ·logσ(x n,c )+(1-y n,c )·log(1-σ(x n,c ))]

[0023] where x n,c represents the predicted value of the nth sample of the cth sentiment category, y n,c represents the true label of the nth sample of the cth emotion category, N represents the number of Batch sizes, and σ represents the Sigmoid activation function. n,c It is the weight corresponding to the loss of different categories in each custom batch. It needs to ensure that its dimension is N×C, p c is the weight corresponding to the positive sample in each category, which is controlled by the ratio of negative samples to positive samples. n,c and p c Both are optional parameters, used to alleviate the imbalance of training sample categories and the imbalance of positive and negative sample distribution within categories.

[0024] In the technical solution provided in the embodiment of the present application, the emotion recognition of audio and text modalities based on the idea of ​​transfer learning is studied, and the emotion features of the two modalities of audio and text are extracted using the pre-trained models VGGish and BERTbase, and the emotions of the target person are analyzed from different angles; and in the text modality, more abundant information is provided for multimodal emotion recognition in the form of text, and the emotion features are extracted through factual description and text description, so as to make more accurate inferences on the current emotional state of the target person in a wide time range. Experiments on the audio and text description data of the SVCEmotion dataset have proved that both the VGGish and BERTbase pre-trained models can achieve good results on the dataset used in this paper, and the parameters learned by the model during the pre-training process can effectively improve its performance on the target task. By comparing VGGish with other audio emotion recognition methods, it is proved that VGGish is more suitable for extracting audio emotion features from the dataset constructed in this paper. The comparative experiment of text description has proved that the situational description introduced in the SVCEmotion dataset for the emotion recognition task under a wide time range can provide clues for emotion recognition, and the combination with the factual description can greatly improve the emotion recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] The methods, systems and / or programs in the accompanying drawings will be further described according to exemplary embodiments. These exemplary embodiments will be described in detail with reference to the drawings. These exemplary embodiments are non-limiting exemplary embodiments, wherein example numbers represent similar mechanisms in the various views of the accompanying drawings.

[0027] Figure 1 This is a schematic diagram of transfer learning provided by an embodiment of the present application;

[0028] Figure 2 It is a schematic diagram of the VGGish network structure provided in an embodiment of the present application;

[0029] Figure 3 It is a schematic diagram of the BERT structure provided in an embodiment of the present application;

[0030] Figure 4 is a schematic diagram of the Transformer encoder structure provided in an embodiment of the present application;

[0031] Figure 5It is a distribution diagram of emotion categories of the training set and the validation set provided in the embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to better understand the above technical scheme, the technical scheme of the present application is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical scheme of the present application, rather than limitations on the technical scheme of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0033] In the following detailed description, numerous specific details are set forth by way of example in order to provide a comprehensive understanding of the relevant guidance. However, it will be apparent to those skilled in the art that the present application may be practiced without these details. In other cases, well-known methods, procedures, systems, compositions and / or circuits have been described at a relatively high level, without detail, in order to avoid unnecessarily obscuring aspects of the present application.

[0034] Flowcharts are used in the present application to illustrate the execution process performed by the system according to the embodiment of the present application. It should be clearly understood that the execution process of the flowchart may not be performed in order. On the contrary, these execution processes may be performed in reverse order or simultaneously. In addition, at least one other execution process may be added to the flowchart. One or more execution processes may be deleted from the flowchart.

[0035] Before further describing the embodiments of the present invention in detail, the nouns and terms involved in the embodiments of the present invention are described. The nouns and terms involved in the embodiments of the present invention are subject to the following interpretations.

[0036] (1) In response, it is used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0037] (2) Based on is used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0038] Transfer learning is a method in machine learning that optimizes the model's learning of knowledge in new tasks by transferring the knowledge that has already been learned to a different but related new task. The existing knowledge is called the source domain, and the new knowledge to be learned is the target domain. This method does not require a large number of data samples, nor does it require the deep learning model to be trained from scratch, which reduces the time cost. For target tasks with relatively scarce data sets, applying transfer learning can make the model have higher performance and help improve the recognition effect of the target task.

[0039] Transfer learning is a method in machine learning that optimizes the model's learning of knowledge in new tasks by transferring the knowledge that has already been learned to a different but related new task. The existing knowledge is called the source domain, and the new knowledge to be learned is the target domain. This method does not require a large number of data samples, nor does it require the deep learning model to be trained from scratch, which reduces the time cost. For target tasks with relatively scarce data sets, applying transfer learning can make the model have higher performance and help improve the recognition effect of the target task. The schematic diagram of the transfer learning principle is shown below. Figure 1 shown.

[0040] The methods of transfer learning include sample transfer, feature transfer, model (parameter) transfer, relationship transfer, etc. The model transfer method is used in this application. By using the model and its parameters pre-trained on a large-scale data set, the feature extraction task is performed on the data set used in this article. This method can reduce the scope of parameter search in the training process of the model for the new task, train with higher initial performance, and help the model improve the recognition accuracy in the scene with smaller samples. Compared with the network trained from scratch, the convergence performance of the pre-trained model is better. In deep neural networks, there are two forms of implementation of transfer learning, namely feature extraction and fine-tuning. The feature extraction form directly uses the pre-trained network as the feature extractor of the target task, freezes all network parameters before the fully connected layer of the model, modifies or replaces the fully connected layer with a classifier suitable for the target task, and randomly initializes its parameters, and only trains the new parameters brought by the classifier. Fine-tuning is to freeze the parameters of a part of the network layer and retrain the parameters of the unfrozen network layer. The shallow features learned by the first few layers of the neural network are general features. As the network deepens, the receptive field gradually increases, and the learned features tend to be high-level features or specific features with rich semantic information for the task. However, it is impossible to determine which layers have learned common features or advanced features. Yosinski et al. explored the transferability of deep neural networks through a series of experiments and quantified the universality and specificity of neurons in each layer of deep convolutional neural networks. The conclusion points out that the parameters of the first three layers of the neural network can be directly transferred, because the extracted features are basically common features, and their parameters can be applied to different tasks. In addition, the network can be initialized with pre-trained network parameters of almost any number of layers, and the network can be fine-tuned to make the effect better than the original model. Because fine-tuning can make part of the network more suitable for the data set of the target task after retraining, the generalization performance of the model is improved. Therefore, when using the fine-tuning method, part of the shallow network is usually frozen, and the deep network is retrained using the data set of the target task to obtain network parameters that are more suitable for the target task and improve the recognition effect of the model.

[0041] This application is based on the idea of ​​transfer learning, using a pre-trained model to extract emotional features of audio and text information and implement emotional classification. Audio emotion recognition uses VGGish as a feature extraction network, and text information contains two parts: fact description and context description of each video. In order to verify the auxiliary effect of context information in a wide time range on emotion recognition, the two text descriptions of each sample are spliced ​​as the third text description. All three text information use BERTbase as the feature extraction network for text emotion classification.

[0042] Audio modal feature extraction and emotion recognition

[0043] In the dataset built based on the real environment, the composition of the audio signal is very complex. The audio data collected under laboratory conditions usually only contains the dialogue sounds of the actors, but in the real environment, in addition to the verbal communication between people, it may also contain the sounds in the environment. The audio of the SVCEmotion dataset collected from film and television dramas often contains some sounds in the scene, and judging the scene category can also help sentiment analysis. Since more than one target person may appear in a video, and the target characters are in the same scene, the audio reflects the scene and atmosphere in the entire video, so the emotion recognition result of the entire audio is used as the global emotion recognition result of the sample. The feature extraction network for audio sentiment analysis will be introduced below.

[0044] Audio emotion recognition based on VGGish

[0045] The VGGis network is a pre-trained model, and the dataset used is AudioSet, a large-scale dataset of manually annotated audio events released by Google. This dataset consists of more than 2 million YouTube audio clips of 10 seconds in length, and a total of 623 audio event classes are annotated, covering most audio events that occur in real-world recordings. Its breadth and data scale are unprecedented, and it is a relatively comprehensive category set. The VGGish model has learned most of the various sound categories in real life by pre-training on this dataset. This section uses this model as the feature extraction network for audio signals in the SVCEmotion dataset. The specific structure of the network is as follows Figure 2 As shown, the module consists of four convolutional and maximum pooling layers and three fully connected layers.

[0046] The input of VGGish is the Mel spectrogram of audio, so the audio signal needs to be converted first. Call the FFmpeg tool to extract the audio from the video, and then resample it to 16kHz mono audio. Next, use a Hann window with a window length of 25ms to intercept the audio segment, and implement short-time Fourier transform on the audio segment with a frame shift of 10ms to capture the frequency domain characteristics of the speech signal. While converting it from the time domain signal to the frequency domain signal, it can ensure that time-related information is not lost. The final result is a spectrogram. By using the Mel scale filter bank, the spectrum is converted into a Mel spectrogram. Each frame has a duration of 10ms and contains 64 Mel frequency bands. The Mel spectrogram of each frame is combined without overlap, and each 0.96s duration is a group. Therefore, the size of each group of input Mel spectrograms is. The converted Mel spectrograms are used as the input of VGGish, and each group of inputs is extracted to obtain a 128-dimensional feature representation. Since the duration of each sample is greater than 0.96s, each sample can obtain multiple feature representations after feature extraction. During the training process, a set of feature representations of the dimensions of the audio sample is randomly selected each time and sent to two fully connected layers to obtain a 28-dimensional classification representation. During the verification process, the mean of all feature representations of each sample is taken as the feature representation at the video level and sent to the classifier to obtain the recognition result.

[0047] Text Sentiment Recognition Based on BERT

[0048] The BERT model uses two large corpora, BooksCorpus and English Wikipedia, to complete two unsupervised pre-training tasks to obtain strong text semantic representation capabilities. The first task is to train the language model using a masked method, randomly masking the tokens of the input sentence at a certain ratio, predicting the masked tokens, and obtaining a deep bidirectional language feature representation. The second task is to determine whether the two input texts are continuous, and train the bidirectional language model to learn the association between longer text sequences. The advantage of this model is that when processing different downstream tasks, the model architecture is unified, and only the output layer of the network needs to be added and fine-tuned for different tasks.

[0049] The BERT model has two model sizes. This article uses the BERTbase model. The network has 12 layers of encoders, the hidden layer size is 768, and the number of multi-head self-attention modules in each layer is 12. The overall scale is larger than the Transformer model. The specific structure is as follows: Figure 3As shown in Figure 1. The input of the model consists of three parts. First, the input sentence is tokenized and each word is converted into a corresponding 768-dimensional word embedding ID, that is, a token for each word. Then, two special tags [CLS] and [SEP] are inserted at the beginning and end of each input sequence, respectively. [CLS] is a tag for classification tasks, and the output state corresponding to its position integrates the semantic information of the entire sentence. The function of the [SEP] tag is to distinguish the input sentence pairs. Since the self-attention mechanism of the Transformer ignores the position of the word in the sentence, position encoding (position embedding) is performed for each word. The dimension of the encoding is the same as the word vector. Finally, segment embedding is added to each token to distinguish the two sentences in the sentence pair. If only one sentence is input, the segment embedding values ​​of all tokens are the same. The three parts of the embedding of each token are added by position to obtain the input vector of BERT. In order to unify the length of the input sequence, the remaining token positions of shorter sentences will be filled with zeros. This paper sets the maximum length of the sequence to 128.

[0050] Text modal feature extraction and sentiment recognition

[0051] In the text annotation content, the dataset constructed in this paper not only describes the events happening in the video, but also describes the events that the target subject experienced before the video content, that is, it introduces situational context information. Therefore, sentiment recognition in text modality contains two types of data, namely, factual description and situational description. By combining the objective content description of the video with the situational context information, we can obtain more abundant sentiment analysis clues with a wider time range, which is more in line with the human thinking logic. First, we learn the sentiment feature representation from these two text descriptions respectively, and then directly splice the contents of the two texts so that the model can extract features from the spliced ​​text, and study the effectiveness of situational description for sentiment analysis.

[0052] Since computers cannot directly recognize text, in order to implement text sentiment analysis, it is necessary to convert text into word vectors that machines can recognize. The data set in this article is annotated in Chinese, so it is necessary to use translation software to translate all the annotations into English before it can be used. The introduction of Transformer has made the self-attention mechanism widely used in the field of natural language processing, changing the previous application of attention mechanism in time series models. The BERT pre-trained model is a bidirectional encoder representation model based on the Transformer Encoder structure. It can combine the left and right contexts in each layer and obtain the correlation between words within the sentence, which can more accurately represent the semantics of the sentence. This section uses the BERT pre-trained model to extract features and classify sentiment of text data.

[0053] Text Sentiment Recognition Based on BERT

[0054] The BERT model uses two large corpora, BooksCorpus and English Wikipedia, to complete two unsupervised pre-training tasks to obtain strong text semantic representation capabilities. The first task is to train the language model using a masked method, randomly masking the tokens of the input sentence at a certain ratio, predicting the masked tokens, and obtaining a deep bidirectional language feature representation. The second task is to determine whether the two input texts are continuous, and train the bidirectional language model to learn the association between longer text sequences. The advantage of this model is that when processing different downstream tasks, the model architecture is unified, and only the output layer of the network needs to be added and fine-tuned for different tasks.

[0055] The BERT model has two model sizes. This article uses the BERTbase model. The network has 12 layers of encoders, the hidden layer size is 768, and the number of multi-head self-attention modules in each layer is 12. The overall scale is larger than the Transformer model. The specific structure is as follows: Figure 3As shown in Figure 1. The input of the model consists of three parts. First, the input sentence is tokenized and each word is converted into a corresponding 768-dimensional word embedding ID, that is, a token for each word. Then, two special tags [CLS] and [SEP] are inserted at the beginning and end of each input sequence, respectively. [CLS] is a tag for classification tasks, and the output state corresponding to its position integrates the semantic information of the entire sentence. The function of the [SEP] tag is to distinguish the input sentence pairs. Since the self-attention mechanism of the Transformer ignores the position of the word in the sentence, position encoding (position embedding) is performed for each word. The dimension of the encoding is the same as the word vector. Finally, segment embedding is added to each token to distinguish the two sentences in the sentence pair. If only one sentence is input, the segment embedding values ​​of all tokens are the same. The three parts of the embedding of each token are added by position to obtain the input vector of BERT. In order to unify the length of the input sequence, the remaining token positions of shorter sentences will be filled with zeros. This paper sets the maximum length of the sequence to 128.

[0056] The core idea of ​​this model is to use the Transformer's self-attention mechanism to achieve parallel processing of the input sequence, while obtaining the correlation of each word in the sequence, capturing the feature dependencies with longer distances within the sequence, and obtaining stronger semantic expression capabilities. The self-attention mechanism is also called internal attention, which is an attention mechanism for the internal elements of the input or output elements.

[0057] The query, key, and value in the self-attention mechanism are all obtained by linearly transforming the input word vector matrix by multiplying it with different weight matrices. By calculating the dot product using the query and key corresponding to the input vector, the correlation between each word vector and other word vectors is obtained. After Softmax normalization, the attention distribution corresponding to the word vector at each position is obtained. The attention value at the corresponding position is multiplied by the value and then summed. The vector representation obtained by each word vector integrates the semantic information of other positions in the input sequence. The calculation process of self-attention is as follows:

[0058]

[0059] where d kThe purpose is to make the elements of the attention map conform to the standard Gaussian distribution with a mean of 0 and a variance of 1, so as to prevent the distribution of the attention map after the softmax function from being too steep. The model uses multi-head self-attention to calculate the self-attention value of the word vector in parallel. Since there is more than one correlation between the vectors of the input sequence, using multiple query vectors can learn different interdependencies in different subspaces.

[0060] For each input sequence, the output dimension after feature extraction is 128×768. The output features corresponding to the classification tag [CLS] are mapped to the output space through the fully connected layer to obtain the classification result of each text. This section extracts features and classifies sentiment for fact description, situation description, and the combination of the two. The classification results are expressed as P t , P c and P tc .

[0061] Loss Function

[0062] The emotion recognition of audio mode, text mode and visual mode in this application are all multi-label classification problems. The loss function is BCEWithLogitsLoss, which combines the Sigmoid activation function and the BCELoss operation. There is no need to add the Sigmoid function operation in the output layer of the model. The log-sum-exp technique is used to make it more stable in terms of numerical performance. The expression of this loss function for multi-label classification with a total number of emotion categories C is as follows:

[0063]

[0064] l n,c =-w n,c [p c y n,c ·logσ(x n,c )+(1-y n,c )·log(1-σ(x n,c ))] (3)

[0066] where x n,c represents the predicted value of the nth sample of the cth sentiment category, y n,c represents the true label of the nth sample of the cth emotion category, N represents the number of Batch sizes, and σ represents the Sigmoid activation function. n,c It is the weight corresponding to the loss of different categories in each custom batch. It needs to ensure that its dimension is N×C, p cis the weight corresponding to the positive sample in each category, which is controlled by the ratio of negative samples to positive samples. n,c and p c Both are optional parameters, used to alleviate the problems of imbalanced training sample categories and imbalanced distribution of positive and negative samples within categories.

[0067] Experimental results and analysis

[0068] This section first introduces the hardware configuration and experimental environment used in the experiments in this paper, then introduces the parameter settings and experimental details of audio modality and text modality emotion recognition, and finally analyzes the recognition effect of the model through the experimental results.

[0069] Table 1 Hardware configuration and experimental environment

[0070]

[0071] Experimental parameters

[0072] All experiments in this application use the SVCEmotion dataset. In this chapter, audio and text data are used. Considering the small size of the dataset, the dataset is divided into a training set and a validation set. The dataset contains clips from 16 films and TV series, 11 of which are used as training sets, and the rest are divided into validation sets. The training set contains 1,218 videos with a total of 1,589 annotations, and the validation set contains 333 videos with a total of 415 annotations. Because the distribution of emotion categories in film and television works of different themes varies, the distribution of emotion categories in the training set and the validation set is kept consistent to the greatest extent possible in the process of dividing the dataset. The distribution of specific emotion categories is as follows: Figure 5 shown.

[0073] The audio modality in this application uses the pre-trained parameters of VGGish as the initial parameters for training, the batch size is set to 6, a total of 300 epochs are trained, and the initial learning rate is set to 0.0001. The optimizer uses the adaptive moment estimation optimizer (Adaptive moment estimation, Adam), and the hyperparameter momentum is set to 0.9. The advantages of Adam are easy to implement, efficient calculation, and the ability to dynamically adjust the learning rate. The weight decay rate is set to 1e-5, and the learning rate is reduced by one tenth when the training reaches the 150th epoch, and the learning rate is reduced by another one tenth when the training reaches the 230th epoch. The factual description, situational description, and combination of the two in the text modality all use the parameters of the BERTbase pre-trained model as the initial parameters for training, the batch size is set to 128, 100 epochs are trained, the initial learning rate is set to 2e-5, and the optimizer also uses Adam, with a weight decay rate of 0.01. Table 2 shows the parameter settings of the two modalities.

[0074] Table 2 Parameter settings

[0075]

[0076] Audio modal test results and analysis

[0077] In view of the problem that the dataset constructed in this paper is small in scale and cannot effectively train deep neural networks, this chapter uses the pre-trained model VGGish to extract audio features and recognize emotions. VGGish has been pre-trained on a large-scale audio event dataset AudioSet, fully learning most of the audio events that will appear in the real environment, and can achieve better feature extraction of audio data, starting training on the dataset of this paper with a better starting point. During the training process, two initial learning rates were used respectively. Table 3 shows the comparison of the recognition effects of the two learning rates.

[0078] Table 3. Audio modality emotion recognition results

[0079]

[0080]

[0081]

[0082] From the results, we can see that the recognition results in the two learning rate experiments are not much different. When the initial learning rate is 0.0001, the effect is slightly higher by about 0.5. The evaluation indicators of most emotional categories are higher than the results of the learning rate of 0.001. The overall effect of audio emotion recognition is effective. This is because the VGGish model used has been trained on the large-scale audio event dataset AudioSet, and the 623 audio event categories in the AudioSet dataset can basically cover the scenes that appear in the dataset used in this chapter. Therefore, after fine-tuning with the SVCEmotion dataset, VGGish can distinguish different audio events. The audio of the dataset in this article contains obvious emotional colors, which enables the model to effectively extract emotional features from the audio. Due to the imbalanced distribution of categories in the dataset used, the recognition effect of emotional categories with a small number of samples is relatively poor, such as "tired", "respectful", and "sensitive". If a sufficient number of samples can be guaranteed, the recognition effect will be improved.

[0083] In addition, the VGGish method used in this application was compared with other speech emotion recognition methods under the same experimental conditions. Table 4 shows the experimental results.

[0084] Table 4 Experimental comparison of VGGish and other methods on the dataset in this paper

[0085]

[0086] In the table, 3D-ACRNN is a three-dimensional convolutional recurrent neural network based on attention. It uses log-Mels with deltas and delta-delta as the input of ACRNN. ​​ACRNN is a combination of CRNN and attention mechanism. The network is used to extract features of the three inputs and finally sent to the classifier to obtain the classification results. It can be observed from the table that VGGish performs slightly better than the other two speech emotion recognition methods, and the overall recognition effect is not much different. Analysis may be due to the fact that VGGish is a pre-training task completed on a large audio dataset, so the recognition effect is slightly higher. It can be shown that in the dataset constructed in this article, the pre-trained model can effectively improve the performance of the model on this dataset. The model pre-trained on a large-scale dataset is more suitable for audio emotion feature extraction and recognition of small samples in this article.

[0087] Text modality experimental results and analysis

[0088] This chapter uses two parts of text modal data, fact description and situation description, from the SVCEmotion dataset, and uses the BERTbase pre-trained model to extract and identify the emotional features of the two text descriptions. In addition, the two text descriptions of each sample are simply concatenated, and the same pre-trained model is used for emotion recognition, and the results of these three emotion recognitions are compared. BERTbase obtains powerful semantic representation capabilities by performing two pre-training tasks on a large corpus. The pre-training task of using random masking input tags enables the model to have deep bidirectional representation capabilities and can obtain richer semantic information by combining left and right contexts. The pre-training task of judging whether the input text pairs are related enables the model to have the ability to mine the internal correlation of longer text sequences. Therefore, the model can effectively learn the association between text descriptions and emotions and realize emotion classification. The parameters and details of the three experiments in the text modality were conducted under the same conditions. Table 5 shows the specific results of the experiments.

[0089] Table 5 Text modality emotion recognition results

[0090]

[0091]

[0092]

[0093] First, we compare factual description and situational description. The two text descriptions are feature extracted by BERTbase, and both achieve sentiment classification. The overall sentiment recognition effect of situational context description is better than that of factual description, with an evaluation index of 7.77 higher. This is because the situational description describes the experience of the target person, and the current possible emotional state can be inferred based on the events that have been experienced. For example, a girl is ridiculed by her boyfriend's family. When she talks to her boyfriend, it can be inferred that her mood may be "sad", "confused" or "tortured". The factual description is just an objective description of the event that the target person is currently in or doing. The text is basically free of emotional color, so it is not easy to be associated with the emotional state, such as "a girl is talking to a boy". In the emotional categories such as "bored", "calm", and "sensitive", the recognition effect of factual description is slightly higher than that of situational recognition. It may be that these emotional categories are mainly expressed through the behavior of the characters. When describing objectively, the events that the characters have done will be described. The situational description cannot highlight what the target person is currently doing, so the recognition effect is poor. For example, "a woman is sitting at a desk", the premise may be that she is relieved about something, but it is easier to infer "calm" from the factual description.

[0094] In order to explore the auxiliary effect of situational context information on general fact description, the two text descriptions were spliced ​​as input, namely "fact + situation description". The recognition effect was significantly improved, with an improvement of 12.64 compared with the fact description. The analysis is because the information of the two text descriptions is complementary, and the combination of the current event and past experience makes it easier for the model to infer the emotions of the target person. Among them, the emotional categories with low recognition rates such as "boredom", "embarrassment" and "confidence" have all been improved to varying degrees, and the RA index of "disagreement" has reached 90.55.

[0095] However, there are also some individual emotion categories where the effects have deteriorated. This may be due to the different contributions of factual descriptions and situational descriptions to emotion classification in different situations. After splicing, their own emotional expression intensity is weakened. Therefore, how to combine the two descriptions more efficiently is an issue that requires in-depth research in the future.

[0096] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0097] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0098] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0099] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0100] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0101] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0102] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0103] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0104] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0105] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for wide temporal range emotion recognition of audio and text based on transfer learning, the method The following steps are involved: Input signal processing; call FFmpeg tool to extract audio from the video, and then resample it to 16kHz mono audio; use Hann window with a window length of 25ms to intercept the audio clip, perform short-time Fourier transform on the audio clip with a frame shift of 10ms to obtain the spectrum, use Mel-scale filter bank to calculate , convert the spectrum into Mel spectrogram, each frame is 10ms long and contains 64 Mel frequency bands, and the Mel spectrograms of each frame are combined without overlap, with each group of 0.96s long, and the input size of each group is Mel spectrogram of Establish a VGGish pre-trained model; the VGGish pre-trained model audio is pre-trained on a large-scale audio event dataset AudioSet dataset, the AudioSet dataset consists of more than 2 million YouTube audio clips with a duration of 10 seconds, and the AudioSet dataset annotates the ontology of 623 audio event classes; The converted Mel spectrogram is used as the input of the VGGish pre-trained model. Each set of input is extracted to obtain a 128-dimensional feature representation. During the training process, a set of feature representations of the dimension of the audio sample is randomly selected each time and sent to two fully connected layers to obtain a 28-dimensional classification representation. ; During the verification process, the mean of all feature representations of each sample is taken as the feature representation at the video level and sent to the classifier to obtain the recognition result; The BERTbase model is used as the feature extraction network for text sentiment classification. The text information includes two parts: the fact description and the context description of each video. In order to verify the auxiliary effect of context information in a wide time range on sentiment recognition, the two text descriptions of each sample are spliced ​​as the third text description. The BERTbase model network has 12 encoding layers, the hidden layer size is 768, the number of multi-head self-attention modules in each layer is 12, and the maximum sequence length is set to 128; By using the query and key corresponding to the input vector to calculate the dot product, we can get the correlation between each word vector and other word vectors. After Softmax normalization, we can get the attention distribution corresponding to the word vector at each position. The attention value at the corresponding position is multiplied by the value and then summed. The vector representation of each word vector integrates the semantic information of other positions in the input sequence. The calculation process of self-attention is as follows: in The purpose is to make the elements of the obtained attention map conform to the standard Gaussian distribution with a mean of 0 and a variance of 1.

2. The method for wide-time-range emotion recognition of audio and text based on transfer learning according to claim 1, Features: For each input sequence, the output dimension after feature extraction is , the output features corresponding to the classification tag [CLS] are mapped to the output space through the fully connected layer to obtain the classification results of each text.

3. The method for audio and text wide time range emotion recognition based on transfer learning according to claim 2, Features: Feature extraction and sentiment classification are performed on fact description, situation description and the combination of the two, and the classification results are expressed as , and .

4. The method for audio and text wide time range emotion recognition based on transfer learning according to claim 3, Features: The BCEWithLogitsLoss loss function is selected. The BCEWithLogitsLoss loss function has the following value for the total number of emotion categories: The expression of multi-label classification is as follows: in Indicates The emotion category The predicted value of samples, Indicates The emotion category The true labels of samples, Indicates the number of Batch sizes. Represents the Sigmoid activation function; It is the weight corresponding to the loss of different categories in each custom batch. It is necessary to ensure that its dimension is , is the weight corresponding to the positive sample in each category, which is controlled by the ratio of negative samples to positive samples; and Both are optional parameters, used to alleviate the imbalance of training sample categories and the imbalance of positive and negative sample distribution within categories.