Emotional estimation device
The emotion estimation device uses a graph neural network to enhance the accuracy of multi-modality emotion estimation by updating features based on modality relationships, outperforming existing methods in integrating facial expressions, audio, and text data.
Patent Information
- Application Number
- JP2020182166
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-10-30
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2040-10-30
AI Technical Summary
Existing multi-modality emotion estimation techniques face challenges in accurately combining the outputs of single-modality models due to the lack of a clear relationship between modalities, leading to insufficient accuracy in emotion estimation.
An emotion estimation device utilizing a graph neural network, specifically a graph attention network (GAT), to update features of multiple modalities based on their relationships, emphasizing the attention between nodes to enhance the accuracy of emotion estimation by combining outputs from facial expressions, uttered audio, and text.
The proposed method significantly improves the accuracy of emotion estimation by effectively integrating features from different modalities, achieving higher performance compared to conventional techniques.
Smart Images

Figure 0007577308000014 
Figure 0007577308000015 
Figure 0007577308000016
Abstract
Description
[Technical field]
[0001] The present invention relates to emotion estimation technology, and more particularly to an emotion estimation device, an emotion estimation method, and a computer program for estimating human emotions using information of multiple types of modalities. [Background technology]
[0002] Recently, research on emotion estimation has been progressing. Emotion estimation is the estimation of a person's internal state based on the person's voice, facial expression, body movements, content of speech, etc. when they are communicating with other people. By estimating a person's emotion, it is possible to control the behavior of an agent that is conversing with a person, for example, or conversely, to convey to the other party what emotion the agent is feeling by controlling the agent's behavior.
[0003] As mentioned above, the channels through which people communicate with other people include voice, face (facial expression), body movements, and speech, each of which is called a modality. Technology that estimates a person's emotion from any one of multiple modalities, such as a person's voice, facial expression, body movements, and speech, is called single-modality emotion estimation technology. Technology that estimates a person's emotion from multiple modalities is called multi-modality emotion estimation technology.
[0004] Various methods have been researched as single-modality emotion estimation technology. For example, Non-Patent Documents 1 and 2, which are listed below, use facial expressions. Non-Patent Documents 3 and 4 use spoken voice. Non-Patent Documents 5 and 6 use body movements. Non-Patent Document 7 uses text, which is the content of the utterance.
[0005] In previous research, data for emotion estimation technology was obtained by, for example, asking actors to speak in a way that expresses a particular emotion. However, such intentionally expressed emotion speech, facial expressions, and actions are often more exaggerated and stronger than expressions seen in everyday communication. Therefore, using data obtained from such speech poses the problem of not being able to accurately capture the emotional expressions that accompany everyday communication. For this reason, more and more researchers have recently started to use images, audio, and other data obtained from everyday communication as basic data for emotion estimation, rather than data obtained from intentionally expressed emotion speech.
[0006] When using such data, there is a problem that the accuracy of emotion estimation from single-modality data as mentioned above is insufficient. In reality, human communication is multi-modal, and it is difficult to estimate emotions with sufficient accuracy using only one of the modalities. Therefore, there is a possibility that accuracy can be improved by estimating emotions using multi-modality.
[0007] Examples of emotion estimation techniques using multi-modality include those disclosed in Non-Patent Documents 8-13 listed below. These propose a multi-modality emotion estimation model that estimates emotions using features that are simply concatenated or averaged from features obtained from emotion estimation models using individual single modalities. [Prior art documents] [Non-patent literature]
[0008] [Non-Patent Document 1] Byeon, YH; Kwak, KC Facial expression recognition using 3d convolutional neural network. Int. J. Adv. Comput. Sci. Appl. 2014, 5.
Outdoor Tool2
Outdoor Tools3
Outdoor Tools 4
Direct Environment 5
Outdoor Configuration6
Direct Environment 7
Outdoor Track 8
Outdoor Tools9
[0009] Multi-modality emotion estimation technology has improved accuracy compared to single-modality. However, in the case of multi-modality, the problem is how to combine the individual outputs of single-modality models to improve accuracy. There should be some relationship between each modality, and if such a relationship can be used to estimate emotions, it may be possible to improve accuracy.
[0010] Therefore, an object of the present invention is to provide an emotion estimation device that can combine the outputs of single-modality models to further improve the accuracy of emotion estimation. [Means for solving the problem]
[0011] A feeling estimation device according to a first aspect of the present invention includes: an updating means for updating each of features of a plurality of types of modalities related to information display and information transmission by a person based on associations existing between features obtained for the plurality of types of modalities; and a feeling estimation means for inferring feelings of the person by using the features of the plurality of types of modalities updated by the updating means.
[0012] Preferably, the emotion estimation device is configured by a neural network.
[0013] More preferably, the updating means includes attention calculation means for calculating the associations existing between features obtained for multiple types of modalities as attention between nodes in a graph neural network in which the features are nodes and edges connect each node, and feature updating means for updating the feature of each node using attention from each of the adjacent nodes adjacent to the node from the attention calculated by the attention calculation means.
[0014] More preferably, the attention between adjacent nodes calculated by the feature updating means differs depending on the direction of attention.
[0015] Preferably, the update means receives, as a feature, a state of a final hidden layer of an individual neural network that calculates a feature vector representing an emotion from a plurality of types of modalities.
[0016] More preferably, the individual neural networks include bidirectional neural networks for time series of multiple types of modalities, and the update means receives as a feature a vector that is a concatenation of a vector representing a state of a final hidden layer in a forward direction and a vector representing a state of a final hidden layer in a backward direction of the bidirectional neural network.
[0017] More preferably, the attention calculation means includes a plurality of attention calculation heads, each of which is composed of a fully connected neural network and each of which calculates the associations existing between features of a plurality of types of modalities as attention between nodes in a graph network in which the features are nodes and which has edges connecting each node, and an average calculation means for each node, which calculates, from the attentions calculated by the plurality of attention calculation heads, the average of the attentions from each of the adjacent nodes adjacent to the node as the attention for the node.
[0018] Preferably, the update means updates the identity of the i-th node by Eh iLet the number of attention calculation heads be K and the number of modality types be N. Then, the feature Eh of the i-th node is calculated by the following formula. i Update.
[0019]
number
[0020] More preferably, the multiple types of modalities include any combination of at least two of a time series of facial expressions, spoken voice, and spoken text.
[0021] More preferably, the multiple types of modalities include a time series of facial expressions, spoken voice, and spoken text.
[0022] An emotion estimation method according to a second aspect of the present invention includes a step of updating each of features of a plurality of types of modalities based on associations existing between features obtained for the plurality of types of modalities related to information display and information transmission by a person, and a step of estimating an emotion of the person using the features of the plurality of types of modalities updated in the updating step.
[0023] A computer program according to a third aspect of the present invention causes a computer to function as an update means for updating each of the features of a plurality of types of modalities based on associations existing between features obtained for a plurality of types of modalities related to human information display and information transmission, and an emotion estimation means for estimating human emotions using the features of the plurality of types of modalities updated by the update means. Effect of the Invention
[0024] According to the present invention, it is possible to provide an emotion estimation device that can improve the accuracy of emotion estimation by combining the outputs of single-modality models. The above and other objects, features, aspects and advantages of the present invention will become apparent from the following detailed description of the present invention taken in conjunction with the accompanying drawings. [Brief description of the drawings]
[0025] [Figure 1] FIG. 1 is a schematic block diagram showing a schematic configuration of a multi-modality emotion estimation system according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing the configuration of the multi-head GAT shown in FIG. [Diagram 3] FIG. 3 is a block diagram showing a more detailed configuration of the update unit shown in FIG. [Figure 4] FIG. 4 is a flowchart showing the training process of the multi-modality emotion estimation system shown in FIG. [Diagram 5] FIG. 5 is a flowchart showing the control structure of a program for training the multi-head GAT shown in FIGS. [Figure 6] FIG. 6 is a flowchart showing a control structure of a program for implementing feature updating by the GAT. [Figure 7] FIG. 7 is a block diagram for explaining the operation of the multi-modality emotion estimation system shown in FIG. 1 during emotion estimation. [Figure 8] FIG. 8 is a diagram showing the appearance of a computer system for implementing each embodiment of the present invention. [Figure 9] FIG. 9 is a block diagram showing the hardware configuration of the computer system whose external appearance is shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0026] In the following description and drawings, the same parts are given the same reference numbers, and therefore detailed descriptions thereof will not be repeated.
[0027] <First embodiment> [composition] In this embodiment, emotion estimation is performed using three modalities: facial expression, speech, and text, which is the content of the speech. GAT is used to estimate emotion by fusing information from these three modalities. By using GAT, the characteristics of each modality are emphasized based on the relationship between these three modalities, improving the accuracy of emotion estimation.
[0028] Fig. 1 shows a schematic configuration diagram of a multi-modality emotion estimation system 50 according to the first embodiment. Referring to Fig. 1, the multi-modality emotion estimation system 50 includes an expression model 60 that receives a time series of expression data when a person to be an emotion estimation target speaks and outputs a time series of expression features 72, 74, etc., a text model 62 that outputs a text feature vector 82 from text when the same person speaks, and a voice model 64 that receives voice data when the same person speaks and extracts a time series of voice features 76, 78, etc.
[0029] The multi-modality emotion estimation system 50 further includes a multi-head GAT 66 consisting of a graph neural network, which is trained to receive an expression feature vector 80 concatenating the expression features 72 and 74, a phonetic feature vector 84 concatenating the phonetic features 76 and 78, and a text feature vector 82, and to perform processing for emotion estimation using a GAT, which will be described later, and a fully connected layer 68 which is trained to receive an output of the multi-head GAT 66 and output a final emotion estimation result 70.
[0030] The facial expression model 60, text model 62, and voice model 64 may be existing models that can output feature vectors. They may be models based on different concepts. They may also be trained separately. The models used in the experiments described below are described in the explanation of the experimental results.
[0031] In this embodiment, an individual neural network is used for each model, and the flattened hidden state of the final layer of the neural network is used as the feature vector.
[0032] Referring to FIG. 2, the multi-head GAT 66 includes update units 100 , 102 and update unit 104 , each of which receives feature vectors from the facial expression model 60 , the text model 62 and the speech model 64 .
[0033] The update units 100, 102, and 104 have the same configuration. However, the update unit 100 updates the output of the facial expression model 60 using the outputs of the text model 62 and the voice model 64. The update unit 102 updates the output of the text model 62 using the outputs of the facial expression model 60 and the voice model 64. The update unit 104 updates the output of the voice model 64 using the outputs of the facial expression model 60 and the text model 62.
[0034] This update is due to GAT. GAT has a different topology than commonly used neural networks. A commonly used neural network contains multiple layers, each of which contains multiple nodes. The number of nodes in each layer may be different or the same. Basically, the connections between the nodes are made between each node in a layer and each node in the adjacent layers above and below it. So the arrangement of the nodes basically forms a lattice.
[0035] However, the relationships between information in the real world are not limited to such lattice-type topologies. For example, networks such as social networks, transportation networks, and road networks are not lattice-type. However, in all cases, they have in common that the network is defined by a set of nodes and edges connecting two nodes. Such figures are generally called graphs. Graph theory is the field that deals with graphs mathematically.
[0036] A neural network that handles networks that form graphs in this way is called a graph neural network (GNN). In this embodiment, we consider a GNN that treats hidden features obtained from hidden layers from different modalities as nodes and relationships between each node as edges, and adopt a graph attention network (GAT) to emphasize the feature values of each modality in this GNN based on the relevance between nodes.
[0037] 2, the facial expression feature obtained from the facial expression model 60 is denoted as A, the text feature obtained from the text model 62 is denoted as B, and the speech feature obtained from the speech model 64 is denoted as C. Consider a GNN with nodes A, B, and C. The update units 100, 102, and 104 all update the features A, B, and C according to the relationship between the modalities in accordance with the equations described below.
[0038] 2 have the same configuration, except for which features they update. Therefore, for simplicity, only the configuration of update unit 100 will be described here.
[0039] 3, the update unit 100 includes multiple heads 120, 122, ..., 124, each of which receives features A, B, and C and updates the value of feature A using features B and C. These have exactly the same configuration. However, since these parameters are initialized by random numbers when the GNN is trained and then trained, they ultimately have different parameters, calculate attention between nodes using different calculations, and update feature A using that attention. The value output by head 120 is feature A' 1 , the value output by the head 122 is the feature A′ 2 Similarly, the value output by the head 124 is used as the feature A′ 8 The heads 120, 122, ..., 124 update the value of feature A according to the following formula. In the formula, K is the number of attention heads possessed by the update units 100, 102, and 104 (K=8 in this embodiment), W is the weight of the linear transformation that learns the parameters by training, and a∈R2nf represents the weight vector, and N represents the number of modalities (N=3 in this embodiment).
[0040]
number
[0041] The update unit 100 further calculates the updated feature A′ calculated by the heads 120, 122, . . . , 124. 1 , A′ 2 , …A′ 8 and outputting the average as feature A'.
[0042] 2 perform the same processing as described above on features B and C, respectively, and updated features A', B', and C' are output from update units 100, 102, and 104. These are concatenated to become the output of multi-head GAT 66, and input to fully connected layer 68. As can be seen from the fact that subscripts i and j are asymmetric in the above formula, attention between nodes differs depending on the direction.
[0043] Training of the multi-modality emotion estimation system 50 is performed as follows. FIG. 4 is a flowchart showing the flow of processing for training the multi-modality emotion estimation system 50. The processing according to this flowchart may be realized by one program, or by multiple programs and a program that executes the multiple programs in a predetermined order or in parallel. Alternatively, each step may be realized by a separate program, and the programs may be executed in order or in parallel as shown in FIG. 4.
[0044] 4, this program is executed after preparing training data for the facial expression model 60, the text model 62, and the voice model 64 shown in Fig. 1 and Fig. 2 in advance and storing them in the training data storage device 150. This program includes a step 160 of increasing (expanding) the number of training data by a predetermined method when the number of training data stored in the training data storage device 150 is small, steps 162, 164, and 166 of training the facial expression model 60, the text model 62, and the voice model 64 in parallel using the training data stored in the training data storage device 150 after step 160, and a step 168 of training the multi-head GAT 66 and the fully connected layer 68 using the facial expression model 60, the text model 62, and the voice model 64 in response to the accuracy of emotion estimation by the facial expression model 60, the text model 62, and the voice model 64 being equal to or higher than a predetermined threshold value after the training is sufficiently performed.
[0045] The training in steps 162, 164, and 166 may be performed using an appropriate existing method depending on each model.
[0046] 5, step 168 in Fig. 4 includes step 200 of executing step 202 described below for all training data, and step 204 of judging whether the parameters of the multi-head GAT 66 and the fully connected layer 68 trained in step 200 have converged, i.e., whether the sum of the absolute values of the differences between the parameters after the previous execution of step 202 and the parameters after the current execution has reached a predetermined threshold value, and if the judgment is negative, control is returned to step 200 to execute training again, and if the judgment is positive, training of the multi-head GAT 66 and the fully connected layer 68 is terminated. Prior to step 200, the parameters of the multi-head GAT 66 and the fully connected layer 68 are initialized, for example, by random numbers.
[0047] Step 202 includes step 220 of randomly sampling a mini-batch consisting of a predetermined number of samples from data that has not yet been used as training data in the current execution of step 202, step 222 of executing step 224 for all training data in the mini-batch sampled in step 220 and accumulating errors between the teacher data and the output of the multi-head GAT 66 that occur for each execution, and step 226 of updating the parameters of the GNN that constitutes the multi-head GAT 66 by the backpropagation method using the errors accumulated in the processing of step 222 after the completion of step 222, and terminating the processing of step 202 for the mini-batch being processed. Note that in step 220, the value of the variable that accumulates the error for the training data of the mini-batch is cleared to 0.
[0048] Step 224 includes step 240 of calculating facial expression features for the training data using the facial expression model 60, step 242 of calculating text features for the training data using the text model 62 in parallel with step 240, and step 244 of calculating phonetic features for the training data using the phonetic model 64 in parallel with steps 240 and 242.
[0049] Step 224 further includes step 246 of updating the features by GAT using the features calculated in steps 240, 242, and 244, respectively, step 248 of estimating an emotion inference result 70 as an output by inputting the features updated in step 246 to the fully connected layer 68, and step 250 of accumulating an error between the emotion inference result 70 estimated by the fully connected layer 68 in step 248 and the teacher data added to the training data, and terminating the execution of step 224. If there is still training data remaining in the mini-batch when the processing of step 250 is completed, step 224 is repeated.
[0050] Fig. 6 shows details of step 246 in Fig. 5. Referring to Fig. 6, the feature update process using the GAT executed in step 246 includes step 280 of updating features using the GAT for facial expression features, text features, and speech features to calculate eight updated features for each feature, step 282 of calculating average values for each of the eight facial expression features, eight text features, and eight speech features calculated in step 280, and step 284 of combining the averages of the eight updated facial expression feature vectors, the averages of the eight text feature vectors, and the averages of the eight speech feature vectors calculated in step 282 to end the process of step 246.
[0051] In step 280, the eight heads 120, 122, ..., 124 of the update unit 100 shown in FIG. 3 perform update processing on the facial expression features in parallel, and the eight updated facial expression features A' 1 , A′ 2 , …, A′ 8 Similarly to step 300, the eight text features B′ are updated by the update unit 102 shown in FIG. 1 , B′ 2 , …, B′ 8 Step 302 calculates the eight phonetic features C′ updated by the update unit 104 shown in FIG. 1 , C′ 2 , …, C′ 8 These processes can be performed in parallel with each other because the update units 100, 102 and 104 shown in FIG.
[0052] Step 282 is a process of dividing the eight facial expression features A′ calculated in step 300. 1 , A′ 2 , …, A′ 8 In step 310, an updated expression feature A′ is calculated, which is the average of the eight text features B′ calculated in step 302. 1 , B′ 2 , …, B′ 8 Step 312 calculates an updated text feature B′, which is the average of the eight phonetic features C′ calculated in step 304.1 , C′ 2 , …, C′ 8 and calculating 314 an updated phonetic feature C', which is the average of
[0053] [Operation] The operation of the multi-modality emotion estimation system 50 has two phases: the first is a training phase, and the second is a testing phase.
[0054] The training phase further includes an individual training phase in which the facial expression model 60, the text model 62, and the voice model 64 are trained individually, and an integrated training phase in which the multi-head GAT 66 is trained using the facial expression model 60, the text model 62, and the voice model 64 that have been trained.
[0055] [Individual training phase] As described above, the facial expression model 60, text model 62, and speech model 64 may be trained according to the model adopted, as shown in steps 162, 164, and 166 of Fig. 4. These may be trained in sequence or in parallel. After sufficient training has been completed, the multi-head GAT 66 can be trained.
[0056] [Integrated training phase] The joint training is performed after sufficient training has been completed for the facial expression model 60, the text model 62, and the speech model 64 (step 168 in FIG. 4).
[0057] 5, in joint training, a mini-batch consisting of a predetermined number of samples is randomly sampled from data that has not yet been used as training data (step 220). Step 224 is executed for all training data in the mini-batch sampled in step 220.
[0058] That is, facial features are calculated from face image data in the training data (step 240), text features are calculated from text data (step 242), and phonetic features are calculated from voice data (step 244). With reference to Fig. 6, in step 300 of step 280, the updating unit 100 shown in Fig. 2 updates facial feature A using text feature B and phonetic feature C, and outputs updated facial feature A'. More specifically, as shown in Fig. 3, facial feature A is updated using text feature B and phonetic feature C in each of eight heads 120, 122, ..., 124, and updated facial feature A' is output. 1 , A′ 2 , …, A′ 8 Similarly, in step 302 of step 280, the updating unit 102 shown in Fig. 2 updates the text feature B using the facial expression feature A and the speech feature C, and outputs the updated text image feature B'. In step 304 of step 280, the updating unit 104 shown in Fig. 2 updates the speech feature A using the facial expression feature A and the text feature B, and outputs the updated speech feature C'.
[0059] Then, the averaged facial expression feature A', text feature B', and speech feature C' are calculated in step 282 of Fig. 6. In step 284, these feature vectors are combined into one feature vector.
[0060] 5, the combined feature vector obtained in step 246 is input to fully connected layer 68 shown in Fig. 2, and emotion estimation result 70 is obtained as the output. In step 250 in Fig. 5, the difference between emotion estimation result 70 and the correct label of the training data is calculated as an error, and the error is accumulated to end the processing of step 224 for this training data.
[0061] As described above, the process of step 224 is executed for all training data in the mini-batch sampled in step 220, and then the process of step 222 is completed. After that, in step 226, the accumulated error is used to update the parameters of the multi-head GAT 66 and the fully connected layer 68 by the error backpropagation method.
[0062] The above process is repeated for all the training data, and when the process for all the training data has been completed, the difference between the updated parameter value and the parameter value before the update (the parameter value at the time when the previous process of step 200 has been completed) is calculated in step 204. If this difference is equal to or smaller than a predetermined threshold, training ends. If the difference is greater than the threshold, control returns to step 200, and the process of step 200 is executed again.
[0063] [experiment] An experiment was conducted under the following conditions for the multi-modality emotion estimation system 50 according to the first embodiment described above.
[0064] 〔audio〕 In the experiment, the maximum length of a user's utterance to be processed was set to 6.5 seconds. Any utterance longer than 6.5 seconds was truncated at 6.5 seconds, and any utterance shorter than 6.5 seconds was padded with zeros at the end of the audio. The sampling rate was 16000Hz. Each frame was subjected to a discrete cosine transform with a length of 8000 and a hop length of 400. The scale step of the mel spectrogram was set to 384.
[0065] To correct the imbalance in the number of training samples, we performed data augmentation based on the entropy weighting method. Let A∈Rm×n be the extracted spectrogram. n indicates the number of time steps and m indicates the mel scale. Each element in A was normalized by the following formula (1). x ij denotes the original element of the mel spectrogram, and p ij indicates the element after normalization.
[0066]
number
[0067] After that, the expanded samples were obtained using the following equations (2) and (3): “alpha” in equation (3) is a constant, which was set to 2 in the following experiments.
[0068]
number
[0069] [Facial expression] Facial features were extracted from each frame in the video of each utterance, and 10 frames were selected at average intervals based on the length of each utterance. In the experiment, the method disclosed in Non-Patent Document 10 was used to extract spatial-temporal features. This method uses the 3D-CNN described in Non-Patent Document 2.
[0070] 3D-CNN is a pre-trained facial expression recognition model that consists of two separate channels of the input stream, a spatial CNN network part that processes static frame-level cropped face images, and a temporal CNN network part that processes optical flow images. After extracting spatial-temporal features from these two parts, they are integrated by a unified network. In the following experiments, we used the output of the hidden layer of this unified network as the features.
[0071] 〔text〕 We used the WordPiece tokenizer (https: / / github.com / google / sentencepiece) to convert each utterance transcript into a vector and tokenize each sentence. This tokenizer is also used in BERT, a well-known language processing model. For more information on BERT, please see the following literature.
[0072] 《Reference 1》 Devlin, J.; Chang, MW; Lee, K.; Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv 2018, arXiv:1810.04805.
[0073] The emotion of each utterance was determined from the entire time series, but there are certain time steps in such sequences that are critical. To select these critical time steps by considering the relationship between each time step of the samples, we used an emotion encoder-decoder (EED), inspired by the following paper in the field of natural language translation:
[0074] 《Reference 2》 Bahdanau, D.; Cho, K.; Bengio, Y. Neural machine translation by jointly learning to align and translate. arXiv 2014, arXiv:1409.0473.
[0075] In the encoding stage, the input to the encoder is a matrix X = [x1, x2, …, x t ], where x t ∈R nf The following equations (4) and (5) express the hidden state h i The flow for calculating → W, → Wz, → Wr∈R nh×nf , → U, → Uz, → Ur∈R nh×nh is the weight matrix (see → " is written in the formula immediately above the letter immediately following this symbol in the above description.) Where n h and n f n indicates the number of hidden units and features from the previous layer, respectively. fNote that varies by modality. See Table 1 below for the number of features for each modality.
[0076] [Table 1] In the above description and in formulas (4) and (5), among the arrows written directly above each character, a forward arrow "→" indicates a forward time-series sequence processing, and a backward arrow "←" indicates a backward time-series sequence processing.
[0077]
number
[0078] n ef is the number (length) of hidden emotion features. To calculate the hidden emotion vector, we flatten the hidden layer obtained from the encoding stage and adopt a fully connected layer as the bottleneck.
[0079] In the experiments, we implemented different neural networks for each modality as uni-modality models (uni-models). Each uni-model was trained separately using the data for time steps t and t-1 for each speaker.
[0080] 〔audio〕 Mel spectrograms for time steps t-1 and t were extracted from the speech and input to a bidirectional LSTM to process the time series information. Next, a multi-head attention layer was used to perform parallel processing to emphasize the relationship between time steps. The hidden layer values after the emphasis processing were input to EED to select the information that contributes most to emotion judgment. Multi-task training was performed to improve performance. In addition to classifying emotion labels, this emotion model also simultaneously recognizes the gender of the speaker, taking into account the impact of pitch differences between females and males.
[0081] [Facial expression] After extracting the speaker's face from the video, the data was input to 3D-CNN to obtain a hidden vector containing spatial-temporal features. In addition to being able to extract appropriate features from each video frame, 3D-CNN can extract spatial-temporal features that consider the facial features in each frame (spatial part) and the optical flow between frames (temporal part) separately. In the facial expression analysis process, a VGG model was adopted to fine-tune the spatial part separately. The temporal part processes the entire video message in sequence. The input in this case is the frame image, as well as the dimensionality of the height and weights. In order to fuse the features of the spatial-temporal CNN after training, a fully connected neural network was adopted to learn discriminative features together. These features were processed by EED to select important information for emotion estimation.
[0082] 〔text〕 For text sentiment analysis, we adopted SeMemNN, described in the following paper, and trained it from scratch.
[0083] 《Reference 3》 Fu, C.; Liu, C.; Ishi, C.; Yoshikawa, Y.; Ishiguro, H. SeMemNN: A Semantic Matrix-Based Memory Neural Network for Text Classification. In Proceedings of the 2020 IEEE 14th International Conference on Semantic Computing (ICSC), San Diego, CA, USA, 3-5 February 2020; pp. 123-127.
[0084] In SeMemNN, two inputs work together to generate an address matrix and a semantic matrix. The address matrix is used to generate an address vector to retrieve corresponding information from the semantic matrix. According to SeMemNN, texts at time steps t-1 and t generate the address matrix, and the semantic matrix is generated separately from the text at time step t. The multi-head design of GAT collects the relationships between speech, facial expressions, and text modalities for various emotions. Since this operation is performed across multiple modalities and each modality is considered as a node, we call it multi-head cross-modality graph attention.
[0085] [Multi-model] Uni-models for speech, facial expression, and text modalities were trained separately in an end-to-end manner on the emotion recognition task. After these uni-models were fully trained, hidden layers were extracted from each modality, and the relationships among these three modalities were calculated by multi-head GAT using the following equation (6), which has already been explained.
[0086]
number
[0087] [Dataset used] In the experiment, we used the Interactive emotional dyadic motion capture database (IEMOCAP) described in the following reference. Its structure is shown in Table 2 below. Each training data is labeled with one of the following:
[0088] [Table 2]
[0089] 《Reference 4》 Busso, C.; Bulut, M.; Lee, CC; Kazemzadeh, A.; Mower, E.; Kim, S.; Chang, JN; Lee, S.; Narayanan, SS IEMOCAP: Interactive emotional dyadic motion capture database. Lang. Resour. Eval. 2008, 42, 335.
[0090] The dataset contains videos of two-way dialogues between 10 speakers. Each video contains a dialogue for a single pair of speakers, segmented by utterance, and labeled with emotion. In our experiments, we split the dataset into training and test sets in an approximately 80 / 20 ratio. The training set contains 5810 samples, and the test set contains 1623 samples.
[0091] In the experiments, we implemented the entire neural network using Keras (https: / / keras.io / ) and used adam (https: / / keras.io / api / optimizers / adam / ) as the optimizer. The learning rate was set to 0.0001. The number of BLSTM cells was 128, and the number of multi-head attention heads was 8.
[0092] For the voice model, the weights for each task were set to 1. Other hyperparameters are as shown in Table 1 above. The number of parameters trained for the voice, facial expression, and text models is 13.6 million, 13.3 million, and 24.8 million, respectively. The number of parameters trained for the final multi-model is 51.8 million.
[0093] 〔targets for comparison〕 In this experiment, results obtained by varying the settings of the above embodiment were compared with the following variations of the embodiment itself and with other techniques.
[0094] Audio sample augmentation. We compare training a model with audio sample augmentation with training without augmentation.
[0095] -Replace EED with LSTM. The EED used in the above embodiment is similar to LSTM. However, in EED, a training matrix U is added to LSTM to analyze time series information and learn linear transformations at each time step. Therefore, for comparison, we created a model in which EED was replaced with a bidirectional LSTM.
[0096] Comparison of SeMemNN and BERT. BERT is known for its outstanding performance in the field of text classification. However, there are few examples of applying BERT to text emotion recognition. Therefore, we compared the performance when replacing SeMemNN with the pre-trained BERT.
[0097] DialogueRNN. This is a recurrent neural network described in Non-Patent Document 10, which uses two GRUs to track speaker states taking into account the global context of the conversation, and another GRU to track the flow of emotional states.
[0098] DialogueGCN. This is described in the following paper, and uses a graph network (GCN) to enhance the dependency of interlocutors on themselves and their counterparts with contextual information.
[0099] 《Reference 5》 Ghosal, D.; Majumder, N.; Poria, S.; Chhaya, N.; Gelbukh, A. Dialoguegcn: A graph convolutional neural network for emotion recognition in conversation. arXiv 2019, arXiv:1908.11540.
[0100] [Experimental Results] ·Voice emotion recognition First, the results for the speech uni-model are shown in Table 3. In Table 3, AM indicates the speech model (speech emotion recognition model). The subscript EED indicates the speech model using EED, and the subscript LSTM indicates the speech model using LSTM. Aug indicates the model trained using the augmented training data, and Ori indicates the model trained using the training data without augmentation.
[0101] [Table 3] From this table, we can see that the method using EED and augmenting the training data performed better than the others.
[0102] When trained with the extended training data, the EED-based approach outperformed the LSTM-based approach on happy, sad, excited, and frustrated by 8.15%, 7%, 1.31%, and 6.97%. The weighted average accuracy also improved from 54.04% to 56.30%, and the F1 score improved from 53.34% to 55.73%.
[0103] When trained on the unaugmented training data, EED outperformed LSTM in neutral, angry, excited, and frustrated by 9.86%, 0.57%, 13.92%, and 5.47%. The weighted average accuracy improved from 47.87% to 52.11%, and the F1 score improved from 46.91% to 51.21%.
[0104] When the training data was expanded, EED showed higher performance in happy, sad, neutral, angry, and frustrated than when it was not expanded, with differences of 3.67%, 15.12%, 0.79%, 0.83%, and 9.67%. The weighted average accuracy improved from 52.11% to 56.30%, and the F1 score also improved from 51.21% to 55.73%.
[0105] When LSTM is applied with augmented training data, it shows higher performance in sad, neutral, angry, excited, and frustrated than without augmentation, with the difference being 0.46%, 11.8%, 8.42%, 4.41%, and 8.17%. The weighted average accuracy improves from 47.87% to 54.04%, and the F1 score also improves from 46.91% to 53.34%.
[0106] ·Facial emotion recognition Table 4 shows the results of the facial emotion recognition model. VM stands for Visual Model. The subscript EED stands for a model using EED, and the subscript LSTM stands for a model replacing EED with LSTM.
[0107] [Table 4] From Table 4, we can see that the model using EED shows better performance. The model using EED showed better performance in sad, neutral, angry, and excited than the model using LSTM, with the difference being 5.25%, 4.01%, 6.17%, and 9.69%. The weighted average precision improved by 1.48%, and the F1 score improved by 3.34%.
[0108] Text Emotion Recognition Table 5 shows the results of the text emotion recognition model. TM indicates the text model, the subscript SeMemNN indicates the text model using SeMemNN, and the subscript BERT indicates the text model using BERT pre-trained on English.
[0109] [Table 5] These results show that the model using SeMemNN performs significantly better than the one using BERT. The weighted average accuracy improved from 39.62% to 52.99%, and the F1 score also improved from 36.13% to 51.51%. In addition, the model using SeMemNN showed accuracy more than 10% higher than the model using BERT for all emotion categories, and the accuracy of Angry in particular improved by about 30 points. As a result, it was found that the model using SeMemNN is more suitable for emotion text recognition than BERT, at least for the IEMOCAP dataset.
[0110] Multi-modality As for the multi-model, the uni-model (VM EED ,AM EED_Aug ,TM SeMemNN The results are shown in Table 6.
[0111] [Table 6]
[0112] Of these, MulM Dense does not use the GAT of the above embodiment. Dense The weighted average accuracy of MulM was 67.26 and the weighted F1 score was 66.74. In comparison, the average accuracy of DialogueRNN was 63.40 and the average F1 score was 62.75, while the average accuracy of DialogueGCN and MulM was 65.25 and the average F1 score was 64.18. Dense Compared to DialogueGCN, it has improved by more than 2% in both average precision and F1 score, and compared to DialogueRNN, it has improved by about 5.5% in both average precision and F1 score. Most of this performance improvement is likely due to the performance of the uni-model.
[0113] In contrast, MulM GAT is MulM DenseThis is a multi-model that adds the GAT described in the above embodiment to the MulM. GAT The performance of MulM is even better, with an average precision of 69.88 and an average F1 score of 68.31. Dense This is an improvement of 2.5% compared to DialogueGCN. When examining the individual emotion labels, happy, neutral, angry, and frustrated showed higher accuracy than the conventional technology. Among these, the emotion label happy showed a large improvement of 47% over DialogueGCN. Dense In comparison, MulM, which uses GAT GAT shows at least 5% improvement for the emotion labels happy, angry, and frustrated.
[0114] From the above, it has been confirmed that the accuracy of multi-modality emotion recognition can be further improved by emphasizing the features of each modality using the correlation between modalities using the GAT described in the above embodiment.
[0115] [Variations] In the above embodiment, three types of modalities, namely, facial expression, voice, and spoken text, are used. However, the present invention is not limited to such an embodiment. Any combination including at least two of these three types of modalities may be used. In addition to the above three types of modalities, or instead of any of them, the movement of the whole body, the movement of the head, arms, hands, or torso, body temperature or its distribution, etc. may be adopted as a modality.
[0116] [Computer implementation] Fig. 8 is an external view of a computer system for implementing each of the above embodiments, and Fig. 9 is a hardware block diagram of the computer system shown in Fig. 8.
[0117] 8, this computer system 950 includes a computer 970 having a DVD drive 1002, and a keyboard 974, a mouse 976, and a monitor 972 for interacting with a user, all of which are connected to the computer 970. Of course, this is just one example of a configuration for user interaction, and any general hardware and software available for user interaction (e.g., a touch panel, voice input, pointing device in general) can be used.
[0118] 9, the computer 970 includes, in addition to the DVD drive 1002, a CPU 990, a GPU 992, a bus 1010 connected to the CPU 990, the GPU 992, and the DVD drive 1002, a ROM 996 connected to the bus 1010 and storing a boot-up program of the computer 970, a RAM 998 connected to the bus 1010 and storing instructions constituting a program, a system program, working data, and the like, and a HDD 1000 which is a non-volatile memory connected to the bus 1010. The HDD 1000 is for storing programs executed by the CPU 990 and the GPU 992, and data used by the programs executed by the CPU 990 and the GPU 992, and the like. The computer 970 further includes a network I / F 1008 which provides a connection to a network 986 which enables communication with other terminals, and a USB port 1006 to which a USB memory 984 can be attached and detached and which provides communication between the USB memory 984 and each part in the computer 970.
[0119] The computer 970 further includes an audio I / F 1004 that is connected to the microphone 982, the speaker 980, and the bus 1010, and that reads out audio signals, video signals, and text data generated by the CPU 990 and stored in the RAM 998 or the HDD 1000 in accordance with instructions from the CPU 990, performs analog conversion and amplification processing to drive the speaker 980, and digitizes the analog audio signal from the microphone 982 and stores it in any address of the RAM 998 or the HDD 1000 specified by the CPU 990.
[0120] In the above embodiment, the data and parameters such as the facial expression model 60, the text model 62, the voice model 64, the multi-head GAT 66, the fully connected layer 68, the speech signal, the video signal, the text data, etc. shown in Fig. 2 are all stored in, for example, the HDD 1000, the RAM 998, the DVD 978, or the USB memory 984 shown in Fig. 9, or in a storage medium of an external device (not shown) connected via the network I / F 1008 and the network 986. Typically, these data and parameters are written into the HDD 1000 from outside, for example, and loaded into the RAM 998 when the computer 970 is executed.
[0121] A computer program for operating the computer system to realize the functions of a training device that individually trains the facial expression model 60, the text model 62, and the voice model 64 shown in FIG. 1, a training device that trains the multi-head GAT 66 and the fully connected layer 68 using the trained facial expression model 60, the text model 62, and the voice model 64, and a multi-modality emotion recognition device using the trained facial expression model 60, the text model 62, the voice model 64, the multi-head GAT 66, and the fully connected layer 68, and each of the components thereof, is stored in a DVD 978 that is inserted in a DVD drive 1002, and transferred from the DVD drive 1002 to the HDD 1000. Alternatively, the program is stored in a USB memory 984, and the USB memory 984 is inserted in a USB port 1006, and the program is transferred to the HDD 1000. Alternatively, the program may be transmitted to the computer 970 through a network 986 and stored in the HDD 1000.
[0122] The program is loaded into RAM 998 when executed. Of course, the source program may be input using keyboard 974, monitor 972, and mouse 976, and the compiled object program may be stored in HDD 1000. In the case of a script language, a script input using keyboard 974 or the like may be stored in HDD 1000. In the case of a program that runs on a virtual machine, a program that functions as a virtual machine must be installed in advance in computer 970.
[0123] The CPU 990 reads a program from the RAM 998 according to an address indicated by an internal register called a program counter (not shown), interprets the instruction, reads data required for executing the instruction from the RAM 998, HDD 1000, or other device according to an address specified by the instruction, and executes the process specified by the instruction. The CPU 990 stores the execution result data in an address specified by the program, such as the RAM 998, HDD 1000, or a register in the CPU 990. At this time, the value of the program counter is also updated by the program. The computer program may be directly loaded into the RAM 998 from the DVD 978, the USB memory 984, or via a network. Note that some tasks (mainly numerical calculations) among the programs executed by the CPU 990 are dispatched to the GPU 992 according to instructions included in the program or according to the analysis results obtained when the CPU 990 executes the instructions.
[0124] The program for implementing the functions of each unit according to each embodiment described above by the computer 970 includes a plurality of instructions written and arranged to operate the computer 970 to implement those functions. Some of the basic functions required to execute the instructions are provided by an operating system (OS) or a third-party program running on the computer 970, or by modules of various tool kits installed on the computer 970. Thus, the program does not necessarily include all of the functions required to implement the system and method of this embodiment. The program may include only instructions to execute the operations of each of the above-mentioned devices and their components by calling appropriate functions or functions of a "programming tool kit" in a controlled manner to obtain a desired result. The method of operating the computer 970 for that purpose is well known, and will not be repeated here.
[0125] The GPU 992 is capable of parallel processing, and can simultaneously execute a large amount of calculations associated with machine learning in parallel or in a pipelined manner. For example, parallel calculation elements found in a program when the program is compiled, or parallel calculation elements found when the program is executed, are dispatched from the CPU 990 to the GPU 992 as needed, and executed, and the results are returned to the CPU 990 directly or via a predetermined address in the RAM 998, and assigned to a predetermined variable in the program.
[0126] The embodiments disclosed herein are merely illustrative, and the present invention is not limited to the above-described embodiments. The scope of the present invention is defined by the claims of the appended claims, taking into consideration the detailed description of the invention, and includes all modifications within the scope and meaning equivalent to the words described therein. [Explanation of symbols]
[0127] 50 Multi-modality emotion estimation system 60 facial expression models 62 Text Model 64 Voice Models 66 Multi-head GAT 68 Fully connected layer 70 Emotion estimation results 72, 74 Facial Expressions 76, 78 Phonetic features 80 facial expression feature vectors 82 Text feature vectors 84 Phonetic feature vectors 100, 102, 104 update section 120, 122, 124 Head 126 Average calculation part 150 Training Data Storage Device
Claims
1. an update means for updating each of the features of the plurality of types of modalities related to information display and information transmission of a person based on associations existing between features obtained for the plurality of types of modalities; and a feeling estimation means for estimating a feeling of the person by using the features of the multiple types of modalities updated by the updating means, the emotion estimation device is configured with a neural network, The update means includes: an attention calculation means for calculating the associations existing between the features obtained for the plurality of types of modalities as attention between nodes in a graph neural network having edges connecting the features as nodes; and a feature updating means for updating, for each of the nodes, a feature of the node by using attention from each of adjacent nodes adjacent to the node among the attentions calculated by the attention calculation means.
2. The emotion estimation device according to claim 1 , wherein the attention between adjacent nodes calculated by the feature update means differs depending on a direction of attention.
3. The emotion estimation device according to claim 2 , wherein the update means receives, as the feature, a state of a final hidden layer of an individual neural network that calculates a feature vector representing an emotion from the multiple types of modalities.
4. the individual neural networks include bidirectional neural networks for the time series of the multiple modalities; 4. The feeling estimation device according to claim 3, wherein the update means receives, as the feature, a concatenation of a vector representing a state of the final hidden layer in a forward direction and a vector representing a state of the final hidden layer in a backward direction of the bidirectional neural network.
5. The attention calculation means a plurality of attention calculation heads, each of which is made of a fully connected neural network, and each of which calculates the associations existing between the features of the plurality of types of modalities as attention between nodes in a graph network having the features as nodes and edges connecting each node; and an average calculation means for calculating, for each of the nodes, an average of attentions from each of adjacent nodes adjacent to the node, among the attentions calculated by the plurality of attention calculation heads, as the attention for the node.
6. The updating means updates the identity of the i-th node by Eh i Let the number of attention calculation heads be K and the number of types of modalities be N. Then, the feature Eh of the i-th node is calculated by the following formula: i The emotion estimation device according to claim 5 , wherein the emotion estimation device updates [0010] Here, σ() represents a nonlinear activation function, W represents the weight of the linear transformation obtained by learning, a represents a weight vector, T on the right side of a represents the transpose of the vector, and the symbol “||” represents the concatenation of vectors.
7. The feeling estimation device according to claim 1 , wherein the multiple types of modalities include any combination of at least two of a time series of facial expressions, a spoken voice, and a spoken text.
8. The feeling estimation device according to claim 1 , wherein the multiple types of modalities include a time series of facial expressions, a spoken voice, and a spoken text.
Citation Information
Patent Citations
Emotion estimation system and emotion estimation model generation system
JP2018072876A
