Emotion estimation device, computer program, and computer
The emotion estimation device enhances accuracy by using a graph neural network with self-attention on complete skeletal graphs, addressing the limitations of existing gesture-based emotion recognition systems.
Patent Information
- Application Number
- JP2021039907
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-03-12
- Publication Date
- 2024-12-04
- Estimated Expiration
- 2041-03-12
AI Technical Summary
Existing emotion estimation technologies struggle to accurately utilize gestures due to the limited information extracted from skeletal data, leading to inefficiencies in training neural networks for emotion recognition.
The proposed emotion estimation device employs a graph neural network that performs graph convolution using self-attention on a complete skeletal graph, integrating spatial and temporal features from three-dimensional skeletal data to enhance emotion estimation accuracy.
The solution significantly improves the accuracy of emotion estimation by effectively extracting and utilizing more information from gestures, outperforming conventional methods in unimodal and multimodal emotion recognition tasks.
Smart Images

Figure 0007594285000015 
Figure 0007594285000016 
Figure 0007594285000017
Abstract
Description
[Technical field]
[0001] The present invention relates to an emotion estimation technique, and more particularly to a technique for estimating the emotion of a speaker by utilizing the gestures of the speaker. [Background technology]
[0002] In controlling robots or operating interactive systems, if the emotions of the people involved can be estimated, information on the emotions can be used in various applications. For example, in the fields of biometric security protection, health care, and games, various options can be taken depending on the emotions of the people involved.
[0003] In emotion estimation, multimodal emotion estimation technology using multiple modalities is considered to be particularly promising. For example, the main modalities are the voice of the utterance, the content of the utterance, and facial expressions. One such technology is disclosed in Non-Patent Document 1, which will be described later. With reference to FIG. 1, the multimodal emotion estimation device 50 disclosed in Non-Patent Document 1 includes a voice-based emotion estimation device 60, a text-based emotion estimation device 62 indicating the content of the utterance, and a fully connected layer 64 that receives parameters output from the voice-based emotion estimation device 60 and parameters output from the text-based emotion estimation device 62. The output of the fully connected layer 64 is an emotion estimation result 66. Each of these parts is composed of a neural network, and the multimodal emotion estimation device 50 as a whole also constitutes one neural network.
[0004] The voice-based emotion estimation device 60 includes an RNN (Recurrent Neural Network) 68 that sequentially receives acoustic features 92, 94, ..., 96 consisting of mel-frequency cepstral coefficients (MFCCs) extracted from the voice in a time series and takes hidden states 82, 84, ..., 86 in order, and a fully connected layer 98 that is connected to receive prosody 80 extracted from the voice and the final hidden state 86 of the RNN 68 as inputs and has its output connected to the input of the fully connected layer 64.
[0005] The text-based emotion estimation device 62 includes an RNN 70 that receives tokens 112, 114, ..., 116 in order and changes its internal state to hidden states 102, 104, ..., 106 in order, and a fully connected layer 118 that receives the hidden state 106 as an input and has its output connected to the input of the fully connected layer 64.
[0006] The emotion estimation devices 60 and 62 are trained separately. Furthermore, the multimodal emotion estimation device 50, which combines the emotion estimation devices 60 and 62 with a fully connected layer 64, is further trained using the speech and text of the utterance and a label indicating the emotion of the speaker of the utterance as training data. As a result, an emotion estimation result 66 consisting of, for example, four values indicating the probability of the emotion of the speaker who made the utterance is obtained as the output of the fully connected layer 64. Here, four types of emotion labels (happiness, anger, sadness, and neutral) are used.
[0007] Meanwhile, in the case of humans, gestures are often used to accompany speech, and scenes in which emotions are emphasized by gestures are often observed. For example, to express anger, people often raise both arms high at the same time or move them up and down vigorously. Unlike voice or facial expressions, such gestures can be clearly recognized from a distance. Therefore, it is desirable to add the speaker's gestures to the information for emotion estimation.
[0008] However, as described in Non-Patent Document 1, emotion estimation using gestures has not been widely used, although emotion estimation using the speaker's voice and text of the speech content is common. The reason for this is that, for example, for voice, facial expression, text of speech content, etc., abundant data for training a neural network for emotion estimation is prepared, whereas such data is scarce for actions. Neural networks have been extremely effective in solving various problems related to identification and classification, but have a problem of requiring a large amount of training data. When training data is scarce, such as for gestures, neural networks cannot be trained efficiently.
[0009] Even in such unfavorable circumstances, research is being conducted to use gestures for emotion estimation, and a proposal for this purpose is made in Non-Patent Document 2, which will be described later. The technology disclosed in Non-Patent Document 2 abstracts the skeletal structure of the human body during speech into a spatial graph. The nodes of this spatial graph correspond to joints, and the edges correspond to the connections of the joints by bones. The emotion estimation device disclosed in Non-Patent Document 2 further constructs a graph neural network using a space-time convolution block and extracts features. The emotion estimation device in Non-Patent Document 2 estimates the speaker's emotion using these features. [Prior art documents] [Non-patent literature]
[0010] [Non-Patent Document 1] Yoon, S.; Byun, S.; Jung, K. Multimodal speech emotion recognition using audio and text. In Proceedings of the 2018 IEEE Spoken Language Technology Workshop (SLT), Athens ,Greece, 18-21 December 2018; pp.112-118. [Non-Patent Document 2] Yan, S.; Xiong, Y.; Lin, D. Spatial temporal graph convolutional networks for skeleton-based action recognition. arXiv, 2018, arXiv:1801.07455. Summary of the Invention [Problem to be solved by the invention]
[0011] The technology disclosed in the above-mentioned Non-Patent Document 2 is noteworthy in the sense that it uses gestures to predict emotions. However, this technology still has problems to be solved.
[0012] That is, the skeletal graph used in Non-Patent Document 2 has a problem in that it uses only a part of the gesture features. In the sense that it focuses on the joints and the bones connecting each joint, the skeletal graph is considered to be a natural representation of the gestures of the human body. However, on the other hand, this means that the technology in Non-Patent Document 2 is restricted in that it can only use limited information about the movements of the human body. If more information could be extracted from gestures and used to estimate emotions, it is believed that the accuracy of the estimation would be higher.
[0013] Therefore, an object of the present invention is to provide an emotion estimation device that can improve accuracy by extracting more information from gestures. [Means for solving the problem]
[0014] (1) A feeling estimation device according to a first aspect of the present invention includes: a feature calculation means for calculating features for each node of a skeleton graph, the skeleton graph including a time series of graphs representing a three-dimensional skeleton of a person while speaking and edges connecting corresponding elements in two graphs at adjacent times in the time series; and a feeling estimation means including a graph neural network pre-trained to perform a first graph convolution in spatial and temporal directions using self-attention on the features for each node of a complete graph in which all nodes of the skeleton graph are connected by one edge, and to estimate the person's emotion when speaking based on a result of the first graph convolution.
[0015] (2) Preferably, the graph neural network of the emotion estimation means includes a plurality of convolution blocks connected in series and a pooling layer connected to receive the final output of the plurality of convolution blocks, an input tensor having features arranged to represent a complete graph as elements is input to the beginning of the plurality of convolution blocks, each of the plurality of convolution blocks includes a spatial convolution block that performs graph convolution using self-attention in the spatial direction of the complete graph and a temporal convolution block that performs graph convolution using self-attention in the time direction of the complete graph, the spatial convolution block and the temporal convolution block are connected in series with each other, and both the spatial convolution block and the temporal convolution block are configured to output a tensor of the same format as the input tensor.
[0016] (3) More preferably, in the emotion estimation device, each of the multiple convolution blocks further includes: convolution means consisting of a graph neural network that performs a second graph convolution on each node of the skeletal graph; and mixing means for mixing an output of the spatial convolution block and the time convolution block present in a subsequent stage with an output of the convolution means in a predetermined ratio to obtain an output of the convolution block.
[0017] (4) More preferably, the nodes of the skeleton graph represent joint positions of the skeleton.
[0018] (5) Preferably, the features include three-dimensional coordinate data of joint positions.
[0019] (6) More preferably, the nodes of the skeletal graph represent bones of a skeleton.
[0020] (7) More preferably, the features include a direction and a length of a vector representing the bone.
[0021] (8) A feeling deduction device according to a second aspect of the present invention includes the first feeling deduction device according to (4) or (5), the second feeling deduction device according to (6) or (7), and a fully connected layer having outputs of the first feeling deduction device and the second feeling deduction device as input, wherein the first feeling deduction device, the second feeling deduction device, and the fully connected layer are trained with training data including skeletal time-series data to which labels indicating emotions have been attached in advance.
[0022] (9) A feeling estimation device according to a third aspect of the present invention includes the video feeling estimation device according to any one of (1) to (7), a voice feeling estimation means pre-trained to estimate a person's feeling from the person's voice data, a text feeling estimation means pre-trained to estimate a person's feeling from text data indicating the content of the person's utterance, and a fully connected layer that receives outputs of the video feeling estimation device, the voice feeling estimation means, and the text feeling estimation means, wherein the video feeling estimation device, the voice feeling estimation means, the text feeling estimation means, and the fully connected layer of the emotion estimation device are further trained with training data consisting of a time series over a plurality of time steps of three-dimensional skeletal data of the person while uttering.
[0023] (10) A computer program according to a fourth aspect of the present invention causes a computer to function as each of the means of any one of the above-mentioned devices.
[0024] (11) A computer according to a fifth aspect of the present invention is programmed to function as any one of the emotion estimation devices (1) to (9).
[0025] The above and other objects, features, aspects and advantages of the present invention will become apparent from the following detailed description of the invention taken in conjunction with the accompanying drawings. [Brief description of the drawings]
[0026] [Figure 1] FIG. 1 is a block diagram showing the configuration of a conventional multi-modal emotion estimation device. [Diagram 2] FIG. 2 is a block diagram showing the configuration of a multimodal feeling estimation device according to the first embodiment of the present invention. [Diagram 3] FIG. 3 is a block diagram showing a process of extracting a skeleton graph from a series of images of the upper body of a person who is speaking in the first embodiment. [Figure 4] FIG. 4 is a schematic diagram showing a graph extracted from the skeleton of a human body at a certain time step. [Diagram 5] FIG. 5 is a schematic diagram showing a complete graph corresponding to a graph extracted from the joint positions of the human skeleton. [Figure 6] FIG. 6 is a schematic diagram showing a process of weighting and adding the results of two types of convolution. [Figure 7] FIG. 7 is a schematic diagram for explaining a method of forming a graph using vectors corresponding to bones connecting joints in the human skeleton. [Figure 8] FIG. 8 is a block diagram showing the configuration of a feeling estimation device according to the second embodiment of the present invention. [Figure 9] FIG. 9 is a block diagram showing the configuration of a graph neural network forming a self-attention enhanced spatial temporal graph convolution network (S-STGCN) used in each embodiment of the present invention. [Figure 10] FIG. 10 is a block diagram showing the configuration of the first-stage convolution block shown in FIG. [Figure 11] FIG. 11 is a block diagram showing the configuration of the S-SGC layer shown in FIG. [Figure 12] FIG. 12 is an external view of a computer system for implementing each embodiment of the present invention. [Figure 13] FIG. 13 is a block diagram showing a hardware configuration of the computer system shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0027] In the following description and drawings, the same parts are given the same reference numbers, and therefore detailed descriptions thereof will not be repeated.
[0028] 1. First embodiment 1. Configuration (1) Overall structure FIG. 2 shows a schematic configuration of a multimodal emotion estimation device 150 according to a first embodiment of the present invention in the form of a block diagram. Referring to FIG. 2, the multimodal emotion estimation device 150 includes the emotion estimation devices 60 and 62 shown in FIG. 1, an emotion estimation device 160 for performing emotion estimation using a skeleton graph obtained from a time series of skeleton information extracted from a video, and a fully connected layer 162 for receiving the emotion estimation devices 60 and 62 and the output of the emotion estimation device 160 and outputting an emotion estimation result 164. The fully connected layer 162 has the same function as the fully connected layer 64 shown in FIG. 1, but differs from the fully connected layer 64 in that it is configured to receive input from the emotion estimation device 160 in addition to the emotion estimation devices 60 and 62.
[0029] The emotion estimation device 160 includes an S-STGCN 174 that performs processing on a time-series skeleton graph 172, which is a graph representing a skeleton time series extracted from a video time series 170 of a person's upper body consisting of a plurality of image sequences, by combining the graph convolution described in Non-Patent Document 2 and graph convolution using self-attention on a complete graph corresponding to the graph time series, and a fully connected layer 176 having an input that receives the output of the S-STGCN 174 and an output connected to the fully connected layer 162.
[0030] The configuration of the S-STGCN 174 will be described later with reference to FIGS.
[0031] (2) Forming a skeleton graph 3, the process of extracting time series skeleton graph 172 from video time series 170 includes a 2D posture estimation unit 200 that estimates a two-dimensional posture of a subject person from video time series 170, and a 3D posture estimation unit 202 that estimates a three-dimensional posture of the person from the two-dimensional posture of the person estimated by 2D posture estimation unit 200 and outputs time series skeleton graph 172 of the person's upper body from the estimated posture. Both of these can be realized by existing open source software.
[0032] (3) Skeleton graph 4 is a schematic diagram of a graph 250 representing a skeleton at a certain point in time. In this example, the graph includes 10 nodes 260, ..., 278 and 9 edges 290, ..., 306. The graph 250 is the same as the skeleton graph adopted in Non-Patent Document 1.
[0033] Nodes 260, ..., 278 each correspond to a joint in the upper body of the person. Edges 290, ..., 306 each correspond to a bone connecting a particular two of these nodes. In this example, edge 290 connects node 260 to node 262. Edge 292 connects node 262 to node 268. Edge 294 connects node 264 to node 266. Edge 296 connects node 266 to node 270. Edge 298 connects node 268 to node 270. Edge 300 connects node 270 to node 272. Edge 302 connects node 270 to node 274. Edge 304 connects node 274 to node 276. Edge 306 connects node 276 to node 278.
[0034] The time series skeleton graph 172 is a graph with a multi-layered structure in which the graphs 250 are arranged in a time series. In the graphs 250 of adjacent time steps, the same nodes are connected to each other by edges (not shown). For example, although not shown in FIG. 4, a node 260 of a certain time step is connected to a node 260 of the time step before and after it by an edge. However, even in adjacent time steps, a node 260 of a certain time step is not connected to another node 262 of the adjacent time step. Moreover, even in the same node 260, nodes 260 of non-adjacent time steps are not connected to each other.
[0035] This skeletal graph is represented as an adjacency matrix, which is divided into multiple sub-matrices according to the division strategy proposed in Non-Patent Document 2.
[0036] (4) Complete graph 5 is a diagram for explaining a complete skeleton graph 350 assumed as a target for graph convolution by self-attention in the first embodiment. Referring to FIG. 5, node 278 is connected to node 276 by edge 306 in FIG. 4, but is not connected to other nodes. However, in the complete skeleton graph 350, node 278 is connected to node 260 by edge 370, to node 264 by edge 372, to node 262 by edge 374, to node 266 by edge 376, to node 268 by edge 378, to node 270 by edge 380, to node 274 by edge 382, and to node 272 by edge 384.
[0037] Edges 370, ..., 384 are of different thicknesses, which indicate the magnitude of the relationship between node 278 and each node that is handled by the graph convolution through self-attention.
[0038] 5 shows the connection relationship between node 278 and other nodes at a certain point in time. However, not only node 278 but also all other nodes are connected to each other node by one edge.
[0039] A graph in which all nodes are connected to all other nodes by one edge is called a complete graph. In the case of the complete skeleton graph 350, not only is the graph of a certain time step a complete graph, but all nodes constituting the time series are also complete graphs in which they are connected to each other node by one edge. As will be described later, in order to extract the relationship between such spatial and temporal joints by graph convolution, in this embodiment, graph convolution is performed in both the spatial and temporal directions.
[0040] In this way, the graph convolution for the complete graph uses self-attention. The mechanism for performing the graph convolution (the configuration of the S-STGCN 174) will be described later with reference to Figs. 9 to 11.
[0041] (5) Integration of two types of graph convolution results As described above, in this embodiment, two graph convolutions are performed: the graph convolution proposed in Non-Patent Document 1 and the graph convolution using self-attention based on a complete graph proposed in this embodiment, and the results are integrated.
[0042] The integration method will be described with reference to FIG. 6. With reference to FIG. 6, the graph 250 is subjected to the graph convolution proposed in Non-Patent Document 1. On the other hand, the complete skeleton graph 350 corresponding to the graph 250 is subjected to the graph convolution using the self-attention proposed in this embodiment. Then, the two results are added together to integrate them. At that time, in this embodiment, the ratio of addition is such that the final result is obtained by adding the value of the graph convolution for the complete skeleton graph 350 multiplied by a constant r to the result of the graph convolution for the graph 250. The constant r is a parameter to be trained.
[0043] 2. Training The emotion estimation devices 60 and 62, the emotion estimation device 160, and the entire multimodal emotion estimation device 150 can be trained using a normal backpropagation method. More specifically, video samples of a person speaking with a certain emotion are prepared, and the samples are labeled with a label indicating the emotion of the person, which serves as training data. The emotion estimation devices 60 and 62, and the emotion estimation device 160 are trained independently. In both of these and the training of the entire multimodal emotion estimation device 150, a certain amount of training data is prepared and given to the target emotion estimation device to obtain its output. The error between the output and the label attached to the training data is calculated, and the parameters of the target emotion estimation device are updated using the backpropagation method.
[0044] 3.Operation The multimodal emotion estimation device 150 operates as follows: The emotion estimation devices 60 and 62 operate in the same manner as the conventional ones shown in FIG.
[0045] In the emotion estimation device 160, the 2D posture estimation unit 200 shown in FIG. 3 estimates a two-dimensional posture of a speaker in a series of images from a video time series 170. From the series of two-dimensional postures, the 3D posture estimation unit 202 estimates a series of three-dimensional postures, and estimates three-dimensional coordinates of joint positions in each of the three-dimensional postures to obtain a time-series skeleton graph 172. Note that when extracting a skeleton from the three-dimensional posture estimation, each joint position and the connection relationships between them are determined in advance. Therefore, it is easy to obtain the time-series skeleton graph 172 from each joint position calculated by the 3D posture estimation unit 202.
[0046] The S-STGCN 174 shown in Fig. 2 performs the prior art graph convolution proposed in Non-Patent Document 2 on the time-series skeletal graph 172. The S-STGCN 174 also performs graph convolution using self-attention as described with reference to Fig. 9 to Fig. 11. The results of both are summed inside the S-STGCN 174 in a ratio of 1:r as described above, and become the output of the S-STGCN 174.
[0047] Second Second Embodiment 1. Configuration (1) Graph using bone vectors In the first embodiment, a graph was considered in which, among the information on the speaker's skeleton, joints were treated as nodes and bones connecting the joints were treated as edges. However, the present invention is not limited to such an embodiment. It is also possible to consider a graph in which bones are treated as nodes and joints corresponding to the end points of the bones are treated as edges.
[0048] 7, in the second embodiment, a vector e , which is a role corresponding to a bone in a skeleton, connects a vertex v1, which is a node 260, and a vertex v2, which is a node 262, in the graph 250. 1,2 are nodes. In other words, the bone vector in this case represents the direction and length of the bone. Similarly, a graph can be considered in which each edge in the graph 250 is a node and each node is an edge. In the second embodiment, this graph is used as the input of the S-STGCN.
[0049] In the second embodiment, emotion estimation is performed based on parameters obtained by two systems: a emotion estimation device using a second S-STGCN that receives as input a time series of such a graph, and the emotion estimation device 160 described in the first embodiment.
[0050] (2) Two-system emotion estimation device FIG. 8 shows a schematic block diagram of the emotion estimation device 450 according to the second embodiment. Referring to FIG. 8, the emotion estimation device 450 includes a joint emotion estimation unit 462 that outputs a score vector for emotion estimation from skeleton information 460 representing a three-dimensional posture extracted from a series of videos by a method similar to that of the emotion estimation device 160 according to the first embodiment, a bone vector emotion estimation unit 464 that receives a time series skeleton graph obtained by extracting information on the time series of the bone vectors from the skeleton information 460 and outputs a score vector for emotion estimation by a method similar to that of the emotion estimation device 160, and an integration unit 466 that is connected to receive the output of the joint emotion estimation unit 462 and the output of the bone vector emotion estimation unit 464 and calculates a weighted sum of the scores output by these. The output of the integration unit 466 gives an output indicating the emotion that is most likely to be the emotion of the speaker among the four emotions described above.
[0051] The joint-based emotion estimation unit 462 and the bone vector-based emotion estimation unit 464 each form a graph neural network. Furthermore, these two functional units and the integration unit 466 are connected to each other to form a neural network as a whole.
[0052] 2. Training The joint-based emotion estimation unit 462 and the bone-vector-based emotion estimation unit 464 are trained independently. Then, the emotion estimation device 450 that combines them with the integration unit 466 is also trained.
[0053] The training of the joint-based emotion estimation unit 462, the bone vector-based emotion estimation unit 464, and the emotion estimation device 450 as a whole can be performed using a normal backpropagation method, as in the first embodiment. More specifically, video samples of a person speaking with a certain emotion are prepared, and the samples are labeled with a label indicating the emotion of the person, which serves as training data. In the training of the joint-based emotion estimation unit 462, the bone vector-based emotion estimation unit 464, and the emotion estimation device 450, a certain amount of training data is prepared and given to the target emotion estimation device to obtain its output. The error between the output and the label attached to the training data is calculated, and the parameters of the emotion estimation device are updated using the backpropagation method.
[0054] 3.Operation 8, in the joint-based emotion estimation unit 462, the joint position information extraction unit 480 extracts joint position information from the skeleton information 460. The S-STGCN 482 generates a time-series skeleton graph based on the joint position information, performs normal graph convolution and graph convolution using self-attention, and outputs a score vector indicating the emotion estimation result to the integration unit 466.
[0055] Similarly, in the bone vector-based emotion estimation unit 464, the bone vector information extraction unit 490 extracts bone vector information representing the length and direction of bones from the skeleton information 460. The S-STGCN 492 generates a time-series skeleton graph based on the bone vector information, performs normal graph convolution and graph convolution using self-attention, and outputs a score vector indicating the emotion estimation result to the integration unit 466.
[0056] The integrating unit 466 integrates the score vector from the joint-based emotion estimation unit 462 and the score vector from the bone-vector-based emotion estimation unit 464 by a weighted sum, and outputs the emotion with the highest probability as a prediction 468.
[0057] 3rd S-STGCN 1. Configuration (1) Overall network configuration The S-STGCN commonly used in the first and second embodiments, for example, S-STGCN482, has the following structure.
[0058] Referring to FIG. 9 , the S-STGCN 482 includes ten serially connected convolution blocks 552, ..., 570 that receive a skeletal graph 550 as input, a pooling layer 572 that performs global average pooling on the output of the convolution blocks 570, and a SoftMax layer 574 that performs a SoftMax operation on the output of the pooling layer 572.
[0059] As described above, the skeleton graph 550 has two directions, the time direction and the space direction. For each of the convolution blocks 552, ..., 570, graph convolution is performed in each of the time direction and the space direction.
[0060] In this embodiment, the value of each node of the skeleton graph 550 is a three-dimensional coordinate in the case of a joint, and therefore the number of input channels is three. This number of channels varies according to the convolution process. However, the convolution blocks 552, ..., 570 basically have the same structure. Therefore, each of them receives an input of the same dimension as the skeleton graph 550, which is the input to the convolution block 552, and outputs data of the same dimension to the next block.
[0061] (2) Configuration of the convolution block Fig. 10 shows a more detailed configuration of the convolution block 552 shown in Fig. 9. The convolution blocks 554, ..., 570 are configured in the same manner. Referring to Fig. 10, the convolution block 552 includes an S-SGC layer 600, a batch normalization layer 602, a Relu 604, a T-Conv layer 606, a batch normalization layer 608, and a Relu 610 connected in series, a residual calculation unit 612 that receives the same input as the input to the S-SGC layer 600, an adder 614 for adding the output of the residual calculation unit 612 to the output of the Relu 610, and a Relu 616 that receives the output of the adder 614.
[0062] The S-SGC layer 600 is for performing graph convolution in the spatial direction of the skeletal graph indicated by the input. That is, the S-SGC layer 600 performs graph convolution between each node of the skeletal graph and a node of the same time step.
[0063] On the other hand, the T-Conv layer 606 is for performing graph convolution in the time direction of the skeletal graph. That is, the T-Conv layer 606 performs graph convolution between each node of the skeletal graph and a node of a different time step.
[0064] (3) Structure of S-SGC layer The S-SGC layer 600 and the T-Conv layer 606 have the same configuration, except for the nodes they target. Therefore, only the configuration of the S-SGC layer 600 will be described below.
[0065] The number of dimensions of tensors, matrices, vectors, etc. that appear in the following description are shown in advance in a list format below. [Table 1]
[0066] A. Overall configuration of S-SGC layer 600 Referring to FIG. 11, the S-SGC layer 600 includes a convolution section 650, a self-attention graph convolution section 652, and a gating section 654.
[0067] The self-attention graph convolution unit 652 has a structure similar to that of the encoder part of the deep learning model using self-attention, called Transformer, described in Reference 1 below. The entire disclosure of Reference 1 is incorporated herein by reference.
[0068] [Reference 1] Vaswani et al., Attention is all you need. In Proceedings of the Advances in neural information processing systems, Long Beach, CA, USA, 4-9 December 2017; pp. 5998-6008.
[0069] B. Self-attention graph convolution unit 652 That is, referring to FIG. 11, the self-attention graph convolution unit 652 converts the input f in (Dimension C in ×T×V tensor, where T is the number of time steps in the skeleton graph and V is the number of joints in the skeleton graph in one time step.) and the dimension is T×V×C in , H (in this embodiment, H=4) heads 670, ..., 676 each receiving a branched output of the batch normalization unit 700, a concatenation unit 714 that concatenates the outputs of these H heads 670, ..., 676, a linear transformation unit 716 that performs a linear transformation by multiplying the output of the concatenation unit 714 by a matrix, a residual calculation unit 718 that performs a predetermined residual calculation on the output of the batch normalization unit 700, and an addition unit 720 that adds the output of the residual calculation unit 718 to the output of the linear transformation unit 716 pointwise.
[0070] All of the H heads 670, ..., 676 have the same configuration. For example, head 670 includes linear transformation units 702 and 704, which each receive the output of a batch normalization unit 700, and a linear transformation unit 706. The configurations of these heads 670, ..., 676 are exactly the same as each other, and they also have the same number of parameters. However, during training, each parameter is independently initialized with a random number. Therefore, the self-attention calculations by each head after training will be different from each other.
[0071] The linear transformation unit 704 is for multiplying the output of the batch normalization unit 700 by a matrix to output a tensor with dimensions T×V×Dk. The output of the linear transformation unit 704 is called a key in Reference 1 and is usually denoted by "K". Dk indicates the number of dimensions that K occupies in the attention obtained by the head 670 for the tensor denoted by "K". In this embodiment, Dk=0.25×C in It is.
[0072] The linear transformation unit 702 is for outputting a tensor of dimensions T×V×Dk by multiplying the output of the batch normalization unit 700 by a matrix. In Reference 1, the output of the linear transformation unit 702 is called a query, denoted by “Q”.
[0073] Similar to the linear transformation units 702 and 704, the linear transformation unit 706 multiplies the output of the batch normalization unit 700 by a matrix, so that the dimensions are T×V×D V The output of the linear transformation unit 706 is called "Value" in Reference 1 and is denoted by "V". V indicates the number of dimensions that V occupies in the attention obtained by head 670.
[0074] The head 670 further includes a scale conversion dot product unit 708 for performing a scale conversion dot product shown in the following equation between the key K from the linear conversion unit 704 and the query Q from the linear conversion unit 702, a SoftMax calculation unit 710 for performing a softmax calculation on the output of the scale conversion dot product unit 708, and a multiplication unit 712 for performing multiplication between the output of the SoftMax calculation unit 710 and the output of the linear conversion unit 706.
[0075] Here, the feature vector of each joint is a i Let V be a set of i = 1, ..., V. Then, according to the disclosure in Reference 1, the similarity e between the i-th line of a query Q and the j-th line of a key K in a certain head is ij head is calculated by the following scaled dot product:
number
[0076] d k The numerator is divided by the square root of (scaled) k This is to prevent a problem in which, when the value of x becomes large, the value of the numerator dot product becomes too large, causing the gradient in the backpropagation of the error to become small and hindering training.
[0077] The element α in the i-th row and j-th column of the output of the SoftMax calculation unit 710 in this head ij head is expressed by the following formula:
number
[0078] The weighted result of these values for a given head is given by the following formula:
number
number
[0079] The output of the concatenation unit 714 is expressed by the following equation.
number
[0080] The residual calculation unit 718 is provided to stabilize the training of the self-attention graph convolution unit 652. The residual calculation unit 718 calculates the input f in For the residual Residual(f in) is used to calculate
number
[0081] The output of the adder 720, i.e., the output f a is expressed by the following formula:
number
[0082] C. Convolution section 650 The convolution unit 650 is for performing the conventional graph convolution described in Non-Patent Document 2. For this convolution, the skeleton graph is input to the convolution unit 650 in the form of an adjacency matrix A. According to the division strategy also proposed in Non-Patent Document 2, the matrix A is divided into K sub-matrices (K here is different from the key input to the attention head and indicates a natural number). That is, A=Σ k K A k Let V be the number of nodes (joints) and A k ∈R V×V (Here, V is different from the V (value) of the input to the attention head, and represents a natural number.) On the other hand, the input f in is also provided to the convolution unit 650.
[0083] The convolution unit 650 receives an input f inand a multiplication unit 732 that multiplies the output of the two-dimensional convolution unit 730 by the adjacent matrix A and provides the output to the gating unit 654.
[0084] Here, the input f in As mentioned above, C in ×T×V tensor. This input f in The convolution operation for is expressed by the following equation.
number
[0085] The output f of the two-dimensional convolution unit 730 is also divided into K subsets as follows, similar to the adjacency matrix A.
number
[0086] Then, the output of the multiplication unit 732, i.e., the output f g is expressed as follows:
number
[0087] D. Gating section 654 Assuming that the output of the convolution unit 650 and the output of the self-attention graph convolution unit 652 have different contributions to updating the state of the node, the weight of the output of the self-attention graph convolution unit 652 is adjusted using the coefficient r to be trained. That is, the gating unit 654 includes a gate 750 that multiplies the output of the self-attention graph convolution unit 652 by r, and a gating unit 654 that adds the output of the gate 750 and the output of the convolution unit 650 to output fout and a summing unit 752 for outputting: r is the training object and is initialized to 1.
[0088] Output f of S-SGC layer 600 out is expressed by the following formula:
number
[0089] The specific number of channels for the 10 convolution blocks shown in Figure 9, the configuration (number of dimensions) of the input and output tensors, and the number of internal attention heads are shown in table format below. In this table, "T" corresponds to the duration of the video, and more specifically, indicates the number of time steps in the time series. As can be seen from this table, the number of dimensions of the input and output tensors is the same for all blocks except for the number of channels. The number of channels changes depending on the number of filters used for convolution within each block. [Table 2]
[0090] Fourth Experiment 1. Experimental Set-up An experiment was conducted on the emotion estimation device of the above embodiment using the database IEMOCAP described in the following Reference 2. The entire disclosure of Reference 2 is incorporated herein by reference.
[0091] [Reference 2] Busso, C.; Bulut, M.; Lee, CC; Kazemzadeh, A.; Mower, E.; Kim, S.; Chang, J .N.; Lee, S.; Narayanan, SS IEMOCAP: Interactive emotional dyadic motion capture database. Lang. Resour. Eval. 2008, 42,335.
[0092] IEMOCAP is a recording of audio and video of two people interacting in a hypothetical or pre-prepared scenario. Each utterance was labeled by three or four annotators into 10 classes: happy, sad, angry, surprised, fear, disgust, frustrated, excited, and others. However, instead of the skeletal data, only head and hand movements were captured and stored in the database, ignoring parts of the body that are important for emotional expression, such as the spine, shoulders, and arms.
[0093] To be consistent with previous studies, we subsumed the excitement label in the happiness label and used four emotion classes: neutral, happy, sad, and angry. The dataset used in the experiment had a total of 5492 samples, of which 1606 were labeled as happy, 1081 as sad, 1102 as angry, and 1703 as neutral.
[0094] In the experiments, we used the OpenSMILE toolkit described in Reference 4 below to extract MFCC and prosody features and initialize tokens with a pre-trained 300-dimensional Glove vector (Reference 3), the entire disclosure of which is hereby incorporated by reference in its entirety.
[0095] [Reference 3] Pennington et al.,: GloVe: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), Doha, Qatar, 25-29 October 2014, pp.1532-1543. [Reference 4] Eyben et al.,: Opensmile: The munich versatile and fast open-source audio feature extractor. In Proceedings of the 18 th ACM international conference on Multimedia, Firenze, Italy, 25-29 October 2010; pp.1459-1462.
[0096] In the experimental training process, samples were divided into a training set, a development set, and a test set in a ratio of 8:0.5:1.5. Mutual entropy loss was used as the loss function for backpropagation.
[0097] 2.Results The performance of the emotion estimation model according to the above embodiment and the baseline model is shown in Table 3 below. [Table 3] The numbers in brackets after each model in Table 3 refer to the following references 5-7.
[0098] [Reference 5] Yoon et al.,: Multimodal speech emotion recognition using audio and text. In Proceedings of the 2018 IEEE Spoken Language Technology Workshop (SLT), Athens, Greece, 18-21 December 2018; pp.112-118. (Non-Patent Document 1) [Reference 6] He et al.,: Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 27-30 June 2016; pp.770-778. [Reference 7] Yan et al.: Spatial temporal graph convolutional networks for skeleton-based action recognition. arXiv, 2018, arXiv:1801.07455. (Non-patent document 2) The disclosure of Reference 7 is incorporated herein by reference in its entirety.
[0099] Table 3 shows the unweighted average recall (UAR) and weighted average recall (WAR) for unimodal and multimodal models using speech signals, text content, and upper body skeletal data. Given the biased sample, we used UAR to evaluate the models by treating each category equally. Because the value of UAR is easily affected by categories with fewer samples, we also used WAR to measure the overall estimation accuracy, since WAR is numerically equal to the accuracy of the model.
[0100] S-STGCN shown in Table 3 is a system that performs emotion estimation using emotion estimation device 160 (FIG. 2) which is a part of the first embodiment, and 2s-S-STGCN corresponds to emotion estimation device 450 (FIG. 8) shown in the second embodiment. SERN is multimodal emotion estimation device 150 (FIG. 2) in the first embodiment, and SERN-2s is obtained by replacing emotion estimation device 160 of multimodal emotion estimation device 150 with emotion estimation device 450 in the second embodiment.
[0101] In Table 3, "A" indicates the voice signal, "T" indicates the text of the utterance, and "S" indicates the gesture (skeletal information). For example, "A+T" indicates multi-modality emotion estimation using the voice signal and the text of the utterance.
[0102] In this experiment, ResNet18 and ST-GCN were used for comparison with the present invention. ResNet is a conventional convolutional network with a residual mechanism added, and is widely used in many tasks. In the experiment, ResNet18 and ST-GCN were applied to skeleton data, and the performance was compared with each embodiment of the present invention.
[0103] As can be seen from Table 3, for the S-STGCN model, the UAR and WAR are both 68.4% and 68.4%, respectively. This performance exceeds that of other networks for emotion estimation, demonstrating the effectiveness of our model. This result indicates that the self-attention graph convolution layer in particular can effectively extract emotion-related features from dynamic skeletal sequences. That is, it can be seen that the self-attention graph convolution unit 652 (FIG. 11) can effectively add auxiliary information to the spatial graph convolution for a given graph.
[0104] The performance is further improved dramatically when a dual-system configuration for joints and bones is adopted, which means that emotion information can be more effectively extracted from skeletal time series.
[0105] For multi-modality emotion estimation, we compare it with MDRE, a basic model that uses speech signals and text. In this example, we also use UAR and WAR as the performance measures of the model.
[0106] The model SERN using the speech signal, text, and skeletal information according to the present invention showed a performance that was 8.4% higher in UAR and 8.6% higher in WAR than MDRE, which does not use skeletal information. This result shows that emotion-related information not contained in speech and text is extracted from skeletal information. Furthermore, SERN-2s, which is the SERN shown in FIG. 2 in which the emotion estimation device 160 (S-STGCN) is replaced with the emotion estimation device 450 (2s-S-STGCN) shown in FIG. 8, was compared with SERN. As a result, SERN-2s showed further improvements in UAR and WAR compared to SERN.
[0107] As described above, it has been found that the emotion estimation model according to the present invention can estimate the emotion of a speaker with higher accuracy than the conventional technology.
[0108] 5. Computer implementation Fig. 12 is an external view of a computer system for implementing each of the above embodiments, and Fig. 13 is a hardware block diagram of the computer system shown in Fig. 12.
[0109] 12, this computer system 950 includes a computer 970 having a DVD (Digital Versatile Disc) drive 1002, and a keyboard 974, a mouse 976, and a monitor 972 for interacting with a user, all of which are connected to the computer 970. Of course, these are just one example of a configuration for when user interaction is required, and any general hardware and software available for user interaction (e.g., a touch panel, voice input, or a general pointing device) can be used.
[0110] 13, the computer 970 includes, in addition to a DVD drive 1002, a CPU (Central Processing Unit) 990, a GPU (Graphics Processing Unit) 992, a bus 1010 connected to the CPU 990, the GPU 992, and the DVD drive 1002, a ROM (Read-Only Memory) 996 connected to the bus 1010 and storing a boot-up program of the computer 970, a RAM (Random Access Memory) 998 connected to the bus 1010 and storing instructions constituting a program, a system program, working data, and the like, and an SSD (Solid State Drive) 1000 which is a non-volatile memory connected to the bus 1010. The SSD 1000 is for storing programs executed by the CPU 990 and the GPU 992, and data used by the programs executed by the CPU 990 and the GPU 992, and the like. The computer 970 further includes a network I / F 1008 that provides a connection to a network 986 enabling communication with other terminals, and a USB port 1006 to which a USB (Universal Serial Bus) memory 984 can be attached / detached and that provides communication between the USB memory 984 and each component within the computer 970.
[0111] The computer 970 further includes an audio I / F 1004 that is connected to the microphone 982 and speaker 980 and the bus 1010, and that reads out audio signals, video signals, and text data generated by the CPU 990 and stored in the RAM 998 or SSD 1000 in accordance with instructions from the CPU 990, performs analog conversion and amplification processing to drive the speaker 980, and digitizes the analog audio signal from the microphone 982 and stores it in any address of the RAM 998 or SSD 1000 specified by the CPU 990.
[0112] In the above embodiment, parameters of the graph neural network for realizing the multimodal emotion estimation device 150 and the emotion estimation device 450, and a program for realizing the graph convolution, etc. are all stored in, for example, the SSD 1000, RAM 998, DVD 978, or USB memory 984 shown in Fig. 13, or a storage medium of an external device (not shown) connected via the network I / F 1008 and the network 986. Typically, these data, parameters, etc. are written to the SSD 1000 from the outside, for example, and loaded into the RAM 998 when the computer 970 is executed.
[0113] 2 and 8, computer programs for operating the computer system to realize the functions of the multimodal emotion estimation device 150 and the emotion estimation device 450 and each of their components are stored in a DVD 978 inserted in a DVD drive 1002 and transferred from the DVD drive 1002 to the SSD 1000. Alternatively, these programs are stored in a USB memory 984, and the USB memory 984 is inserted in a USB port 1006 and the programs are transferred to the SSD 1000. Alternatively, the programs may be transmitted to the computer 970 via a network 986 and stored in the SSD 1000.
[0114] The program is loaded into the RAM 998 when executed. Of course, the source program may be input using the keyboard 974, monitor 972, and mouse 976, and the compiled object program may be stored in the SSD 1000. In the case of a script language, a script input using the keyboard 974 or the like may be stored in the SSD 1000. In the case of a program that runs on a virtual machine, a program that functions as a virtual machine must be installed in the computer 970 in advance. Since training and testing of a neural network involves a large amount of calculation, it is preferable to realize each part of the embodiment of the present invention as an object program consisting of native computer code rather than a script language.
[0115] The CPU 990 reads a program from the RAM 998 according to an address indicated by a register (not shown) called a program counter inside the CPU 990, interprets the instruction, reads data required for executing the instruction from the RAM 998, the SSD 1000, or other devices according to an address specified by the instruction, and executes the process specified by the instruction. The CPU 990 stores the execution result data in an address specified by the program, such as the RAM 998, the SSD 1000, or a register in the CPU 990. At this time, the value of the program counter is also updated by the program. The computer program may be directly loaded into the RAM 998 from the DVD 978, the USB memory 984, or via a network. Note that some tasks (mainly numerical calculations) among the programs executed by the CPU 990 are dispatched to the GPU 992 according to instructions included in the program or according to the analysis results when the CPU 990 executes the instructions.
[0116] The program for implementing the functions of each unit according to each embodiment described above by the computer 970 includes a plurality of instructions written and arranged to operate the computer 970 to implement those functions. Some of the basic functions required to execute these instructions are provided by an operating system (OS) or a third-party program running on the computer 970, or by modules of various tool kits installed on the computer 970. Thus, this program does not necessarily include all of the functions required to implement the system and method of this embodiment. This program may include only instructions to perform the operations of each of the above-mentioned devices and their components by statically linking or dynamically calling appropriate functions or functions of a "programming tool kit" in a controlled manner to obtain the desired results. The operation method of the computer 970 for this purpose is well known, so it will not be repeated here.
[0117] The GPU 992 is capable of parallel processing, and can simultaneously execute a large amount of calculations associated with machine learning in parallel or in a pipelined manner. For example, parallel calculation elements found in a program when the program is compiled, or parallel calculation elements found when the program is executed, are dispatched from the CPU 990 to the GPU 992 as needed, and executed, and the results are returned to the CPU 990 directly or via a predetermined address in the RAM 998, and assigned to a predetermined variable in the program.
[0118] 6th Variation 1. Single-track emotion estimation device using gestures The first embodiment relates to a multi-modality emotion estimation system, and the second embodiment relates to an emotion estimation system (FIG. 2) by two-system graph convolution processing using gestures. However, as shown in the above experimental results, emotion estimation can be performed with higher accuracy than the conventional technology even if only the emotion estimation device 160 of the first embodiment is used. In addition, in the case of the second embodiment (FIG. 8), each of the two systems can be used independently as an emotion estimation device.
[0119] 2. Emotion estimation device with added modality In the above embodiment, emotion estimation is performed using a voice signal, text, and skeletal information of the upper body. However, the information to be used is not limited to this. For example, facial expressions or body movements other than those of the upper body may also be used.
[0120] 3. Dimensional Variations The dimensions of each matrix, tensor, and vector given in the above embodiment are just an example. The number of dimensions is not limited to those adopted in the embodiment. Other numbers of dimensions may be used. In addition, the number of attention heads (H) used to calculate self-attention is 4 in the above embodiment. However, the number of heads is not limited to 4, of course. For example, as disclosed in Reference 1, the number of heads may be set to 8, or may be another number.
[0121] 4. Variations in skeletal information In the above embodiment, due to the limitations of available training data, skeletal information is extracted from an image of the speaker's upper body. However, the present invention is not limited to such an embodiment. If an image of the whole body is available, it is also possible to use the image of the whole body. Alternatively, various information can be used, such as skeletal information from the knees up, the lower body only, or separated into the upper and lower body.
[0122] The embodiments disclosed herein are merely illustrative, and the present invention is not limited to the above-described embodiments. The scope of the present invention is defined by the claims of the appended claims, taking into consideration the detailed description of the invention, and includes all modifications within the scope and meaning equivalent to the words described therein. [Explanation of symbols]
[0123] 50, 150 Multimodal emotion estimation device 60, 62, 160, 450 emotion estimation device 66, 164 Emotion estimation results 68, 70 RNN 170 Video Time Series 172 Time series skeleton graph 174, 482, 492 S-STGCN 200 2D pose estimation section 202 3D pose estimation section 250 graphs 350 Complete Skeleton Graphs 460 Skeleton Information 462 Emotion Estimation by Joints 464 Emotion Estimation Using Bone Vectors 466 Integration Department 468 Predictions 480 Joint position information extraction unit 490 Bone Vector Information Extraction Unit 550 Skeleton Graph 552, 554, 556, 558, 560, 562, 564, 566, 568, 570 Convolution Blocks 600 S-SGC layer 606 T-Conv layer 612, 718 Residual calculation section 650 Convolution Section 652 Self-Attention Graph Convolution Part 654 Gating section 670 Head 708 Scale conversion dot product part 714 Connection part 730 2D Convolution Section
Claims
1. a feature calculation means for calculating features of each node of a skeleton graph including a time series of graphs representing a three-dimensional skeleton of a person who is speaking and edges connecting corresponding elements in two graphs at adjacent times in the time series; and an emotion estimation means including a graph neural network that is trained in advance to perform a first graph convolution in spatial and temporal directions using self-attention on the features for each node of a complete graph in which all nodes of the skeletal graph are connected by a single edge, and to estimate the emotion of the person when speaking based on a result of the first graph convolution.
2. The graph neural network of the emotion estimation means includes: A plurality of convolution blocks connected in series; a pooling layer connected to receive a final output of the plurality of convolution blocks; an input tensor having the features arranged so as to represent the complete graph as elements is input to the beginning of the plurality of convolution blocks; Each of the plurality of convolution blocks comprises: a spatial convolution block that performs graph convolution using self-attention in a spatial direction of the complete graph; a time convolution block that performs graph convolution using self-attention in the time direction of the complete graph; the spatial convolution block and the temporal convolution block are connected in series with each other; The emotion estimation device according to claim 1 , wherein the spatial convolution block and the temporal convolution block are both configured to output a tensor having the same format as the input tensor.
3. Each of the plurality of convolution blocks further comprises: a convolution means comprising a graph neural network for performing a second graph convolution on each node of the skeleton graph; 3. The feeling estimation device according to claim 2, further comprising a mixing means for mixing an output of one of the spatial convolution block and the temporal convolution block present in a subsequent stage with an output of the convolution means at a predetermined ratio to obtain an output of the convolution block.
4. The emotion estimation device according to claim 1 , wherein the nodes of the skeletal graph represent joint positions of the skeleton.
5. The emotion estimation device according to claim 4 , wherein the features include three-dimensional coordinate data of the joint positions.
6. The emotion estimation device according to claim 1 , wherein the nodes of the skeletal graph represent bones of the skeleton.
7. The emotion estimation device according to claim 6 , wherein the features include a direction and a length of a vector representing the bone.
8. A first feeling estimation device according to claim 4 or claim 5; A second feeling deduction device according to claim 6 or claim 7; a fully connected layer that receives outputs of the first feeling estimation device and the second feeling estimation device as inputs, the first emotion estimation device, the second emotion estimation device, and the fully connected layer are trained with training data including time-series data of skeletons to which labels indicating emotions have been attached in advance.
9. The video emotion estimation device according to any one of claims 1 to 7, A voice emotion estimation means that is trained in advance to estimate the emotion of the person from voice data of the person while he / she is speaking; A text-based emotion estimation means that is trained in advance to estimate the emotion of the person from text data indicating the content of the utterance of the person; a fully connected layer that receives outputs of the video emotion estimation device, the voice emotion estimation means, and the text emotion estimation means, the video emotion estimation device, the audio emotion estimation means, the text emotion estimation means, and the fully connected layer of the emotion estimation device are further trained with training data consisting of a time series of three-dimensional skeletal data of the person across a plurality of time steps.
10. A computer program causing a computer to function as the emotion estimation device according to any one of claims 1 to 9.
11. A computer programmed to function as the emotion estimation device according to any one of claims 1 to 9.
Citation Information
Patent Citations
Method and system for emotion and behavior recognition
JP2015130151A
Information processor and program
JP2018152004A
Detecting device and detecting method
JP2019071008A
