Estimation method, estimation device, and estimation program

Abstract

In the present invention, an acquisition unit (15a) acquires training data (14a) that includes: nonverbal information or paralanguage information; and correct-answer labels added by a plurality of workers, the correct-answer labels representing the state of mind expressed in the nonverbal information or paralanguage information. A calculation unit (15b) calculates the posterior probability of the state of mind with regard to the acquired nonverbal information or paralanguage information. A training unit (15c) trains, using the training data (14a) and the calculated posterior probability of the state of mind, model parameters (14b) for a model that estimates the state of mind expressed in inputted nonverbal information or paralanguage information.

Description

Technical Field

[0001] The present invention relates to an estimation method, an estimation device, and an estimation program.

Background Art

[0002] Conventionally, research and development have been carried out on technologies for automatically estimating the mental state expressed in non-verbal and para-verbal information such as human voice, face, gestures, etc. For example, in conversations with agents or robots, it is expected to reflect the mental state of the conversation partner when generating their responses, utilize the estimation results as part of mental health care, or quantify the states of participants in web conferences, etc. to make it easier to grasp.

[0003] In general, the estimation of the mental state expressed in such non-verbal and para-verbal information is defined as supervised learning that outputs the posterior probability, etc. of each label representing the defined mental state for inputs such as feature amounts and the data itself extracted from voice and moving images (see Non-Patent Document 1).

[0004] Here, emotion and facial expression recognition are classified into several classes such as normal, happy, sad, surprised, scared, disgusted, angry, contemptuous, etc. Also, a specific degree such as the degree of understanding is classified into arbitrary stages. In supervised learning, labels corresponding to the classes defined in this way are annotated by one or more workers.

[0005] However, such annotation, that is, the work of label assignment, becomes more difficult as the granularity of the label becomes finer. For example, in terms of the degree of understanding, when the granularity is made finer from three stages of not understood, normal, understood to five stages of not understood, slightly not understood, normal, slightly understood, understood, the difficulty of label assignment increases.

[0006] For such fine-grained classification problems, the annotation results by workers may not match locally even if they generally match. For example, regarding the understanding level in five grades, when multiple workers perform annotation, even if all workers agree that the understanding level is low, there may be a slight difference in the evaluation between "not understood" and "slightly not understood". In this case, if the biases such as the fatigue, experience, and judgment criteria of the workers change, the results may also change. In this way, in supervised learning, accurate but incorrect correct labels containing noise will be mixed, which will have an adverse effect on learning and evaluation.

[0007] Therefore, conventionally, label correction techniques such as relabeling are known (see Non-Patent Documents 2 and 3).

Prior Art Documents

Non-Patent Documents

[0008]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0009] However, in the prior art, it has been difficult to accurately correct the labels representing the mental states shown in non-verbal and para-verbal information. For example, in the prior art, only the labeling result of one operator or the maximum value of the voting results of multiple operators is held as a label, and it is hard to say that human knowledge is fully utilized.

[0010] The present invention has been made in view of the above, and an object thereof is to accurately correct the labels representing the mental states shown in non-verbal and para-verbal information.

Means for Solving the Problems

[0011] In order to solve the above-described problems and achieve the object, an estimation method according to the present invention is an estimation method executed by an estimation device, and includes: an acquisition step of acquiring learning data including non-verbal information or para-verbal information and correct labels representing mental states shown in the non-verbal information or para-verbal information assigned by a plurality of workers; a calculation step of calculating a posterior probability of the mental state for the acquired non-verbal information or para-verbal information; and a learning step of learning model parameters of a model for estimating a mental state shown in input non-verbal information or para-verbal information using the learning data and the calculated posterior probability of the mental state.

Effects of the Invention

[0012] According to the present invention, it is possible to accurately correct the labels representing the mental states shown in non-verbal and para-verbal information.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

[0014] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited by this embodiment. Also, in the description of the drawings, the same parts are denoted by the same reference numerals.

[0015] [Configuration of Estimation Device] FIG. 1 is a schematic diagram illustrating a schematic configuration of the estimation device. Also, FIG. 2 is a diagram for explaining the processing of the estimation device. The estimation device 10 of the present embodiment estimates, in five stages, the degree of understanding as a mental state represented by non-verbal and paralanguage information, using a neural network, for a video in which the upper body of a subject, which is non-verbal and paralanguage information, is reflected. The degree of understanding is defined, for example, as 1. not understood, 2. slightly not understood, 3. normal state, 4. slightly understood, 5. understood, and the larger the number, the more understood it is.

[0016] First, as illustrated in FIG. 1, the estimation device 10 of the present embodiment is realized by a general-purpose computer such as a personal computer, and includes an input unit 11, an output unit 12, a communication control unit 13, a storage unit 14, and a control unit 15.

[0017] The input unit 11 is implemented using an input device such as a keyboard or a mouse, and inputs various instruction information such as a processing start to the control unit 15 in response to an input operation by the operator. The output unit 12 is implemented by a display device such as a liquid crystal display, a printing device such as a printer, an information communication device, or the like. The communication control unit 13 is implemented by a NIC (Network Interface Card) or the like, and controls communication between the control unit 15 and an external device such as a server or a device that manages learning data via a network.

[0018] The storage unit 14 is implemented by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. Note that the storage unit 14 may be configured to communicate with the control unit 15 via the communication control unit 13. In the present embodiment, the storage unit 14 stores, for example, learning data 14a used for the estimation process described later, model parameters 14b generated and updated in the estimation process, and the like.

[0019] Here, FIG. 3 is a diagram illustrating the data configuration of the learning data. As shown in FIG. 3, the learning data 14a includes at least video data in which the upper body of the target person appears as non-verbal and para-verbal information, a data ID for identifying each video data, and a correct label representing a mental state such as the degree of understanding appearing in each video data. In the example shown in FIG. 3, a degree of understanding label as a correct label is included.

[0020] The learning data 14a may include a personal ID that can identify an individual, labels representing attributes of a person such as age and gender. Further, if necessary, learning, development, or division into an evaluation set or data expansion of the learning data 14a may be performed.

[0021] Note that preprocessing such as contrast normalization and face detection may be performed so that only a certain area of the video data is used. Also, the codec or the like of the input data (video data) is not particularly limited. Further, the learning data 14a holds the correct label of the degree of understanding updated by the process of the update unit 15d described later.

[0022] Specifically, when estimating the degree of understanding from video data in the estimation process described later, for example, video data in the H264 format recorded at 30 frames per second by a web camera may be resized so that one side becomes 224 pixels. Each of the X pieces of video data is given the personal ID of S subjects, the correct label of the degree of understanding given by A names, and the correct label of the degree of understanding updated by the process of the update unit 15d. In the example shown in FIG. 3, the updated correct label includes the updated degree of understanding label.

[0023] Returning to the description of FIG. 1. The control unit 15 is realized using a CPU (Central Processing Unit), an NP (Network Processor), an FPGA (Field Programmable Gate Array), or the like, and executes a processing program stored in a memory. Thereby, the control unit 15 functions as an acquisition unit 15a, a calculation unit 15b, a learning unit 15c, and an update unit 15d as illustrated in FIG. 1. Note that these functional units may be implemented on different hardware. For example, the acquisition unit 15a may be implemented on hardware different from other functional units. Also, the control unit 15 may include other functional units.

[0024] The acquisition unit 15a acquires learning data 14a including non-verbal information or para-verbal information and a correct label representing the mental state represented by the non-verbal information or para-verbal information given by a plurality of operators. Specifically, the acquisition unit 15a acquires, via the input unit 11 or via the communication control unit 13 from a device that generates learning data or the like, video data in which the upper body of a subject appears as non-verbal / para-verbal information, a data ID for identifying each video data, and a correct label representing a mental state such as the degree of understanding appearing in each video data, as the learning data 14a.

[0025] The acquisition unit 15a stores the learning data 14a acquired in advance prior to the following processing in the storage unit 14. Note that the acquisition unit 15a may transfer the acquired learning data 14a to the calculation unit 15b shown below without storing it in the storage unit 14.

[0026] The calculation unit 15b calculates the posterior probability of the mental state for the acquired non-verbal information or para-verbal information. For example, the calculation unit 15b calculates the posterior probability regarding the mental state such as the degree of understanding represented in the video data, which is the event to be predicted, for the video data of the learning data 14a using a neural network.

[0027] Note that the processing using the neural network described below is not limited to this embodiment. For example, elements of well-known techniques such as Batch Normalization, dropout, L1 / L2 regularization, etc. may be added at arbitrary locations.

[0028] Specifically, the calculation unit 15b extracts the feature amount for each frame from the video data x with the frame length T 1:T using a 2D CNN (Convolutional Neural Network). Next, the calculation unit 15b calculates the embedding representation tensor H in the time direction as shown in the following equation (1) using an RNN (Recurrent Neural Network) having an output dimension of D. Here, θ is the set of parameters of the CNN, and φ is the set of parameters of the RNN. x

[0029]

Equation

[0030] Next, the calculation unit 15b calculates the time to be focused on in the time direction and calculates the weighted sum vector v in the time direction using a multi-head self-attention mechanism as shown in the following equation (2).

[0031] ​ [Number]

[0032] In the above formula (2), the calculation unit 15b calculates the attention weight from query Q i and key K i and applies it to value V i , and finally calculates the total in the time direction.

[0033] Here, d is the number of attention heads, i is each attention head, and W i Q , W i K , W i V represent the weights for Query, key, and value in each attention head, respectively.

[0034] Finally, the calculation unit 15b calculates the posterior probability p(C|x 1:T ) for each of the five levels of understanding using a two-layer fully connected layer as shown in the following formula (3).

[0035] [Number]

[0036] Here, W1 FC , W2 FC represent the weights of the two-layer fully connected layer, D FC represents the output dimension number of the first-layer fully connected layer, and C represents the number of prediction labels (C = 5 in this embodiment). Also, the ReLU function is used as the activation function of the first-layer fully connected layer.

[0037] The learning unit 15c learns the model parameters 14b of the model that estimates the mental state represented by the input non-verbal information or para-verbal information using the learning data 14a and the calculated posterior probability of the mental state.

[0038] Specifically, the learning unit 15c updates the model parameter set Ω and obtains the learned model parameter set Ω'. The learning unit 15c can apply well-known loss functions and update methods. For example, the model parameter set Ω may include those pre-trained for any other task, or the initial values may be generated by any random numbers, or some model parameters may not be updated.

[0039] For example, the learning unit 15c uses the Stochastic Gradient Descent (SGD) method to update the model parameter set Ω with the cross-entropy L shown in the following equation (4) as the loss function. At this time, arbitrary values are used for hyperparameters such as the learning rate.

[0040]

Equation

[0041] Here, m x is the correct distribution of the input video data x 1:T . The expression method of the correct distribution is not particularly limited. For example, it may be expressed as a one-hot vector using the understanding label L x illustrated in FIG. 3. Alternatively, the correct distribution may be represented by approximating a normal distribution centered on the correct class, or the annotation result may be directly used as a soft-label for representation.

[0042] Note that the learning unit 15c stores the obtained learned model parameter set Ω' in the storage unit 14 as the model parameter 14b.

[0043] Returning to the description of FIG. 1, the update unit 15d updates the correct label of the training data 14a using the learned model parameter 14b. Specifically, the update unit 15d updates the correct label of the training data 14a when the similarity between the posterior probability of the mental state calculated using the learned model parameter 14b and the correct label of the training data 14a is equal to or greater than a predetermined threshold.

[0044] For example, the update unit 15d updates the correct label L obtained by normalizing the distribution of the labels of the learning data 14a given by a plurality of workers. First, the update unit 15d predicts the posterior probability of the degree of understanding for the learning data 14a using the learned model parameter set Ω'. Then, the update unit 15d calculates the similarity between the correct label and the posterior probability, and updates the correct label if the calculated similarity is equal to or greater than a predetermined threshold value.

[0045] When the update unit 15d updates the correct label L of the degree of understanding of the input video data x using the learned model parameter Ω', first, the posterior probability p(C|x x ,Ω') for each degree of understanding is calculated. Next, the update unit 15d calculates the similarity between L 1:T and p(C|x x ,Ω'). The similarity calculated by the update unit 15d is not particularly limited. For example, it is calculated using an algorithm capable of calculating the distance or similarity between vectors, such as cross entropy, Kullback–Leibler divergence, cosine similarity, Euclidean distance, etc. 1:T ,Ω')

[0046] For example, as shown in the following formula (5), the update unit 15d calculates the cosine similarity c x (-1 ≦ c x ≦ 1).

[0047]

Equation

[0048] In addition, the update unit 15d can also determine whether to update based on whether a specific condition is satisfied. For example, the update unit 15d may set the update determination for L 1:T to TRUE when max(p(C|x x ,Ω')) is equal to or greater than a predetermined threshold value.

[0049] Alternatively, if the top two values of Lx are adjacent and the ratio is within the range of 4:6 to 6:4, the update determination when the correct answer changes within those two classes may be set to TRUE. For example, in the comprehension label of the data with data ID = 0000002 shown in FIG. 3, the top two values (0.4, 0.6) are adjacent and the ratio is within the range of 4:6 to 6:4, so the update determination is set to TRUE. On the other hand, in the comprehension label of the data with data ID = 0001459, although the top two values (0.8, 0.2) are adjacent, the ratio is not within the range of 4:6 to 6:4, so the update determination is set to FALSE.

[0050] Next, the update unit 15d determines whether the calculated similarity is equal to or greater than a predetermined threshold value. If it is true, L x is updated, and after the updated comprehension label L x ’, p(C|x 1:T ,Ω’) is substituted. At this time, the update unit 15d may perform update determination based on a single condition, or may perform update determination by combining a plurality of AND conditions or OR conditions.

[0051] In addition, the update unit 15d may perform preprocessing such as normalizing a small value less than 0.1 to 0 again, and then substitute p(C|x x ’,Ω’) into L 1:T ’. L x ’ is used in the calculation of the loss function instead of L x as the correct label when continuously learning the model parameter 14b.

[0052] Note that the process of the update unit 15d can be activated at any timing of the learning of the model parameter 14b by the learning unit 15c. For example, the update unit 15d may be configured to activate the process when the number of updates of the model parameter 14b by the learning unit 15c reaches a predetermined threshold value or more. Alternatively, the update unit 15d may be more complex. For example, the first activation of the process may be performed after 1000 updates of the model parameter 14b by the learning unit 15c, and the activation of the process after the second time may be performed after 100 updates of the model parameter 14b by the learning unit 15c.

[0053] [Estimation Process] Next, the estimation process by the estimation device 10 will be described. FIGS. 4 and 5 are flowcharts showing the estimation process procedure. The estimation process of this embodiment includes a learning process and an update process. First, FIG. 4 shows the learning process procedure. The flowchart of FIG. 4 starts, for example, at the timing when an input instructing the start of the learning process is received.

[0054] First, the acquisition unit 15a acquires learning data 14a including non-verbal information or paralanguage information and a correct label representing the mental state represented by the non-verbal information or paralanguage information given by a plurality of operators (step S1). The acquisition unit 15a stores the acquired learning data 14a in the storage unit 14. Alternatively, the acquisition unit 15a may transfer the acquired learning data 14a to the calculation unit 15b without storing it in the storage unit 14.

[0055] Also, the calculation unit 15b calculates the posterior probability of the mental state for the acquired non-verbal information or paralanguage information (step S2).

[0056] Next, the learning unit 15c learns the model parameters 14b of a model for estimating the mental state represented by the input non-verbal information or paralanguage information using the learning data 14a and the calculated posterior probability of the mental state (step S3). Thereby, a series of learning processes is completed.

[0057] Next, FIG. 5 shows the update process procedure. The flowchart of FIG. 5 starts, for example, at the timing when an input instructing the start of the update process is received.

[0058] First, the update unit 15d calculates the posterior probability of the degree of understanding with respect to the learning data 14a using the learned model parameters 14b (step S11).

[0059] Next, when the similarity between the calculated posterior probability of the mental state and the correct label of the learning data 14a is equal to or greater than a predetermined threshold, the update unit 15d updates the correct label of the learning data 14a (step S12). Thereby, a series of update processes is completed.

[0060] [Effect] As described above, in the estimation device 10 of the present embodiment, the acquisition unit 15a acquires learning data 14a including non-verbal information or para-verbal information and a correct label representing the mental state represented by the non-verbal information or para-verbal information given by a plurality of workers. The calculation unit 15b calculates the posterior probability of the mental state for the acquired non-verbal information or para-verbal information. The learning unit 15c learns the model parameters 14b of the model for estimating the mental state represented by the input non-verbal information or para-verbal information using the learning data 14a and the calculated posterior probability of the mental state.

[0061] Thereby, the estimation device 10 can accurately estimate the mental state represented by the non-verbal information or para-verbal information by learning global labeling using the correct labels given by a plurality of persons. Therefore, the estimation device 10 can accurately assign a label representing the mental state using the estimated result. Thus, according to the estimation device 10, it is possible to accurately correct the label representing the mental state represented by the non-verbal / para-verbal information.

[0062] Further, the update unit 15d updates the correct label of the learning data 14a using the learned model parameters 14b. Specifically, the update unit 15d updates the correct label of the learning data 14a when the similarity between the posterior probability of the mental state calculated using the learned model parameters 14b and the correct label of the learning data 14a is equal to or greater than a predetermined threshold.

[0063] As a result, the estimation device 10 can correct the correct label only when the distribution of the correct label assignment results by multiple persons is somewhat similar. Therefore, it is possible to eliminate the possibility of erroneously assigning a label of a class with a distant meaning and to locally correct the label without changing the global labeling. Also, by referring to the tendency of human labeling, certain restrictions are imposed, and it becomes possible to stably correct the label with a small amount of data. Thus, according to the estimation device 10, it is possible to accurately correct the label even for data for which labeling is difficult.

[0064] When the number of update times due to the learning of the model parameter 14b in the learning unit 15c reaches a predetermined threshold or more, the process of the update unit 15d is activated. As a result, it becomes possible to correct the label with higher accuracy.

[0065] [Program] It is also possible to create a program that describes the process executed by the estimation device 10 according to the above embodiment in a computer-executable language. As one embodiment, the estimation device 10 can be implemented by installing an estimation program that executes the above-described estimation process as package software or online software on a desired computer. For example, by causing the information processing device to execute the above-described estimation program, the information processing device can be made to function as the estimation device 10. In addition, the information processing device includes mobile communication terminals such as smartphones, mobile phones, and PHSs (Personal Handyphone System), and further slate terminals such as PDAs (Personal Digital Assistant) in its category. Also, the function of the estimation device 10 may be implemented in a cloud server.

[0066] FIG. 6 is a diagram showing an example of a computer that executes an estimation program. The computer 1000 includes, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0067] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System), for example. The hard disk drive interface 1030 is connected to a hard disk drive 1031. The disk drive interface 1040 is connected to a disk drive 1041. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1041, for example. A mouse 1051 and a keyboard 1052 are connected to the serial port interface 1050, for example. A display 1061 is connected to the video adapter 1060, for example.

[0068] Here, the hard disk drive 1031 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. Each piece of information described in the above embodiment is stored in the hard disk drive 1031 or the memory 1010, for example.

[0069] Also, the estimation program is stored in the hard disk drive 1031 as a program module 1093 in which instructions executed by the computer 1000 are described, for example. Specifically, a program module 1093 in which each process executed by the estimation device 10 described in the above embodiment is described is stored in the hard disk drive 1031.

[0070] In addition, the data used for information processing by the estimation program is stored, for example, as program data 1094 in the hard disk drive 1031. Then, the CPU 1020 reads out the program module 1093 and the program data 1094 stored in the hard disk drive 1031 into the RAM 1012 as necessary, and executes each of the above-described procedures.

[0071] Note that the program module 1093 and the program data 1094 related to the estimation program are not limited to being stored in the hard disk drive 1031. For example, they may be stored in a removable storage medium and read out by the CPU 1020 via the disk drive 1041 or the like. Alternatively, the program module 1093 and the program data 1094 related to the estimation program may be stored in another computer connected via a network such as a LAN (Local Area Network) or a WAN (Wide Area Network), and read out by the CPU 1020 via the network interface 1070.

[0072] As described above, the embodiments to which the invention made by the present inventor is applied have been described. However, the present invention is not limited by the description and the drawings that form a part of the disclosure of the present invention according to the present embodiment. That is, all other embodiments, examples, operation techniques, etc. made by those skilled in the art based on the present embodiment are included in the scope of the present invention.

Explanation of Reference Numerals

[0073] 10 Estimation device 11 Input unit 12 Output unit 13 Communication control unit 14 Storage unit 14a Learning data 14b Model parameter 15 Control unit 15a Acquisition unit 15b Calculation unit 15c Learning unit 15d Update unit

Claims

1. An estimation method executed by an estimation device, comprising: an acquisition step of acquiring learning data including non-verbal information or para-verbal information and a correct label representing a mental state represented by the non-verbal information or para-verbal information assigned by a plurality of workers; a calculation step of calculating a posterior probability of the mental state for the acquired non-verbal information or para-verbal information; a learning step of learning model parameters of a model for estimating a mental state represented by input non-verbal information or para-verbal information using the learning data and the calculated posterior probability of the mental state; an update step of updating the correct label of the learning data using the learned model parameters; The estimation method characterized by including the above.

2. The update step updates the correct label of the learning data when a similarity between the posterior probability of the mental state calculated using the learned model parameters and the correct label of the learning data is equal to or greater than a predetermined threshold. The estimation method according to Claim 1.

3. The estimation method according to Claim 1, characterized in that the update step is activated when the number of updates by learning of the model parameters in the learning step reaches a predetermined threshold or more.

4. An estimation device comprising: an acquisition unit that acquires learning data including non-verbal information or para-verbal information and a correct label representing a mental state represented by the non-verbal information or para-verbal information assigned by a plurality of workers; a calculation unit that calculates a posterior probability of the mental state for the acquired non-verbal information or para-verbal information; a learning unit that learns model parameters of a model for estimating a mental state represented by input non-verbal information or para-verbal information using the learning data and the calculated posterior probability of the mental state; an update unit that updates the correct label of the learning data using the learned model parameters; The estimation device characterized by including the above.

5. An acquisition step of acquiring learning data including non-verbal information or para-verbal information and a correct label representing a mental state represented by the non-verbal information or para-verbal information assigned by a plurality of workers; a calculation step of calculating a posterior probability of the mental state for the acquired non-verbal information or para-verbal information; A learning step of learning model parameters of a model that estimates a mental state represented by input non-verbal information or para-verbal information using the learning data and the calculated posterior probability of the mental state; An update step of updating the correct label of the learning data using the learned model parameters; An estimation program for causing a computer to execute the above.

Images