Emotional inference device

Abstract

To provide technology for quickly estimating emotion of a speaker.SOLUTION: An emotion estimation device includes: an acquisition unit for acquiring sound information including utterances of a speaker; a segmentation unit for segmenting utterances of a speaker into a series of groups from sound information; and an estimation unit for estimating emotion of the speaker by inputting a part of the group starting from the top group of utterance into a learning model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technique for estimating the emotion of a speaker. [Background technology]

[0002] Patent Document 1 discloses an information transmission device that includes a feature extraction unit that extracts feature values ​​of a series of utterances by a speaker based on an acoustic signal detected by a microphone; a voice signal generation unit that generates a voice signal to be uttered so as to have the same feature values ​​based on the extracted feature values; a voice output unit that speaks the voice signal generated by the voice signal generation unit; an emotion estimation unit that calculates feature amounts used to estimate emotions from the feature values ​​and estimates the speaker's emotions based on these feature amounts; and a first color output unit that displays a color corresponding to the emotion estimated by the emotion estimation unit in synchronization with the voice output from the voice output unit. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-113546 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 analyzes a series of utterances in order to estimate the speaker's emotions, which takes time.

[0005] An object of the present invention is to provide a technique that can quickly estimate the emotion of a speaker. [Means for solving the problem]

[0006] In order to solve the above problem, a feeling estimation device according to an embodiment of the present invention includes: an acquisition unit that acquires sound information including an utterance of a speaker; a segmentation unit that segments the utterance of the speaker from the sound information into a series of blocks; and an estimation unit that estimates the emotion of the speaker by inputting a portion of the block starting from the beginning of the block of utterance into a learning model. The estimation unit divides the speech chunk at predetermined time intervals, estimates the speaker's emotion value for each divided area in order from the beginning of the speech chunk, and when the total number of estimated divided areas exceeds a predetermined proportion of the speech chunk, terminates the emotion estimation for that speech chunk, and when there are multiple emotion estimation results for the divided areas, averages the emotion values ​​to estimate the emotion. [Effects of the Invention]

[0007] According to the present invention, a technique that can quickly estimate the emotion of a speaker can be provided. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 10 is a diagram showing an image displayed on a terminal device when multiple people are having a conversation. [Figure 2] FIG. 1 is a diagram illustrating a functional configuration of an emotion estimation system. [Figure 3] 10 is a flowchart of an emotion estimation process according to an embodiment. [Figure 4] 10 is a flowchart of a process for training a learning model for emotion estimation. DETAILED DESCRIPTION OF THE INVENTION

[0009] 1 shows an image displayed on the terminal device 12 when multiple people are having a conversation. The terminal device 12 has a display and a speaker, and outputs the voice of the conversation while displaying the images of the participants in the conversation.

[0010] 1 shows a speaker image 40 and an interlocutor image 44, depicting an image of two people having a conversation. The interlocutor shown in the interlocutor image 44 is the owner of the terminal device 12. The display of the terminal device 12 also shows an emotion display image 42, which is the result of estimating the emotion of the speaker 40a, and indicates that the emotion of the speaker 40a is positive.

[0011] In this way, the emotion estimation system estimates the emotion of the speaker 40a and notifies the interlocutor. When conversing via the terminal device 12, it can be more difficult to grasp the other person's emotion than when conversing face-to-face, but the emotion estimation system can help the conversation proceed smoothly by estimating the emotion of the conversation partner.

[0012] FIG. 2 shows the functional configuration of the emotion estimation system 1. Each function of the emotion estimation system 1 can be configured in hardware using circuit blocks, memory, and other LSIs, and is realized in software using system software or application programs loaded into memory. Therefore, it will be understood by those skilled in the art that each function of the emotion estimation system 1 can be realized in various forms using only hardware, only software, or a combination thereof, and is not limited to any one of them.

[0013] The emotion estimation system 1 includes an emotion estimation device 10 and a terminal device 12. The terminal device 12 is connected to the emotion estimation device 10 via a network, and although only one terminal device 12 is shown in FIG. 2, the system is actually made up of multiple terminal devices.

[0014] The terminal device 12 includes an application program for linking with the emotion estimation device 10. The terminal device 12 includes a camera 14, a microphone 15, a speaker state detection sensor 16, an output unit 17, and a communication unit 18.

[0015] The camera 14 captures images of the periphery of the terminal device 12 and captures images of the speaker. The camera 14 can acquire captured images arranged in chronological order. The microphone 15 collects sounds around the terminal device 12 and acquires sound information including the speaker's speech. The detection results of the camera 14 and the microphone 15 are video with sound. The speaker state detection sensor 16 detects the speaker's state, such as the speaker's heart rate and brain waves. The detection results of the camera 14, the microphone 15, and the speaker state detection sensor 16 are transmitted from the communication unit 18 to the emotion estimation device 10. The detection results of the camera 14, the microphone 15, and the speaker state detection sensor 16 are assigned timestamps and associated based on the timestamps. The user ID of the terminal device 12 is assigned to the data transmitted from the communication unit 18, allowing the emotion estimation device 10 to distinguish which terminal device 12 transmitted the data.

[0016] The output unit 17 is, for example, a display, and displays the emotion estimation result received from the emotion estimation device 10. The emotion estimation result may be displayed as text as shown in FIG. 1, or may be displayed as a color or a picture. For example, if the speaker's emotion is positive, it is displayed in yellow or with a smiling face icon; if it is negative, it is displayed in red or with an angry face icon; and if it is normal, it is displayed in a neutral color, orange, or with a neutral icon. In this way, the speaker's emotion is displayed in text, color, and / or a picture. Furthermore, the speaker's emotion is not limited to three levels, positive, normal, and negative, and may be set to more levels.

[0017] A communication unit 18 of the terminal device 12 transmits detection results from the camera 14, the microphone 15, and the speaker state detection sensor 16 to the feeling estimation device 10, and receives a speaker's emotion estimation result from the feeling estimation device 10.

[0018] The feeling estimation device 10 includes an acquisition unit 20, an estimation processing unit 22, a learning processing unit 24, and an output control unit 38. The acquisition unit 20 acquires a captured image of a speaker from a camera 14, acquires sound information including the speaker's voice from a microphone 15, and acquires speaker state information from a speaker state detection sensor 16. In other words, the acquisition unit 20 acquires information about the speaker.

[0019] The estimation processing unit 22 receives information about the speaker from the acquisition unit 20 and executes processing to estimate the emotion of the speaker. The estimation processing unit 22 includes a division unit 26, an estimation unit 28, and an additional processing unit 30.

[0020] The segmentation unit 26 extracts speech from sound information and segments the extracted speech into a series of chunks. A series of chunks in a speech is a unit into which the speech is segmented, and may be segmented by sentence or clause. Alternatively, a series of chunks in a speech may be segmented not by sentence or clause, but at the timing when the speech is interrupted for a predetermined period of time, for example, at the timing when the speech is interrupted for one second or more. In this way, the segmentation unit 26 segments the extracted speech into a series of chunks based on a predetermined rule. Alternatively, the segmentation unit 26 may input the extracted speech into a learning model using a neural network technique and segment the speech into a series of chunks. Segmenting the speech by the timing of interruptions is easy because it does not require understanding the content of the utterance.

[0021] The segmentation unit 26 segments an utterance into chunks, sends a portion of the segmented utterance chunk to the estimation unit 28, and does not send the remainder of the utterance chunk. A portion of a speech chunk refers to a portion of the utterance chunk from the beginning up to a predetermined percentage. The predetermined percentage may be any value between 10% and 50%, and is set to an appropriate value through experiments, etc. If the predetermined percentage is 30%, the segmentation unit 26 may send 30% of the utterance chunk to the estimation unit 28, and not send the remaining 70% to the estimation unit 28. This allows the speaker's emotions to be estimated efficiently and the estimation result to be output quickly. Whether the predetermined percentage is reached or exceeded may be determined by time.

[0022] The segmentation unit 26 may divide an utterance chunk every second, and send a predetermined number or a predetermined ratio of divided regions from the beginning of the chunk to the estimation unit 28. Before dividing the utterance into chunks, the segmentation unit 26 may divide the utterance every predetermined number of seconds and send them to the estimation unit 28.

[0023] The segmentation unit 26 may send a part of an utterance chunk to the estimation unit 28 if the utterance chunk contains multiple words, and may send the entire utterance chunk to the estimation unit 28 if the utterance chunk contains one word. Furthermore, the segmentation unit 26 may send a part of an utterance chunk to the estimation unit 28 if the utterance chunk is longer than a predetermined number of seconds.

[0024] Part of the utterance chunks sent by the segmentation unit 26 to the estimation unit 28 may include a series of associated captured images, or may send a video to the estimation unit 28. In addition, the segmentation unit 26 may send to the estimation unit 28 the detection result of the speaker state detection sensor 16 associated with the utterance.

[0025] The estimation unit 28 inputs a portion of the beginning of the utterance block received from the classification unit 26 into a learning model and outputs an emotion estimation result. The learning model is generated using a machine learning technique, takes the utterance as input, and outputs an emotion value. A larger emotion value indicates a more positive emotion, and a smaller emotion value indicates a more negative emotion. In this way, the estimation unit 28 estimates the speaker's emotion by inputting a portion of the beginning of the utterance block into the learning model. This reduces the processing load compared to when the estimation unit 28 estimates the emotion of the entire utterance, and enables the emotion estimation result to be output more quickly. Furthermore, because emotions are more likely to be expressed at the beginning of an utterance, using information from the beginning of the utterance can prevent the accuracy of the estimation result from decreasing even when only a portion of the utterance is used.

[0026] The data input to the learning model may include not only videos but also the detection results of the speaker state detection sensor 16. The estimation unit 28 associates and stores the video corresponding to the utterance and the estimation result of its emotion, and uses these as learning data for the learning processing unit 24. The estimation result of the emotion is associated with the video by a timestamp.

[0027] For example, if the predetermined percentage is 30% and the speech chunk is 10 seconds long, the estimation unit 28 completes emotion estimation by outputting emotion values ​​for the speech divided into 1-second segments three times. The estimation unit 28 may derive the speaker's emotion value for each 1-second segment of the speech and average the derived emotion values ​​to obtain the estimation result. The estimation unit 28 divides the speech chunk into predetermined time segments, sequentially estimating emotions for each segment starting from the beginning of the speech chunk, and terminates emotion estimation for the speech chunk when the total number of estimated segments exceeds the predetermined percentage of the speech chunk. If there are multiple emotion estimation results for the speech chunk, the estimation unit 28 averages these estimation results.

[0028] The additional processing unit 30 determines to display an emotion display image 42 according to the emotion value of the speaker derived by the learning model, and generates information related to the emotion display image 42. For example, the additional processing unit 30 determines to display one of the emotion display images 42, "positive," "normal," or "negative," according to the emotion value of the speaker. The information related to the emotion display image 42 may be the emotion display image 42 itself, or may be identification information indicating the emotion display image 42.

[0029] The output control unit 38 sends information about the emotion-displaying image 42 generated by the additional processing unit 30 to the terminal device 12 and controls the terminal device 12 to output the information. As a result, the output unit 17 displays the emotion-displaying image 42 shown in FIG.

[0030] The learning processing unit 24 receives an utterance as input and generates a learning model that outputs the emotion value of the utterance. The learning processing unit 24 includes a data generation unit 32, a feature extraction unit 34, and a model generation unit 36.

[0031] The data generation unit 32 receives information about the speaker from the acquisition unit 20, receives the emotion estimation result from the estimation processing unit 22, and generates utterances tagged with the emotion estimation result as training data. The training data includes a series of audio information, captured images, and emotion values. The training data may also include the detection results of the speaker state detection sensor 16.

[0032] The feature extraction unit 34 extracts speech features such as MFCC (Mel Frequency Cepstrum Coefficients), pitch, and line spectrum pairs from the training data generated by the data generation unit 32, and includes the extracted speech features in the training data.

[0033] The model generation unit 36 ​​updates the learning model using the learning data. The learning model is designed based on a machine learning technique based on gradient boosting, and for example, uses LightGBM (Light Gradient Boosting Machine) as a framework, with the learning data as the explanatory variable and the emotional value of the utterance as the objective variable. The updated learning model is used by the estimation processing unit 22. In this way, the learning processing unit 24 updates the learning model based on the estimation result of the estimation processing unit 22.

[0034] FIG. 3 is a flowchart of the emotion estimation process according to the embodiment. The acquisition unit 20 acquires captured images and sound information of the speaker, i.e., video, from the terminal device 12 (S10). The segmentation unit 26 extracts speech from the sound information (S12) and segments the speech into chunks (S14). The segmentation unit 26 segments the extracted speech into sentences or clauses to create speech chunks. The segmentation unit 26 divides the speech chunks into one-second intervals and sends them to the estimation unit 28 in order, starting from the beginning of the speech (S16).

[0035] The estimation unit 28 inputs a part of the utterance chunk into the learning model (S18), outputs the emotion value of the utterance, and estimates the emotion (S20). The estimation unit 28 inputs the utterance every second into the learning model and outputs the emotion value.

[0036] The estimation unit 28 determines whether the total duration of the utterances input to the learning model exceeds a predetermined percentage of the total duration of the utterance chunk (S22). If the total duration of the utterances input to the learning model does not exceed the predetermined percentage of the total duration of the utterance chunk (N in S22), the estimation unit 28 repeats emotion estimation of the utterances every second (S18).

[0037] If the total duration of the utterances input to the learning model exceeds a predetermined percentage of the total duration of the utterance chunk (Y in S22), the estimation unit 28 averages the emotion values ​​of the utterances per second and calculates the average emotion value as the emotion estimation result (S24). This allows the emotion to be estimated using only a portion of the utterance chunk, allowing the estimation result to be output quickly. The estimation unit 28 stores the utterance chunk and the emotion value as the estimation result (S26).

[0038] 4 is a flowchart of the process of training a learning model for emotion estimation. The data generation unit 32 of the learning processing unit 24 acquires the captured image and sound information of the speaker, i.e., the video with audio, from the acquisition unit 20 (S30), and acquires the emotion estimation result from the estimation processing unit 22 (S32). The data generation unit 32 generates learning data using the video with audio as a setting variable and the emotion value as a target variable (S34).

[0039] The feature extraction unit 34 extracts features from the utterances included in the training data and includes them in the training data (S36). The model generation unit 36 ​​updates the training model based on the training data (S38).

[0040] The present disclosure has been described above based on examples. The present disclosure is not limited to the above examples, and various modifications such as design changes may be made based on the knowledge of those skilled in the art.

[0041] For example, in the embodiment, the feeling estimation device 10 includes the estimation processing unit 22 and the learning processing unit 24, but this is not limiting. For example, the estimation processing unit 22 and the learning processing unit 24 may be included in separate server devices.

[0042] Furthermore, in the embodiment, the estimation processing unit 22 is provided in the feeling estimation device 10, but the present invention is not limited to this, and the estimation processing unit 22 may be provided in the terminal device 12. [Explanation of symbols]

[0043] 1 emotion estimation system, 10 emotion estimation device, 12 terminal device, 14 camera, 15 microphone, 16 speaker state detection sensor, 17 output unit, 18 communication unit, 20 acquisition unit, 22 estimation processing unit, 24 learning processing unit, 26 classification unit, 28 estimation unit, 30 additional processing unit, 32 data generation unit, 34 feature extraction unit, 36 model generation unit, 38 output control unit, 40 speaker image, 40a speaker, 42 emotion display image, 44 interlocutor image.

Claims

[Claim 1] an acquisition unit that acquires sound information including an utterance of a speaker; a segmentation unit that segments a speaker's utterance into a series of chunks based on the sound information; an estimation unit that estimates the emotion of a speaker by inputting a part of the chunk of speech starting from the beginning of the chunk of speech into a learning model, The estimation unit divides the chunk of speech at predetermined time intervals, estimates the emotion value of the speaker for each divided area starting from the beginning of the chunk of speech, terminates the estimation of emotion for the chunk of speech when the total number of estimated divided areas exceeds a predetermined proportion of the chunk of speech, and when there are multiple emotion estimation results for the divided areas, averages the emotion values ​​to estimate the emotion.

Images