Facial expression conversion device, facial expression conversion method, and program

Abstract

This facial expression conversion device for converting facial expression data indicating a listener's facial expression comprises: an emotion inference unit that infers the emotion of a speaker on the basis of text indicating a content spoken by the speaker; a feature point change amount calculation unit that calculates the change amount of a feature point corresponding to the inferred emotion; and a facial expression conversion unit that outputs converted facial expression data obtained by changing the feature point of the facial expression data on the basis of the calculated change amount.

Description

[Technical field]

[0001] The present invention relates to a facial expression conversion device, a facial expression conversion method, and a program. [Background technology]

[0002] A technology that converts facial expressions based on emotions has been developed. For example, Non-Patent Document 1 discloses a technology that converts the facial expression of a listener into a smile during a video conference. In this technology, the speaker is always shown a smile, which evokes a pleasant emotion from the speaker toward the listener, making communication smoother. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Sakurai, S., Nakazato, N., Yoshida, S., Narumi, T., Tanigawa, T., & Hirose, M. (2015). Creativity Improvement Support in Remote Collaborative Work Using Facial Expression Feedback (Special Issue: Education, Training, and Support). Transactions of the Virtual Reality Society of Japan, 20(4), 323-332. Summary of the Invention [Problem to be solved by the invention]

[0004] In technology that always converts the listener's facial expression into a smile, the smile often does not match the text that the speaker is speaking. For example, if the speaker smiles while talking about something that the listener does not want to smile about, such as self-deprecating content or a sad episode, the speaker will not evoke a pleasant feeling toward the listener, and communication will be hindered.

[0005] The disclosed technology aims to change the facial expression of a listener to match what the speaker is saying. [Means for solving the problem]

[0006] The disclosed technology is a facial expression conversion device for converting facial expression data indicating a listener's facial expression, the device including: an emotion estimation unit that estimates an emotion of a speaker based on a text indicating the content of speech by the speaker; a feature point change amount calculation unit that calculates an amount of change in a feature point corresponding to the estimated emotion; and a facial expression conversion unit that outputs converted facial expression data in which the feature points of the facial expression data are changed based on the calculated amount of change. The emotion estimation unit calculates a score for each emotion expressed by the text and estimates the emotion as a combination of emotions in proportions according to the calculated scores. The feature point change amount calculation unit calculates an amount of change by multiplying an amount of change for each emotion in feature point change amount data indicating an amount of change in feature points of facial expressions corresponding to each emotion, using an emotionless facial expression as a reference, by the proportion of each estimated emotion, and calculates an amount of change by summing up the calculated amounts of change for all emotions as an amount of change in feature points corresponding to the estimated emotion. It is a facial expression conversion device. Effect of the Invention

[0007] The listener's facial expression can be changed to match what the speaker is saying. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a functional configuration diagram of the facial expression conversion device. [Diagram 2] 13 is a flowchart showing an example of the flow of a facial expression conversion process. [Diagram 3] FIG. 2 illustrates an example of a hardware configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0010] (Outline of this embodiment) The facial expression conversion device according to the present embodiment is a device for converting facial expression data showing the facial expression of a listener into facial expression data showing a facial expression that matches the content of what the speaker is saying. Specifically, the facial expression conversion device estimates the emotion expressed by the speaker, converts the facial expression of the listener into an expression that represents the estimated emotion, and presents it to the speaker.

[0011] (Example of functional configuration of facial expression conversion device) 1 is a functional block diagram of a facial expression conversion device 10. The facial expression conversion device 10 includes a storage unit 11, a voice recognition unit 12, a feeling estimation unit 13, a feature point change amount calculation unit 14, and a facial expression conversion unit 15.

[0012] The storage unit 11 stores the feature point change amount data 100. The feature point change amount data 100 is data indicating the amount of change in the feature points of facial expressions corresponding to each emotion based on a neutral facial expression, and is stored in advance. For example, the feature point change amount data 100 may be generated by collecting images of a listener making an expression of each emotion and calculating the amount of change in the feature points from the images. The feature point change amount data 100 may be data generated based on the listener's facial expression, or may be data generated based on the facial expressions of other people who have attributes or characteristics similar to the listener, or on statistics of the facial expressions of an unspecified number of people.

[0013] An example of feature point change amount data 100 is data showing that, for an emotionless facial expression, the feature points at the corners of the mouth rise by 4.3 mm for the emotion of “happiness”, rise by 1.2 mm for the emotion of “goodwill”, fall by 2.4 mm for the emotion of “fear”, fall by 3.2 mm for the emotion of “sadness”, and fall by 1.6 mm for the emotion of “anger”.

[0014] The speech recognition unit 12 acquires speech data indicating a speech uttered by a speaker and converts the acquired speech data into text. The converted text is an example of text indicating the contents of what the speaker said. The speech recognition unit 12 may apply a technique such as that described in reference [1] as a speech recognition method.

[0015] The emotion estimation unit 13 estimates the emotion expressed by the text indicating the contents of what the speaker said. An example of the text indicating the contents of what the speaker said is a text converted from the voice uttered by the speaker by the voice recognition unit 12. Alternatively, the text indicating the contents of what the speaker said may be a text written by the speaker himself or a text written down after listening to someone else speak.

[0016] The emotion estimation unit 13 may apply a technique such as that described in reference [2] as an emotion estimation method. Specifically, the emotion estimation unit 13 calculates a score for each emotion expressed by the text, and estimates the emotion based on the calculated score. For example, the emotion estimation unit 13 calculates the score for each emotion expressed by the text "I had a good breakfast this morning" as follows: "joy" 0.68, "goodwill" 0.66, "fear" 0.29, "sadness" 0.14, and "anger" 0.34. The estimated emotion may be a combination of emotions in proportions according to the calculated scores, or may be the emotion with the maximum score.

[0017] In addition, the emotion estimation unit 13 estimates emotions for every predetermined number of characters (e.g., 300 characters). Alternatively, the emotion estimation unit 13 may apply a conventional technique to divide the text and estimate emotions for each divided text. For example, the emotion estimation unit 13 may use the Python library of reference [3].

[0018] The feature point change amount calculation unit 14 calculates the amount of change in the feature point of the facial expression corresponding to the estimated emotion based on the feature point change amount data 100. Specifically, the feature point change amount calculation unit 14 calculates the amount of change by multiplying the amount of change for each emotion in the feature point change amount data 100 by the proportion of each estimated emotion, and calculates the amount of change by summing up the calculated amounts of change for all emotions as the amount of change in the feature point.

[0019] For example, the estimated emotion is assumed to be a combination of emotions with the ratios of "happiness" 0.68, "goodwill" 0.66, "fear" 0.29, "sadness" 0.14, and "anger" 0.34. The feature point change amount data 100 is assumed to be data indicating that the feature points of the corners of the mouth rise 4.3 mm for the emotion of "happiness", rise 1.2 mm for the emotion of "goodwill", fall 2.4 mm for the emotion of "fear", fall 3.2 mm for the emotion of "sadness", and fall 1.6 mm for the emotion of "anger".

[0020] In this case, the feature point change amount calculation unit 14 calculates the amount of change in the feature points of the corners of the mouth for each emotion as follows. "Joy": Raise by 4.3mm x 0.68 = 2.9mm "Favour": Raise by 1.2mm x 0.66 = 0.79mm "Fear": Lower 2.4mm x 0.29 = 0.70mm "Sadness": Lower 3.2mm x 0.14 = 0.45mm "Anger": Lowered by 1.6mm x 0.34 = 0.54mm

[0021] Then, the feature point change amount calculation unit 14 calculates the sum of all these emotions, "raise by 2.0 mm," as the change amount of the feature point of the corner of the mouth.

[0022] The facial expression conversion unit 15 acquires facial expression data indicating the facial expression of the listener, and changes the feature points of the acquired facial expression data based on the amount of change calculated by the feature point change amount calculation unit 14. Then, the facial expression conversion unit 15 outputs the converted facial expression data with the changed feature points.

[0023] (Example of the operation of the facial expression conversion device) Next, a description will be given with reference to the drawings of an example of the operation of the facial expression conversion device 10. The facial expression conversion device 10 starts a facial expression conversion process in response to a user's operation or the like.

[0024] 2 is a flowchart showing an example of the flow of the facial expression conversion process. The voice recognition unit 12 converts the voice of a speaker into text (step S11).

[0025] Next, the emotion estimation unit 13 estimates the emotion expressed by the text (step S12).

[0026] Next, the feature point change amount calculation unit 14 calculates the change amount of the feature points of the emotion expressed by the text (step S13). Then, the facial expression conversion unit 15 changes the feature points of the listener's facial expression (step S14). The facial expression conversion unit 15 outputs the converted facial expression data with the changed feature points.

[0027] (Example of Hardware Configuration According to the Present Embodiment) The facial expression conversion device 10 can be realized, for example, by making a computer execute a program in which the processing contents described in this embodiment are described. Note that this "computer" may be a physical machine or a virtual machine on the cloud. When a virtual machine is used, the "hardware" described here is virtual hardware.

[0028] The above program can be recorded on a computer-readable recording medium (such as a portable memory) and can be stored or distributed. The above program can also be provided via a network such as the Internet or e-mail.

[0029] Fig. 3 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 3 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B.

[0030] The program for realizing the processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0031] When a program start instruction is received, the memory device 1003 reads out and stores the program from the auxiliary storage device 1002. The CPU 1004 realizes the functions related to the device according to the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network. The display device 1006 displays a GUI (Graphical User Interface) or the like according to a program. The input device 1007 is composed of a keyboard, a mouse, a button, a touch panel, or the like, and is used to input various operation instructions. The output device 1008 outputs the calculation result. Note that the above computer may be equipped with a GPU (Graphics Processing Unit) or a TPU (Tensor processing unit) instead of the CPU 1004, or may be equipped with a GPU or a TPU in addition to the CPU 1004. In that case, the processes may be shared and executed, for example, the GPU or the TPU executes processes that require special calculations, and the CPU 1004 executes other processes.

[0032] (Effects of this embodiment) According to the facial expression conversion device 10 of this embodiment, the emotion expressed by the speaker is estimated, and the listener's facial expression is converted to the facial expression of the estimated emotion. This makes it possible to convert the listener's facial expression to match the content of the speaker's speech. In addition, the speaker is more likely to feel sympathized with, and communication becomes smoother. Furthermore, the burden of emotional labor on the listener, such as suppressing one's own emotions and expressing a different emotion while considering the speaker's emotions, is reduced.

[0033] (References) [1]:Speech to text conversion, [online], Internet<URL:https: / / www.textfromtospeech.com / ja / voice-to-text / > [2]:User Local,Inc.,Voice Emotion Recognition AI,[online],Internet<URL:https: / / emotion-voice-ai.userlocal.jp / > [3]: I created a library that divides Japanese text nicely, [online], Internet<URL:https: / / qiita.com / wwwcojp / items / 3535985007aa4269009c>

[0034] (Summary of the embodiment) This specification describes at least the facial expression conversion device, the facial expression conversion method, and the program described in the following sections. (Section 1) A facial expression conversion device for converting facial expression data indicating a listener's facial expression, comprising: an emotion estimation unit that estimates an emotion of a speaker based on a text indicating the content of what the speaker said; a feature point change amount calculation unit that calculates a change amount of a feature point corresponding to the estimated emotion; a facial expression conversion unit that outputs facial expression data after conversion in which the feature points of the facial expression data are changed based on the calculated change amount, Facial expression conversion device. (Section 2) the feature point change amount calculation unit calculates an amount of change in a feature point of a facial expression corresponding to the estimated emotion based on feature point change amount data indicating an amount of change in a feature point of a facial expression corresponding to each emotion, using an emotionless facial expression as a reference; 2. The facial expression conversion device according to claim 1. (Section 3) The emotion estimation unit calculates a score for each emotion expressed by the text, and estimates the emotion as a combination of emotions in proportions according to the calculated scores; the feature point change amount calculation unit calculates an amount of change by multiplying an amount of change for each emotion in the feature point change amount data by a proportion of each estimated emotion, and calculates, as an amount of change in the feature point, an amount of change obtained by summing up the calculated amounts of change for all emotions; 3. The facial expression conversion device according to claim 2. (Section 4) The feature point change amount data is data generated based on facial expression data of the listener. 4. The facial expression conversion device according to claim 2 or 3. (Section 5) a speech recognition unit that acquires speech data representing a speech uttered by the speaker and converts the acquired speech data into text; The emotion estimation unit estimates an emotion of the speaker based on the converted text. 5. The facial expression conversion device according to any one of claims 1 to 4. (Section 6) A facial expression conversion method executed by a facial expression conversion device for converting facial expression data indicating a listener's facial expression, comprising the steps of: estimating a speaker's emotion based on a text indicating what the speaker said; calculating an amount of change in a feature point corresponding to the estimated emotion; and outputting converted facial expression data in which the feature points of the facial expression data are changed based on the calculated amount of change. Facial expression transformation method. (Section 7) A program for causing a computer to function as each unit in the facial expression conversion device according to any one of claims 1 to 5.

[0035] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0036] 10 Facial Expression Conversion Device 11 Storage section 12 Voice Recognition Unit 13 Emotion estimation part 14 Feature point change amount calculation unit 15 Facial Expression Conversion Unit 100 feature point change data 1000 Drive Device 1001 Recording media 1002 Auxiliary storage 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device 1008 Output device

Claims

1. A facial expression conversion device for converting facial expression data indicating a listener's facial expression, comprising: an emotion estimation unit that estimates an emotion of a speaker based on a text indicating the content of what the speaker said; a feature point change amount calculation unit that calculates a change amount of a feature point corresponding to the estimated emotion; a facial expression conversion unit that outputs facial expression data after conversion in which the feature points of the facial expression data are changed based on the calculated change amount, The emotion estimation unit calculates a score for each emotion expressed by the text, and estimates the emotion as a combination of emotions in proportions according to the calculated scores; the feature point change amount calculation unit calculates an amount of change by multiplying an amount of change for each emotion in feature point change amount data indicating an amount of change in feature points of facial expressions corresponding to each emotion, using an emotionless facial expression as a reference, by a proportion of each estimated emotion, and calculates an amount of change by summing up the calculated amounts of change for all emotions as an amount of change in the feature points corresponding to the estimated emotion. Facial expression conversion device.

2. The feature point change amount data is data generated based on facial expression data of the listener. The facial expression conversion device according to claim 1 .

3. a speech recognition unit that acquires speech data representing a speech uttered by the speaker and converts the acquired speech data into text; The emotion estimation unit estimates an emotion of the speaker based on the converted text. The facial expression conversion device according to claim 1 .

4. A facial expression conversion method executed by a facial expression conversion device for converting facial expression data indicating a listener's facial expression, comprising the steps of: an emotion estimation step of estimating an emotion of a speaker based on a text indicating the content of what the speaker said; a feature point change amount calculation step of calculating an amount of change in a feature point corresponding to the estimated emotion; and a facial expression conversion step of outputting converted facial expression data in which the feature points of the facial expression data have been changed based on the calculated amount of change, The emotion estimation step includes calculating a score for each emotion expressed by the text, and estimating the emotion as a combination of emotions in proportions according to the calculated scores; the feature point change amount calculation step calculates an amount of change by multiplying an amount of change for each emotion in feature point change amount data indicating an amount of change in feature points of facial expressions corresponding to each emotion, using an emotionless facial expression as a reference, by a proportion of each estimated emotion, and calculates an amount of change by summing up the calculated amounts of change for all emotions as an amount of change in feature points corresponding to the estimated emotion; Facial expression transformation method.

5. A program for causing a computer to function as each unit in the facial expression conversion device according to any one of claims 1 to 3.

Images