Emotion analysis device, emotion analysis method, and emotion analysis program

Abstract

In an emotion analysis device (10), a question generation unit (15b) causes a trained LLM (14a) to generate a question for identifying an emotion of a speaker at the time of the utterance, using time-series data of utterance text representing content of a conversation to be processed, and utterance text of the speaker to be processed. An answer generation unit (15c) uses the time-series data and the generated question to cause the trained LLM (14a) to generate an answer to the question. An emotion estimation unit (15d) uses the time-series data and the utterance text, and the question and the answer that are generated, to cause the trained LLM (14a) to estimate the emotion with respect to the utterance text.

Description

Emotion analysis device, emotion analysis method, and emotion analysis program

[0001] The present invention relates to an emotion analysis device, an emotion analysis method, and an emotion analysis program.

[0002] In recent years, technologies for capturing the emotions of speakers have been attracting attention in order to ensure smooth communication between speakers in online remote conferences and conversations on social networks. For example, a known technology estimates the emotions of speakers during a conversation by using a dataset in which emotion labels are manually assigned to dialogue scripts, dialogue videos, audio, and biometric information, and a pre-trained model (see Non-Patent Document 1).

[0003] Additionally, LLMs (Large Language Models) are known that are pre-trained based on massive amounts of text data and have general-purpose knowledge (see Non-Patent Documents 2 and 3). LLMs exhibit high performance in natural language processing tasks without additional training, and can be used as general-purpose emotion estimation models that are not limited to specific domains.

[0004] “Supervised Prototypical Contrastive Learning for Emotion Recognition in Conversation”, [online], December 2022, Association for Computational Linguistics, [Retrieved December 28, 2023], Internet<URL:https: / / aclanthology.org / 2022.emnlp-main.347.pdf> “Language Models are Few-Shot Learners”, [online], 2020, OpenAI, [Retrieved December 28, 2023], Internet<URL:https: / / ar5iv.labs.arxiv.org / html / 2005.14165> “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”, [online], January 2023, 36th Conference on Neural Information Processing Systems (NeurIPS 2022), [Retrieved December 28, 2023], Internet<URL:https: / / arxiv.org / abs / 2201.11903>

[0005] However, with conventional techniques, it can be difficult to estimate the emotions of a speaker during a conversation while keeping costs down. For example, with conventional techniques, collecting the datasets used to train the model is costly, and whenever the scope of application of the estimation changes, re-training with a new dataset is required.

[0006] Furthermore, since LLM is a model that originally learns the characteristics of general language, it is difficult to extract emotional information about the speaker as it is.

[0007] The present invention has been made in view of the above, and has an object to estimate the emotion of a speaker during a conversation at a reduced cost.

[0008] In order to solve the above-described problems and achieve the object, the emotion analysis device according to the present invention includes: a question generation unit that causes an LLM (Large Language Model) to generate a question for identifying the emotion of the speaker at the time of speaking, using time-series data of utterance text representing the content of a conversation to be processed and utterance text of the speaker to be processed; an answer generation unit that causes the LLM to generate an answer to the question, using the time-series data and the generated question; and an emotion estimation unit that causes the LLM to estimate an emotion toward the utterance text, using the time-series data, the utterance text, the generated question, and the answer.

[0009] According to the present invention, it is possible to estimate the emotion of a speaker during a conversation at a reduced cost.

[0010] FIG. 1 is a diagram for explaining an overview of the emotion analysis device of this embodiment. FIG. 2 is a schematic diagram illustrating an example of the overall configuration of the emotion analysis device of this embodiment. FIG. 3 is a diagram for explaining emotion analysis processing. FIG. 4 is a flowchart showing the emotion analysis processing procedure. FIG. 5 is a diagram showing an example of a computer that executes an emotion analysis program.

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are designated by the same reference numerals.

[0012] [Outline of the Emotion Analysis Apparatus] Fig. 1 is a diagram for explaining an outline of the emotion analysis apparatus of this embodiment. The emotion analysis apparatus of this embodiment estimates the emotion of a speaker during a dialogue by extracting emotion information using a trained LLM and multi-step prompt input.

[0013] Specifically, as shown in Figure 1, a question prompt is input to the LLM, which then generates a question to deepen understanding of the conversation context. The LLM generates a question that links the flavor of the conversation content with the speaker's emotions. This provides versatile emotion analysis capabilities that can be applied to a variety of situations.

[0014] Next, based on the generated list of questions, an answer prompt is input to the LLM, causing it to generate answers. Then, based on the question list and answer list generated by the LLM, an emotion estimation prompt is input, causing the LLM to estimate emotions and output results such as emotion labels. In this way, the emotion analysis device uses the LLM to estimate the speaker's emotions at a given time when speaking, without the cost of collecting a dataset for emotion analysis according to the situation and the cost of training a model.

[0015] [Configuration of the emotion analysis device] Fig. 2 is a schematic diagram illustrating the overall configuration of the emotion analysis device of this embodiment. As illustrated in Fig. 2, the emotion analysis device 10 of this embodiment is realized by a general-purpose computer such as a personal computer, and includes an input unit 11, an output unit 12, a communication control unit 13, a storage unit 14, and a control unit 15.

[0016] The input unit 11 is realized using input devices such as a keyboard and a mouse, and inputs various instruction information such as a command to start processing to the control unit 15 in response to input operations by an operator. The output unit 12 is realized by a display device such as a liquid crystal display, a printing device such as a printer, etc. For example, the output unit 12 displays the results of the emotion analysis processing described below.

[0017] The communication control unit 13 is realized by a NIC (Network Interface Card) or the like, and controls communication between the control unit 15 and an external device via a telecommunication line such as a LAN (Local Area Network) or the Internet. For example, the communication control unit 13 controls communication between the control unit 15 and a management device that manages the learned LLM, a terminal used by a speaker, or the like.

[0018] The storage unit 14 is realized by a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 14 stores in advance the processing program that operates the emotion analysis device 10, data used during execution of the processing program, and the like, or temporarily stores the data each time processing is performed. In this embodiment, the storage unit 14 may store a trained LLM 14a that is used in the emotion analysis process described below. The storage unit 14 may be configured to communicate with the control unit 15 via the communication control unit 13.

[0019] The control unit 15 is realized using a CPU (Central Processing Unit), an NP (Network Processor), an FPGA (Field Programmable Gate Array), a GPU (Graphics Processing Unit), or the like, and executes a processing program stored in memory. As a result, the control unit 15 functions as an acquisition unit 15a, a question generation unit 15b, an answer generation unit 15c, and a feeling estimation unit 15d, as illustrated in FIG. 2, to perform feeling analysis processing. Note that these functional units may be implemented individually or in part in different hardware. The control unit 15 may also include other functional units.

[0020] The acquiring unit 15a acquires time-series data T of utterance text representing the content of the conversation to be processed and the i-th utterance text Ti to be processed. For example, the acquiring unit 15a acquires the time-series data T of utterance text representing the content of the conversation and the utterance text Ti of the speaker via the input unit 11 or the communication control unit 13.

[0021] The acquisition unit 15 a also acquires the trained LLM 14 a. For example, prior to the emotion analysis process described below, the acquisition unit 15 a acquires the LLM from a management device that manages the trained LLM via the communication control unit 13 and stores the LLM in the storage unit 14.

[0022] Note that the present invention is not limited to the case where the acquisition unit 15a acquires a trained LLM in advance and stores it in the storage unit 14. For example, the question generation unit 15b, the answer generation unit 15c, and the feeling estimation unit 15d, which will be described later, may use the LLM by communicating with a management device that manages the trained LLM via the communication control unit 13 each time processing is performed.

[0023] The question generator 15b uses time-series data of the speech text representing the conversation content to be processed and the speech text of the speaker to be processed to cause the LLM 14a to generate a question for identifying the emotion of the speaker at the time of the speech. Specifically, the question generator 15b creates a prompt that causes the LLM 14a to generate a question useful for estimating the emotion of the speaker at the time of the speech, and inputs the prompt to the LLM 14a to cause the LLM 14a to generate the question.

[0024] 3 is a diagram illustrating the emotion analysis process. The question generator 15b generates a question prompt as shown in FIG. 3(a) and inputs it to the LLM 14a. In the example shown in FIG. 3(a), the prompt is generated by inputting the acquired time-series data T of the utterance text and the utterance text Ti of the speaker to be processed into a pre-designed prompt format.

[0025] This prompt is used as an API call prompt for an external LLM such as Chat-GPT or an LLM in a local environment. In response to this prompt, the LLM 14a returns a list of important questions in text format for estimating Ti's emotions.

[0026] The question generator 15b may also cause the LLM 14a to add additional information and then use this information to generate a question. For example, the question generator 15b may collect information about the speaker or search for external knowledge, and add an instruction to the prompt to use the information about the speaker when generating a question, thereby instructing the LLM 14a.

[0027] The answer generator 15c uses the time-series data and the generated question to cause the LLM 14a to generate an answer to the question. Specifically, the answer generator 15c creates a prompt that causes the LLM 14a to generate an answer to the question, and inputs the prompt to the LLM 14a to cause the LLM 14a to generate an answer.

[0028] For example, the answer generator 15c generates an answer prompt as shown in Fig. 3(b) and inputs it to the LLM 14a. In the example shown in Fig. 3(b), the prompt is created by inputting the acquired time-series data T of the utterance text and the generated question list into a pre-designed prompt format. The LLM 14a returns a list of answers to each question in the question list in text format.

[0029] The LLM used by the answer generator 15c does not necessarily have to be the same as the LLM used by the question generator 15b. For example, the answer generator 15c may use a trained LLM specialized for answering questions, which can improve accuracy.

[0030] The feeling estimation unit 15d causes the LLM 14a to estimate the feeling for the utterance text Ti by using the time-series data T, the utterance text Ti, and the generated question and answer. Specifically, the feeling estimation unit 15d creates a prompt for causing the LLM 14a to estimate the feeling for the utterance text Ti and inputs the prompt to the LLM 14a, thereby causing the LLM 14a to estimate the feeling.

[0031] For example, the feeling estimation unit 15 d generates a feeling estimation prompt as illustrated in Fig. 3(c) and inputs it to the LLM 14 a. In the example illustrated in Fig. 3(c), the prompt is created by inputting the generated question list, answer list, time-series data T of the utterance text, and utterance text Ti into a pre-designed prompt format.

[0032] Then, the LLM 14a returns an emotion label expressed by words such as joy, sadness, anger, etc. This enables the emotion estimation unit 15d to output the emotion of the speaker of the utterance text Ti in the form of an emotion label.

[0033] The feeling estimation unit 15 d can also select a word that expresses the feeling for the utterance text Ti from predetermined word candidates that express feelings. For example, the prompt illustrated in FIG. 3D can be created to cause the LLM 14 a to select an appropriate emotion label from emotion labels that are limited to a predetermined number of types.

[0034] The feeling deduction unit 15 d also checks whether the formats of the question list, answer list, and emotion labels are properly maintained by using, for example, regular expressions, etc. Then, the feeling deduction unit 15 d outputs the emotion of the speaker of the utterance text Ti in the form of an emotion label to, for example, the output unit 12.

[0035] In this way, in the emotion analysis device 10, the time-series data T of the utterance text of the conversation to be processed is composed of series data of utterances from multiple people, and the emotion of the person who uttered the i-th utterance text Ti in the series data at the time of utterance is estimated by the above-mentioned emotion analysis process.

[0036] The conversation to be processed may be a one-on-one dialogue or a conference of three or more people. Alternatively, a speech by one person may be segmented into sentences and the utterance to be analyzed may be designated as the processing target. The conversation may also contain multimodal information such as audio and images. In this case, sentiment analysis processing can be applied by first converting the data into sequential text data via a multimodal LLM.

[0037] Furthermore, the above-described sentiment analysis process estimates the sentiment for the i-th utterance text in the conversational utterance text. By repeatedly performing the process while changing the Ti input to this prompt, it is possible to estimate the sentiment for all utterance texts in the conversational utterance text. Alternatively, by changing the prompt so that processing is performed for all Ti, it is possible to simultaneously estimate the sentiment for all utterance texts in the conversational utterance text.

[0038] [Emotion Analysis Processing] Next, the emotion analysis processing performed by the emotion analysis device 10 according to this embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the procedure of the emotion analysis processing. The flowchart in Fig. 4 starts, for example, when the user performs an operation input to instruct the start of the processing.

[0039] First, the acquiring unit 15a acquires time-series data T of utterance text representing the conversation content to be processed and the i-th utterance text Ti to be processed (step S1).

[0040] Next, the question generator 15b uses the time-series data of the speech text representing the conversation content to be processed and the speech text of the speaker to be processed to cause the trained LLM 14a to generate a question for identifying the emotion of the speaker at the time of the speech (step S2). Specifically, the question generator 15b creates a prompt that causes the LLM 14a to generate a question useful for estimating the emotion of the speaker at the time of the speech, and inputs the prompt to the LLM 14a to cause the LLM to generate a question.

[0041] Next, the answer generator 15c uses the time-series data T and the generated question to cause the LLM 14a to generate an answer to the question (step S3). Specifically, the answer generator 15c creates a prompt that causes the LLM 14a to generate an answer to the question, and inputs the prompt to the LLM 14a to cause the LLM 14a to generate an answer.

[0042] Then, the feeling estimation unit 15d causes the LLM 14a to estimate the feeling for the utterance text Ti by using the time-series data T, the utterance text Ti, and the generated question and answer (step S4). Specifically, the feeling estimation unit 15d creates a prompt for causing the LLM 14a to estimate the feeling for the utterance text Ti, and inputs the prompt to the LLM 14a to cause the LLM 14a to estimate the feeling.

[0043] Furthermore, the emotion estimation unit 15d outputs the emotion of the speaker of the spoken text Ti in the form of an emotion label, for example, to the output unit 12. This completes the emotion analysis process.

[0044] [Effects] As described above, in the emotion analysis device 10 of this embodiment, the question generator 15b uses time-series data T of utterance text representing the content of the conversation to be processed and the utterance text Ti of the speaker to be processed to cause the trained LLM 14a to generate a question for identifying the emotion of the speaker at the time of the utterance. The answer generator 15c uses the time-series data T and the generated question to cause the trained LLM 14a to generate an answer to the question. The emotion estimation unit 15d uses the time-series data T, the utterance text Ti, and the generated question and answer to cause the trained LLM 14a to estimate the emotion toward the utterance text Ti.

[0045] Specifically, the question generation unit 15b creates a question prompt that causes the trained LLM 14a to generate a question that is useful for estimating the speaker's emotions at the time of speaking, and inputs the question prompt into the LLM 14a, causing the LLM 14a to generate a question.

[0046] The answer generating unit 15c also creates an answer prompt that causes the trained LLM 14a to generate an answer to the question, and inputs the answer prompt to the LLM 14a, causing the LLM 14a to generate an answer.

[0047] Furthermore, the feeling estimation unit 15d creates a feeling estimation prompt that causes the trained LLM 14a to estimate the feeling for the utterance text Ti, and inputs the prompt to the LLM 14a, thereby causing the LLM 14a to estimate the feeling.

[0048] In this way, the sentiment analysis device 10 utilizes a trained LLM capable of understanding general linguistic features, eliminating the need to collect additional datasets for sentiment analysis tailored to the situation or retrain the model through the intermediate step of question generation for linking conversation content text with emotional information. The question-answering process also improves the accuracy of emotion recognition by the LLM. Furthermore, it aids in linguistic understanding, such as interpreting noteworthy utterances and contexts, in deriving emotion estimation results. This makes it possible to estimate the emotion of a speaker at a given point in time while reducing the cost of collecting training datasets and the cost of training the model.

[0049] This makes it possible to predict emotions from the messages and comments of others regardless of the situation, such as on social media or in conference calls. Furthermore, it becomes possible to display information that helps understand why a person feels a certain emotion. Furthermore, this technology can be used in educational systems that improve empathy, and as a filter to prevent posts on social media that offend others.

[0050] The feeling estimation unit 15 d may also select a word that expresses the feeling toward the utterance text from predetermined word candidates that express feelings, thereby limiting the types of feelings to be estimated and enabling more accurate feeling estimation.

[0051] The question generator 15b may also add information to the LLM 14a and generate a question using this information. This adds information that is more useful for estimating the speaker's emotion, making it possible to estimate the emotion with high accuracy.

[0052] [Program] A program written in a computer-executable language may be generated to execute the processes performed by the emotion analysis device 10 according to the above embodiment. In one embodiment, the emotion analysis device 10 can be implemented by installing a sentiment analysis program that executes the sentiment analysis process as package software or online software on a desired computer. For example, by executing the sentiment analysis program on an information processing device, the information processing device can function as the sentiment analysis device 10. The information processing device referred to here includes desktop and notebook personal computers. Other examples of information processing devices include mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as slate terminals such as PDAs (Personal Digital Assistants). The functions of the emotion analysis device 10 may also be implemented on a cloud server.

[0053] 5 is a diagram showing an example of a computer that executes an emotion analysis program. The computer 1000 includes, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0054] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1031. The disk drive interface 1040 is connected to a disk drive 1041. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1041. The serial port interface 1050 is connected to a mouse 1051 and a keyboard 1052, for example. The video adapter 1060 is connected to a display 1061, for example.

[0055] Here, the hard disk drive 1031 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. The various pieces of information described in the above embodiments are stored in the hard disk drive 1031 or the memory 1010, for example.

[0056] The emotion analysis program is stored in the hard disk drive 1031 as a program module 1093 in which instructions to be executed by the computer 1000 are written. Specifically, the program module 1093 in which each process executed by the emotion analysis device 10 described in the above embodiment is written is stored in the hard disk drive 1031.

[0057] Furthermore, data used for information processing by the emotion analysis program is stored as program data 1094, for example, in the hard disk drive 1031. Then, the CPU 1020 reads the program module 1093 and program data 1094 stored in the hard disk drive 1031 into the RAM 1012 as necessary, and executes each of the above-described procedures.

[0058] The program module 1093 and program data 1094 related to the emotion analysis program are not limited to being stored in the hard disk drive 1031, and may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1041. Alternatively, the program module 1093 and program data 1094 related to the emotion analysis program may be stored in another computer connected via a network such as a LAN or a WAN (Wide Area Network), and read by the CPU 1020 via the network interface 1070.

[0059] Although the present invention has been described above as an embodiment, the present invention is not limited to the description and drawings that form part of the disclosure of the present invention. In other words, other embodiments, examples, and operational techniques that can be made by those skilled in the art based on the present invention are all included in the scope of the present invention.

[0060] REFERENCE SIGNS LIST 10 Emotion analysis device 11 Input unit 12 Output unit 13 Communication control unit 14 Memory unit 14a LLM 15 Control unit 15a Acquisition unit 15b Question generation unit 15c Answer generation unit 15d Emotion estimation unit

Claims

1. A sentiment analysis device comprising: a question generation unit that uses time-series data of spoken text representing the content of a conversation to be processed and spoken text of a speaker to be processed to cause an LLM (Large Language Model) to generate a question for identifying the emotion of the speaker at the time of speaking; an answer generation unit that uses the time-series data and the generated question to cause the LLM to generate an answer to the question; and an emotion estimation unit that uses the time-series data, the spoken text, the generated question, and the answer to cause the LLM to estimate the emotion toward the spoken text.

2. The emotion analysis device described in claim 1, characterized in that the question generation unit creates a prompt that causes the LLM to generate a question that is useful for estimating the speaker's emotion at the time of the utterance, and inputs the prompt into the LLM, thereby causing the LLM to generate the question.

3. The sentiment analysis device described in claim 1, characterized in that the answer generation unit creates a prompt that causes the LLM to generate an answer to the question, and inputs the prompt into the LLM, thereby causing the LLM to generate the answer.

4. The emotion analysis device according to claim 1, characterized in that the emotion estimation unit creates a prompt that causes the LLM to estimate the emotion for the spoken text, and inputs the prompt into the LLM to cause the LLM to estimate the emotion.

5. The emotion analysis device according to claim 1, wherein the emotion estimation unit selects a word that expresses emotion toward the utterance text from predetermined word candidates that express emotion.

6. The sentiment analysis device according to claim 1, wherein the question generation unit adds information to the LLM and generates questions using the information.

7. A sentiment analysis method executed by a sentiment analysis device, comprising: a question generation step of causing an LLM (Large Language Model) to generate a question for identifying the emotion of the speaker at the time of speaking, using time-series data of utterance text representing the content of a conversation to be processed and the utterance text of the speaker to be processed; an answer generation step of causing the LLM to generate an answer to the question, using the time-series data and the generated question; and an emotion estimation step of causing the LLM to estimate the emotion of the utterance text, using the time-series data, the utterance text, the generated question, and the answer.

8. A sentiment analysis program for causing a computer to execute the following steps: a question generation step of causing an LLM (Large Language Model) to generate a question for identifying the emotion of the speaker at the time of speaking, using time series data of utterance text representing the content of a conversation to be processed and the utterance text of the speaker to be processed; an answer generation step of causing an LLM to generate an answer to the question, using the time series data and the generated question; and an emotion estimation step of causing an LLM to estimate the emotion of the utterance text, using the time series data, the utterance text, the generated question, and the answer.

Images