Terminal device and prediction method

Abstract

To improve a technique for estimating feeling from utterance details of a user.SOLUTION: A terminal device 10 comprises a control unit 15 configured to: continuously acquire character information uttered by a user 3 during a conversation using a conversation application actuated on the device; store in a database 14A reference data including the character information uttered by the user 3 and a label indicative of feeling of the user during a predetermined period before and after the time when the label is posted by the user 3 each time the user 3 posts the label indicative of the feeling of the user 3; extract the reference data including the label corresponding to the feeling that the user 3 shows from the database 14A by the use of artificial intelligence on detecting the feeling that the user 3 shows when the amount of stored reference data exceeds a threshold; and estimate character information that the user 3 utters by learning the extracted reference data by the use of artificial intelligence.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a terminal device and a prediction method. [Background technology]

[0002] Conventionally, there are known techniques for estimating the content of a user's comments from their emotions. For example, Patent Literature 1 discloses a technique for estimating the emotions of a user by analyzing the user's biometric information and the content of posts on social media. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-142137 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, artificial intelligence (AI) using large-scale language models (LLMs) has been actively utilized. LLMs are language models constructed using large amounts of data and deep learning technology. For example, users manually label their own utterances and store them as data. The users then use the stored data to perform deep learning on the LLM. However, manual labeling is cumbersome. For this reason, it is difficult to accumulate a sufficient amount of data through manual labeling. Therefore, there is room for improvement in technology for inferring user utterances from their emotions.

[0005] In view of the above circumstances, an object of the present disclosure is to improve the technology for estimating the content of a user's utterances from their emotions. [Means for solving the problem]

[0006] A terminal device according to one embodiment of the present disclosure is a terminal device having a database and a control unit using artificial intelligence capable of searching the database, wherein the control unit continuously acquires text information uttered by a user during a conversation using a conversation app launched on the terminal device, and each time the user posts a label indicating the user's emotion, stores reference data in the database including the text information uttered by the user and the label indicating the user's emotion for a predetermined period before and after the time the label was posted, and when the amount of stored reference data exceeds a threshold, extracts reference data including a label corresponding to the emotion expressed by the user from the database using the artificial intelligence when the emotion expressed by the user is detected, and estimates the text information uttered by the user by learning the extracted reference data using the artificial intelligence.

[0007] A prediction method according to one embodiment of the present disclosure includes: continuously acquiring, by a terminal device, text information uttered by a user during a conversation using a conversation app launched on the terminal device; each time the user posts a label indicating the user's emotion, storing reference data in a database including the text information uttered by the user and the label indicating the user's emotion for a predetermined period before and after the time the label was posted; if the amount of the stored reference data exceeds a threshold, when the emotion expressed by the user is detected, extracting from the database using artificial intelligence reference data including a label corresponding to the emotion expressed by the user; and estimating the text information uttered by the user by learning the extracted reference data using the artificial intelligence. [Effects of the Invention]

[0008] According to an embodiment of the present disclosure, a technology for estimating the content of a user's speech from emotions is improved. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram illustrating a schematic configuration example of a system according to an embodiment of the present disclosure. [Figure 2] 10 is a flowchart illustrating an example of the operation of the terminal device. [Figure 3] FIG. 1 is a schematic diagram illustrating a large-scale language model in which RAG is implemented. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described.

[0011] (Outline of the embodiment) An overview of a system 1 according to an embodiment of the present disclosure will be described with reference to Fig. 1. The system 1 includes a terminal device 10 and a server 20. The terminal device 10 and the server 20 are communicably connected to a network 2 including, for example, the Internet and a mobile communication network.

[0012] The terminal device 10 is a personal computer (PC) or the like owned by the user 3. However, the terminal device 10 is not limited to a personal computer and may be any information processing terminal. The terminal device 10 is capable of communicating with the server 20 via the network 2.

[0013] The server 20 is a computer owned by, for example, a business operator that provides AI services. The server 20 is capable of communicating with the terminal device 10 via the network 2.

[0014] First, an outline of the present embodiment will be described, and details will be described later. The terminal device 10 continuously acquires text information uttered by the user 3 during a conversation using a conversation app launched on the device, and each time the user 3 posts a label indicating the user 3's emotion, stores reference data including the text information uttered by the user 3 and the label indicating the user 3's emotion in a database 14A for a predetermined period around the time the label was posted. If the amount of stored reference data exceeds a threshold, when the emotion expressed by the user 3 is detected, the terminal device 10 extracts reference data including the label corresponding to the emotion expressed by the user 3 from the database 14A using artificial intelligence, and learns the extracted reference data using artificial intelligence to estimate the text information uttered by the user 3.

[0015] As described above, according to this embodiment, in order to estimate the text information uttered by user 3, it is necessary for user 3 to post a label indicating the emotion of user 3 during a conversation using a chat app. This makes it possible to prevent the text information uttered by user 3 from being arbitrarily estimated by others. Furthermore, the only operation performed by user 3 is to post a label indicating the emotion of user 3 during a conversation using a chat app. This reduces the hassle of user 3 having to manually input a large amount of reference data. This improves the technology for estimating the content of a user's utterances from their emotions.

[0016] Next, each component of the system 1 will be described in detail.

[0017] (Terminal Device Configuration) As shown in FIG. 1, the terminal device 10 includes a communication unit 11, an input unit 12, an output unit 13, a storage unit 14, and a control unit 15.

[0018] The communication unit 11 includes at least one communication module that can be connected to the network 2. The communication module is, for example, a communication module compatible with a mobile communication standard such as LTE (Long Term Evolution), 4G, or 5G, a wired LAN standard, or a wireless LAN standard. The communication module may also be a communication module compatible with a short-range wireless communication standard such as Bluetooth (registered trademark). However, the communication module is not limited to these. The communication module may be compatible with any communication standard. In this embodiment, the terminal device 10 communicates with the server 20 via the communication unit 11 and the network 2.

[0019] The input unit 12 is configured to include at least one input interface capable of accepting input from the user 3. The input interface may be an input screen that is integrated with a display and into which the user 3 inputs text data. The input interface may also be a camera that captures an image of the user 3's face, etc. Furthermore, the input interface may also be a microphone that picks up the speech of the user 3. However, the input interface is not limited to this.

[0020] The output unit 13 is capable of outputting data. The output unit 13 is capable of outputting data. The output interface is configured to include one output interface, for example. These include a speaker and a display. The display may be, for example, an LCD or an organic EL display. However, the output interface is not limited to this.

[0021] The storage unit 14 includes one or more memories. The memories may be, for example, semiconductor memories, magnetic memories, or optical memories, but are not limited to these. Each memory included in the storage unit 14 may function as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 14 stores any information used in the operation of the terminal device 10. For example, the storage unit 14 may store system programs, application programs, embedded software, conversation apps, and AI engines. Furthermore, the storage unit 14 includes a database 14A that stores reference data for the AI ​​to learn. The information stored in the storage unit 14 may be updatable with information obtained from the network 2 via the communication unit 11, for example.

[0022] The control unit 15 includes one or more processors, one or more programmable circuits, one or more dedicated circuits, or a combination thereof. The processor is, for example, a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for a specific process. However, the processor is not limited to these. The programmable circuit is, for example, an FPGA (Field-Programmable Gate Array). However, the programmable circuit is not limited to an FPGA. The dedicated circuit is, for example, an ASIC (Application Specific Integrated Circuit). However, the dedicated circuit is not limited to an ASIC. The control unit 15 executes information processing related to the operation of the terminal device 10.

[0023] (Server configuration) As shown in FIG. 1, the server 20 includes a communication unit 21, a storage unit 22, and a control unit .

[0024] The communication unit 21 includes one or more communication interfaces connected to the network 2. The communication interfaces correspond to, for example, a mobile communication standard, a wired LAN standard, or a wireless LAN standard. However, the communication interfaces are not limited to these and may correspond to any communication standard. In this embodiment, the server 20 communicates with the terminal device 10 via the communication unit 21 and the network 2.

[0025] The storage unit 22 includes one or more memories. Each memory included in the storage unit 22 may function as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 22 stores any information used in the operation of the server 20. For example, the storage unit 22 may store a system program, a database, an application program, an AI engine, and the like. Furthermore, the storage unit 22 may include a database 22A that stores reference data learned by the AI ​​and information containing vocabulary of natural language used by humans. The information stored in the storage unit 22 may be updatable with information obtained from the network 2 via the communication unit 21, for example.

[0026] The control unit 23 includes one or more processors, one or more programmable circuits, one or more dedicated circuits, or a combination thereof. The control unit 23 executes information processing related to the operation of the server 20.

[0027] (Operation flow of terminal device) Fig. 2 is a flowchart showing an example of the operation of the terminal device 10. The operation of the terminal device 10 according to this embodiment will be described with reference to Fig. 2. This operation relates to the estimation of text information ci (i is from 1 to n) uttered by the user 3.

[0028] S101: The control unit 15 continuously acquires text information ci uttered by the user 3 during a conversation using a conversation application started on the terminal device 10.

[0029] The conversation application is, for example, LINE (registered trademark). However, the conversation application is not limited to LINE. The text information ci is a word, a phrase, or a sentence. However, the text information ci is not limited to these.

[0030] S102: The control unit 15 acquires a label emi (where i ranges from 1 to n) that indicates the emotion of the user 3 and that is posted by the user 3.

[0031] One method of classifying emotions is the theory of basic emotions, which proposes that there are six basic emotions, such as joy, fear, surprise, disgust, anger, and sadness. Another method of classifying emotions is the Russell Circumplex model. The Russell Circumplex model has two axes: the vertical axis represents the degree of cheerfulness, and the horizontal axis represents the degree of arousal. The Russell Circumplex model represents 19 types of human emotions, such as surprise, happiness, boredom, depression, anger, and fear, on the circumference of the circle. In the present disclosure, the emotions of user 3 are classified into the six basic emotions of joy, fear, surprise, disgust, anger, and sadness. However, the emotions of user 3 are not limited to this, and may be classified into 19 types of emotions according to the Russell Circumplex model.

[0032] The label emi indicating the emotion of user 3 is, for example, a stamp used on LINE. In the present disclosure, the control unit 15 classifies, for example, a label corresponding to the emotion of joy as em1, a label corresponding to the emotion of fear as em2, a label corresponding to the emotion of surprise as em3, a label corresponding to the emotion of disgust as em4, a label corresponding to the emotion of anger as em5, and a label corresponding to the emotion of sadness as em6. In this case, when user 3 feels one of the six emotions during a conversation, he or she posts a stamp corresponding to one of the labels em1 to em6.

[0033] S103: The control unit 15 generates reference data di (i is from 1 to n) including the text information ci spoken by the user 3 and the label emi indicating the emotion of the user 3 during a predetermined period T around the time ti (i is from 1 to n) when the label emi is posted.

[0034] The predetermined period T is, for example, a period T (ti±180 seconds) of 3 minutes (180 seconds) before and after the time ti when the label emi is posted. However, the predetermined period T is not limited to 3 minutes and may be defined arbitrarily.

[0035] S104: The control unit 15 stores the generated reference data di in the database 14A one by one.

[0036] S105: The control unit 15 checks whether the amount of stored reference data di exceeds the threshold value α. If the amount of reference data di exceeds the threshold value α, the control unit 15 proceeds to S106 and executes information processing. If the amount of reference data di is equal to or less than the threshold value α, the control unit 15 proceeds to S101 and executes information processing.

[0037] The control unit 15 generates the reference data di until the amount of the reference data di to be referred to when learning is performed using artificial intelligence (AI) in the database 14A reaches a required amount (threshold α).

[0038] S106: The control unit 15 detects the emotion expressed by the user 3.

[0039] The control unit 15 may detect the emotion expressed by the user 3 by receiving a post of the label emi by the user 3 during a conversation using a conversation app. The control unit 15 may also receive the expression of emotion by the user 3's voice picked up by a microphone during a web conference. The control unit 15 may also receive the expression of emotion by the user 3's facial expression captured by a camera during a web conference. However, the means for detecting the emotion expressed by the user 3 are not limited to these.

[0040] S107: The control unit 15 extracts reference data di including the label emi corresponding to the emotion expressed by the user 3 from the database 14A using artificial intelligence.

[0041] The control unit 15 selects a label em i corresponding to the emotion expressed by the user 3 from among labels em1, em2, em3, em4, em5, and em6 corresponding to the emotions of joy, fear, surprise, disgust, anger, and sadness, respectively. As shown in Fig. 3, when the emotion expressed by the user 3 is anger, the control unit 15 selects the label em5 corresponding to the emotion of anger and extracts reference data di including the label em5 from the database 14A.

[0042] S108: The control unit 15 estimates the text information ci uttered by the user 3 by learning the extracted reference data di using artificial intelligence.

[0043] Artificial intelligence (AI) is a large-scale language model (LLM) that implements RAG (Retrieval Augmented Generation). LLM is a type of artificial intelligence that is part of AI and is capable of understanding and conversing with human language. LLM learns from a huge amount of text data and generates and understands natural language in the same way that humans do. LLM has the advantage of being able to have natural conversations, as if speaking with a human. On the other hand, LLM has the disadvantage that it can only handle information that has already been learned, i.e., it cannot handle closed information.

[0044] RAG complements the disadvantage of LLM, which cannot handle closed information, by combining the generation of character information (text data) by LLM with the search of external information. RAG stands for Search Augmentation Generation.

[0045] 3 is a schematic diagram illustrating a large-scale language model in which RAG is implemented. As shown in FIG. 3, RAG can search a database 14A and performs the following information processing (i) to (vii). (i) The RAG receives a label em5 indicating the emotion of anger posted by the user 3 from the control unit 15. (ii) RAG accesses database 14A to search for reference data containing label em5. (iv) RAG extracts reference data d3 containing label em5 from database 14A. In the example of Figure 3, only reference data d3 is extracted. However, in reality, a huge number of reference data items, such as 1,000, 10,000, etc., may be extracted. (v) The RAG communicates the extracted reference data d3 to the LLM. (vi) The RAG receives the character information c8 estimated by the LLM. (vii) The RAG presents the character information c8 to the control unit 15.

[0046] In addition to database 14A, RAG may be configured to be able to search other databases (for example, database 22A shown in FIG. 1) that contain vocabulary in natural language used by humans. In the example of FIG. 1, the location of the other database that contains vocabulary in natural language used by humans is server 20. However, the location of the other database may be any location that can be communicated with via network 2.

[0047] The control unit 15 may acquire an LLM engine with RAG implemented via the network 2 from a server 20 owned by a business providing AI services. The engine is a collection of programs and algorithms that form the core of artificial intelligence. On the other hand, the control unit 15 may use a cloud service of an LLM with RAG implemented, which is provided by the server 20 via the network 2. A cloud service is a service that provides desktop virtualization, hardware such as shared disks, and infrastructure functions via the Internet.

[0048] S109: The control unit 15 confirms with the user 3 whether the estimation result of the text information ci spoken by the user 3 is correct. If the estimation result is correct, the control unit 15 proceeds to S111 and executes information processing. If the estimation result is incorrect, the control unit 15 proceeds to S110 and executes information processing.

[0049] S110: If the estimation result is correct, the control unit 15 further stores reference data di including the estimated character information ci and the label emi corresponding to the emotion expressed by the user 3 in the database 14A.

[0050] Reference data including character information ci for which the estimation result has been determined to be correct by user 3 is further stored in database 14A. By learning using such highly accurate reference data, LLM improves the accuracy of estimating character information that user 3 is likely to utter when feeling a particular emotion.

[0051] S111: If the estimation result is incorrect, the control unit 15 deletes the estimated character information ci.

[0052] S112: The control unit 15 determines whether or not to continue the information processing. If the information processing is to be continued, the control unit 15 proceeds to S106 and executes the information processing. If the information processing is not to be continued, the control unit 15 ends the information processing.

[0053] The control unit 15 may store in the database 14A reference data di′ that further includes data on the number of conversation partners with the user 3 during the predetermined period T or data on the attributes of the conversation partners, in addition to the text information uttered by the user 3 during the predetermined period T and the label em indicating the emotion of the user 3. In this way, it becomes possible to estimate text information ci that the user 3 is likely to utter when feeling a particular emotion, depending on individual conversation situations that vary depending on the number of conversation partners or the attributes of the conversation partners (family, friends, superiors at work, customers, etc.).

[0054] As described above, the terminal device 10 according to this embodiment continuously acquires text information uttered by the user 3 during a conversation using a conversation app launched on the device itself, and each time the user 3 posts a label indicating the emotion of the user 3, stores reference data including the text information uttered by the user 3 and the label indicating the emotion of the user 3 in the database 14A for a predetermined period around the time the label was posted. If the amount of stored reference data exceeds a threshold, when the emotion expressed by the user 3 is detected, the terminal device 10 extracts reference data including the label corresponding to the emotion expressed by the user 3 from the database 14A using artificial intelligence, and estimates the text information uttered by the user 3 by learning the extracted reference data using artificial intelligence.

[0055] According to this configuration, in order to estimate the text information uttered by user 3, it is necessary for user 3 to post a label indicating the emotion of user 3 during a conversation using the chat app. This makes it possible to prevent the text information uttered by user 3 from being arbitrarily estimated by others. Furthermore, the only operation user 3 must perform is to post a label indicating the emotion of user 3 during a conversation using the chat app. This reduces the hassle of user 3 having to manually input a large amount of reference data. This improves the technology for estimating the content of a user's utterances from their emotions.

[0056] Although the present disclosure has been described based on the drawings and examples, it should be noted that those skilled in the art may make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included in the scope of the present disclosure. For example, the functions included in each component or step can be rearranged so as not to be logically inconsistent, and multiple components or steps can be combined or divided into one.

[0057] For example, in the above-described embodiment, the configuration and operation of the terminal device 10 may be distributed among multiple computers that can communicate with each other. Also, for example, an embodiment in which some or all of the components of the terminal device 10 are provided in the server 20 may be possible. For example, the terminal device 10 may transmit reference data to the server 20, and then entrust the control unit 15 with estimating the text information uttered by the user 3 to the server 20.

[0058] Also, for example, an embodiment is possible in which a general-purpose computer functions as the terminal device 10 according to the above-described embodiment. Specifically, a program describing the processing content for realizing each function of the terminal device 10 according to the above-described embodiment is stored in the memory of the general-purpose computer, and the program is read and executed by a processor. Therefore, the present disclosure can also be realized as a program executable by a processor or a non-transitory computer-readable medium storing the program. [Explanation of symbols]

[0059] 1 System 2 Network 3 User 10 Terminal Equipment 11 Communications Department 12 Input section 13 Output section 14 Storage section 14A, 22A Database (DB) 15 Control Unit 20 servers 21 Communications Department 22 Memory section 23 Control Unit

Claims

1. A database, A terminal device having a control unit using artificial intelligence capable of searching the database, The control unit continuously acquires text information uttered by a user during a conversation using a conversation app launched on the terminal device, and each time the user posts a label indicating the user's emotion, stores reference data in the database including the text information uttered by the user and the label indicating the user's emotion for a predetermined period before and after the time the label was posted, and if the amount of the stored reference data exceeds a threshold, when the emotion expressed by the user is detected, extracts reference data including a label corresponding to the emotion expressed by the user from the database using the artificial intelligence, and learns the extracted reference data using the artificial intelligence to estimate the text information uttered by the user.

2. The terminal device according to claim 1, The artificial intelligence is a large-scale language model implemented with RAG, The RAG is a terminal device capable of searching the database, extracting the reference data stored in the database, and transmitting the extracted reference data to the large-scale language model.

3. The terminal device according to claim 1, The control unit confirms with the user whether the estimation result of the text information uttered by the user is correct, and if the estimation result is correct, further stores reference data including the estimated text information and a label corresponding to the emotion expressed by the user in the database.

4. The terminal device according to claim 1, A terminal device wherein the reference data includes, in addition to text information uttered by the user during the specified period and a label indicating the user's emotions, data on the number of conversation partners with whom the user spoke during the specified period or data on the attributes of the conversation partners.

5. Depending on the terminal device, continuously acquiring text information uttered by a user during a conversation using a conversation application activated on the terminal device; Each time a label indicating the user's emotion is posted by the user, reference data including text information uttered by the user and the label indicating the user's emotion is stored in a database during a predetermined period around the time the label was posted; extracting, using artificial intelligence, reference data from the database, the reference data including a label corresponding to an emotion expressed by the user when the emotion expressed by the user is detected, if the amount of stored reference data exceeds a threshold; using the artificial intelligence to learn the extracted reference data, thereby estimating the text information uttered by the user; A prediction method to perform.

Images