Interaction system and program

Abstract

To provide an interaction system with which a user may have a sense of affinity or a sense of immersion.SOLUTION: A system for interacting with an avatar is configured to: transmit a text or voice data for interaction input by a user to a language analysis site; receive a response comment calculated by the language analysis site and information on emotion category; and present, by the avatar, the response comment with texts or voice, while showing motions and expression of the avatar or performing effects on the basis of data on the motions and the expression of the avatar or the effects associated in advance with the emotion category.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a dialogue system and a program. [Background technology]

[0002] As a conventional technique, a dialogue system is known in which an avatar is displayed as a dialogue partner of a user and the user can dialogue with the avatar.

[0003] Patent Document 1 describes an invention of an avatar providing device that allows a user to select one of two or more avatars and communicate with the avatar by transmitting information of the avatar selected by the user to a terminal device. This avatar providing device has a dialogue model learned by machine learning, and can execute processes such as purchases and store reservations through dialogue with the user.

[0004] Patent document 2 describes an invention of an electronic order taking system in which a user can log in to a matching site via a network and interact with an avatar, an agent of their choice, from the screen of their terminal, which provides support for matching.

[0005] Although the dialogue systems in both Patent Documents 1 and 2 enable a dialogue between the user and avatar by the avatar responding to the user, they do not take into consideration the display of the avatar's emotional expressions, and therefore the user is unable to feel a sense of closeness or immersion in the avatar. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2023-1577 A [Patent Document 2] JP 2018-116427 A Summary of the Invention [Problem to be solved by the invention]

[0007] The present invention has been made in consideration of the above circumstances, and has an object to provide a dialogue system that gives the user a sense of familiarity and immersion. [Means for solving the problem]

[0008] The invention described in claim 1 is a system for interacting with an avatar, comprising one or more processors that transmit dialogue information input by a user to a language analysis site, receive information regarding a response comment and an emotion category calculated by the language analysis site, and express the response comment in text or voice while directing the avatar's movements, expressions, or effects based on data regarding the avatar's movements, expressions, or effects that are pre-associated with the emotion category. The invention described in claim 2 is the dialogue system described in claim 1, wherein the emotion categories include at least one of sadness, anger, surprise, joy, worry, fear, excitement, confusion, curiosity, favoritism, discomfort, dependability, expectation, gratitude, seriousness, tension, relief, hope, impatience, mourning, indifference, sense of accomplishment, criticism, sympathy, understanding, humility, caution, affirmation, denial, understanding, doubt, shyness, and relaxation. The invention described in claim 3 is the dialogue system described in claim 1, wherein the one or more processors count the number of interactions with the user or the number of interactions by emotional category, and change data regarding the avatar's movements, facial expressions, or effects based on the number of interactions. The invention described in claim 4 is a program that enables a computer to realize a function of acquiring information regarding a response comment and an emotion category calculated from dialogue information input by a user, and a function of expressing the response comment in text or voice while directing the movement, facial expression, or effect of an avatar based on data regarding the movement, facial expression, or effect of the avatar that is pre-associated with the emotion category. The invention described in claim 5 is the program described in claim 4, wherein the emotion categories include at least one of sadness, anger, surprise, joy, worry, fear, excitement, confusion, curiosity, favoritism, discomfort, dependability, expectation, gratitude, seriousness, tension, relief, hope, impatience, mourning, indifference, sense of accomplishment, criticism, sympathy, understanding, humility, caution, affirmation, denial, understanding, doubt, shyness, and relaxation. The invention described in claim 6 is the program described in claim 4 that realizes a function of counting the number of interactions with the user or the number of interactions by emotional category, and a function of changing data regarding the avatar's motion, facial expression, or effects based on the number of interactions. Effect of the Invention

[0009] According to the present invention, a dialogue system is provided that allows a user to feel a sense of familiarity and immersion by endowing an avatar with emotional expression. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram showing an overall configuration of a dialogue system to which an embodiment of the present invention is applied; [Diagram 2] FIG. 2 is a diagram illustrating a functional configuration of a terminal device of the dialogue system. [Diagram 3] FIG. 1A is a diagram showing a screen of the dialogue system, FIG. 1B is a diagram showing an avatar expressing the emotion of joy, and FIG. 1C is a diagram showing an avatar expressing the emotion of anger. [Figure 4] FIG. 2 is a diagram showing a history of a dialogue between a user and an avatar. [Diagram 5] FIG. 13 is a diagram showing a data table in which emotion categories are associated with data on actions, effects, and facial expressions. [Figure 6] FIG. 13 is a diagram showing data recording the number of interactions. [Figure 7] FIG. 2 is a diagram showing a processing flow of a terminal device constituting the dialogue system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The dialogue system according to the present invention will be described with reference to the drawings. Note that the present invention is not limited to the following embodiments.

[0012] First Embodiment FIG. 1 is a diagram showing the overall configuration of a dialogue system to which the present embodiment is applied. The dialogue system 1 in this embodiment is a system in which a terminal device 10 used for a user to interact with an avatar, a server 30 that manages a dialogue application program, and a language processing server 40 are connected via a network 50. The network 50 is, for example, a network such as the Internet, and the user can access various service providing sites and the language processing server 40 from the terminal device 10 via the Internet. The dialogue system 1 exemplified in this embodiment is a system that can provide a dialogue service by, for example, transmitting dialogue information input by the user to the language processing server 40 when the user interacts with an avatar, and receiving the result of language analysis performed by the language processing server by the terminal device 10. The server 30 stores a program of an application that executes the dialogue service, and transmits the program to the terminal device 10 via the network 50. In other words, the user can receive the dialogue service of this embodiment only by downloading the application program that executes the dialogue service to the terminal device 10.

[0013] An avatar is a virtual character that represents an alter ego of a person in a virtual space displayed by a computer. However, the appearance of the avatar is not limited to an alter ego of a person, and in this specification, avatars include characters that have animal elements, animal characters, and even non-living inorganic characters such as robot characters, as long as the characters can have a conversation with the user. The user can input conversation information to the terminal device 10, hear the response from the avatar by voice, and have a conversation while looking at the emotional expressions of the displayed avatar. The purpose of the conversation may be to receive information on recommended products or travel destinations, to learn about unknown information, or to simply enjoy a conversation with the avatar without any specific purpose. In this embodiment, the dialogue information is linguistic information related to the dialogue, and may be text such as sentences, words, and symbols, or may be audio.

[0014] The terminal device 10 is composed of a computer terminal device such as a smartphone, tablet, etc. The terminal device 10 has a control unit 11 which is a processor (CPU (Central Processing Unit)) that controls the entire device, a memory 12 such as a RAM (Random Access Memory) used as a working area during calculations, and a storage unit 13 which is a storage device such as an HDD (Hard Disk Drive) or semiconductor memory used to store programs, various setting data, etc. The terminal device 10 also has a communication unit 14 that transmits and receives voice and text data via the network 50. It also has an operation unit 15 such as a touch panel, keyboard, pointing device, etc. that accepts input operations from a user of the terminal device 10, a display unit 16 consisting of a liquid crystal display or the like that displays images and text information to the user, and a display control unit 17 that controls the display unit 16. The terminal device 10 also has an imaging unit 18 that captures an image of the user, etc., and an audio input / output unit 19 that inputs and outputs audio.

[0015] The server 30 is composed of a computer device such as a workstation, a desktop PC, or a notebook PC. The server 30 has a control unit 31, which is a processor (CPU (Central Processing Unit)) that controls the entire device, a memory 32 such as a RAM (Random Access Memory) used as a working area during calculations, and a storage unit 33, which is a storage device such as a HDD (Hard Disk Drive) or a semiconductor memory used to store programs and various setting data. The server 30 also has a communication unit 34 that transmits and receives data via a network 50. The server 30 also has an operation unit 35 such as a keyboard, a pointing device, or a touch panel that accepts input operations from a user on the server 30 side, a display unit 36 ​​consisting of a liquid crystal display or the like that displays images and text information to server users, and a display control unit 37 that controls the display unit 36. The storage unit 33 stores an application program of the dialogue system, and the application program of the dialogue system is downloaded to the terminal device 10 via the network 50 in response to a request from a user. Here, the application program of the dialogue system does not necessarily have to be downloaded from the storage unit 33 of the server 30. For example, the dialogue application program stored in the storage unit 33 of the server 30 can be uploaded in advance to a site from which various applications can be downloaded, and can be downloaded to the terminal device 10 by using an application download service in response to a request from a user.

[0016] The language processing server 40 includes a large language model. The large language model can generate text of a conversation content (response comment) in response to input conversation text, and generate emotion categories by performing emotion analysis on the input text. The large language model can use known technology. For example, ChatGPT (Chat Generative Pretrained Transformer) by OpenAI can be used. The language processing server 40 performs emotion analysis on the input text and generates one of the emotion categories, for example, sadness, anger, surprise, joy, worry, fear, excitement, confusion, curiosity, favoritism, discomfort, dependability, expectation, gratitude, earnestness, tension, relief, hope, impatience, mourning, indifference, sense of accomplishment, criticism, sympathy, understanding, humility, caution, affirmation, negation, understanding, doubt, shyness, and relaxation.

[0017] In this embodiment, the dialogue information is transmitted to the language processing server 40 and language analysis is performed. However, since language analysis is usually performed based on text-based information, the voice data input by the user is converted to text before language analysis. In this embodiment, the input voice is converted to text in the function control unit 26 (see FIG. 2) of the terminal device 10. The voice-to-text conversion function can use a known conversion model. For example, voice data can be transmitted to a site that provides a voice transcription service via the network 50, and text data can be received. There are many known techniques for converting voice data to text, and in particular, the conversion can be performed by the language processing server 40, the terminal device 10, or a conversion service provided on the cloud via the network 50. For example, Whisper by OpenAI can be used. Therefore, the phrase "transmitting the dialogue information to the language analysis site" in the claims includes both the case where, when the user input is voice, the dialogue information is transmitted as data obtained by converting voice data into text and the case where the voice data is transmitted to the language processing server. Of course, when the user input is text, the case where the dialogue information is transmitted as is to the language processing server is also included in "transmitting the dialogue information to the language analysis site".

[0018] The "one or more processors" in this case are constituted by CPUs used in the terminal device 10, the server 30, and various devices connected to the terminal device 10 and the server 30, and realize various functions in this embodiment. 1 do not necessarily have to be in the same housing, and may be understood as a system. When the main body and the housing are different, they are connected by wire or wirelessly.

[0019] 2 is a diagram showing a functional configuration of the terminal device 10. The terminal device 10 has a reception unit 20 that receives user input, a storage unit 13 that stores the input information, etc., a communication unit 14 that transmits and receives data such as voice and text, a display unit 16 that displays the text of the dialogue and an avatar, etc., an imaging unit 18 that captures an image of the user's face, etc., a function control unit 26 that performs drawing and function control, and an output unit 21 that outputs the stored information as data or voice.

[0020] The storage unit 13 includes a motion data storage unit 22 , an effect data storage unit 23 , a facial expression data storage unit 24 , and a program storage unit 25 . The movement data storage unit 22 stores information relating to the movement of the avatar, the effect data storage unit 23 stores information relating to the effects of the avatar, the facial expression data storage unit 24 stores information relating to the facial expressions of the avatar, and the program storage unit 25 stores data for the application program of the dialogue system. An avatar's "movement data" refers to data related to the avatar's general movements, not just upper body movements such as hand raising and lowering gestures, palm movements, and bowing. An avatar's "effect data" refers to data such as marks that are displayed around the avatar and have the effect of expressing the avatar's emotions, such as musical note symbols or angry marks. An avatar's "expression data" refers to information about the avatar's facial expressions, such as the degree to which the eyebrows are raised, the shape of the mouth, the opening of the eyes, and other facial elements that express emotions.

[0021] The program storage unit 25 stores the application program of the interactive system received by the communication unit 14 from the server 30 or the application download site via the network 50. Other information relating to the user is stored in the storage unit 13.

[0022] The function control unit 26 controls various functions of the terminal device 10. For example, the function control unit 26 starts an application of a dialogue system and controls an avatar. Here, the application of the dialogue system is, for example, software that can interact with a user and is installed in the terminal device 10. The function control unit 26 includes an avatar control unit 27 and an interaction count unit 28. For example, the avatar control unit 27 controls the avatar's movements based on information on the avatar's movements, controls the avatar's facial expression based on data on the avatar's expressions, and controls the avatar's performance based on data on the avatar's effects. In controlling the avatar described above, the avatar control unit 27 controls the avatar based on the emotion category received from the language processing server 40. That is, when the emotion category received from the language processing server 40 is sadness, the avatar control unit 27 controls the avatar to perform sad movements, sad expressions, and sad performances. The conversation count unit 28 counts the number of conversations between the user and the avatar. In this case, the conversation count is not only the total number of conversations but also the number of conversations for each emotion category. By counting the conversation count, it becomes possible to control the avatar so that it expresses more intimacy as the conversation count increases, and expresses more strongly the emotion of anger when the number of anger conversations reaches a predetermined threshold value.

[0023] The text data of the response message received from the language processing server 40 is converted into voice data by the function control unit 26. A known technique can be used for the text-to-voice conversion. It is not necessary for the function control unit 26 of the terminal device 10 to perform the text-to-voice conversion process, and for example, a service such as Google's Text to Speech can be used. The voice input / output unit 19 outputs voice data obtained by converting the response text into voice. The output unit 21 causes the display unit 16 to display the response text received from the language processing server 40.

[0024] FIG. 3(A) is a diagram showing a screen of the dialogue system, (B) is a diagram showing an avatar expressing an emotion of joy, and (C) is a diagram showing an avatar expressing an emotion of anger. As shown in Fig. 3(A), when an avatar is displayed on the screen of a terminal device of the dialogue system, the user can input the contents of the dialogue for the avatar by voice or text. When the icon on the left side of the lower part of the screen is touched, a text input screen is displayed and the user can input the contents of the dialogue in text. Also, when the icon on the right side of the lower part of the screen is touched, the user can input the contents of the dialogue by voice. The input voice is converted into text by the function control unit 26 of the terminal device 10. The voice-to-text conversion function can use a known conversion model. For example, voice data can be sent to a site that provides a voice transcription service via the network 50, and text data can be received. For example, Whisper by OpenAI can be used.

[0025] The text input by the user or the text converted from the voice is transmitted to the language processing server 40 via the network 50. The terminal device 10 receives the conversation text for response and the emotion category from the language processing server 40. The function control unit 26 converts the conversation text for response to the user received by the terminal device 10 from the language processing server 40 into voice, which is then output by the voice input / output unit 19, allowing the user to feel as if he or she is having a conversation with an avatar.

[0026] Furthermore, the terminal device 10 acquires not only the response conversation text but also emotion category data from the language processing server 40, and the avatar control unit 27 controls the avatar based on the emotion category. For example, when the terminal device 10 acquires the emotion category of joy, the avatar control unit 27 makes the avatar take an action that is a joyful pose, changes the avatar's facial expression to a smile, and displays musical note symbols representing joy around the avatar, as shown in FIG. 3(B). For example, when the terminal device 10 acquires the emotion category of anger, the avatar control unit 27 makes the avatar take an action that is an angry pose, changes the avatar's face to an angry expression, and displays marks representing anger around the head, as shown in FIG. 3(C).

[0027] FIG. 4 is a diagram showing the history of a dialogue between a user and an avatar. Even if the dialogue between the user and the avatar is by voice, the contents of the dialogue are stored as text in the storage unit 13 and can be viewed as history information. The speech bubble from the right side shows the contents of the utterance from the user, and the contents written on the right side show the response text, which is the utterance of the avatar. For example, when conveying information on a URL that displays a recommended site, if it is difficult to convey the information by voice alone, it is also possible to obtain the URL from the history information and access the linked destination. In this way, the display screen of the dialogue history has the effect of easily obtaining the desired information by obtaining the information from the screen that displays the dialogue history, with respect to information that is difficult to convey by voice but easy to convey by text.

[0028] FIG. 5 is a diagram showing a data table in which emotion categories are associated with data on actions, effects, and facial expressions. The first column on the far left stores emotion categories such as sadness, anger, surprise, joy, worry, fear, excitement, confusion, curiosity, favor, discomfort, dependability, expectation, gratitude, earnestness, tension, relief, hope, impatience, mourning, indifference, sense of accomplishment, criticism, sympathy, understanding, humility, caution, affirmation, denial, consent, doubt, shyness, and relaxation. The second column lists the location (link) of information on the action data of the avatar corresponding to the emotion category in the first column. The third column, Effects, lists the location of information on effects for producing the avatar. The fourth column, Data on facial expressions, stores the location of information on facial expressions associated with the emotion categories. As described above, data on actions, effects, and facial expressions are associated with each emotion category, but it is not necessary to use all three types of data, namely, action, effect, and facial expression, when expressing each emotion. In other words, avatar control unit 27 can use an appropriate combination of data, such as expressing an avatar using only action data, or expressing an avatar using effects and facial expressions.

[0029] Based on the emotion category received from the language processing server 40, the avatar control unit 27 refers to the data table shown in FIG. 5 stored in the memory unit 13 to obtain the location (link) of the data of actions, effects, and facial expressions corresponding to the emotion to which the response should be made, obtains the action data stored in the action data memory unit 22, the effect data stored in the effect data memory unit 23, and the facial expression data stored in the facial expression data memory unit 24, and controls the avatar actions, effects, and facial expressions corresponding to the emotion. The movement data stored in the movement data storage unit 22 is data related to expressing the movement of the avatar, and is, for example, still image or video data related to the movement of the avatar corresponding to each emotion, such as a movement of joy with both hands raised or an angry pose with hands on hips, or data of movement parameters for displaying the movement. The effect data stored in the effect data storage unit 23 is data related to the production effects that express the emotions of the avatar, such as images or videos of marks expressing anger or sadness, or data related to the method of displaying the emotion marks. The facial expression data stored in the facial expression data storage unit 24 is data that expresses the facial expression of the avatar, and is, for example, still image or video data of the facial expression itself, or parameter file data that expresses the shape and movement of the eyebrows, mouth, eyes, and cheeks. 5 describes the location of the data of the actions, effects, and facial expressions corresponding to the emotions, but in addition to the location information of the data, parameters related to the actions, effects, and facial expressions may be directly stored. It is sufficient if the table can obtain the data of the actions, effects, and facial expressions associated with the responding emotions.

[0030] FIG. 6 is a diagram showing data recording the number of conversations. The conversation counting unit 28 counts the number of conversations between the user and the avatar. The user's utterance and the avatar's response may be counted as one set, or the number of conversations for each may be counted. Furthermore, the conversation counting unit 28 counts the number of conversations by emotion category and stores the number of conversations in the storage unit 13 as a record of the number of conversations. When the number of conversations exceeds a predetermined threshold, the avatar responds with a familiar expression to the content of the conversation to which it responds, or expresses a familiar action during the user's utterance, or produces an effect that creates a sense of familiarity. Furthermore, when the number of times of an emotion category exceeds a predetermined threshold, the avatar control unit 27 controls the emotion expression to be stronger and more direct. For example, when the number of times of the emotion category of anger exceeds a threshold, the avatar control unit 27 controls the emotion expression to be expressed with a greater anger by setting the parameter of the angry expression to a larger value. When one session ends, the number of conversations themselves may continue to be accumulated while the number of emotion categories may be reset to zero before the next conversation begins, or the number of conversations may be reset to zero and then counted again in the next conversation session.

[0031] [Dialogue system processing] FIG. 7 is a diagram showing a processing flow of the terminal device 10 constituting the dialogue system. The following dialogue is executed on the assumption that the user has downloaded an application program for executing the dialogue service to the terminal device 10 in advance. The application program may be downloaded from the server 30 or from a download site for the application program on the network. The function control unit 26 shown in FIG. 2 starts processing of the dialogue application installed in the terminal device 10 of the dialogue system upon receiving a start instruction from the user (START). Next, the function control unit 26 displays a registration screen (step 701). Next, the storage unit 13 stores the user information input by the user and acquired by the reception unit 20 (step 702). The avatar control unit 27 displays an avatar on the display unit 16 of the terminal device 10 (step 703). Then, the user inputs the dialogue content for the avatar by voice or text (step 704). The function control unit 26 converts the input voice data into text (step 705). The communication unit 14 transmits the text data to the language processing server 40 (step 706). Next, the communication unit 14 receives the text and emotion category of the response message from the language processing server 40 (step 707). The function control unit 26 converts the response text into voice data (step 708). The avatar control unit 27 displays an avatar on the display unit 16 based on the received emotion category and the action, effect, and emotion data stored in the action data storage unit 22, the effect data storage unit 23, the emotion data storage unit 24, and the storage unit 13, respectively (step 709). When the avatar control unit 27 starts displaying the avatar on the display unit 16 in step 709, the output unit 21 outputs the voice data of the response message together with the display (step 710). The interaction count unit 28 counts the number of interactions and the number of interactions by emotion category (step 711). If the interaction is to be continued, the process returns to step 704 and continues the interaction (step 712). If the interaction is to be ended, the process ends by touching an end button or the like on the screen (END).

[0032] In this embodiment, it is described that the conversion from voice to text and the conversion from text to voice are performed by the function control unit 26 of the terminal device 10, but this does not need to be limited to processing within the terminal device, and it is also possible to use a well-known conversion service on the cloud via the network 50.

[0033] In this embodiment, the language used in the dialogue is also converted as appropriate. Even if the language input by the user is different from the language handled by the language processing server 40, it is sufficient to use either the terminal device 10, the language processing server 40, or another translation site on the cloud.

[0034] Furthermore, in the present embodiment, an embodiment in which transmission and reception is performed via the language processing server 40 and the network 50 has been described as an example, but if the server 30 is equipped with a language processing model that can return response text and emotion categories, the language processing server 40 and the server 30 can also be integrated. [Explanation of symbols]

[0035] 1... dialogue system, 10... terminal device, 11... control unit, 12... memory, 13... storage unit, 14... communication unit, 15... operation unit, 16... display unit, 17... display control unit, 18... imaging unit, 19... voice input / output unit, 20... reception unit, 21... output unit, 22... action data storage unit, 23... effect data storage unit, 24... facial expression data storage unit, 25... program storage unit, 26... function control unit, 27... avatar control unit, 28... dialogue count unit, 30... server, 31... control unit, 32... memory, 33... storage unit, 34... communication unit, 35... operation unit, 36... display unit, 37... display control unit, 40... language processing server, 50... network

Claims

1. One or more processors; The one or more processors: Transmit the dialogue information entered by the user to a language analysis site, receiving information about the response comment and the emotion category calculated by the language analysis site; expressing the response comment in text or voice while directing a movement, expression, or effect of the avatar based on data related to the movement, expression, or effect of the avatar that is previously associated with the emotion category; A system for interacting with the avatar.

2. The emotion category includes at least one of sadness, anger, surprise, joy, worry, fear, excitement, confusion, curiosity, favor, discomfort, dependability, expectation, gratitude, seriousness, tension, relief, hope, impatience, mourning, indifference, sense of accomplishment, criticism, sympathy, understanding, humility, caution, affirmation, denial, understanding, doubt, shyness, and relaxation. The dialogue system according to claim 1 .

3. The one or more processors: Counting the number of interactions with the user or the number of interactions by emotion category; Changing data related to the movement, facial expression, or effect of the avatar based on the number of interactions. The dialogue system according to claim 1 .

4. On the computer, A function of acquiring information regarding a response comment and an emotion category calculated from dialogue information input by a user; a function of expressing a response comment in text or voice while directing a movement, expression, or effect of an avatar based on data related to the movement, expression, or effect of the avatar that is previously associated with the emotion category; A program to achieve this.

5. The emotion category includes at least one of sadness, anger, surprise, joy, worry, fear, excitement, confusion, curiosity, favor, discomfort, dependability, expectation, gratitude, seriousness, tension, relief, hope, impatience, mourning, indifference, sense of accomplishment, criticism, sympathy, understanding, humility, caution, affirmation, denial, understanding, doubt, shyness, and relaxation. The program according to claim 4.

6. A function of counting the number of interactions with the user or the number of interactions by emotion category; A function of changing data related to the motion, facial expression, or effect of the avatar based on the number of interactions; The program according to claim 4, which realizes the above.

Images