Conversation device, conversation method, and computer program

Abstract

To provide a conversation system that can provide, at an appropriate time in a conversation with a user, visual materials suited to promoting the conversation, and can promote the conversation on the basis of discretionary visual materials.SOLUTION: A conversation device is for carrying out a conversation with a user, including: a conversation engine 390 for generating a system speech text on the basis of a conversation history with a user; an image generation AI module 160 for generating visual materials on the basis of either the system speech text or user speech text; and an image generation text generation module 392 for generating text with which the image generation AI module 160 can generate an appropriate visual material for an input, on the basis of either the system speech text or the user speech text, and giving the text to the image generation AI module 160.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] The present invention relates to natural language processing technology, and more particularly to a dialogue device that dialogues with people using voice or the like. [Background technology]

[0002] In spoken dialogue systems or text-based dialogue systems, it is known that presenting visual materials (photos, illustrations, videos, etc.) related to the topic can increase familiarity and make the dialogue more memorable. For this reason, dialogue systems that use visual materials are used.

[0003] For example, there are voice dialogue systems that collect visual materials in advance and present visual materials that are searched for using words obtained from the user's speech as keywords. Another example is a dialogue system that uses a dialogue scenario. A voice dialogue system that uses a dialogue scenario presents visual materials that are determined according to the dialogue scenario.

[0004] Conversely, there are systems that generate responses based on visual materials used in dialogue systems. Examples include VisualQA (see Non-Patent Document 1) and Visual Dialog (see Non-Patent Document 2). Both of these systems perform the task of providing an answer when they are given a question about a photograph, which is an example of visual material, and an object shown in the photograph.

[0005] One field in which such dialogue systems using visual materials are suitable is the prevention and improvement of dementia in the elderly. A method applied to the elderly in this field is called reminiscence therapy. Reminiscence therapy is a method that aims to prevent dementia or improve its symptoms by having the elderly reminisce about past events. It is said that the brain is activated when the elderly reminisce about past events, and in reminiscence therapy, a human practitioner presents visual materials to the elderly subject and engages in a dialogue. It would be preferable if the dialogue system could carry out this dialogue on behalf of the practitioner. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2022-133570 A [Patent Document 2] JP 2013-175097 A [Non-patent literature]

[0007] [Non-Patent Document 1] Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D.: Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017) [Non-Patent Document 2] Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, JMF, Parikh, D., and Batra, D.: Visual Dialog, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017) Summary of the Invention [Problem to be solved by the invention]

[0008] Both the system disclosed in Non-Patent Document 1 and the system disclosed in Non-Patent Document 2 are only compatible with the special case of answering questions about objects present (visible) in visual materials. Therefore, the situations in which these systems can be applied are very limited. For example, when these systems are used for reminiscence therapy, it is necessary to collect a large number of visual materials related to the past of each elderly person in advance. Furthermore, when a dialogue is executed, it is necessary to search and display any of the visual materials appropriate for the progress of the dialogue at that time at the appropriate time. If keywords are assigned to the visual materials in advance, it may be possible to search for visual materials. However, it is time-consuming to assign keywords to each visual material. Furthermore, there is no guarantee that the selected visual materials will be appropriate for developing the dialogue.

[0009] Therefore, an object of the present invention is to provide a dialogue system that can provide visual materials appropriate for developing a dialogue with a user in a timely manner and that can develop the dialogue based on any visual materials. [Means for solving the problem]

[0010] An interaction device according to a first aspect of the present invention is an interaction device for interacting with a user, and includes: a system utterance text generation means for generating system utterance text based on a dialogue history with the user; a visual material generation means for generating visual material based on either the system utterance text or the user's utterance text; and a text generation means for generating visual material generation text based on either the system utterance text or the user's utterance text so that the visual material generation means can generate visual material appropriate for the input, and providing the text for visual material generation to the visual material generation means.

[0011] Preferably, the means for generating text for generating visual materials includes a machine learning model trained to generate text for generating visual materials using an interaction history with a user as input.

[0012] More preferably, the dialogue device further includes a text generation means for generating relevant text related to the dialogue with the user by verbalizing the content of the visual material generated by the visual material generation means, and the system speech text generation means, the visual material generation means, or the text generation means for generating visual material further receives the relevant text as input.

[0013] More preferably, the dialogue history includes non-verbalized information indicating at least a portion of the dialogue situation in a non-verbal manner, and the visual material generation means or the text generation means for generating visual material converts the non-verbalized information of the dialogue history into text and links it with other text included in the dialogue history to input it.

[0014] Preferably, the dialogue context represents attributes of the user's utterance.

[0015] More preferably, the dialogue context represents an attribute of the user.

[0016] A computer program according to a second aspect of the present invention is a computer program that causes a computer to function as an interactive device for interacting with a user, and causes the computer to function as: system utterance text generation means for generating system utterance text based on a dialogue history with the user; visual material generation means for generating visual material based on either the system utterance text or the user's utterance text; and visual material generation text generation means for generating text for visual material generation based on either the system utterance text or the user's utterance text so that the visual material generation means can generate visual material appropriate for the input, and providing the text for visual material generation to the visual material generation means.

[0017] An interaction method according to a third aspect of the present invention is a computer-implemented interaction method for interacting with a user, comprising the steps of: generating a system utterance text by the computer based on a dialogue history with the user; generating visual material by the computer based on either the system utterance text or the user's utterance text; and generating visual material generation text based on either the system utterance text or the user's utterance text so that appropriate visual material can be generated in the visual material generating step for the input, and using the text as input to the visual material generating step.

[0018] The above and other objects, features, aspects and advantages of the present invention will become apparent from the following detailed description of the invention taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a block diagram showing a schematic configuration of a dialogue system according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a schematic diagram for explaining the general operation of the dialogue system shown in FIG. [Diagram 3] FIG. 3 is a functional block diagram of the dialogue server shown in FIG. [Figure 4]FIG. 4 is a diagram showing a schematic configuration of a dialogue scenario used by the dialogue server shown in FIG. [Diagram 5] FIG. 5 is a flowchart showing a control structure of a main routine of a program executed by the dialogue server shown in FIG. 3 for dialogue processing. [Figure 6] FIG. 6 is a flowchart showing a control structure of a program for implementing the dialogue start process among the programs shown in FIG. [Figure 7] FIG. 7 is a flowchart showing a control structure of a program for implementing response processing based on a dialogue scenario, among the programs shown in FIG. [Figure 8] FIG. 8 is a flowchart showing a control structure of a program for implementing the chat process among the programs shown in FIG. [Figure 9] FIG. 9 is a flowchart showing a control structure of a program for achieving a process of acquiring chat text, out of the programs shown in FIG. [Figure 10] FIG. 10 is a schematic diagram showing an example of a dialogue in the dialogue system according to the second embodiment of the present invention. [Figure 11] FIG. 11 is a block diagram showing the functional configuration of a dialogue server according to the second embodiment of the present invention. [Figure 12] FIG. 12 is a diagram showing a learning process of a text generation module for image generation used by the dialogue server according to the second embodiment. [Figure 13] FIG. 13 is a flowchart showing a control structure of a main routine of a program executed for dialogue processing by the dialogue server in the second embodiment. [Figure 14] FIG. 14 is a flowchart showing a control structure of a program for implementing the dialogue start process shown in FIG. [Figure 15] FIG. 15 is a flowchart showing a control structure of a program for implementing the response process in accordance with the dialogue scenario shown in FIG. [Figure 16] FIG. 16 is a flowchart showing a control structure of a program for implementing the chat process shown in FIG. [Figure 17] FIG. 17 is an external view of the dialogue server shown in FIG. [Figure 18] FIG. 18 is a hardware block diagram of the dialogue server shown in FIG. [Figure 19] FIG. 19 is a hardware block diagram of the terminal shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] In the following description and drawings, the same parts are given the same reference numbers, and therefore detailed descriptions thereof will not be repeated.

[0021] 1. First embodiment 1.1 Overview Fig. 1 shows a schematic configuration of a dialogue system 50 according to a first embodiment of the present invention. Referring to Fig. 1, the dialogue system 50 includes a stuffed toy-shaped terminal 60 and a dialogue server 62 for receiving a voice signal 64 of a dialogue partner (user) from the terminal 60, generating a speech signal 66 including a voice signal and an image signal for dialogue with the dialogue partner, and transmitting the speech signal 66 to the terminal 60.

[0022] The terminal 60 includes a microphone and a speaker not shown in FIG. 1, and an LCD (Liquid Crystal Display) 68 for displaying an image received from the dialogue server 62. The terminal 60 has the external shape of a stuffed toy so that a natural dialogue with a dialogue partner can be realized. Of course, the terminal 60 is not limited to a stuffed toy shape, and may be a so-called smartphone, a panel type computer, or some kind of figure. The hardware configuration of the terminal 60 will be described later with reference to FIG. 19.

[0023] The dialogue server 62 used in this embodiment is a general-purpose computer. The hardware configuration thereof will be described later with reference to FIGS.

[0024] FIG. 2 shows a schematic diagram of the general operation of the dialogue system 50. In FIG. 2, a dialogue by this dialogue system 50 is carried out between a system 100 and a user 102. First, the system 100 utters an opening utterance 104. In this example, the opening utterance 104 is "Did you have a school trip in junior high school?" This opening utterance 104 is specified as the first utterance of the dialogue in a dialogue scenario prepared in advance. The dialogue scenario will be described later. Most of the utterances by the system uttered according to the dialogue scenario are in the form of questions.

[0025] In this embodiment, the system 100 extracts words from the opening utterance 104 when the opening utterance 104 is spoken, and generates 106 an image 108 related to the words as visual information using a so-called image generation AI (Artificial Intelligence) based on the words. The image 108 is displayed by the LCD 68 shown in FIG. 1. Any image generation AI may be used as long as it can generate an image from text, such as one based on a diffusion model or one based on a GAN (Generative Adversarial Network). The image 108 may be a realistic one like a so-called photograph, or an illustration that imitates handwriting. Furthermore, the image may not be a still one like a photograph, but may be a moving image that imitates a live-action moving image, or may be a moving image that imitates an animation.

[0026] The system 100 generates the image 108 and also generates 112 text 114 that verbalizes the image 108 from the contents of the image 108. A so-called caption generation AI can be used to generate 112 the text 114. In this example, the caption generated for the text 114 is "This is an image of students buying lots of souvenirs."

[0027] By displaying the image 108 together with the opening utterance 104 in this way, the user 102 is inspired by both the opening utterance 104 and the image 108 to recall the past, and utters a response utterance 110 such as, for example, "Yes, I went there when I was in the third year of junior high school." This response utterance 110 indicates some kind of positive response to the opening utterance 104, since it starts with "Yes."

[0028] In this embodiment, the system 100 performs the following process in response to receiving the response utterance 110 from the user 102. Some words (e.g., "souvenir", "student", etc.) are extracted from the text 114, and a new system utterance 118, "Do you remember what souvenir you bought?" is generated 116 using one of them (e.g., "souvenir") and uttered to the user 102. In the example shown in FIG. 2, an image such as the image 108 is not generated for this system utterance 118, but an image may be generated from the system utterance 118.

[0029] Assume that the user 102, in response to the system utterance 118 and the image 108, remembers that he bought manju as a souvenir on a school trip he took in his third year of junior high school. As a result, the user 102 makes a response utterance 120 saying, "It was manju. I love sweet things." Since the response utterance 120 contains the word "manju," the system 100 generates 122 an image 124 of manju and presents it to the user 102. Thereafter, the system 100 makes further utterances based on the generation 122. Thereafter, the dialogue between the user 102 and the system 100 develops in a similar manner.

[0030] 1.2 Configuration 3, the dialogue server 62 has the following functional configuration. That is, the dialogue server 62 includes a communication device 150 that communicates voice signals and image signals with the terminal 60 (see FIG. 1), a storage device 152 for storing dialogue scenarios prepared in advance, a voice recognition engine 156 for performing voice recognition on the user's voice received by the communication device 150 from the terminal 60 and converting it into text, and a dialogue engine 154 for generating a system utterance text that is a response utterance to the user utterance, using the content of the user utterance output by the voice recognition engine 156 and the dialogue scenario stored in the storage device 152. The dialogue engine 154 includes a user utterance semantic interpretation module (not shown) for interpreting the meaning of the user utterance by natural language processing.

[0031] Referring to FIG. 4, the dialogue scenario basically has a binary tree structure with an opening utterance (Question 1) as the root and question utterances (Question 1.1, Question 1.2, ...) assigned to each node. The first branch from a certain node is a branch when the user's response to the question utterance at that node is affirmative, and the second branch is a branch when the user's response is negative. For example, in the dialogue scenario shown in FIG. 4, if the user's response to Question 1 is affirmative, the system next utters Question 1.1 to the user. If the user's response to Question 1 is negative, the system next utters Question 1.2 to the user. The dialogue ends when it reaches the leaf at the end of the binary tree structure.

[0032] In this embodiment, the questions assigned to each node of the dialogue scenario are of a type that can be answered by either yes or no. In the case of this type of question, if the user's response utterance contains either "yes" or "no" or a similar phrase, the branch destination can be easily determined. In this embodiment, if these phrases are not included in the response utterance, or if the system 100 determines that it is appropriate to move to casual conversation, a casual conversation utterance is generated as described below, and the conversation follows a different course from the scenario. The system 100 always records where the dialogue was located in the dialogue scenario. Therefore, even during casual conversation, the system 100 can always return to the dialogue scenario based on the recorded information. In this case, control may be performed to return to the dialogue by selecting any node of the scenario at any time and uttering a question again.

[0033] The following process is the same whether the system utterance is an opening utterance or a subsequent utterance. Therefore, in the following description, the utterance generated by the dialogue engine 154 is simply called "system utterance" and the text is called "system utterance text". In principle, the system utterance specified by the dialogue scenario is a question type that can be answered by "yes" or "no". In this specification, this type of question is called "multiple choice type". As the name multiple choice type suggests, the question here may include not only two options such as "yes" and "no" but also three or more options. In contrast, the system utterance in the form of a chat may be a so-called in-depth question in response to the previous question, but is not limited to a question. In a chat, there is a possibility that there will be more utterances other than questions.

[0034] The dialogue server 62 further includes a speech synthesis engine 158 for synthesizing a speech signal from the spoken text generated by the dialogue engine 154 and providing the speech signal to the dialogue engine 154 .

[0035] The dialogue server 62 further includes an image generation AI module 160 for receiving system utterance text or user utterance text from the dialogue engine 154, generating an image that is a kind of visual material that visualizes the contents of the input utterance, and providing the image to the dialogue engine 154, a caption generation AI module 162 for verbalizing the image generated by the image generation AI module 160 and generating a caption that is a related text related to the image, and a chat response generation module 164 including a machine learning model for generating a chat response text with the user that is different from the dialogue scenario, using the dialogue history recorded so far in the dialogue history DB (DateBase) 396, and providing the chat response to the dialogue engine 154. This chat response is also a part of the utterance by the dialogue engine 154, and is a kind of system utterance. The decision as to when to perform image generation by image generation AI module 160 and whether to provide system spoken text or user spoken text to image generation AI module 160 may be determined based on rules based on the content of the text, or a machine learning model may be used that takes the speech history, system spoken text, and user spoken text as input and determines whether to perform image generation and, if image generation is to be performed, whether to select the system spoken text or the user spoken text.

[0036] In this embodiment, the image generation AI module 160 and the caption generation AI module 162 are part of the dialogue server 62. However, the present invention is not limited to such an embodiment. Either or both of the image generation AI module 160 and the caption generation AI module 162 may be provided as a network service outside the dialogue server 62. In that case, the dialogue server 62 uses the image generation AI module 160 or the caption generation AI module 162 via the network.

[0037] The dialogue server 62 further includes an attribute information DB 170 connected to the dialogue engine 154, which stores the attributes of the user (age, sex, family, address, lifestyle, work history, etc.) and outputs attribute information on the user in response to a query from the dialogue engine 154, a dialogue history DB 166 for storing the dialogue history between the system 100 and the user 102 and the caption output by the caption generation AI module 162 as part of the dialogue history, and a chat response candidate DB 168 for storing a large number of candidates that can be used as chat responses and that the chat response generation module 164 refers to when generating chats. In this embodiment, the dialogue history includes a log of the semantic analysis of the user's utterance, and specifically includes a semantic classification of the user's utterance. The text of the user's utterance itself is also saved as the dialogue history.

[0038] The dialogue engine 154 manages dialogue with the user in accordance with the outputs of the voice recognition engine 156, the voice synthesis engine 158, the image generation AI module 160, and the small talk response generation module 164, the dialogue scenario stored in the storage device 152, the user's attribute information stored in the attribute information DB 170, and the dialogue history stored in the dialogue history DB 166. On the other hand, the small talk response generation module 164 generates small talk responses using the output of the caption generation AI module 162, the dialogue history stored in the dialogue history DB 166, and the small talk response candidates stored in the small talk response candidate DB 168.

[0039] 1.3 Program Structure Fig. 5 shows in the form of a flowchart the control structure of a program for enabling the dialogue server 62 shown in Fig. 3 to dialogue with a user. Referring to Fig. 5, this program includes step 200 for reading a dialogue scenario from the storage device 152, connecting to the communication device 150, the voice recognition engine 156, the voice synthesis engine 158, the image generation AI module 160, the caption generation AI module 162, and the chat response generation module 164 shown in Fig. 3, and initializing memory areas.

[0040] This program further includes step 202 of starting a dialogue in accordance with the dialogue scenario read out in step 200, step 204 of waiting until a voice signal is received from terminal 60, and upon receiving the voice signal, accepting the voice signal as a voice signal of a user's utterance, step 206 of performing speech recognition on the voice signal accepted in step 204 and converting it into text, step 208 of performing semantic interpretation on the text obtained in step 206, and step 210 of branching the flow of control in accordance with the semantic interpretation performed in step 208.

[0041] In this embodiment, there are three branch destinations based on the meaning determination in step 210: step 212, step 214, and step 216. However, the present invention is not limited to such an embodiment. There may be four or more options. For example, if the user utterance contains "I don't understand" or a similar expression, it may be determined that the user does not fully understand the meaning of the system utterance, and a process of repeating or paraphrasing the system utterance may be introduced.

[0042] This program further includes step 212, which is executed when it is determined in step 210 that the user's utterance is unclear, and which requests the user to repeat the utterance and returns the flow of control to step 204; step 214, which is executed when it is determined in step 210 that the user has rejected the dialogue, and which issues an utterance to end the dialogue and terminates the execution of the program; and step 216, which determines whether or not the user's utterance should be responded to by chatting in step 210, and branches the flow of control according to the result. The semantic interpretation in step 208 is performed by a user utterance semantic interpretation module provided in the dialogue engine 154, based on whether or not the user's response utterance can be interpreted by natural language processing, and whether or not there is an expression in the user's response utterance that indicates refusal to answer. In addition, the determination in step 216 is performed based on the dialogue history, etc., including the system utterance and the user utterance. This determination is performed by the user utterance semantic interpretation module, based on information including the dialogue history, etc. The determination by the user utterance meaning interpretation module may be rule-based, or may be performed by a machine learning model that has been trained in advance to input a dialogue history and output a determination result.

[0043] The program further includes step 220, when it is determined in step 216 that the system should respond to the user's utterance with casual conversation, generating and outputting the casual conversation utterance and returning the flow of control to step 204, and step 218, when it is determined in step 216 that the system should respond in accordance with the dialogue scenario, making an utterance defined by the dialogue scenario and returning control to step 204.

[0044] Fig. 6 shows details of a program for realizing step 202 in Fig. 5. Referring to Fig. 6, step 202 includes step 240 of reading a dialogue scenario from storage device 152 shown in Fig. 3 and loading it into a memory, and step 242 of generating a voice signal of the first utterance (opening utterance) by providing an initial utterance text specified by a root node of the dialogue scenario to speech synthesis engine 158 shown in Fig. 3 and transmitting the generated voice signal to dialogue server 62 via communication device 150 shown in Fig. 3.

[0045] This program further includes step 244 of reading out the user's attributes from the attribute information DB 170 shown in FIG. 3, step 246 of inputting the spoken text used for the voice synthesis in step 242 and the user's attributes acquired in step 244 to the image generation AI module 160 shown in FIG. 3, and step 248 of transmitting an image generated and output by the image generation AI module 160 in response to the spoken text and the user's attributes to the terminal 60 via the communication device 150. By these steps 240, 242, 244, 246 and 248, for example, the opening utterance 104 shown in FIG. 2 is generated and uttered by the terminal 60 shown in FIG. 1, and an image corresponding to the opening utterance 104 is displayed by the LCD 68. Note that in this embodiment, text is assumed as the input to the image generation AI module 160. On the other hand, the user's attributes are often held by numerical values ​​or symbols other than normal text, for example, numbers such as 0 and 1, each of which is assigned a meaning of each attribute. There is no point in inputting such information to the image generation AI module 160. Therefore, in this embodiment, the value indicating the user's attribute is converted into text representing the content of the attribute, concatenated with other text with a separation token in between, and input to the image generation AI module 160.

[0046] The processing of step 218, which is executed when the determination in step 216 is that a response based on the scenario should be made (the determination in step 216 is negative), is realized by a program having a control structure represented by the flowchart shown in FIG.

[0047] 7, this program includes step 260 for branching the control flow depending on whether the current dialogue mode is chatting or not, step 262 for branching the control flow depending on whether the user's response utterance includes "Yes" or an expression equivalent thereto when the determination in step 260 is negative, step 264 for selecting a branch corresponding to "Yes" of the node currently being executed in the dialogue scenario and synthesizing a question corresponding to a new node at the branch destination when the determination in step 262 is positive, and step 266 for selecting a branch corresponding to "No" of the node currently being executed in the dialogue scenario and synthesizing a question corresponding to a new node at the branch destination when the determination in step 262 is negative. Note that the utterance text of the dialogue scenario that is the source of the voice synthesis in steps 264 and 266 may include a variable. By replacing this variable with a specific word included in the previous utterance of the user 102, the system 100 can respond according to the content of the utterance of the user 102 while following the dialogue scenario.

[0048] This program further includes, when the determination in step 260 is positive, a step 268 of identifying a node in the dialogue scenario that has been stored in advance as a point at which the dialogue according to the dialogue scenario is to be resumed, and a step 270 of obtaining from the node identified in step 268 the spoken text of the question assigned to that node and synthesizing the text by voice.

[0049] This program further includes step 272 of transmitting the voice signal synthesized in step 264, step 266 or step 270 to terminal 60, step 274 of searching and acquiring the dialogue history in dialogue history DB 166 shown in FIG. 3, and step 276 of branching the control flow depending on whether or not to generate an image based on the current spoken text, based on the dialogue history acquired in step 274 and the current spoken text.

[0050] This program further includes step 278, in which, when the determination in step 276 is positive, one of the system utterance text and the user utterance text is selected, inputted to the image generation AI module 160 shown in FIG. 1, and the image generation AI module 160 is made to generate an image. The text to be selected in step 278 may be determined based on a rule base, or may be determined by a machine learning model that inputs both texts, user attributes, and dialogue history. As an example of the rule, a simple rule such as which text contains a larger number of content words or which text is longer may be used. In this case, the user utterance text may basically be given priority, and the system utterance text may be selected only when the number of content words in the system utterance text is greater than the number of content words in the user utterance text.

[0051] This program further includes step 280 of presenting the image generated and output by the image generation AI module 160 to the terminal 60 to the user, and step 282 of inputting the image output by the image generation AI module 160 to the caption generation AI module 162 to generate a caption for the image. This program further includes step 284 of registering the image generated in step 276 and the caption generated in step 280 in the dialogue history DB 166 as a dialogue history. In this way, the dialogue history includes the image generated by the image generation AI module 160 and the caption generated from the image. Therefore, when the next system utterance text is generated according to the dialogue history, the content of the image generated previously is also reflected.

[0052] The program further includes step 286, which is executed when the determination in step 276 is negative, or when the determination in step 276 is positive and the processing of steps 278 to 284 is completed, and which saves the node positions in the current scenario in a storage device and terminates execution of the program.

[0053] Fig. 8 shows a detailed control structure of step 220 shown in Fig. 5. Referring to Fig. 8, step 220 includes step 300 of searching and acquiring a dialogue history in the dialogue history DB 166 shown in Fig. 3, step 302 of inputting a user attribute and a dialogue history to the chat response generation module 164 shown in Fig. 3 and acquiring a chat text from the chat response generation module 164, step 304 of synthesizing a voice of the chat text acquired in step 302 and transmitting it to the terminal 60, and step 306 of branching the flow of control depending on whether or not to end the chat and return to the dialogue scenario.

[0054] The program further includes a step 308 for determining, when the determination in step 306 is positive, a node at which the dialogue scenario should be resumed in accordance with pre-stored information, and a step 310 for performing speech synthesis on the text of the dialogue scenario assigned to the node determined in step 308, and transmitting the synthesized speech signal to the terminal 60.

[0055] The program further includes step 312, which branches the flow of control according to whether or not an image should be generated when the determination in step 306 is negative, and when the determination in step 306 is positive and the processing up to step 310 is completed. In this embodiment, an image is generated from the chat text obtained in step 302, regardless of whether the determination in step 306 is positive or negative.

[0056] This program further includes step 314 of inputting the system utterance text or the user utterance text to the image generation AI module 160 shown in Fig. 3 when the determination in step 312 is positive, step 316 of transmitting an image generated and output by the image generation AI module 160 as a result of the processing in step 314 to the terminal 60, step 318 of inputting the image obtained in step 316 to the caption generation AI module 162, and step 320 of saving the caption generated and output by the caption generation AI module 162 as a result of the processing in step 318, together with the text of the chat response and the text generated in step 310, if any, in the dialogue history DB 166 as a dialogue history, and terminating the execution of this program. When the determination in step 312 is negative, the execution of this program is immediately terminated.

[0057] In this embodiment, the determination in step 306 is made by a pre-trained machine learning model based on the user attributes, the final chat text, and the spoken text and caption in the dialogue history. Of course, this determination may be made by rule-based processing based on the user attributes, the final chat text, and the spoken text and caption in the dialogue history. The determination in step 312 is similar. In addition, which text to select in step 314 may be determined based on a rule base, or may be determined by a machine learning model that inputs both texts, user attributes, and dialogue history.

[0058] Fig. 9 is a flowchart showing a control structure of a program for implementing step 302 in Fig. 8. Referring to Fig. 9, this program uses chat response candidate DB 168 to generate chat response candidates.

[0059] A plurality of chat response candidates are stored in advance in the chat response candidate DB 168. These candidates are classified based on various viewpoints such as whether they are likely to attract the user's interest as a chat response and whether they contain so-called hate speech by applying a language model to text collected from the web, and are determined to be appropriate as a chat response. A large number of these chat response candidates are created and organized into a database so that they can be searched using key words as keywords.

[0060] This program includes step 350 of extracting important keywords from the user's utterances in the dialogue history, and obtaining one or more chat response candidates from the chat response candidate DB 168 by searching the chat response candidate DB 168 using the important keywords as a key, step 352 of scoring and ranking the chat response candidates obtained in step 350, and step 354 of selecting the chat candidate with the highest score from the scores obtained for each chat candidate in step 352, and returning to the processing of step 220.

[0061] The scoring of each chat candidate in step 352 uses a pre-trained machine learning model (such as BERT (Bidirectional Encoder Representations from Transformers)) to input a pair of the dialogue history and the chat candidate and output the probability that the chat candidate is appropriate in the context. Of course, scoring may be performed according to some rules.

[0062] 1.4 Operation Hereinafter, the operation of the dialogue system 50 (FIG. 1) according to the first embodiment will be described mainly with reference to the flowcharts shown in FIGS.

[0063] When the dialogue system 50 starts operating, the dialogue server 62 executes an initial process (step 200 in FIG. 5). In the initial process, the dialogue engine 154 of the dialogue server 62 establishes a connection with the terminal 60 via the communication device 150 and checks the connections with each part of the dialogue server 62.

[0064] In the following step 202, the dialogue engine 154 (see FIG. 3) of the dialogue server 62 reads the dialogue scenario from the storage device 152 shown in FIG. 3 and loads it into memory (step 240 in FIG. 6). In step 202, the dialogue engine 154 reads from memory the text of an opening utterance (question) assigned to the root node of the binary tree constituting the dialogue scenario. The dialogue engine 154 provides this text to the speech synthesis engine 158 shown in FIG. 3. The speech synthesis engine 158 performs speech synthesis based on the provided text, and provides the synthesized speech signal to the dialogue engine 154. The dialogue engine 154 transmits this speech signal to the terminal 60 (step 242).

[0065] The dialogue engine 154 reads out the user's attributes from the attribute information DB 170 (FIG. 3) (step 244). The dialogue engine 154 further provides the text of the opening utterance of the dialogue scenario and the user's attributes to the image generation AI module 160 (FIG. 3) (step 246). The image generation AI module 160 generates an image based on this text and provides it to the dialogue engine 154.

[0066] The dialogue engine 154 provides the image signal output by the image generation AI module 160 to the terminal 60 via the communication device 150 (step 248). The dialogue engine 154 then goes into a standby state for a user utterance (step 204).

[0067] 1, the terminal 60 reproduces, by an amplifier and speaker (not shown) and an LCD 68, audio and images from a speech signal 66 representing the audio and images received from the dialogue server 62. When the user makes some utterance in response to the audio and images, the terminal 60 transmits an audio signal of the utterance to a communication device 150 of the dialogue server 62, and the communication device 150 provides the received audio signal to a dialogue engine 154.

[0068] 5, the dialogue engine 154 receives the voice signal of the user utterance in step 204. The dialogue engine 154 provides the voice signal to the speech recognition engine 156. The speech recognition engine 156 performs speech recognition on the voice signal and outputs text representing the user utterance to the dialogue engine 154.

[0069] In the next step 208, the dialogue engine 154 performs semantic interpretation on the text of the user utterance received from the voice recognition engine 156. The dialogue engine 154 branches the flow of control according to the result of this semantic interpretation (step 210) as follows. That is, if the content of the user utterance is unclear, the dialogue engine 154 synthesizes a voice signal of an utterance requesting the user to repeat the utterance in step 212 and transmits it to the terminal 60. After this, the dialogue engine 154 returns to the standby state for a user utterance (step 204). If the dialogue engine 154 determines in step 210 that the user refuses the dialogue as a result of the semantic interpretation of the user utterance, it performs a process required to end the dialogue in step 214 and ends the execution of the program.

[0070] On the other hand, when it is determined in step 210 that the content of the user utterance is neither unclear nor refusing dialogue, the dialogue engine 154 determines in step 216 whether or not to respond to the user utterance with a chat. This determination is made based on the user's attributes and dialogue history, as described above. When it is determined that a chat response should not be made, the dialogue engine 154 executes a response process based on a scenario in step 218 and returns control to step 204. On the other hand, when it is determined that a chat response should be made, the dialogue engine 154 executes a chat process in step 220 and returns control to step 204 after the chat process is completed.

[0071] In step 218, the dialogue server 62 operates as follows: Referring to Fig. 7, if the current dialogue mode is chatting (the determination in step 260 is positive), the dialogue engine 154 identifies a resumption part of the dialogue scenario based on pre-stored information (268). The dialogue engine 154 further obtains the spoken text of the question assigned to the node identified in step 268 from the node and synthesizes it into voice (step 270).

[0072] On the other hand, when the determination in step 260 is negative, the dialogue engine 154 branches the flow of control according to whether the user's response utterance includes "Yes" or an expression equivalent thereto (step 262). That is, when the determination in step 262 is positive, the dialogue engine 154 selects a branch corresponding to "Yes" of the node currently being executed in the dialogue scenario, and voice-synthesizes a question corresponding to the new node at the branch destination (step 264). On the other hand, when the determination in step 262 is negative, the dialogue engine 154 selects a branch corresponding to "No" of the node currently being executed in the dialogue scenario, and voice-synthesizes a question corresponding to the new node at the branch destination (step 220). The synthesized voice is transmitted to the terminal 60 and played back.

[0073] Regardless of the outcome of the decisions in steps 260 and 262, the dialogue engine 154 transmits the voice signals synthesized in these steps to the terminal 60 (step 272). These voice signals are played back on the terminal 60.

[0074] The dialogue engine 154 further searches and acquires the dialogue history from the dialogue history DB 166 shown in FIG. 3 (step 274). The dialogue engine 154 determines whether or not to generate an image based on the current spoken text based on the dialogue history acquired in step 274 (step 276). When the determination in step 276 is positive, the dialogue engine 154 inputs the system spoken text or the user spoken text to the image generation AI module 160 shown in FIG. 1 (step 278). The image generation AI module 160 generates and outputs an image in response to this input. The image generation AI module 160 is trained in advance so that this image is suitable for the input. The dialogue engine 154 transmits the image generated and output by the image generation AI module 160 to the terminal 60 (step 280). As a result, this image is presented to the user by the terminal 60. The user is expected to be stimulated by the speech generated by the synthesized speech signal in step 264, step 266, or step 270 and the image generated in step 280 to make a further response speech.

[0075] Meanwhile, the dialogue engine 154 further inputs the image output by the image generation AI module 160 to the caption generation AI module 162 (step 282). In response to this input, the caption generation AI module 162 generates a caption for the input image (step 282). The dialogue engine 154 registers the image generated in step 276 and the caption generated in step 282 as a dialogue history in the dialogue history DB 166 (step 284). Thereafter, the dialogue engine 154 stores the current node position in the scenario in memory as the resume position of the scenario (step 286), and ends the processing of step 218.

[0076] On the other hand, if the determination in step 274 is negative, the dialogue engine 154 does not execute steps 276 to 284, but immediately executes step 286 and ends the processing of step 218.

[0077] On the other hand, when the determination in step 216 in Fig. 5 is positive, the dialogue engine 154 operates as follows. With reference to Fig. 8, the dialogue engine 154 first searches and acquires a dialogue history from the dialogue history DB 166 shown in Fig. 3 (step 300). The dialogue engine 154 further inputs the user attributes and the dialogue history to the chat response generation module 164 shown in Fig. 3, and acquires a chat text from the chat response generation module 164 (step 302).

[0078] In step 302, the dialogue engine 154 operates as follows. With reference to Fig. 9, the dialogue engine 154 extracts important keywords from the user's utterance in the dialogue history, and acquires one or more chat response candidates from the chat response candidate DB 168 by searching the chat response candidate DB 168 using the important keywords as a key (step 350). The dialogue engine 154 inputs these chat response candidates and the dialogue history into a machine learning model, thereby scoring and ranking each chat candidate (step 352). The dialogue engine 154 selects the chat candidate with the highest score from the scores obtained for each chat candidate (step 354). This candidate is selected as the chat text in step 302 of Fig. 8.

[0079] Referring back to FIG. 8, the dialogue engine 154 synthesizes voice by providing the chat text obtained in step 302 to the voice synthesis engine 158 in step 304 , and transmits the synthesized voice signal to the terminal 60 .

[0080] After this, the dialogue engine 154 judges whether or not to end the chat and return to the dialogue scenario (step 306). If the judgment in step 306 is positive, the dialogue engine 154 determines the node at which the dialogue scenario is to be resumed according to pre-stored information (step 308). The dialogue engine 154 then performs voice synthesis on the text assigned to the node determined in step 308 in the dialogue scenario, and transmits the synthesized voice signal to the terminal 60 (step 310). The dialogue engine 154 then executes step 312 in which it judges whether or not an image should be generated. On the other hand, if the judgment in step 306 is negative, the dialogue engine 154 immediately executes the judgment in step 312 without executing steps 308 and 310.

[0081] When the determination in step 312 is positive, the dialogue engine 154 inputs the system utterance text or the user utterance text to the image generation AI module 160 shown in FIG. 3 (step 314). As a result of the process in step 314, the image generation AI module 160 generates and outputs an image appropriate for the input text. The dialogue engine 154 transmits this image to the terminal 60 (step 316). The dialogue engine 154 also inputs the image obtained in step 316 to the caption generation AI module 162 (step 318). As a result of this process, the caption generation AI module 162 generates and outputs a caption that describes the image obtained in step 316. The dialogue engine 154 saves the caption output by the caption generation AI module 162, together with the text of the chat response and the text generated in step 310, if any, in the dialogue history DB 166 as a dialogue history, and ends the execution of this program (step 320).

[0082] On the other hand, if the determination in step 312 is negative, that is, if it is determined that an image should not be generated, the dialogue engine 154 immediately terminates the execution of this program.

[0083] As described above, the dialogue system 50 according to this embodiment utilizes a generation AI to generate visual materials on demand from user responses and system responses and provide them to the user. As a result, the dialogue system 50 makes it possible to present visual materials such as images to the user from any utterance. In addition, the dialogue system 50 provides a procedure for generating text from any generated visual materials and generating a response to the system utterance based on the text.

[0084] Specifically, the dialogue system 50 verbalizes the visual materials using an AI that converts the contents of the visual materials into text, and extracts words contained therein that are likely to become topics for chat. The dialogue system 50 further generates chat-like utterances consisting of questions that broaden the dialogue, taking into account the context of the dialogue or the user's previous response. Furthermore, by generating probing questions from the chat-like utterances, questions that ask deeper questions related to the questions generated from the visual materials can also be generated. As a result, the dialogue system 50 can further develop the dialogue with the user.

[0085] The dialogue system 50 can provide the user with visual materials corresponding to the contents of any user response and system response. In addition, the dialogue system 50 also generates chatty responses based on the visual materials. In other words, the dialogue system 50 is a dialogue system that utilizes a generation AI in both directions. Therefore, it is easy to apply the dialogue system 50 to a task-oriented dialogue system that uses a dialogue scenario as in the above embodiment. In addition, the dialogue system 50 can also be applied to a non-task-oriented dialogue system that specializes in chatting.

[0086] As can be seen from this configuration, the dialogue system 50 can enhance the sense of realism of the dialogue with the user and generate chat-like responses on a wide range of topics based on the generated visual materials. The dialogue system 50 can contribute to expanding the topics of the dialogue with the user and livening up the dialogue itself. By using such a dialogue system as a system for implementing reminiscence therapy for elderly people, for example, it is expected to be highly effective in enabling elderly people to enjoy better lives.

[0087] 2. Second embodiment 2.1 Overview The dialogue engine 154 of the dialogue server 62 according to the first embodiment has a function of generating visual materials related to user utterances or system utterances and presenting them to the user. However, the dialogue engine 154 mainly uses the text of the user utterances and system utterances and the dialogue history based on them, in addition to the attributes of the user, to proceed with the dialogue. However, when engaging in dialogue with a user, it is desirable to consider not only the content of the utterance but also the user's emotions when speaking and the influence of the topic of the dialogue on the user. This second embodiment does not simply perform speech recognition of the user utterance, but also generates the content of the system utterance and visual materials in consideration of the situation of the dialogue, such as the user's emotions expressed by the user utterance according to the prosody of the user utterance, and the polarity of the content of the user utterance (active / passive, good / dislike, positive / negative), the attributes of the user or the user's voice at the time of utterance, etc.

[0088] With reference to FIG. 10, an outline of an example of the progress of a dialogue in the second embodiment will be described. For example, assume that the system 370 according to this embodiment dialogues with the user 372. In this example, the user 372 has attributes of being in his 80s, male, and living alone. In response to the system 370's utterance of "What are your memories of elementary school?", the user 372 answers "The athletic meet." In response to the system 370's utterance of "What sport did you play?", the user 372 responds with "I wanted to play a foot race, but I was too small." From the text of the utterance of the user 372 alone, it is not immediately clear what emotion the user 372 is expressing in the utterance. Therefore, in the second embodiment, the system 370 detects the emotion of the user 372 by analyzing the prosodic features of the speech of the user 372, in addition to performing a simple semantic analysis of the text.

[0089] In this embodiment, the prosodic features of the speech refer to phonological features other than the text of the speech, such as the pitch, tone, intonation, accent, volume, emphasis, speed, rhythm, beat, and pause of the speech. By analyzing the prosodic features, the emotion of the speaker can be estimated. As a method for estimating the emotion, for example, the technology described in Patent Document 1 can be used. In this embodiment, the emotions are limited to two, "bright" and "dark". Of course, the emotions do not need to be divided into these two states, and they may be classified into a wider variety of emotions.

[0090] Also, by merely analyzing the text of the utterance of the user 372 superficially, it is not immediately clear what meaning the content of the utterance by the user 372 has for the user 372. Therefore, in this second embodiment, a polarity analysis is performed on the text of the utterance of the user 372. The polarity in this second embodiment is a concept that indicates whether the content of the utterance is positive or negative overall. Specifically, the polarity of the utterance is determined from a combination of the polarity of the noun that forms the core of the utterance and its predicate. The polarity of a noun is a concept that indicates whether the thing represented by the noun represents a problem or burden (for example, "influenza", "mistake", etc.) or not (for example, "medicine", "food", "correction", etc.). The polarity of a predicate is a concept that indicates whether the predicate has a meaning that activates (for example, "cause", "progress", etc.) or inactivates (for example, "reduce", "destroy", etc.) the function represented by the subject of the predicate. If the noun polarity is positive and the predicate polarity is positive, the utterance is considered to have a positive overall content. If the noun polarity is negative and the predicate polarity is negative, the utterance is considered to have a negative overall content. If the noun polarity is negative and the predicate polarity is positive, the utterance is considered to have a negative overall content. If the noun polarity is negative and the predicate polarity is negative, the utterance is considered to have a positive overall content.

[0091] For such polarity determination, for example, the technology disclosed in Patent Document 2 can be used.

[0092] In the example shown in FIG. 10, the polarity detected for the second utterance of the user 372 is negative, and the emotion is "dark." When the system responds with "I see," the user 372 continues to speak, "But you were fast, so you got first place." The polarity of this utterance is then reversed to positive, and the emotion is also "cheerful." If the dialogue server 380 praises the content of the user 372's speech by saying "That's wonderful," and the user 372 further utters, "I was so happy," the polarity is also positive, and the emotion is "cheerful." At this time, if the dialogue server 380 displays an image 384 showing an elementary school student who won first place in a running race, the user 372 is likely to remember something from the past and continue speaking. As a result, it is expected that the reminiscence method will be more effective.

[0093] On the other hand, even if the emotion of the user 372 and the polarity of the utterance of the user 372 are obtained, if the content of the image 384 generated according to the content of the utterance is ambiguous or shows a content different from the content of the utterance, the effect of the reminiscence therapy is likely to be reduced. Therefore, in this second embodiment, as described later, a measure is taken to more reliably obtain an image that correctly reflects the content of the utterance.

[0094] The above-mentioned user's emotions, dialogue situations such as speech polarity, and user attributes are stored in the dialogue history. However, in this embodiment, all of these pieces of information are finite, so they are represented not by text but by non-verbal information (e.g., numerical values ​​such as 0 and 1) and then stored in the dialogue history. As will be described later, when generating system utterance text or generating visual materials that visualize system utterance text, it is assumed that the input to the machine learning model is text. Therefore, the information stored non-verbally in the dialogue history as described above is converted to text when it is input to the machine learning model.

[0095] 2.2 Configuration Fig. 11 shows, in block diagram form, a schematic configuration of a dialogue server 380 used in a system 370 according to the second embodiment. Referring to Fig. 11, the dialogue server 380 includes a communication device 150, a storage device 152 for storing dialogue scenarios, a voice recognition engine 156, a voice synthesis engine 158, an image generation AI module 160, and a caption generation AI module 162, similar to the dialogue server 62 shown in Fig. 3.

[0096] The dialogue server 380 further includes a dialogue engine 154 for generating a text of an utterance in response to the user utterance, using the content of the user utterance output by the speech recognition engine 156 and a dialogue scenario stored in the storage device 152 .

[0097] The dialogue server 380 differs from the dialogue server 62 in the first embodiment in that it further includes an emotion detection unit 393 for receiving a voice signal of a user's utterance, performing prosodic analysis on the voice signal, and estimating the emotion of the speaker, and a polarity determination unit 391 for determining the polarity of the content of the user's utterance upon receiving a voice recognition result by the voice recognition engine 156. The dialogue server 380 includes a dialogue history DB 396 for storing, as a dialogue history, the voice recognition result by the voice recognition engine 156, the emotion detection result by the emotion detection unit 393, and the polarity determination result by the polarity determination unit 391, etc. The emotion detection unit 393 includes a user utterance semantic interpretation module (not shown) for interpreting the semantic meaning of the user utterance by natural language processing, similar to the dialogue engine 154 in the first embodiment.

[0098] 3, the dialogue server 380 also includes a small talk response generation module 394 for generating small talk responses for small talk with a user, using the dialogue history stored in the dialogue history DB 396 and the user attributes stored in the attribute information DB 170. The dialogue server 380 further includes a large-scale language model 398 that the small talk response generation module 394 uses when generating small talk from the dialogue history.

[0099] The dialogue server 380 further includes a text generation module for image generation 392 for formatting the spoken text based on the dialogue scenario received from the dialogue engine or the text for image generation received from the chat response generation module 394 so that the image generation AI module 160 can generate an appropriate image for the input, and providing the formatted text to the image generation AI module 160. The caption output from the caption generation AI module 162 is stored as part of the dialogue history in the dialogue history DB 396, and is used for generating chat responses from the next time onwards.

[0100] Furthermore, the dialogue server 380 includes a dialogue engine 390, instead of the dialogue engine 154 shown in FIG. 3, which is connected to the storage device 152, the attribute information DB 170, the voice synthesis engine 158, the voice recognition engine 156, the emotion detection unit 393, the text generation module for image generation 392, and the chat response generation module 394, for carrying out a dialogue with the user 372 in cooperation with each part of the dialogue server 380.

[0101] The text generation module for image generation 392 is a text generation model by machine learning that has a function of generating input text to the image generation AI module 160 from a combination of input utterance text and dialogue history so that an image generated by the image generation AI module 160 is suitable for the input obtained from a combination of utterance text and dialogue history according to a dialogue scenario output from the dialogue engine 390, or a combination of a chat response and dialogue history input from the chat response generation module 394. The text generation module for image generation 392 has been trained in advance using dialogue data prepared in advance and training data created using the image generation AI module 160.

[0102] 12 shows a schematic configuration of a training system 410 for training the text generation module 392 for image generation. Referring to FIG. 12, the training system 410 includes a storage device 420 storing a large amount of dialogue history, an image generation unit 422 for reading out a certain spoken text and a dialogue history associated with the spoken text from dialogue data stored in the storage device 420 and inputting the read out spoken text to the image generation AI module 160 to generate an image, a verification unit 424 for displaying the image generated by the image generation AI module 160, and outputting the images after verification by an operator into images appropriate for the input and images that are not appropriate, and an annotation unit 426 for the operator to perform annotation processing such as adaptation and correction on the spoken text input to the image generation AI module 160 when it is determined that the image output by the image generation AI module 160 is not appropriate. The spoken text corrected by the annotation unit 426 is provided to the image generation unit 422 together with the original speech history, and is further provided to the image generation AI module 160.

[0103] The training system 410 further includes a learning data creation unit 428 for creating learning data for the text generation module for image generation 392 by combining, as teacher data, the spoken text determined by the verification unit 424 to be suitable as learning data, and as input, the original spoken text read from the storage unit 420 and the speech history, and a learning data storage device 430 for storing the learning data created by the learning data creation unit 428. The learning data storage device 430 stores a large amount of learning data created by the above-mentioned process.

[0104] In this embodiment, the dialogue history includes information such as system spoken text, user spoken text, user attributes, emotions associated with the speech, speech polarity, images generated by the image generation AI module 160, and captions created for the image generation AI module 160.

[0105] The training system 410 further includes a learning processing unit 432 for learning the text generation module for image generation 392 using the learning data stored in the learning data storage device 430. The text generation module for image generation 392 is a machine learning model having a function of generating and outputting spoken text based on an input spoken text and a speech history accompanying the spoken text, so that an image related to the spoken text and a correct and clear image is generated by the image generation AI module 160, and BERT or the like can be used.

[0106] Fig. 13 is a flowchart showing a control structure of a main routine of a computer program for implementing dialogue server 380 according to the second embodiment by a computer. Fig. 13 corresponds to Fig. 5 of the first embodiment.

[0107] Referring to Figure 13, this program includes step 448 for executing initial processing such as reading a dialogue scenario from the storage device 152 shown in Figure 11, connecting with the communication device 150, the voice recognition engine 156, the voice synthesis engine 158, the image generation AI module 160, the caption generation AI module 162, the emotion detection unit 393, the polarity determination unit 391, the chat response generation module 394, and the text generation module for image generation 392, and initializing the memory area.

[0108] This program further includes step 450 of starting a dialogue in accordance with the dialogue scenario read out in step 448, step 204 of waiting until a voice signal is received from terminal 60, and upon receiving the voice signal, accepting the voice signal as a voice signal of a user's utterance, step 206 of performing voice recognition on the voice signal accepted in step 204 and converting it into text, step 452 of performing semantic interpretation of the text obtained in step 206, emotion detection, and polarity determination on the voice signal accepted in step 204, and step 454 of branching the flow of control in accordance with the semantic interpretation performed in step 452.

[0109] In the second embodiment, the options for meaning determination in step 454 are not limited to three, but may be four or more.

[0110] The program further includes step 212, which is executed when it is determined in step 454 that the user's utterance is unclear, and which asks the user to repeat the utterance and returns the flow of control to step 204; step 214, which is executed when it is determined in step 454 that the user has refused the dialogue, and which makes a utterance to end the dialogue and terminates execution of the program; and step 456, which determines whether or not to respond to the user's utterance in step 454 with casual conversation, and branches the flow of control according to the result.

[0111] The semantic interpretation in step 452 is performed by a user utterance semantic interpretation module included in the dialogue engine 390, based on whether the user's response utterance can be interpreted by natural language processing, and whether the user's response utterance contains an expression that rejects the answer. In addition, the determination in step 456 is performed based on the dialogue history including the system utterance and the user utterance. This determination is performed by the user utterance semantic interpretation module based on information including the dialogue history. As in the first embodiment, the determination by the user utterance semantic interpretation module may be rule-based, or may be performed by a machine learning model that has been trained in advance to input the dialogue history and output a determination result.

[0112] The program further includes step 460, when it is determined in step 456 that the system should respond to the user's utterance with casual conversation, generating and outputting a casual conversation utterance using the large-scale language model 398 based on the dialogue history stored in the dialogue history DB 396, including the utterance in response to the immediately previous user utterance and the caption by the caption generation AI module 162, and returning the flow of control to step 204; and step 458, when it is determined in step 456 that the system should respond in accordance with the dialogue scenario, making an utterance defined by the dialogue scenario and returning control to step 204.

[0113] Fig. 14 is a flowchart showing a control structure of a program for realizing step 450 of Fig. 13. Referring to Fig. 14, this program includes steps 240, 242, 244, and 248 similar to those shown in Fig. 6 relating to the first embodiment, and step 500, which is inserted in order between steps 244 and 248 instead of step 246 shown in Fig. 6, for concatenating an utterance text of an opening utterance specified by a dialogue scenario with a text obtained by converting attributes of user 372 into text and inputting the concatenated text to image generation text generation module 392, and step 502 for inputting the text output by image generation text generation module 392 as a result of the process in step 500 to image generation AI module 160.

[0114] FIG. 15 shows a control structure of a program for implementing step 458 in FIG. 13. Referring to FIG. 15, step 458 includes steps 260 to 276, and steps 280 and 282, similar to those shown in FIG. 7. This program further includes step 550, which is inserted between step 276 and step 278 in order instead of step 278 shown in FIG. 7, for selecting either the latest user utterance text or the system utterance text, concatenating the text, user attributes, and utterance history (including the system utterance text, the user utterance text, the emotion and polarity of the user utterance, and the image and caption generated from the utterance) and inputting them to the text generation module for image generation 392, and step 552 for inputting the text output from the text generation module for image generation 392 to the image generation AI module 160. Among the inputs to the text generation module for image generation 392, those that are not text are converted into text (converted into text).

[0115] Step 458 further includes step 554, after step 282, of storing the text generated in step 550, the image generated in step 552, and the caption generated in step 282 in the dialogue history DB 396 as part of the dialogue history before executing step 286.

[0116] Fig. 16 is a flowchart showing a control structure of a program for realizing step 460 of Fig. 13. Referring to Fig. 16, this program includes a step 600 of reading out a dialogue history from dialogue history DB 396, and a step 602 of inputting user attributes, utterance text in the dialogue history, emotion, polarity, and caption into a chat response generation module to obtain chat text. The chat response generation module converts these into text as necessary and then concatenates them, inputs them into large-scale language model 398 shown in Fig. 11, and obtains the output as chat text.

[0117] This program further includes steps 304 to 312 and steps 316 to 320 similar to those shown in FIG. 8 for the first embodiment, and step 604 which is inserted in order between step 312 and step 316 instead of step 314 shown in FIG. 8, in which either the current user utterance text or the system utterance text selected therefrom, user attributes, and information in the dialogue history, which have been converted into text as necessary, are concatenated and input to text generation module 392 for image generation shown in FIG. 11, and step 606 which inputs the text output by text generation module 392 for image generation to image generation AI module 160.

[0118] 2.3 Operation The operation of the dialogue server 380 is basically the same as that of the dialogue server 62 in the first embodiment. However, the operation of the dialogue server 380 differs from that of the dialogue server 62 in that not only the voice recognition process by the voice recognition engine 156 but also the emotion detection process by the emotion detection unit 393 and the polarity determination process by the polarity determination unit 391 are performed on the utterance of the user 372, these are stored in the dialogue history DB 396, the chat response generation module 394 generates a chat response using the dialogue history DB 396 and the large-scale language model 398, and either the system utterance text or the user utterance including the chat response and the dialogue history, etc. are input to the text for image generation generation module 392 instead of being input directly to the image generation AI module 160, and the text generated by the text for image generation generation module 392 is provided to the image generation AI module 160.

[0119] Among the dialogue history stored in the dialogue history DB 396, the information on the user utterance includes not only the text of the utterance by the user 372, but also the emotion of the user 372 at that time and the polarity of the content of the utterance, the image generated by the image generation AI module 160, and the caption generated by the caption generation AI module 162. When the dialogue server 380 generates a response, the chat response generation module 394 uses not only the system utterance text generated by the dialogue server 380 but also the dialogue history of the user utterance and the system utterance previously stored in the dialogue history DB 396. The information provided from the chat response generation module 394 to the text generation module for image generation 392 and the information provided from the dialogue engine 390 to the text generation module for image generation 392 include all of these information in addition to the current system utterance text and the user utterance text. Therefore, the image generated by the image generation AI module 160 reflects not only the text of the past user utterance but also the dialogue history such as the emotion of the user at that time and in the past and the polarity of the content of the user utterance.

[0120] Furthermore, the system utterance text and the user utterance text are not immediately given to the image generation AI module 160. The system utterance text and the user utterance text are given to the image generation text generation module 392 together with the dialogue history, and the text generated by the image generation text generation module 392 is given to the image generation AI module 160. The image generation text generation module 392 selects either the input system utterance text or the user utterance text, and converts the utterance text based on the selected utterance text and the dialogue history so that the image generation AI module 160 generates a clear image that better reflects the content represented by the input utterance text. Therefore, the output of the image generation AI module 160 is an image that appropriately reflects the content of either the system utterance text or the user utterance text and the dialogue history, and is a clear image. As a result, the user is inspired by the image, and is more likely to remember past events and make new utterances. As a result, it is expected that the effect of the reminiscence method can be enhanced.

[0121] 3. Hardware Configuration Fig. 17 is an external view of the dialogue server 62 according to the first embodiment shown in Fig. 1, for example. Fig. 18 is a hardware block diagram of the dialogue server 62. Figs. 17 and 18 also apply to the dialogue server 380 according to the second embodiment. Therefore, in the following explanation, the hardware configuration of the dialogue server 62 will be mainly explained based on the first embodiment.

[0122] 17, the dialogue server 62 includes a computer 650 having a DVD (Digital Versatile Disc) drive 662, and a keyboard 654, a mouse 656, and a monitor 652 for dialogue with a user, all of which are connected to the computer 650. Of course, these are just one example of a configuration for when dialogue with an operator becomes necessary, and any general hardware and software (e.g., a touch panel, voice input, pointing device in general) that can be used for dialogue with an operator can be used.

[0123] 17 and 18, the computer 650 includes a central processing unit (CPU) 710, a graphics processing unit (GPU) 712, and a bus 720 connected to the CPU 710, the GPU 712, and the DVD drive 662, in addition to a DVD drive 662. The computer 650 further includes a read-only memory (ROM) 714 connected to the bus 720 and storing a boot-up program of the computer 650, a random access memory (RAM) 716 connected to the bus 720 and storing instructions constituting a program, a system program, working data, and the like, and a solid state drive (SSD) 718, which is a non-volatile memory, connected to the bus 720. The SSD 718 is for storing programs executed by the CPU 710 and the GPU 712, and data used by the programs executed by the CPU 710 and the GPU 712, and the like. The computer 650 further includes a network I / F (Interface) 726 that provides a connection to a network enabling communication with other terminals, and a USB port 664 to which a USB (Universal Serial Bus) memory 702 can be attached / detached and that provides communication between the USB memory 702 and each part within the computer 650.

[0124] The computer 650 further includes an audio I / F 722 that is connected to the microphone 660, the speaker 658, and the bus 720, and has the functions of reading out audio signals, video signals, and text data generated by the CPU 710 and stored in the RAM 716 or the SSD 718 according to an instruction from the CPU 710, converting the audio signals to analog, amplifying the audio signals, and driving the speaker 658, and digitizing the analog audio signals from the microphone 660 and storing them in an arbitrary address in the RAM 716 or the SSD 718 designated by the CPU 710. In the embodiment of the present invention, a dialogue with a user is performed using a terminal 60 as shown in FIG. 1. For this purpose, the keyboard 654 and the microphone 660 are not used. However, the dialogue server 62 may directly perform the dialogue using the microphone 660, the keyboard 654, and the audio I / F 722.

[0125] In the above embodiment, the programs and the like for realizing the functions of the dialogue server 62 shown in Fig. 1 and the dialogue server 380 shown in Fig. 11 are all stored in, for example, the ROM 714, SSD 718, DVD 700 or USB memory 702 shown in Fig. 18, or a storage medium of an external device (not shown) connected via the network I / F 726 and the network 704. Typically, these data and parameters are written, for example, from outside into the SSD 718, and loaded into the RAM 716 when the computer 650 is executed.

[0126] Computer programs for operating this computer system to realize the functions of each component of the dialogue server 62 shown in Fig. 1 and the dialogue server 380 shown in Fig. 11 are stored in a DVD 700 mounted in a DVD drive 662, and transferred from the DVD drive 662 to the SSD 718. Alternatively, these programs are stored in a USB memory 702, which is mounted in a USB port 664 and the programs are transferred to the SSD 718. Alternatively, the programs may be transmitted to the computer 650 via the network 704 and stored in the SSD 718. Of course, a source program may be input using the keyboard 654, the monitor 652, and the mouse 656, and the compiled object program may be stored in the SSD 718.

[0127] The program is loaded into the RAM 716 at the time of execution. If the program is written in a script language, the script input by the operator using the keyboard 654 or the like may be stored in the SSD 718. In the case of a program that runs on a virtual machine, a program that functions as a virtual machine must be installed in the computer 650 in advance. A machine learning model such as a deep neural network is used for the dialogue engine 154, the voice recognition engine 156, the voice synthesis engine 158, the image generation AI module 160, the caption generation AI module 162, and the chat response generation module 164 shown in FIG. 2. In the dialogue server 62, a machine learning model that has been trained in another device may be used, or the dialogue server 62 may be used as a learning device to train the machine learning model. The same applies to the dialogue server 380 according to the second embodiment.

[0128] The CPU 710 reads a program from the RAM 716 according to an address indicated by a register (not shown) called a program counter inside the CPU 710 and interprets the instruction. The CPU 710 reads data required for executing the instruction from the RAM 716, the SSD 718, or other devices according to an address specified by the instruction, and executes the process specified by the instruction. The CPU 710 stores the execution result data in an address specified by the program, such as the RAM 716, the SSD 718, or a register in the CPU 710. Depending on the address, the execution result data is output from the computer to the outside via, for example, the network I / F 726. The output destination is, for example, the terminal 60 shown in FIG. 1. At this time, the value of the program counter is also updated by the program. The computer program may be directly loaded into the RAM 716 from the DVD 700, the USB memory 702, or via the network 704. Note that some tasks (mainly numerical calculations) among the programs executed by the CPU 710 are issued to the GPU 712 according to instructions included in the program or according to the analysis results when the CPU 710 executes the instructions.

[0129] The program for implementing the functions of each part of the dialogue server 62 and the dialogue server 380 according to the above-described embodiment by the computer 650 includes a plurality of instructions written and arranged to operate the computer 650 to implement those functions. Some of the basic functions required to execute the instructions may be provided by an OS (Operating System) or a third-party program running on the computer 650, various toolkit modules installed on the computer 650, or an execution environment of the program. Therefore, the program does not necessarily include all of the functions required to implement the system and method of this embodiment. The program may include only instructions to execute the operations of each of the above-described devices and their components by statically linking appropriate functions or modules at compile time or dynamically calling them at run time in a controlled manner so as to obtain a desired result. The operation method of the computer 650 for this purpose is well known. Therefore, the description of the operation method of the computer 650 will not be repeated in this section.

[0130] The GPU 712 is capable of parallel processing, and can simultaneously execute a large amount of calculations associated with machine learning and inference in a parallel or pipelined manner. For example, parallel calculation elements found in a program when the program is compiled, or parallel calculation elements found when the program is executed, are dispatched from the CPU 710 to the GPU 712 as needed, and executed. The results are returned to the CPU 710 directly or via a predetermined address in the RAM 716, and assigned to a predetermined variable in the program.

[0131] A hardware configuration of the terminal 60 commonly used in the first and second embodiments is shown in Fig. 19. Referring to Fig. 19, the terminal 60 includes a processor 750 including a CPU and a memory (not shown), and a communication device 752 connected to the processor 750 for providing a connection with the dialogue server 62 or the dialogue server 380 via the network 704 shown in Fig. 18.

[0132] 1, a display control device 758 connected to the processor 750 and the LCD 68, a speaker 754 and a microphone 756 provided in positions not visible to the user in the terminal 60 shown in Fig. 1, and an audio processing device 760 connected to the processor 750, the speaker 754, and the microphone 756. The audio processing device 760 has a function of digitizing an audio signal from the microphone 756 and converting it into a format suitable for communication via the communication device 752, and a function of converting a voice signal generated by voice synthesis received from the dialogue server 62 via the communication device 752 into an analog signal, amplifying the obtained analog voice signal, and providing it to the speaker 754 to generate a voice.

[0133] The embodiments disclosed herein are merely illustrative, and the present invention is not limited to the above-described embodiments. The scope of the present invention is defined by the claims of the appended claims, taking into consideration the detailed description of the invention, and includes all modifications within the scope and meaning equivalent to the words described therein. [Explanation of symbols]

[0134] 50 Dialogue Systems 60 terminals 62,380 conversation server 64 Audio Signals 66 Speech Signals 68 LCD 100, 370 systems 102,372 users 104 Opening Speech 108, 124, 384 images 110, 120 Response utterances 114 Text 118 System Speech 150, 752 Communication equipment 152, 420 storage device 154, 390 Dialogue Engine 156 Voice Recognition Engine 158 Text-to-Speech Engine 160 Image Generation AI Module 162 Caption Generation AI Module 164, 394 Chat response generation module 166,396 Dialogue history DB 168 Chat Response Candidate DB 170 Attribute information DB 391 Polarity judgment section 392 Text Generation Module for Image Generation 393 Emotion Detection Unit 398 Large-scale Language Models 410 Training System 422 Image Generation Unit 424 Verification Department 426 Annotation Department 428 Learning Data Creation Department 430 Learning Data Storage Device 432 Learning Processing Unit

Claims

1. An interaction device for interacting with a user, comprising: a system utterance text generating means for generating a system utterance text based on a dialogue history with the user; a visual aid generating means for generating visual aids based on either the system spoken text or the user spoken text; and a text generation means for generating text for visual material generation based on either the system spoken text or the user spoken text, so that the visual material generation means can generate appropriate visual material for the input, and providing the generated text to the visual material generation means.

2. The dialogue device according to claim 1 , wherein the means for generating text for generating visual materials includes a machine learning model that has been trained to generate the text for generating visual materials using a dialogue history with the user as an input.

3. a text generating means for generating a related text related to the dialogue with the user by verbalizing the content of the visual material generated by the visual material generating means, 2. The dialogue apparatus according to claim 1, wherein the system utterance text generating means, the visual material generating means, or the text for visual material generation generating means further receives the associated text as an input.

4. the dialogue history includes non-verbalized information indicating at least a part of a dialogue situation in a non-verbal manner; 2. The dialogue device according to claim 1, wherein the visual material generating means or the text generating means for generating visual material converts the non-verbalized information of the dialogue history into text, and inputs the text by linking it with other text included in the dialogue history.

5. A computer program for causing a computer to function as an interactive device for interacting with a user, the computer comprising: a system utterance text generating means for generating a system utterance text based on a dialogue history with the user; a visual aid generating means for generating visual aids based on either the system spoken text or the user spoken text; A computer program that functions as a text generation means for visual material generation, which generates text for visual material generation based on either the system spoken text or the user spoken text so that the visual material generation means can generate appropriate visual material for the input, and provides the text for visual material generation to the visual material generation means.

6. 1. A computer-implemented method for interacting with a user, comprising: A step of generating a system utterance text by a computer based on a dialogue history with the user; generating visual materials based on the system spoken text and the user spoken text by a computer; and generating text for generating visual material based on either the system spoken text or the user spoken text, such that appropriate visual material for the input can be generated in the visual material generating step, and using the generated text as input to the visual material generating step.

Images