Conversation device, conversation method, and computer program

Abstract

This conversation device is for carrying out a conversation with a user, and provides a conversation system that can provide, at an appropriate time in a conversation with a user, visual materials suited to promoting the conversation, and can promote the conversation on the basis of discretionary visual materials, the conversation device including: a conversation engine 390 for generating system speech text on the basis of a conversation history with a user; an image generation AI module 160 for generating visual materials on the basis of either the system speech text or user speech text; and an image generation text generation module 392 for generating text with which the image generation AI module 160 can generate an appropriate visual material for the input, on the basis of either the system speech text or the user speech text, and giving the text to the image generation AI module 160.

Description

Dialogue device, dialogue method, and computer program

[0001] This invention relates to natural language processing technology, and in particular to a dialogue device that dialogues with people using voice, etc. This application claims priority to Japanese Application No. 2023-178602, filed October 17, 2023, the entire disclosure of which is incorporated herein by reference.

[0002] In voice dialogue systems or text-based dialogue systems, it is known that presenting visual materials (photos, illustrations, videos, etc.) related to the topic can increase familiarity and make the dialogue more memorable. For this reason, dialogue systems that use visual materials are widely used.

[0003] For example, there are voice dialogue systems that collect visual materials in advance and then present visual materials retrieved from the collected visual materials using keywords derived from user utterances. Another example is a dialogue system that uses a dialogue scenario. A voice dialogue system that uses a dialogue scenario presents visual materials that are determined according to the dialogue scenario.

[0004] Conversely, there are systems that generate responses based on visual materials used in dialogue systems. Examples include VisualQA (see Non-Patent Document 1) and Visual Dialog (see Non-Patent Document 2). Both of these systems perform the task of providing an answer when a question about a photograph, which is an example of visual material, and an object shown in the photograph are given.

[0005] One field in which such dialogue systems using visual materials are suitable is the prevention and improvement of dementia in the elderly and others. Reminiscence therapy is a method applied to elderly people in this field. Reminiscence therapy is a technique that aims to prevent dementia or improve its symptoms by having elderly people reminisce about past events. It is said that the brain is activated when elderly people reminisce about past events, and in reminiscence therapy, a human practitioner engages in a dialogue with the elderly target while presenting visual materials. It would be desirable if the dialogue system could carry out this dialogue on behalf of the practitioner.

[0006] JP 2022-133570 A JP 2013-175097 A

[0007] Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D.: Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J. M. F., Parikh, D., and Batra, D.: Visual Dialog, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

[0008] Both the systems disclosed in Non-Patent Document 1 and Non-Patent Document 2 address only the special case of answering questions about (visible) objects present in visual materials. Therefore, the application of these systems is extremely limited. For example, when using these systems for reminiscence therapy, it is necessary to collect a large amount of visual materials related to the elderly person's past in advance for each target elderly person. Furthermore, when a dialogue is being conducted, it is necessary to search and display from among these visual materials at the appropriate time, those that are appropriate for the progress of the dialogue at that time. Searching for visual materials may be possible if keywords are assigned to the visual materials in advance. However, assigning keywords to each visual material is time-consuming. Furthermore, there is no guarantee that the selected visual materials will be appropriate for developing the dialogue.

[0009] Therefore, an object of the present invention is to provide a dialogue system that can provide visual materials suitable for developing a dialogue with a user in a timely manner and that can develop the dialogue based on any visual materials.

[0010] An interaction device according to a first aspect of the present invention is an interaction device for interacting with a user, and includes: a system utterance text generation means for generating system utterance text based on an interaction history with the user; a visual material generation means for generating visual material based on either the system utterance text or the user utterance text; and a visual material generation text generation means for generating visual material generation text based on either the system utterance text or the user utterance text, so that the visual material generation means can generate visual material appropriate for the input, and providing the visual material generation means with the text.

[0011] Preferably, the means for generating text for generating visual materials includes a machine learning model that has been trained to generate text for generating visual materials using a dialogue history with a user as input.

[0012] More preferably, the dialogue device further includes a text generation means for generating related text related to the dialogue with the user by verbalizing the content of the visual material generated by the visual material generation means, and the system speech text generation means, the visual material generation means, or the text generation means for generating visual material further receives the related text as input.

[0013] More preferably, the dialogue history includes non-verbalized information that non-verbalizes at least a portion of the dialogue situation, and the visual material generation means or the text generation means for generating visual material converts the non-verbalized information of the dialogue history into text and combines it with other text included in the dialogue history as input.

[0014] Preferably, the dialogue context represents attributes of the user's utterance.

[0015] More preferably, the dialogue situation represents an attribute of the user.

[0016] A computer program according to a second aspect of the present invention is a computer program that causes a computer to function as an interactive device for interacting with a user, and causes the computer to function as: system utterance text generation means for generating system utterance text based on a dialogue history with the user; visual material generation means for generating visual material based on either the system utterance text or the user utterance text; and visual material generation text generation means for generating visual material generation text based on either the system utterance text or the user utterance text, so that the visual material generation means can generate visual material appropriate for the input, and providing the visual material generation means with the text.

[0017] A dialogue method according to a third aspect of the present invention is a dialogue method implemented by a computer for dialogue with a user, comprising the steps of: generating system utterance text by the computer based on a dialogue history with the user; generating visual material by the computer based on either the system utterance text or the user utterance text; and generating visual material generation text based on either the system utterance text or the user utterance text so that appropriate visual material can be generated in the visual material generating step, and using the text as input to the visual material generating step.

[0018] The above and other objects, features, aspects and advantages of the present invention will become apparent from the following detailed description of the invention taken in conjunction with the accompanying drawings.

[0019] FIG. 1 is a block diagram showing a schematic configuration of a dialogue system according to a first embodiment of the present invention. FIG. 2 is a schematic diagram for explaining the general operation of the dialogue system shown in FIG. 1. FIG. 3 is a functional block diagram of the dialogue server shown in FIG. 1. FIG. 4 is a schematic diagram showing the configuration of a dialogue scenario used by the dialogue server shown in FIG. 1. FIG. 5 is a flowchart showing the control structure of a main routine of a program executed for dialogue processing by the dialogue server shown in FIG. 3. FIG. 6 is a flowchart showing the control structure of a program that realizes dialogue start processing among the programs shown in FIG. 5. FIG. 7 is a flowchart showing the control structure of a program that realizes response processing based on a dialogue scenario among the programs shown in FIG. 5. FIG. 8 is a flowchart showing the control structure of a program that realizes chat processing among the programs shown in FIG. 5. FIG. 9 is a flowchart showing the control structure of a program that realizes chat text acquisition processing among the programs shown in FIG. 8. FIG. 10 is a schematic diagram showing a dialogue example in a dialogue system according to a second embodiment of the present invention. FIG. 11 is a block diagram showing the functional configuration of the dialogue server according to the second embodiment of the present invention. FIG. 12 is a diagram showing the learning process of a text generation module for image generation used by the dialogue server according to the second embodiment. Fig. 13 is a flowchart showing the control structure of a main routine of a program executed for dialogue processing by a dialogue server according to the second embodiment. Fig. 14 is a flowchart showing the control structure of a program for realizing the dialogue start processing shown in Fig. 13. Fig. 15 is a flowchart showing the control structure of a program for realizing response processing in accordance with a dialogue scenario shown in Fig. 13. Fig. 16 is a flowchart showing the control structure of a program for realizing the chat processing shown in Fig. 13. Fig. 17 is an external view of the dialogue server shown in Fig. 1. Fig. 18 is a hardware block diagram of the dialogue server shown in Fig. 17. Fig. 19 is a hardware block diagram of the terminal shown in Fig. 1.

[0020] In the following description and drawings, the same parts are designated by the same reference numerals, and therefore detailed description thereof will not be repeated.

[0021] 1 shows a schematic configuration of a dialogue system 50 according to a first embodiment of the present invention. Referring to Fig. 1, the dialogue system 50 includes a stuffed toy-shaped terminal 60 and a dialogue server 62 that receives a voice signal 64 of a dialogue partner (user) from the terminal 60, generates a speech signal 66 including a voice signal and an image signal for dialogue with the dialogue partner, and transmits the speech signal 66 to the terminal 60.

[0022] The terminal 60 includes a microphone and a speaker (not shown in FIG. 1 ), and an LCD (Liquid Crystal Display) 68 for displaying images received from the dialogue server 62. The terminal 60 has the external shape of a stuffed toy so that natural dialogue with the dialogue partner can be realized. Of course, the terminal 60 is not limited to being in the shape of a stuffed toy, and may be in the shape of a so-called smartphone, a panel computer, or some kind of figurine. The hardware configuration of the terminal 60 will be described later with reference to FIG. 19 .

[0023] The dialogue server 62 used in this embodiment is a general-purpose computer, the hardware configuration of which will be described later with reference to Figures 17 and 18.

[0024] FIG. 2 shows a schematic diagram of the overall operation of the dialogue system 50. In FIG. 2, a dialogue by this dialogue system 50 takes place between a system 100 and a user 102. First, the system 100 utters an opening utterance 104. In this example, the opening utterance 104 is "Did you have a school trip in junior high school?" This opening utterance 104 is specified as the first utterance of the dialogue in a dialogue scenario prepared in advance. The dialogue scenario will be described later. Most of the utterances from the system uttered according to the dialogue scenario are in the form of questions.

[0025] In this embodiment, when an opening utterance 104 is spoken, the system 100 extracts words from the opening utterance 104 and generates 106 an image 108 related to the words as visual information using a so-called image generation AI (Artificial Intelligence) based on the words. The image 108 is displayed on the LCD 68 shown in FIG. 1 . Any image generation AI capable of generating an image from text may be used, such as one based on a diffusion model or a Generative Adversarial Network (GAN). The image 108 may be a realistic image, such as a photograph, or an illustration that imitates hand-drawn illustrations. Furthermore, instead of a still image like a photograph, the image may be a moving image that imitates a live-action moving image or a moving image that imitates an animation.

[0026] The system 100 generates the image 108 and also generates 112 text 114 that verbalizes the image 108 from the content of the image 108. A so-called caption generation AI can be used to generate 112 the text 114. In this example, the caption generated for the text 114 is "This is an image of students buying lots of souvenirs."

[0027] When the image 108 is displayed together with the opening utterance 104, the user 102 is inspired by both the opening utterance 104 and the image 108 to recall the past, and utters a response utterance 110 such as, for example, "Yes, I went there when I was in the third year of junior high school." This response utterance 110 indicates some kind of affirmative response to the opening utterance 104, since it begins with "Yes."

[0028] In this embodiment, the system 100 performs the following process in response to receiving the response utterance 110 from the user 102: Several words (e.g., "souvenir," "student," etc.) are extracted from the text 114, and a new system utterance 118, "Do you remember what souvenir you bought?" is generated 116 using one of the words (e.g., "souvenir") and uttered to the user 102. In the example shown in FIG. 2 , an image such as the image 108 is not generated in response to this system utterance 118, but an image may be generated from the system utterance 118.

[0029] Assume that the user 102, in response to the system utterance 118 and the image 108, remembers buying steamed buns as a souvenir on a school trip he or she took in the third year of junior high school. As a result, the user 102 makes a response utterance 120, saying, "It was steamed buns. I love sweet things." Since the response utterance 120 contains the word "steamed buns," the system 100 generates 122 an image 124 of steamed buns and presents it to the user 102. Thereafter, the system 100 makes further utterances based on the generation 122. Thereafter, the dialogue between the user 102 and the system 100 develops in a similar manner.

[0030] 3, the dialogue server 62 has the following functional configuration. That is, the dialogue server 62 includes a communication device 150 that communicates voice signals and image signals with the terminal 60 (see FIG. 1), a storage device 152 for storing dialogue scenarios prepared in advance, a voice recognition engine 156 that performs voice recognition on the user's voice received by the communication device 150 from the terminal 60 and converts it into text, and a dialogue engine 154 that generates system utterance text, which is a response utterance to the user utterance, using the content of the user utterance output by the voice recognition engine 156 and the dialogue scenario stored in the storage device 152. The dialogue engine 154 includes a user utterance semantic interpretation module (not shown) that interprets the meaning of the user utterance by natural language processing.

[0031] Referring to FIG. 4 , a dialogue scenario basically has a binary tree structure with an opening utterance (Question 1) as the root and question utterances (Question 1.1, Question 1.2, ...) assigned to each node. The first branch from a certain node is a branch when the user's response to the question utterance at that node is affirmative, and the second branch is a branch when the user's response is negative. For example, in the dialogue scenario shown in FIG. 4 , if the user's response to Question 1 is affirmative, the system next utters Question 1.1 to the user. If the user's response to Question 1 is negative, the system next utters Question 1.2 to the user. The dialogue ends when it reaches the terminal leaf of the binary tree structure.

[0032] In this embodiment, the questions assigned to each node of the dialogue scenario are questions that can be answered with either "yes" or "no." For this type of question, the branch destination can be easily determined if the user's response utterance includes either "yes" or "no," or similar phrases. In this embodiment, if these phrases are not included in the response utterance, or if the system 100 determines that it is appropriate to transition to casual conversation, a casual conversation utterance is generated as described below, and the conversation follows a different course from the scenario. The system 100 always records where the dialogue is located in the dialogue scenario. Therefore, even during casual conversation, the system 100 can always return to the dialogue scenario based on the recorded information. In this case, control to return to the dialogue may be performed by selecting any node in the scenario at any time and uttering a question again.

[0033] Note that the following process is the same whether the system utterance is an opening utterance or a subsequent utterance. Therefore, in the following description, the utterance generated by the dialogue engine 154 will be simply referred to as a "system utterance," and the text thereof will be referred to as a "system utterance text." The system utterance specified by the dialogue scenario is, in principle, a question format that can be answered with "yes" or "no." In this specification, this type of question is referred to as a "multiple choice" question. As the term "multiple choice" suggests, the question referred to here does not only include two choices, such as "yes" and "no," but also three or more choices. In contrast, a chat-style system utterance may be a so-called probing question in response to the previous question, but is not necessarily a question. In chat, there is a possibility that utterances other than questions will prevail.

[0034] The dialogue server 62 further includes a speech synthesis engine 158 for synthesizing a speech signal from the spoken text generated by the dialogue engine 154 and providing the speech signal to the dialogue engine 154 .

[0035] The dialogue server 62 further includes an image generation AI module 160 that receives system utterance text or user utterance text from the dialogue engine 154, generates an image that is a type of visual material that visualizes the content of the input utterance, and provides the image to the dialogue engine 154, a caption generation AI module 162 that verbalizes the image generated by the image generation AI module 160 and generates a caption that is associated text related to the image, and a chat response generation module 164 that includes a machine learning model that uses the dialogue history recorded in a dialogue history DB (DataBase) 396 to generate text for a chat response with the user that is different from that based on the dialogue scenario, and provides the text to the dialogue engine 154. This chat response is also part of the utterance by the dialogue engine 154 and is a type of system utterance. The decision as to when to generate an image using the image generation AI module 160 and whether to provide the image generation AI module 160 with system-spoken text or user-spoken text may be determined based on rules based on the content of the text, or a machine learning model may be used that uses the speech history, system-spoken text, and user-spoken text as input and makes a decision as to whether to generate an image, and if so, whether to select system-spoken text or user-spoken text.

[0036] In this embodiment, the image generation AI module 160 and the caption generation AI module 162 are part of the dialogue server 62. However, the present invention is not limited to such an embodiment. Either or both of the image generation AI module 160 and the caption generation AI module 162 may be provided as a network service outside the dialogue server 62. In that case, the dialogue server 62 uses the image generation AI module 160 or the caption generation AI module 162 via the network.

[0037] The dialogue server 62 further includes an attribute information DB 170 connected to the dialogue engine 154, which stores user attributes (age, gender, whether or not there is a family, address, lifestyle, work history, etc.) and outputs attribute information about the user in response to a query from the dialogue engine 154; a dialogue history DB 166 which stores the dialogue history between the system 100 and the user 102 and the captions output by the caption generation AI module 162 as part of the dialogue history; and a chat response candidate DB 168 which stores a large number of candidates that can be used as chat responses and are referenced by the chat response generation module 164 when generating chat responses. In this embodiment, the dialogue history includes a log of semantic analysis of user utterances, specifically, a semantic classification of the user utterances. The text of the user utterances themselves is also saved as dialogue history.

[0038] The dialogue engine 154 manages dialogue with the user in accordance with the outputs of the speech recognition engine 156, the voice synthesis engine 158, the image generation AI module 160, and the chat response generation module 164, the dialogue scenario stored in the storage device 152, the user's attribute information stored in the attribute information DB 170, and the dialogue history stored in the dialogue history DB 166. On the other hand, the chat response generation module 164 generates chat responses using the output of the caption generation AI module 162, the dialogue history stored in the dialogue history DB 166, and chat response candidates stored in the chat response candidate DB 168.

[0039] 1.3 Program Structure The control structure of the program by which the dialogue server 62 shown in Fig. 3 realizes a dialogue with a user is shown in the form of a flowchart in Fig. 5. Referring to Fig. 5, this program includes step 200 for reading a dialogue scenario from the storage device 152, connecting with the communication device 150, speech recognition engine 156, speech synthesis engine 158, image generation AI module 160, caption generation AI module 162, and chat response generation module 164 shown in Fig. 3, and initializing memory areas.

[0040] This program further includes step 202 of starting a dialogue in accordance with the dialogue scenario read out in step 200, step 204 of waiting until a voice signal is received from terminal 60, and upon receiving the voice signal, accepting the voice signal as a voice signal of a user utterance, step 206 of performing speech recognition on the voice signal received in step 204 and converting it into text, step 208 of performing semantic interpretation on the text obtained in step 206, and step 210 of branching the flow of control in accordance with the semantic interpretation performed in step 208.

[0041] In this embodiment, the branch destinations based on the meaning determination in step 210 are three: step 212, step 214, and step 216. However, the present invention is not limited to such an embodiment. There may be four or more options. For example, if the user utterance includes an expression such as "I don't understand" or a similar expression, it may be determined that the user does not fully understand the meaning of the system utterance, and a process of repeating or paraphrasing the system utterance may be introduced.

[0042] This program further includes step 212, which is executed when it is determined in step 210 that the content of the user's utterance is unclear, and which requests the user to repeat the utterance and returns the flow of control to step 204; and step 214, which is executed when it is determined in step 210 that the user has rejected the dialogue, and which issues an utterance indicating the end of the dialogue and terminates execution of the program. This program also includes step 216, which determines whether or not to respond to the user's utterance in step 210 with casual conversation and branches the flow of control according to the result. The semantic interpretation in step 208 is performed by a user utterance semantic interpretation module included in the dialogue engine 154, based on whether or not the user's response utterance can be interpreted using natural language processing and whether or not the user's response utterance contains an expression that indicates a refusal to answer. The determination in step 216 is also made based on the dialogue history, including system utterances and user utterances. This determination is made by the user utterance semantic interpretation module based on information including the dialogue history. The determination by the user utterance semantic interpretation module may be rule-based, or may be performed by a machine learning model that has been trained in advance to input a dialogue history and output a determination result.

[0043] The program further includes step 220, when it is determined in step 216 that the system should respond to the user's utterance with casual conversation, generating and outputting a casual conversation utterance and returning the flow of control to step 204; and step 218, when it is determined in step 216 that the system should respond in accordance with the dialogue scenario, making an utterance defined by the dialogue scenario and returning the control to step 204.

[0044] Fig. 6 shows details of a program for realizing step 202 in Fig. 5. Referring to Fig. 6, step 202 includes step 240 of reading a dialogue scenario from the storage device 152 shown in Fig. 3 and loading it into memory, and step 242 of providing the initial utterance text specified by the root node of the dialogue scenario to the speech synthesis engine 158 shown in Fig. 3 to generate a voice signal of the initial utterance (opening utterance) and transmit it to the dialogue server 62 via the communication device 150 shown in Fig. 3.

[0045] This program further includes step 244 of reading user attributes from attribute information DB 170 shown in FIG. 3 ; step 246 of inputting the spoken text used for speech synthesis in step 242 and the user attributes acquired in step 244 into image generation AI module 160 shown in FIG. 3 ; and step 248 of transmitting an image generated and output by image generation AI module 160 in response to the spoken text and user attributes to terminal 60 via communication device 150. Through steps 240, 242, 244, 246, and 248, for example, opening utterance 104 shown in FIG. 2 is generated and spoken by terminal 60 shown in FIG. 1 , and an image corresponding to opening utterance 104 is displayed on LCD 68. Note that in this embodiment, text is assumed as input to image generation AI module 160. On the other hand, user attributes are often represented by numbers or symbols other than ordinary text, with the meaning of each attribute assigned to a number such as 0 or 1. Inputting such information to image generation AI module 160 is meaningless. Therefore, in this embodiment, the value indicating the user's attribute is converted into text representing the content of the attribute, concatenated with other text with a separation token in between, and input to the image generation AI module 160.

[0046] The processing of step 218, which is executed when the determination in step 216 is that a response based on the scenario should be made (the determination in step 216 is negative), is realized by a program having a control structure represented by the flowchart shown in FIG.

[0047] 7 , this program includes step 260, which branches control flow depending on whether the current dialogue mode is chat mode or not, and step 262, which branches control flow depending on whether the user's response utterance includes "yes" or an expression equivalent thereto when the determination in step 260 is negative. This program further includes step 264, which is executed when the determination in step 262 is positive, for selecting a branch corresponding to "yes" in the currently executed node in the dialogue scenario and synthesizing a question corresponding to a new node to which the branch is to be branched, and step 266, which is executed when the determination in step 262 is negative, for selecting a branch corresponding to "no" in the currently executed node in the dialogue scenario and synthesizing a question corresponding to a new node to which the branch is to be branched. Note that the utterance text of the dialogue scenario that is the basis for speech synthesis in steps 264 and 266 may include variables. By replacing these variables with specific words included in the previous utterance of the user 102, the system 100 can respond in accordance with the content of the user 102's utterance while following the dialogue scenario.

[0048] This program further includes, when the determination in step 260 is positive, step 268 of identifying a node in the dialogue scenario that has been stored in advance as a point at which the dialogue according to the dialogue scenario will be resumed, and step 270 of obtaining the spoken text of the question assigned to the node identified in step 268 and synthesizing the text into voice.

[0049] This program further includes step 272 of transmitting the voice signal synthesized in step 264, step 266 or step 270 to terminal 60, step 274 of searching and acquiring the dialogue history in dialogue history DB 166 shown in FIG. 3, and step 276 of branching the flow of control depending on whether or not to generate an image based on the current utterance text, based on the dialogue history acquired in step 274 and the current utterance text.

[0050] This program further includes step 278, in which, when the determination in step 276 is affirmative, one of the system-uttered text and the user-uttered text is selected, input to the image generation AI module 160 shown in FIG. 1 , and the image generation AI module 160 generates an image. The text to be selected in step 278 may be determined using a rule-based determination, or may be determined using a machine learning model that inputs both texts, user attributes, and dialogue history. As an example of the rule, a simple rule such as which text contains a larger number of content words or which text is longer may be used. In this case, priority may basically be given to the user-uttered text, and the system-uttered text may be selected only if the number of content words in the system-uttered text is greater than the number of content words in the user-uttered text.

[0051] The program further includes step 280 of presenting the image generated and output by image generation AI module 160 to the user by transmitting it to terminal 60, and step 282 of inputting the image output by image generation AI module 160 to caption generation AI module 162 to generate a caption for the image. The program further includes step 284 of registering the image generated in step 276 and the caption generated in step 280 as a dialogue history in dialogue history DB 166. In this way, the dialogue history includes the image generated by image generation AI module 160 and the caption generated from that image. Therefore, when the next system utterance text is generated according to the dialogue history, the content of the image generated previously is also reflected.

[0052] This program further includes step 286, which is executed when the determination in step 276 is negative, or when the determination in step 276 is positive and the processing of steps 278 to 284 is completed, and which saves the node positions in the current scenario to a memory device and terminates execution of this program.

[0053] Fig. 8 shows a detailed control structure of step 220 shown in Fig. 5. Referring to Fig. 8, step 220 includes step 300 of searching for and acquiring a dialogue history from dialogue history DB 166 shown in Fig. 3, step 302 of inputting user attributes and the dialogue history into chat response generation module 164 shown in Fig. 3 and acquiring a chat text from chat response generation module 164, step 304 of synthesizing speech of the chat text acquired in step 302 and transmitting it to terminal 60, and step 306 of branching the flow of control depending on whether or not to end the chat and return to the dialogue scenario.

[0054] This program further includes step 308, when the determination in step 306 is positive, determining the node at which the dialogue scenario should be resumed in accordance with pre-stored information, and step 310, performing speech synthesis on the text of the dialogue scenario assigned to the node determined in step 308, and transmitting the synthesized speech signal to terminal 60.

[0055] The program further includes step 312, which branches the flow of control depending on whether an image should be generated when the determination in step 306 is negative, and when the determination in step 306 is positive and the processing up to step 310 has been completed. In this embodiment, an image is generated from the chat text obtained in step 302, regardless of whether the determination in step 306 is positive or negative.

[0056] 3 when the determination in step 312 is affirmative, and step 316 of transmitting an image generated and output by the image generation AI module 160 as a result of the processing in step 314 to the terminal 60. The program further includes step 318 of inputting the image obtained in step 316 to the caption generation AI module 162, and step 320 of saving the caption generated and output by the caption generation AI module 162 as a result of the processing in step 318, together with the text of the chat response and the text generated in step 310, if any, in the dialogue history DB 166 as a dialogue history, and terminating execution of the program. When the determination in step 312 is negative, execution of the program is immediately terminated.

[0057] In this embodiment, the determination in step 306 is made by a pre-trained machine learning model based on the user attributes, the final chat text, and the spoken text and captions in the dialogue history. Of course, this determination may also be made by rule-based processing based on the user attributes, the final chat text, and the spoken text and captions in the dialogue history. The determination in step 312 is similar. Furthermore, which text to select in step 314 may be determined based on a rule base, or may be determined by a machine learning model that inputs both texts, user attributes, and the dialogue history.

[0058] Fig. 9 is a flowchart showing the control structure of a program for realizing step 302 in Fig. 8. Referring to Fig. 9, this program uses chat response candidate DB 168 to create chat response candidates.

[0059] The chat response candidate DB 168 stores a plurality of chat response candidates in advance. These candidates are determined to be appropriate chat responses by applying a language model to text collected from the web and classifying them based on various perspectives, such as whether they are likely to attract the user's interest as a chat response and whether they contain so-called hate speech. A large number of these chat response candidates are created and organized into a database that can be searched using key words as keywords.

[0060] This program includes step 350 of extracting important keywords from user utterances in the dialogue history and searching chat response candidate DB 168 using the important keywords as a key to obtain one or more chat response candidates from chat response candidate DB 168; step 352 of scoring and ranking the chat response candidates obtained in step 350; and step 354 of selecting the chat candidate with the highest score from the scores obtained for each chat candidate in step 352 and returning to the processing of step 220.

[0061] The scoring of each chat candidate in step 352 uses a pre-trained machine learning model (such as BERT (Bidirectional Encoder Representations from Transformers)) that takes a pair of a dialogue history and a chat candidate as input and outputs the probability that the chat candidate is appropriate in light of the context. Of course, scoring may be performed according to some rules.

[0062] 1.4 Operation Hereinafter, the operation of the dialogue system 50 (FIG. 1) according to the first embodiment will be described mainly with reference to the flowcharts shown in FIGS.

[0063] When the dialogue system 50 starts operating, the dialogue server 62 executes an initial process (step 200 in FIG. 5). In the initial process, the dialogue engine 154 of the dialogue server 62 establishes a connection with the terminal 60 via the communication device 150 and checks the connection with each part of the dialogue server 62.

[0064] In the following step 202, the dialogue engine 154 (see FIG. 3) of the dialogue server 62 reads the dialogue scenario from the storage device 152 shown in FIG. 3 and loads it into memory (step 240 in FIG. 6). In step 202, the dialogue engine 154 reads from memory the text of the opening utterance (question) assigned to the root node of the binary tree that constitutes the dialogue scenario. The dialogue engine 154 provides this text to the speech synthesis engine 158 shown in FIG. 3. The speech synthesis engine 158 performs speech synthesis based on the provided text and provides the synthesized speech signal to the dialogue engine 154. The dialogue engine 154 transmits this speech signal to the terminal 60 (step 242).

[0065] The dialogue engine 154 reads the user's attributes from the attribute information DB 170 (FIG. 3) (step 244). The dialogue engine 154 further provides the text of the opening utterance of the dialogue scenario and the user's attributes to the image generation AI module 160 (FIG. 3) (step 246). The image generation AI module 160 generates an image based on this text and provides it to the dialogue engine 154.

[0066] The dialogue engine 154 provides the image signal output by the image generation AI module 160 to the terminal 60 via the communication device 150 (step 248). The dialogue engine 154 then waits for a user utterance (step 204).

[0067] 1 , the terminal 60 uses an amplifier and speaker (not shown) and an LCD 68 to reproduce audio and images from a speech signal 66 representing the audio and images received from the dialogue server 62. When the user makes an utterance in response to the audio and images, the terminal 60 transmits the audio signal of the utterance to a communication device 150 of the dialogue server 62, and the communication device 150 provides the received audio signal to a dialogue engine 154.

[0068] 5, the dialogue engine 154 receives the audio signal of the user's utterance in step 204. The dialogue engine 154 provides the audio signal to the speech recognition engine 156. The speech recognition engine 156 performs speech recognition on the audio signal and outputs text representing the user's utterance to the dialogue engine 154.

[0069] In the following step 208, the dialogue engine 154 performs a semantic interpretation of the text of the user utterance received from the speech recognition engine 156. The dialogue engine 154 branches the control flow as follows according to the result of this semantic interpretation (step 210). That is, if the content of the user utterance is unclear, the dialogue engine 154 synthesizes a speech signal of an utterance requesting the user to repeat the utterance in step 212 and transmits it to the terminal 60. Thereafter, the dialogue engine 154 returns to the standby state for a user utterance (step 204). If the dialogue engine 154 determines in step 210 as a result of the semantic interpretation of the user utterance that the user is refusing the dialogue, it performs the processing required to end the dialogue in step 214 and terminates program execution.

[0070] On the other hand, if it is determined in step 210 that the content of the user utterance is neither unclear nor indicates a refusal to engage in dialogue, the dialogue engine 154 determines in step 216 whether or not to provide a chat response to the user utterance. This determination is made based on the user's attributes and dialogue history, as described above. If it is determined that a chat response should not be provided, the dialogue engine 154 executes a response process based on the scenario in step 218 and returns control to step 204. On the other hand, if it is determined that a chat response should be provided, the dialogue engine 154 executes a chat process in step 220 and returns control to step 204 after the process is completed.

[0071] In step 218, the dialogue server 62 operates as follows: Referring to Fig. 7, if the current dialogue mode is chatting (the determination in step 260 is affirmative), the dialogue engine 154 identifies a resumption portion of the dialogue scenario based on pre-stored information (268). The dialogue engine 154 further obtains the spoken text of the question assigned to the node identified in step 268 from the node and synthesizes it into voice (step 270).

[0072] On the other hand, if the determination in step 260 is negative, the dialogue engine 154 branches the flow of control according to whether the user's response utterance includes "yes" or an expression equivalent thereto (step 262). That is, if the determination in step 262 is positive, the dialogue engine 154 selects the branch corresponding to "yes" of the currently executed node in the dialogue scenario, and voice synthesizes a question corresponding to the new node to which the branch is to be branched (step 264). On the other hand, if the determination in step 262 is negative, the dialogue engine 154 selects the branch corresponding to "no" of the currently executed node in the dialogue scenario, and voice synthesizes a question corresponding to the new node to which the branch is to be branched (step 220). The synthesized voice signal is transmitted to the terminal 60 and played back.

[0073] Regardless of the results of the determinations in steps 260 and 262, the dialogue engine 154 transmits the voice signals synthesized in these steps to the terminal 60 (step 272). These voice signals are played back on the terminal 60.

[0074] The dialogue engine 154 further searches and acquires the dialogue history from the dialogue history DB 166 shown in FIG. 3 (step 274). Based on the dialogue history acquired in step 274, the dialogue engine 154 determines whether to generate an image based on the current spoken text (step 276). If the determination in step 276 is affirmative, the dialogue engine 154 inputs the system-spoken text or the user-spoken text to the image generation AI module 160 shown in FIG. 1 (step 278). The image generation AI module 160 generates and outputs an image in response to this input. The image generation AI module 160 is pre-trained to generate an image appropriate for the input. The dialogue engine 154 transmits the image generated and output by the image generation AI module 160 to the terminal 60 (step 280). As a result, this image is presented to the user by the terminal 60. The user is expected to be prompted by the speech generated by the synthesized speech signal in step 264, step 266 or step 270 and the image generated in step 280 to make a further response speech.

[0075] Meanwhile, the dialogue engine 154 further inputs the image output by the image generation AI module 160 to the caption generation AI module 162 (step 282). In response to this input, the caption generation AI module 162 generates a caption for the input image (step 282). The dialogue engine 154 registers the image generated in step 276 and the caption generated in step 282 as a dialogue history in the dialogue history DB 166 (step 284). Thereafter, the dialogue engine 154 stores the current node position in the scenario in memory as the scenario resume position (step 286), and the dialogue engine 154 ends the processing of step 218.

[0076] On the other hand, if the determination in step 274 is negative, the dialogue engine 154 does not execute steps 276 to 284, but immediately executes step 286 and ends the processing of step 218.

[0077] On the other hand, when the determination in step 216 in Fig. 5 is positive, the dialogue engine 154 operates as follows: Referring to Fig. 8, the dialogue engine 154 first searches and acquires a dialogue history from the dialogue history DB 166 shown in Fig. 3 (step 300). The dialogue engine 154 further inputs the user attributes and the dialogue history to the chat response generation module 164 shown in Fig. 3, and acquires a chat text from the chat response generation module 164 (step 302).

[0078] In step 302, the dialogue engine 154 operates as follows. Referring to FIG. 9 , the dialogue engine 154 extracts important keywords from the user's utterances in the dialogue history. The dialogue engine 154 then searches the chat response candidate DB 168 using the important keywords as a key to obtain one or more chat response candidates from the chat response candidate DB 168 (step 350). The dialogue engine 154 inputs these chat response candidates and the dialogue history into a machine learning model, thereby scoring and ranking each chat candidate (step 352). The dialogue engine 154 selects the chat candidate with the highest score from the scores obtained for each chat candidate (step 354). This candidate is selected as the chat text in step 302 of FIG. 8 .

[0079] Referring again to FIG. 8, the dialogue engine 154 synthesizes the chat text obtained in step 302 by providing it to the speech synthesis engine 158 in step 304 , and transmits the synthesized speech signal to the terminal 60 .

[0080] Thereafter, the dialogue engine 154 determines whether to end the chat and return to the dialogue scenario (step 306). If the determination in step 306 is positive, the dialogue engine 154 determines the node at which the dialogue scenario should be resumed in accordance with pre-stored information (step 308). The dialogue engine 154 then performs speech synthesis on the text assigned to the node determined in step 308 in the dialogue scenario, and transmits the synthesized speech signal to the terminal 60 (step 310). The dialogue engine 154 then executes step 312, in which it determines whether an image should be generated. On the other hand, if the determination in step 306 is negative, the dialogue engine 154 immediately executes the determination in step 312 without executing steps 308 and 310.

[0081] If the determination in step 312 is positive, the dialogue engine 154 inputs the system-uttered text or the user-uttered text to the image generation AI module 160 shown in FIG. 3 (step 314). As a result of the processing in step 314, the image generation AI module 160 generates and outputs an image appropriate for the input text. The dialogue engine 154 transmits this image to the terminal 60 (step 316). The dialogue engine 154 also inputs the image obtained in step 316 to the caption generation AI module 162 (step 318). As a result of this processing, the caption generation AI module 162 generates and outputs a caption that describes the image obtained in step 316. The dialogue engine 154 saves the caption output by the caption generation AI module 162, together with the text of the chat response and the text generated in step 310, if any, as a dialogue history in the dialogue history DB 166, and terminates execution of this program (step 320).

[0082] On the other hand, if the determination in step 312 is negative, that is, if it is determined that an image is not to be generated, the dialogue engine 154 immediately terminates the execution of this program.

[0083] As described above, the dialogue system 50 according to this embodiment utilizes generative AI to generate visual materials on demand from user responses and system responses and provide them to the user. As a result, the dialogue system 50 can present visual materials such as images to the user from any utterance. In addition, the dialogue system 50 provides a procedure for generating text from any generated visual materials and generating a response to the system utterance based on the generated text.

[0084] Specifically, the dialogue system 50 verbalizes the visual materials using AI that converts the contents of the visual materials into text and extracts words contained therein that are likely to become topics for casual conversation. The dialogue system 50 then generates chatty utterances consisting of questions that broaden the dialogue, taking into account the context of the dialogue or the user's immediate response. Furthermore, by generating probing questions from the chatty utterances, questions that are related to the questions generated from the visual materials but ask deeper questions can also be generated. As a result, the dialogue system 50 can further develop the dialogue with the user.

[0085] The dialogue system 50 can provide the user with visual materials corresponding to the content of any user response and system response. Furthermore, the dialogue system 50 also generates chatty responses based on the visual materials. In other words, the dialogue system 50 is a dialogue system that utilizes generative AI in both directions. Therefore, it can be easily applied to task-oriented dialogue systems that use dialogue scenarios such as those in the above-described embodiments. Furthermore, the dialogue system 50 can also be applied to non-task-oriented dialogue systems that specialize in chatting.

[0086] As can be seen from this configuration, the dialogue system 50 can enhance the sense of realism of the dialogue with the user and generate chatty responses on a wide range of topics based on the generated visual materials. The dialogue system 50 can contribute to expanding the topics of the dialogue with the user and livening up the dialogue itself. For example, by using such a dialogue system as a system for implementing reminiscence therapy for elderly people, it is expected to be highly effective in helping elderly people enjoy better lives.

[0087] 2. Second Embodiment 2.1 Overview The dialogue engine 154 of the dialogue server 62 according to the first embodiment has the function of generating visual materials related to user utterances or system utterances and presenting them to the user. To advance the dialogue, the dialogue engine 154 primarily uses the text of user utterances and system utterances and the dialogue history based on them, in addition to the user's attributes. However, when engaging in dialogue with a user, it is desirable to consider not only the content of the utterance but also the user's emotions at the time of speaking and the impact of the dialogue topic on the user. This second embodiment not only performs speech recognition of the user's utterance, but also generates the content of the system utterance and visual materials while taking into consideration the context of the dialogue, such as the user's emotions expressed by the user's utterance through the prosody of the user's utterance, and the polarity of the content of the user's utterance (positive / negative, like / dislike, positive / negative).

[0088] An example of the progression of a dialogue in the second embodiment will be outlined with reference to FIG. 10 . For example, assume that a system 370 according to this embodiment is engaged in a dialogue with a user 372. In this example, the user 372 is in his 80s, male, and lives alone. In response to the system 370's utterance, "What are your memories of elementary school?", the user 372 responds with, "Sports day." In response to the system 370's further utterance, "What sports did you play?", the user 372 responds with, "I played a foot race. I actually wanted to play tug-of-war, but I was too small." From the text of the user 372's utterance alone, it is not immediately clear what emotion the user 372 is expressing. Therefore, in the second embodiment, the system 370 detects the emotion of the user 372 by analyzing the prosodic features of the user 372's speech, in addition to simple semantic analysis of the text.

[0089] In this embodiment, the prosodic features of the speech refer to phonological features other than the text of the speech, such as the pitch, tone, intonation, accent, volume, emphasis, speed, rhythm, beat, and pause of the speech. By analyzing the prosodic features, the emotion of the speaker can be estimated. As a method for estimating the emotion, for example, the technology described in Patent Document 1 can be used. In this embodiment, the emotions are limited to two, "cheerful" and "dark." Of course, emotions do not need to be divided into these two states, and a wider variety of emotions can be classified.

[0090] Furthermore, merely superficially analyzing the text of the user's utterance does not immediately reveal what meaning the content of the utterance by the user 372 has for the user 372. Therefore, in the second embodiment, a polarity analysis is performed on the text of the user's utterance. In the second embodiment, polarity is a concept indicating whether the content of the utterance is positive or negative overall. Specifically, the polarity of an utterance is determined from the combination of the polarity of the noun that forms the core of the utterance and its predicate. The polarity of a noun is a concept indicating whether the thing represented by the noun represents a problem or a burden (e.g., "influenza" or "mistake") or not (e.g., "medicine," "food," "correction"). The polarity of a predicate is a concept indicating whether the predicate activates (e.g., "causes" or "progresses") or inactivates (e.g., "decreases" or "destroys") the function represented by the subject of the predicate. If the noun polarity is positive and the predicate polarity is positive, the utterance is considered to have a positive overall content. If the noun polarity is positive and the predicate polarity is negative, the utterance is considered to have a negative overall content. If the noun polarity is negative and the predicate polarity is positive, the utterance is considered to have a negative overall content. Also, if the noun polarity is negative and the predicate polarity is negative, the utterance is considered to have a positive overall content.

[0091] For such polarity determination, for example, the technology disclosed in Patent Document 2 can be used.

[0092] In the example shown in FIG. 10 , the polarity detected for the second utterance by user 372 is negative, and the emotion is “gloomy.” Assume that the system responds with “I see,” and user 372 continues, “But you were a fast runner, so you got first place.” The polarity of this utterance suddenly changes to positive, and the emotion is “cheerful.” Assume that the dialogue server 380 praises the content of user 372’s utterance by saying, “That’s wonderful,” and user 372 further utters, “I was so happy,” and the polarity is also positive, and the emotion is “cheerful.” At this timing, if the dialogue server 380 displays an image 384 showing an elementary school student who won first place in a running race, user 372 is likely to remember even more past events and continue talking. As a result, it is expected that the reminiscence method will be more effective.

[0093] On the other hand, even if the emotion of the user 372 and the polarity of the speech of the user 372 are obtained, if the content of the image 384 generated in accordance with the content of the speech is ambiguous or shows content different from the content of the speech, the effect of the reminiscence therapy is likely to be reduced. Therefore, in this second embodiment, as will be described later, efforts are made to more reliably obtain images that correctly reflect the content of the speech.

[0094] The above-mentioned user emotions, dialogue situations such as speech polarity, and user attributes are stored in the dialogue history. However, in this embodiment, since all of this information is finite, it is represented as non-verbal information (e.g., numerical values ​​such as 0 and 1) rather than text and then stored in the dialogue history. As will be described later, when generating system utterance text or generating visual materials that visualize the system utterance text, it is assumed that the input to the machine learning model is text. Therefore, the information stored non-verbally in the dialogue history as described above is converted into text when input to the machine learning model.

[0095] 2.2 Configuration Fig. 11 shows, in block diagram form, the schematic configuration of a dialogue server 380 used in a system 370 according to the second embodiment. Referring to Fig. 11, the dialogue server 380 includes, similar to the dialogue server 62 shown in Fig. 3, a communication device 150, a storage device 152 for storing dialogue scenarios, a speech recognition engine 156, a speech synthesis engine 158, an image generation AI module 160, and a caption generation AI module 162.

[0096] The dialogue server 380 further includes a dialogue engine 154 for generating a text of an utterance in response to the user utterance, using the content of the user utterance output by the speech recognition engine 156 and the dialogue scenario stored in the storage device 152 .

[0097] The dialogue server 380 differs from the dialogue server 62 of the first embodiment in that it further includes an emotion detection unit 393 that receives a voice signal of a user's utterance, performs prosodic analysis on the voice signal, and estimates the emotion of the speaker, and a polarity determination unit 391 that receives a voice recognition result from the voice recognition engine 156 and determines the polarity of the content of the user's utterance. Instead of the dialogue history DB 166 of the first embodiment, the dialogue server 380 includes a dialogue history DB 396 that stores, as a dialogue history, the voice recognition result from the voice recognition engine 156, the emotion detection result from the emotion detection unit 393, the polarity determination result from the polarity determination unit 391, and the like. Like the dialogue engine 154 of the first embodiment, the emotion detection unit 393 includes a user utterance semantic interpretation module (not shown) that interprets the meaning of the user utterance through natural language processing.

[0098] 3, the dialogue server 380 also includes a small talk response generation module 394 for generating small talk responses for a small talk conversation with a user, using the dialogue history stored in the dialogue history DB 396 and the user attributes stored in the attribute information DB 170. The dialogue server 380 further includes a large-scale language model 398 that the small talk response generation module 394 uses when generating small talk from the dialogue history.

[0099] The dialogue server 380 further includes a text generation module for image generation 392 for formatting the speech text based on the dialogue scenario received from the dialogue engine or the text for image generation received from the chat response generation module 394 so that the image generation AI module 160 can generate an appropriate image for the input, and providing the formatted text to the image generation AI module 160. The caption output from the caption generation AI module 162 is stored in the dialogue history DB 396 as part of the dialogue history, and is used for generating chat responses from the next time onwards.

[0100] Furthermore, instead of the dialogue engine 154 shown in FIG. 3 , the dialogue server 380 includes a dialogue engine 390 that is connected to the storage device 152, the attribute information DB 170, the speech synthesis engine 158, the speech recognition engine 156, the emotion detection unit 393, the text generation module for image generation 392, and the chat response generation module 394, and that cooperates with each part of the dialogue server 380 to execute a dialogue with the user 372.

[0101] The text generation module for image generation 392 is a machine learning text generation model that has the function of generating input text to the image generation AI module 160 from a combination of input utterance text and dialogue history so that the image generated by the image generation AI module 160 is appropriate for the input obtained from the combination of utterance text and dialogue history in accordance with the dialogue scenario output from the dialogue engine 390, or the combination of the chat response and dialogue history input from the chat response generation module 394. The text generation module for image generation 392 has been trained in advance using dialogue data prepared in advance and training data created using the image generation AI module 160.

[0102] 12 shows a schematic configuration of a training system 410 for training the text generation module 392 for image generation. Referring to FIG. 12, the training system 410 includes a storage device 420 storing a large amount of dialogue history, and an image generation unit 422 for reading out a certain spoken text and the dialogue history associated with the spoken text from the dialogue data stored in the storage device 420 and inputting the read-out spoken text into the image generation AI module 160 to generate an image. The training system 410 further includes a verification unit 424 for displaying the images generated by the image generation AI module 160, verifying the images by an operator, and outputting the images as appropriate for the input and as inappropriate images, and an annotation unit 426 for the operator to perform annotating processes such as adaptation and correction on the spoken text input to the image generation AI module 160 when it is determined that the image output by the image generation AI module 160 is inappropriate. The speech text corrected by the annotate unit 426 is provided to the image generation unit 422 together with the original speech history, and further provided to the image generation AI module 160.

[0103] The training system 410 further includes a training data creation unit 428 for creating training data for the text generation module for image generation 392 by combining, as teacher data, the spoken text determined by the verification unit 424 to be suitable as training data, and, as input, the original spoken text and the speech history read from the storage unit 420, and a training data storage device 430 for storing the training data created by the training data creation unit 428. The training data storage device 430 stores a large amount of training data created by the above-mentioned process.

[0104] In this embodiment, the dialogue history includes information such as system spoken text, user spoken text, user attributes, emotions associated with the speech, speech polarity, images generated by the image generation AI module 160, and captions created for the image generation AI module 160.

[0105] The training system 410 further includes a learning processing unit 432 for training the text generation module for image generation 392 using the learning data stored in the learning data storage device 430. The text generation module for image generation 392 is a machine learning model that has a function of generating and outputting spoken text based on input spoken text and the speech history accompanying the spoken text, so that the image generation AI module 160 generates a correct and clear image related to the spoken text, and BERT, etc., can be used.

[0106] 13 is a flowchart showing a control structure of a main routine of a computer program for implementing dialogue server 380 according to the second embodiment by a computer. Fig. 13 corresponds to Fig. 5 of the first embodiment.

[0107] Referring to Figure 13, this program includes step 448 for performing initial processing such as reading a dialogue scenario from the storage device 152 shown in Figure 11, connecting with the communication device 150, the voice recognition engine 156, the voice synthesis engine 158, the image generation AI module 160, the caption generation AI module 162, the emotion detection unit 393, the polarity determination unit 391, the chat response generation module 394, and the text generation module for image generation 392, and initializing the memory area.

[0108] This program further includes step 450 of starting a dialogue in accordance with the dialogue scenario read out in step 448, and step 204 of waiting until a voice signal is received from terminal 60, and upon receiving the voice signal, accepting the voice signal as a voice signal of a user utterance. This program further includes step 206 of performing speech recognition on the voice signal accepted in step 204 and converting it into text, step 452 of performing semantic interpretation and emotion detection on the text obtained in step 206, and polarity determination on the voice signal accepted in step 204, and step 454 of branching the flow of control in accordance with the semantic interpretation performed in step 452.

[0109] In the second embodiment, the number of options for meaning determination in step 454 is not limited to three, but may be four or more.

[0110] This program further includes step 212, which is executed when it is determined in step 454 that the content of the user's utterance is unclear, and which requests the user to repeat the utterance and returns the flow of control to step 204; step 214, which is executed when it is determined in step 454 that the user has refused the dialogue, and which makes a utterance to end the dialogue and terminates execution of the program; and step 456, which determines whether or not to respond to the user's utterance in step 454 with casual conversation and branches the flow of control according to the result.

[0111] The semantic interpretation in step 452 is performed by a user utterance semantic interpretation module included in the dialogue engine 390, using natural language processing to determine whether the user's response utterance can be interpreted and whether the user's response utterance contains an expression that indicates refusal to answer. Furthermore, the determination in step 456 is made based on the dialogue history, including system utterances and user utterances. This determination is made by the user utterance semantic interpretation module based on information including the dialogue history. Similar to the first embodiment, the determination by the user utterance semantic interpretation module may be rule-based, or may be made using a machine learning model that has been trained in advance to input the dialogue history and output a determination result.

[0112] The program further includes step 460, in which, when it is determined in step 456 that the system should respond to the user's utterance with casual conversation, the program generates and outputs a casual conversation utterance using the large-scale language model 398 based on the dialogue history stored in the dialogue history DB 396, including the utterance for the previous user utterance and the caption generated by the caption generation AI module 162, and returns the flow of control to step 204; and step 458, in which, when it is determined in step 456 that the system should respond in accordance with the dialogue scenario, the program makes an utterance defined by the dialogue scenario and returns control to step 204.

[0113] Fig. 14 is a flowchart showing a control structure of a program for implementing step 450 of Fig. 13. Referring to Fig. 14, this program includes steps 240, 242, 244, and 248 similar to those shown in Fig. 6 relating to the first embodiment, and step 500 inserted in order between steps 244 and 248 instead of step 246 shown in Fig. 6, for concatenating an utterance text of an opening utterance specified by the dialogue scenario with a text representing attributes of user 372, and inputting the concatenated text to text generation module for image generation 392; and step 502 for inputting the text output by text generation module for image generation 392 as a result of the processing in step 500 to image generation AI module 160.

[0114] FIG. 15 shows the control structure of a program for implementing step 458 in FIG. 13 . Referring to FIG. 15 , step 458 includes steps 260 to 276, as well as steps 280 and 282, similar to those shown in FIG. 7 . This program further includes step 550, which is inserted in order between steps 276 and 278 in place of step 278 shown in FIG. 7 , for selecting either the latest user utterance text or the system utterance text, concatenating the selected text, user attributes, and utterance history (including the system utterance text, the user utterance text, the emotion and polarity of the user utterance, and the image and caption generated from the utterance) and inputting the concatenated text into the text for image generation module 392, and step 552 for inputting the text output by the text for image generation module 392 into the image generation AI module 160. Note that, of the inputs to the text for image generation module 392 that are not text, they are converted into text (converted into text).

[0115] Step 458 further includes step 554, after step 282, of storing the text generated in step 550, the image generated in step 552, and the caption generated in step 282 in the dialogue history DB 396 as part of the dialogue history, and then executing step 286.

[0116] Fig. 16 is a flowchart showing a control structure of a program for implementing step 460 of Fig. 13. Referring to Fig. 16, this program includes step 600 of reading out a dialogue history from dialogue history DB 396, and step 602 of inputting user attributes, utterance text in the dialogue history, emotion, polarity, and captions into a chat response generation module to acquire chat text. The chat response generation module converts these into text as necessary and then concatenates the text, inputs it into large-scale language model 398 shown in Fig. 11, and acquires the output as chat text.

[0117] This program further includes steps 304 to 312 and steps 316 to 320 similar to those shown in Figure 8 for the first embodiment, and step 604 inserted in order between steps 312 and 316 instead of step 314 shown in Figure 8, in which either the current user utterance text or the system utterance text selected therefrom, user attributes, and information in the dialogue history converted into text as necessary are concatenated and input into the image generation text generation module 392 shown in Figure 11, and step 606 in which the text output by the image generation text generation module 392 is input into the image generation AI module 160.

[0118] 2.3 Operation The operation of the dialogue server 380 is basically the same as that of the dialogue server 62 in the first embodiment. However, the operation of the dialogue server 380 differs from that of the dialogue server 62 in that not only is speech recognition processing by the speech recognition engine 156 performed on the utterance of the user 372, but also emotion detection processing by the emotion detection unit 393 and polarity determination processing by the polarity determination unit 391 are performed, these are stored in the dialogue history DB 396, the chat response generation module 394 generates chat responses using the dialogue history DB 396 and the large-scale language model 398, and either system utterance text or user utterance including a chat response, the dialogue history, etc. are input to the text for image generation module 392 rather than directly to the image generation AI module 160, and the text generated by the text for image generation module 392 is provided to the image generation AI module 160.

[0119] Among the dialogue history stored in the dialogue history DB 396, information related to user utterances includes not only the text of the utterance by the user 372, but also the user 372's emotions at that time, the polarity of the content of the utterance, the image generated by the image generation AI module 160, and the caption generated by the caption generation AI module 162. When the dialogue server 380 generates a response, the chat response generation module 394 uses not only the system utterance text generated by the dialogue server 380, but also the dialogue history of user utterances and system utterances previously stored in the dialogue history DB 396. The information provided from the chat response generation module 394 to the text for image generation generation module 392 and the information provided from the dialogue engine 390 to the text for image generation generation module 392 include all of this information in addition to the current system utterance text and user utterance text. Therefore, the image generated by the image generation AI module 160 reflects not only the text of past user utterances, but also the dialogue history, such as the user's emotions at that time and in the past and the polarity of the content of the user utterances.

[0120] Furthermore, the system utterance text and the user utterance text are not immediately provided to the image generation AI module 160. Instead, the system utterance text and the user utterance text, together with the dialogue history, are provided to the image generation text generation module 392, and the text generated by the image generation text generation module 392 is provided to the image generation AI module 160. The image generation text generation module 392 selects either the input system utterance text or the input user utterance text, and converts the input text based on the selected text and the dialogue history so that the image generation AI module 160 generates a clear image that better reflects the content of the input text. Therefore, the output of the image generation AI module 160 is a clear image that appropriately reflects the content of either the system utterance text or the user utterance text and the dialogue history. As a result, the user is more likely to be inspired by the image, recall past events, and make new utterances. This is expected to enhance the effectiveness of reminiscence therapy.

[0121] 3. Hardware Configuration Fig. 17 is an external view of the dialogue server 62 according to the first embodiment shown in Fig. 1, for example. Fig. 18 is a hardware block diagram of the dialogue server 62. Figs. 17 and 18 also apply to the dialogue server 380 according to the second embodiment. Therefore, in the following explanation, the hardware configuration of the dialogue server 62 will be mainly explained based on the first embodiment.

[0122] 17, this dialogue server 62 includes a computer 650 having a DVD (Digital Versatile Disc) drive 662, and a keyboard 654, a mouse 656, and a monitor 652 for dialogue with a user, all of which are connected to the computer 650. Of course, these are just one example of a configuration for when dialogue with an operator becomes necessary, and any general hardware and software (for example, a touch panel, voice input, or a general pointing device) that can be used for dialogue with an operator can also be used.

[0123] 17 and 18 , computer 650 includes, in addition to a DVD drive 662, a CPU (Central Processing Unit) 710, a GPU (Graphics Processing Unit) 712, and a bus 720 connected to CPU 710, GPU 712, and DVD drive 662. Computer 650 further includes bus 720, a ROM (Read-Only Memory) 714 connected to bus 720 and storing a boot-up program of computer 650, etc., a RAM (Random Access Memory) 716 connected to bus 720 and storing instructions constituting a program, system programs, working data, etc., and an SSD (Solid State Drive) 718 which is a non-volatile memory connected to bus 720. The SSD 718 is for storing programs executed by the CPU 710 and the GPU 712, as well as data used by the programs executed by the CPU 710 and the GPU 712. The computer 650 further includes a network I / F (Interface) 726 that provides connection to a network that enables communication with other terminals, and a USB port 664 to which a USB (Universal Serial Bus) memory 702 is detachable and that provides communication between the USB memory 702 and each unit within the computer 650.

[0124] The computer 650 further includes an audio I / F 722 connected to the microphone 660, the speaker 658, and the bus 720. The audio I / F 722 reads audio signals, video signals, and text data generated by the CPU 710 and stored in the RAM 716 or the SSD 718 according to instructions from the CPU 710, converts the audio signals to analog, amplifies the audio signals, and drives the speaker 658. The computer 650 also includes a digital audio signal from the microphone 660 and stores the digital audio signal at an address in the RAM 716 or the SSD 718 specified by the CPU 710. In this embodiment of the present invention, a conversation with a user is conducted using the terminal 60 as shown in FIG. 1 . For this purpose, the keyboard 654 and the microphone 660 are not used. However, the conversation server 62 may directly conduct the conversation using the microphone 660, the keyboard 654, and the audio I / F 722.

[0125] In the above embodiment, the programs and the like that realize the functions of the dialogue server 62 shown in Fig. 1 and the dialogue server 380 shown in Fig. 11 are all stored, for example, in the ROM 714, SSD 718, DVD 700, or USB memory 702 shown in Fig. 18, or in a storage medium of an external device (not shown) connected via the network I / F 726 and the network 704. Typically, these data and parameters are written to the SSD 718 from the outside, for example, and loaded into the RAM 716 when the computer 650 is executed.

[0126] Computer programs for operating this computer system to realize the functions of each component of the interaction server 62 shown in Figure 1 and the interaction server 380 shown in Figure 11 are stored on a DVD 700 inserted into the DVD drive 662 and transferred from the DVD drive 662 to the SSD 718. Alternatively, these programs may be stored in a USB memory 702, which may be inserted into a USB port 664 and transferred to the SSD 718. Alternatively, the programs may be transmitted to the computer 650 via the network 704 and stored in the SSD 718. Of course, a source program may be input using the keyboard 654, the monitor 652, and the mouse 656, and the compiled object program may be stored in the SSD 718.

[0127] The program is loaded into RAM 716 when executed. If the program is written in a scripting language, the script entered by the operator using the keyboard 654 or the like may be stored in the SSD 718. In the case of a program that runs on a virtual machine, a program that functions as a virtual machine must be installed in the computer 650 in advance. Machine learning models such as deep neural networks are used for the dialogue engine 154, speech recognition engine 156, speech synthesis engine 158, image generation AI module 160, caption generation AI module 162, and chat response generation module 164 shown in FIG. 2 . The dialogue server 62 may use a machine learning model that has been trained in another device, or the dialogue server 62 may be used as a learning device to train the machine learning model. The same applies to the dialogue server 380 according to the second embodiment.

[0128] The CPU 710 reads a program from the RAM 716 according to an address indicated by an internal register called a program counter (not shown) and interprets the instructions. The CPU 710 reads data necessary to execute the instructions from the RAM 716, the SSD 718, or another device according to the address specified by the instruction, and executes the processing specified by the instruction. The CPU 710 stores the execution result data at an address specified by the program, such as the RAM 716, the SSD 718, or a register within the CPU 710. Depending on the address, the execution result data is output from the computer to an external device, for example, via the network I / F 726. The output destination is, for example, the terminal 60 shown in FIG. 1. At this time, the program counter value is also updated by the program. The computer program may be loaded directly into the RAM 716 from the DVD 700, the USB memory 702, or via the network 704. Of the programs executed by the CPU 710, some tasks (mainly numerical calculations) are issued to the GPU 712 according to instructions contained in the programs or according to the analysis results obtained when the CPU 710 executes the instructions.

[0129] The program that enables the computer 650 to realize the functions of each unit of the dialogue server 62 and the dialogue server 380 according to the above-described embodiment includes a plurality of instructions written and arranged to cause the computer 650 to operate to realize those functions. Some of the basic functions required to execute these instructions may be provided by an operating system (OS) or third-party program running on the computer 650, various toolkit modules installed on the computer 650, or the program execution environment. Therefore, the program does not necessarily include all of the functions required to realize the system and method according to this embodiment. The program may include only instructions that execute the operations of the above-described devices and their components by statically linking appropriate functions or modules at compile time or by dynamically calling them at runtime in a controlled manner to achieve the desired results. The method for operating the computer 650 for this purpose is well known. Therefore, a description of the method for operating the computer 650 will not be repeated here.

[0130] The GPU 712 is capable of parallel processing, and can execute a large amount of calculations associated with machine learning and inference simultaneously in parallel or in a pipelined manner. For example, parallel calculation elements discovered in a program when the program is compiled or when the program is executed are dispatched from the CPU 710 to the GPU 712 as needed, and executed. The results are returned to the CPU 710 directly or via a predetermined address in the RAM 716 and assigned to a predetermined variable in the program.

[0131] The hardware configuration of the terminal 60 commonly used in the first and second embodiments is shown in Fig. 19. Referring to Fig. 19, the terminal 60 includes a processor 750 including a CPU and a memory (not shown), and a communication device 752 connected to the processor 750 for providing a connection with the dialogue server 62 or the dialogue server 380 via the network 704 shown in Fig. 18.

[0132] 1 , a display control device 758 connected to the processor 750 and the LCD 68, a speaker 754 and a microphone 756 provided in positions not visible to the user in the terminal 60 shown in Fig. 1 , and an audio processing device 760 connected to the processor 750, the speaker 754, and the microphone 756. The audio processing device 760 has the functions of digitizing an audio signal from the microphone 756 and converting it into a format suitable for communication via the communication device 752, and converting a synthesized audio signal received from the dialogue server 62 via the communication device 752 into an analog signal, amplifying the obtained analog audio signal, and providing it to the speaker 754 to generate audio.

[0133] The embodiments disclosed herein are merely examples, and the present invention is not limited to the above-described embodiments. The scope of the present invention is defined by the claims in the scope of the claims, taking into consideration the description of the detailed description of the invention, and includes all modifications within the meaning and scope equivalent to the wordings described therein.

[0134] 50 Dialogue system 60 Terminal 62, 380 Dialogue server 64 Audio signal 66 Speech signal 68 LCD 100, 370 System 102, 372 User 104 Opening utterance 108, 124, 384 Image 110, 120 Response utterance 114 Text 118 System utterance 150, 752 Communication device 152, 420 Storage device 154, 390 Dialogue engine 156 Speech recognition engine 158 Speech synthesis engine 160 Image generation AI module 162 Caption generation AI module 164, 394 Chat response generation module 166, 396 Dialogue history DB 168 Chat response candidate DB 170 Attribute information DB 391 Polarity determination unit 392 Text generation module for image generation 393 Emotion detection unit 398 Large-scale language model 410 Training system 422 Image generation unit 424 Verification unit 426 Annotation unit 428 Training data creation unit 430 Training data storage device 432 Training processing unit

Claims

1. An interactive device for interacting with a user, comprising: a system utterance text generation means for generating a system utterance text based on a dialogue history with the user; a visual material generation means for generating visual material based on either the system utterance text or a text uttered by the user; and a text generation means for visual material generation for generating text for visual material generation based on either the system utterance text or the text uttered by the user, so that the visual material generation means can generate visual material appropriate for an input, and providing the text for visual material generation to the visual material generation means.

2. The dialogue device according to claim 1, wherein the means for generating text for generating visual materials includes a machine learning model trained to generate the text for generating visual materials using a dialogue history with the user as input.

3. The dialogue device as described in claim 1, further comprising a text generation means for generating related text related to the dialogue with the user by verbalizing the content of the visual material generated by the visual material generation means, wherein the system utterance text generation means, the visual material generation means, or the text generation means for generating visual material further receives the related text as an input.

4. The dialogue device of claim 1, wherein the dialogue history includes non-verbalized information that non-verbalizes at least a portion of the dialogue situation, and the visual material generation means or the text generation means for generating visual material converts the non-verbalized information of the dialogue history into text, and inputs the text by linking it with other text included in the dialogue history.

5. A computer program that causes a computer to function as an interactive device for interacting with a user, the computer program causing the computer to function as: a system utterance text generation means for generating system utterance text based on an interaction history with the user; a visual material generation means for generating visual material based on either the system utterance text or the user's utterance text; and a text generation means for visual material generation that generates text for visual material generation based on either the system utterance text or the user's utterance text so that the visual material generation means can generate visual material appropriate to the input, and provides the text for visual material generation to the visual material generation means.

6. A computer-implemented method for interacting with a user, comprising: a step of a computer generating a system utterance text based on a dialogue history with the user; a step of a computer generating visual material based on either the system utterance text or the user utterance text; and a step of generating visual material generation text based on either the system utterance text or the user utterance text so that appropriate visual material for the input can be generated in the visual material generating step, and using the generated text as input to the visual material generating step.

Images