Dialogue-adaptive chatbot

Abstract

This invention provides a chatbot, method, and program that are computer systems capable of more deeply reflecting the context of user interactions and enabling more natural dialogue. [Solution] A method performed by a computer in interaction between a user and a large-scale language model, comprising: a first score calculation step of calculating a first score based on the user's facial expressions acquired by a camera; a prompt generation step of generating a prompt based on the first score; and a prompt addition step of adding a prompt to the user's input to the large-scale language model.

Description

[Technical Field]

[0001] The present application generally relates to chatbots, and more particularly to a chatbot that utilizes LLM (Large Language Model) and CNN (Convolutional Neural Network) to better reflect and adapt to the context of interaction with users. [Background Art]

[0002] Technological innovation in generative AI-based chatbots (computer programs that automatically respond to user questions) that utilize LLMs (Large Language Models) represented by ChatGPT has been remarkable, and they are being used in various applications such as customer support, online education, text writing, text correction, machine translation, automatic program code generation, and program bug checking.

[0003] On the other hand, CNN (Convolutional Neural Network) is a type of neural network that has the advantage of being able to process image data with high real-time performance, and its unique architecture enables effective extraction of image features. For this reason, it is widely applied in various fields dealing with images and videos, such as face recognition, object detection, video analysis, and image classification.

[0004] Various technologies have been utilized and proposed to make chatbot dialogue more natural. For example, Patent Document 1 discloses a technology for generating dialogue content of a dialogue agent by combining user's linguistic information and non-linguistic information.

[0005] However, in the prior art, problems remain in achieving more natural dialogue, that is, in easily providing interaction close to communication between humans. In particular, further improvement has been demanded in terms of ease of system implementation and reduction of development and operation costs by enabling the direct use of currently widely available chatbot services such as ChatGPT, instead of custom-developing everything from scratch. The present invention has been made in view of such circumstances. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2024-112283 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] We provide a chatbot that more deeply reflects the context of user interactions and enables more natural conversations. [Means for solving the problem]

[0008] The present invention solves the above problem by providing a method performed by a computer in an interaction between a user and a large-scale language model, the method comprising: a first score calculation step of calculating a first score based on the user's facial expressions acquired by a camera; a prompt generation step of generating a prompt based on the first score; and a prompt addition step of adding the prompt to the user's input to the large-scale language model.

[0009] Furthermore, the present invention solves the above problem by providing the method described in paragraph 0008, wherein the first score is a score relating to one or more of the user's level of understanding, level of interest, and level of emotional positivity.

[0010] Furthermore, the present invention solves the above problems by providing a method performed by a computer in an interaction between a user and a large-scale language model, which includes a second score calculation step of calculating a second score based on the user's input to the large-scale language model or the output from the large-scale language model to the user; a prompt generation step of generating a prompt based on the second score; and a prompt addition step of adding the prompt to the user's input to the large-scale language model.

[0011] Furthermore, the present invention solves the above problem by providing the method described in paragraph 0010, wherein the first score is a score relating to one or more of the user's level of understanding, level of interest, and level of emotional positivity.

[0012] Furthermore, the present invention solves the above problem by providing the method described in paragraph 0010, wherein the second score calculation step calculates a second score based on one or more of the following: matching with a predetermined keyword, semantic similarity with an expected answer, or the output of a trained machine learning model.

[0013] Furthermore, the present invention solves the above problems by providing a method performed in interaction between a user and a large-scale language model, which includes a stage determination step of determining the stage of the interaction based on the user's input to the large-scale language model or the output from the large-scale language model to the user; a prompt generation step of generating a prompt based on the stage; and a prompt addition step of adding the prompt to the user's input to the large-scale language model, and is performed by a computer.

[0014] Furthermore, the present invention solves the above problem by providing the method described in paragraph 0013, wherein the dialogue stage is one or more of the following: opening, needs hearing, presentation, resolving customer anxieties and questions, closing, and after-sales follow-up.

[0015] Furthermore, the present invention solves the above problem by providing the method described in paragraph 0013, wherein the stage determination step determines the stage of the dialogue based on one or more of the following: matching with a predetermined keyword, semantic similarity with an expected answer, or the output of a trained machine learning model.

[0016] Furthermore, the present invention solves the above problem by providing a program that is executed in a dialogue between a user and a large-scale language model, which causes a computer to execute a group of first score calculation commands that calculate a first score based on the user's facial expressions acquired by a camera, a group of prompt generation commands that generate prompts based on the first score, and a group of prompt addition commands that add the prompts to the user's input to the large-scale language model.

[0017] Furthermore, the present invention solves the above problem by providing the program described in paragraph 0016, wherein the first score is a score relating to one or more of the user's level of understanding, level of interest, and level of emotional positivity.

[0018] Furthermore, the present invention solves the above problems by providing a program that is executed in interaction between a user and a large-scale language model, which causes a computer to execute a group of second score calculation commands that calculate a second score based on the user's input to the large-scale language model or the output from the large-scale language model to the user, a group of prompt generation commands that generate a prompt based on the second score, and a group of prompt addition commands that add the prompt to the user's input to the large-scale language model.

[0019] Furthermore, the present invention solves the above problem by providing the program described in paragraph 0018, wherein the first score is a score relating to one or more of the user's level of understanding, level of interest, and level of emotional positivity.

[0020] Furthermore, the present invention solves the above problem by providing the program according to paragraph 0018, wherein the second score calculation instruction group includes an instruction group that calculates the second score based on any one or more of matching with a predetermined keyword, semantic similarity with an expected answer, or an output of a trained machine learning model.

[0021] Furthermore, the present invention solves the above problem by providing a program that is executed in a dialogue between a user and a large language model, which causes a computer to execute: a stage determination instruction group that determines the stage of the dialogue based on the user's input to the large language model or the output from the large language model to the user; a prompt generation instruction group that generates a prompt based on the stage; and a prompt addition instruction group that adds the prompt to the user's input to the large language model.

[0022] Furthermore, the present invention solves the above problem by providing the program according to paragraph 0021, wherein the dialogue stage is any one or more of opening, needs hearing, presentation, resolving customer concerns and questions, closing, and after-sales follow-up.

[0023] Furthermore, the present invention solves the above problem by providing the program according to paragraph 0021, wherein the second stage determination instruction group includes an instruction group that determines the dialogue stage based on any one or more of matching with a predetermined keyword, semantic similarity with an expected answer, or an output of a trained machine learning model.

[0024] Furthermore, the present invention solves the above problem by providing a computer system configured to execute the method according to any one of paragraphs 0008 to 0015. Effects of the Invention

[0025] A chatbot that can more deeply reflect the state of dialogue with a user and realize more natural dialogue is provided. [BRIEF DESCRIPTION OF THE DRAWINGS]

[0026] [Figure 1] It is a component of one embodiment of a system that executes the chatbot according to the present invention. [Figure 2] It is an example of a dialogue showing the effect of prompt engineering in an LLM. [Figure 3] It is one embodiment of a processing flow of a prompt generation unit according to the present invention. [Figure 4] It is one embodiment of a dialogue with a user of the chatbot according to the present invention. [MODES FOR CARRYING OUT THE INVENTION]

[0027] Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 shows one embodiment of a functional configuration of a system that executes the chatbot according to the present invention.

[0028] A user (101) is a natural person who conducts a dialogue with the chatbot according to the present invention. The dialogue may be performed via character input means such as a keyboard, or may be performed via direct voice input using a microphone. Alternatively, the dialogue may be performed, for example, by image recognition of sign language. Output of information from the system may be performed by character display on a screen, or may be performed by output of synthetic speech. Alternatively, the output may be performed via a braille display, for example. When outputting information, an avatar may be displayed on the screen, and the dialogue may be conducted in an image where the avatar speaks. A (physical) robot may be operated as part of the dialogue output. In this case, the movements and facial expressions of the avatar or the robot may be changed in accordance with the context of the dialogue.

[0029] The terminal device (102) is a means for the user (101) to input and output information to the chatbot according to the present invention, and may be any computer device such as a personal computer, smartphone, tablet terminal, smartwatch, smart glasses, HMD (head-mounted display), kiosk terminal, car navigation equipment, car automation equipment, robot, etc. The terminal device (102) may be equipped with a microphone. Furthermore, it is preferable that the terminal device (102) is equipped with a camera (103) for acquiring images of the user's (101) facial expressions, but the camera (103) may be provided as a device independent of the terminal device (102).

[0030] The chat interface provider (104) is a group of programs that provide an interactive user interface using a chatbot. For example, it may be based on ChatGPT provided by OpenAI, but it may also be a group of programs custom-developed by an organization for its own use. For example, it may be a group of programs developed using LangChain, an LLM integration library for the Python language. The chat interface provider (104) runs on a server (typically a server managed as a cloud), but some or all of its functions may run on a terminal device (102) or a locally managed computer device.

[0031] The dialogue generation unit (105) is a machine learning model trained in advance with a large amount of language data to enable natural responses to user (101) input, and is typically a neural network based on the Transformer architecture. The dialogue generation unit (105) may be, for example, GPT-4 provided by OpenAI, Claude provided by Anthropic, PaLM2 provided by Google, LalaMa2 provided by Meta, etc. It may also be a neural network developed and trained in-house. The dialogue generation unit (105) provides intelligent chatbot functionality in cooperation with the chat interface provider unit (104). Multimodal interaction (enabling exchange of voice and images in addition to text) may be implemented in the LLM of the dialogue generation unit (105). As an alternative configuration, a group of programs (which may be implemented by a neural network) that perform speech recognition, image recognition, and speech synthesis and image synthesis may be added as a front-end to the LLM to perform conversion between voice / image and text information.

[0032] The user response scoring unit (106) and the response scoring unit (107) are a group of programs and their execution environment that provide a function to monitor the state of a user (101) interacting with a chatbot in real time and calculate a score. These scores may include an understanding score (the degree to which the user is responding to or asking questions that demonstrate an understanding of the conversation with the chatbot), an interest score (the degree to which the user is interested in the conversation with the chatbot, or has a high level of purchasing intent or favorability towards the service), and an emotional positivity score (the degree to which the user has positive feelings towards the chatbot with which they are interacting). The average value, weighted average value, or even the normalized value thereof of these scores may be treated as the overall score.

[0033] The user response scoring unit (106) is a group of programs that calculate a score for the user's response based on the user's (101) facial expressions captured using the camera (103), and it is desirable that a CNN (Convolutional Neural Network), which is excellent for real-time image processing, is used. In addition, the user response scoring unit (106) may also calculate a score based on the voice spoken by the user (101).

[0034] The user response scoring unit (106) may calculate scores to increase the comprehension score, interest score, and emotional positivity score if positive signs such as nodding or smiling are recognized in the user's (101) facial expression, or if the tone of voice is high or stable. Conversely, these scores may be calculated to decrease if there is little change in facial expression, furrowed brow, downturned corners of the mouth, low tone of voice, wandering eyes, increased hesitant words (such as "um"), or a long delay before responding.

[0035] The response scoring unit (107) is a group of programs that monitor and score the text or voice input from the user to the dialogue generation unit (105), the text or voice output from the dialogue generation unit (105) to the user, or both. It may be configured as a neural network (especially an LLM), but it may also be a rule-based program (script) that calculates the score.

[0036] The response scoring unit (107) may calculate the comprehension score based on whether key concepts (e.g., 2.5%, payment, 30 years) (or their synonyms) in the expected responses that the chatbot designer expects from the user are included in the user's response. The expected responses that the user will likely provide may be generated using a neural network-based generative AI. In calculating the comprehension score, scoring may be performed based on keyword matching, semantic similarity with expected responses through embedding, calculation using a machine learning model (typically a neural network) that has been pre-trained on a large amount of data, or a combination thereof (such as a weighted average).

[0037] The response scoring unit (107) may calculate the interest score in a way that increases the score when the user's (101) input includes phrases that suggest a high level of interest, such as "tell me more," "I want to know more," or "sounds interesting," and when there are many questions. Conversely, it may calculate the score in a way that decreases the score when negative expressions such as "I don't understand" are used, or when the frequency of questions is low. In the calculation, scoring may be performed based on keyword matching, semantic similarity with expected answers calculated by embedding, machine learning models (typically neural networks) that have been trained on a large amount of data in advance, or a combination of these (such as a weighted average).

[0038] The response scoring unit (107) may calculate the emotional positivity score by increasing the score when positive words such as "happy" or "fun" are included. It may also calculate the score by increasing the score when negative words such as "hate" or "complicated" are included. It may also calculate the score by setting it to the median when neutral words such as "okay" are included. In the calculation, scoring may be performed based on keyword matching, semantic similarity with expected responses calculated by embedding, machine learning models (typically neural networks) that have been trained on a large amount of data in advance, or a combination thereof (such as a weighted average).

[0039] The dialogue flow monitoring unit (108) is a program or neural network for determining the stage of the dialogue between the user (101) and the dialogue generation unit (105) (for example, whether the dialogue has just begun, whether it will continue for a while, or whether it is nearing its end). For example, in a dialogue between a salesperson and a customer in product sales, the dialogue stages may be defined as: 1. Opening, 2. Needs assessment, 3. Presentation, 4. Addressing customer concerns and questions, 5. Closing, and 6. After-sales follow-up. By explicitly understanding these stages, the dialogue flow monitoring unit (108) can enable more natural dialogue with the customer. For example, it can prevent situations where detailed product information is suddenly provided during the opening or needs assessment stage, which could lead to customer distrust.

[0040] Stage determination may be made by keyword matching (e.g., "Welcome," "How can I help you?", "What function?", "What are you looking for?", "This function is," "Recommended product"), by semantic similarity with a predetermined reference sentence through embedding into the semantic space, by a machine learning model (typically a neural network) pre-trained on a large amount of data, or by scoring based on a combination of these (e.g., weighted average). For example, if the output from the dialogue generation unit (105) includes "How can I help you?", the dialogue stage may be determined to be the opening stage. If the semantic similarity with a pre-prepared reference sentence (e.g., "I'll consider it.") exceeds a predetermined value, the dialogue stage may be determined to be the closing stage. In addition, a large-scale language model may be pre-trained, fine-tuned, or trained in context so that the current dialogue stage can be output by inputting a dialogue log.

[0041] The prompt generation unit (109) is a group of programs that generate prompts to make the interaction with the user (101) more natural, based on at least one of the scores calculated by the user response scoring unit (106), the response scoring unit (107), and the dialogue flow monitoring unit (108). In addition, prompts may be generated based on stage information determined by the dialogue flow monitoring unit (108). The prompt generation unit (109) may be implemented as a script (a program that follows a predetermined procedure), or it may be implemented by a machine learning model (typically a neural network) that has been trained in advance with training data. Details of the processing of the prompt generation unit (109) will be described later.

[0042] Note that the functional configuration shown in Figure 1 is a logical configuration and does not represent a physical configuration. In other words, it does not mean that one function is implemented by one device, and one function may be implemented by multiple devices. Here, the device that performs the processing may be any device, such as a server managed by the organization, a group of servers managed in the cloud, or a computer managed by the user (101) (including a terminal device (102)). The configuration may also not include one or more of the user response scoring unit (106), response scoring unit (107), or dialogue flow monitoring unit (108).

[0043] Figure 2 shows examples of questions and responses when using ChatGPT as a chatbot and GPT4 o1 (preview) as the LLM. Even when asking the same question, adding a phrase like "Please explain in as simple terms as possible" to the input makes the LLM's response clearer. This process of tailoring the input to the LLM to bring its response closer to the desired outcome is generally called prompt engineering. By appropriately utilizing prompt engineering, it is possible to customize the output from the LLM and achieve a more natural dialogue for the user without retraining the LLM or adjusting its parameters.

[0044] Figure 3 shows an example of the processing flow of the program constituting the prompt generation unit (109). This example shows the case where the prompt generation unit (109) is implemented as a script (a program that follows a predetermined procedure). However, if the prompt generation unit (109) is implemented by a machine learning model (typically a neural network), the neural network may be trained to obtain results equivalent to those obtained by the script. Reinforcement learning may be performed with humans as training data, or reinforcement learning may be performed using the processing results from the script as training data.

[0045] When a user (101) provides input to the dialogue generation unit (105) via the chat interface provider unit (104), the response scoring unit (107) and the dialogue flow monitoring unit (108) also read the input and the output from the dialogue generation unit (105), calculate the score, and pass the score information to the prompt generation unit (109). At the same time, the user response scoring unit (106) scores the user (101) based on their facial expressions and inputs this score to the prompt generation unit (109).

[0046] Figure 3 shows an example flowchart of the prompt generation unit (109) (and the program that constitutes it). The prompt generation unit (109) obtains a score from the user response scoring unit (106) (S301). Next, the prompt generation unit (109) obtains a score from the response scoring unit (107) (S302). Next, the prompt generation unit (109) obtains information on the stage of the dialogue from the dialogue flow monitoring unit (108) (S303). The input of score and stage information may be performed periodically at regular intervals, but the prompt generation unit (109) may also query the user response scoring unit (106), the response scoring unit (107), and the dialogue flow monitoring unit (108) at the time of input from the user (101) to the chat interface provider unit (104). The score may not be the value at the time of the query, but may be a value that reflects past conditions, such as the average value of scores from a predetermined time (for example, 1 minute) before the query.

[0047] It is not necessary to execute all of steps S301, S302, and S303; a configuration in which one or more of them are executed is acceptable. Next, the prompt generation unit (109) generates or selects a prompt (S304) according to the acquired score and stage information. For example, if the comprehension value falls below a threshold, the logic may add the prompt "Please explain in simpler terms" to the user's (101) input. If the stage is determined to be "closing," the logic may add the prompt "Please show me the order screen." It is preferable that the administrator can customize what prompts are generated under what conditions. For example, a configuration may be adopted in which a correspondence table between score value conditions and the prompts to be added is stored on persistent storage. Prompt generation may also be performed by a trained machine learning model. In this case, the machine learning model may be reinforced using human judgment or the results of script processing as training data.

[0048] Next, the prompt generation unit (109) instructs the chat interface provider unit (104) to add the prompt to the user's (101) input and make it the input for the dialogue generation unit (105) (S304). This process may be performed, for example, via a library such as LangChain. Instead of adding a prompt, or in addition to adding a prompt, the system may also adjust the parameters of the dialogue generation unit (105) (for example, a parameter called temperature that adjusts the randomness of response generation) via the API of the dialogue generation unit (105). It is preferable that the chat interface provider unit (104) is configured such that the added prompt is not displayed to the user (101) (or the user can choose to display or hide it). This is to avoid giving the user a sense of distrust that their conversation is being manipulated in the background.

[0049] In addition to the above, if the score meets a predetermined condition (for example, if the comprehension score and the emotional positivity score exceed a predetermined value), the prompt generation unit (109) may generate a prompt to encourage the end of the dialogue between the user (101) and the dialogue generation unit (105) (for example, "Please proceed to closing the dialogue."). In this case, it is desirable to also use conditions other than the score, such as a predetermined amount of time having elapsed since the start of the dialogue, or the number of dialogue exchanges exceeding a predetermined number.

[0050] Figure 4 shows an example of a conversation between a user (101) and a chatbot (dialogue generation unit (105)) when the method according to the present invention is applied to mobile phone sales operations, compared with an example using the prior art. Figure 4-a is an example when the method according to the present invention is not used. Although the conversation is coherent, it cannot be said that an effective sales response was provided. Figure 4-b is an example when the method according to the present invention is used, and a conversation that is appropriate to the user's needs and the stage of the conversation is realized. The response scoring unit (107) calculates a score in response to the input from the user (101) or the output from the chatbot (dialogue generation unit (105)), and the dialogue flow monitoring unit (108) determines the stage of the conversation. The prompt generation unit (105) generates a predetermined prompt (or one generated by a machine learning model) based on the score or stage information and adds it to the user's (101) input to the dialogue generation unit (105) (the added prompt is not visible to the user (101)). This enables natural dialogue without causing the user (101) any unnecessary awareness. This example is merely to illustrate the concept, and the effectiveness of the method described in this application is even greater in real-world, more complex, and less predictable dialogues.

Claims

1. A method that is performed in the interaction between a user and a large-scale language model, A first score calculation step, which calculates a first score based on the user's facial expression acquired by the camera, A prompt generation step that generates a prompt based on the first score, The process includes a prompting step of adding the prompt to the user's input to the large-scale language model, A method performed by a computer.

2. The first score is a score relating to one or more of the user's level of understanding, level of interest, and level of emotional positivity. The method according to claim 1.

3. A method that is performed in the interaction between a user and a large-scale language model, A second score calculation step, which calculates a second score based on the user's input to the large-scale language model, or the output from the large-scale language model to the user; A prompt generation step that generates a prompt based on the second score, The process includes a prompting step of adding the prompt to the user's input to the large-scale language model, A method performed by a computer.

4. The second score is a score relating to one or more of the user's level of understanding, level of interest, and level of emotional positivity. The method according to claim 3.

5. The second score calculation step calculates a second score based on one or more of the following: matching with a predetermined keyword, semantic similarity with an expected answer, or the output of a trained machine learning model. The method according to claim 3.

6. A method that is performed in the interaction between a user and a large-scale language model, A stage determination step that determines the stage of the aforementioned dialogue based on the user's input to the large-scale language model, or the output from the large-scale language model to the user, A prompt generation step that generates a prompt based on the aforementioned stage, The process includes a prompting step of adding the prompt to the user's input to the large-scale language model, A method performed by a computer.

7. The aforementioned dialogue stages include one or more of the following: opening, needs assessment, presentation, addressing customer concerns and questions, closing, and after-sales follow-up. The method according to claim 6.

8. The aforementioned stage determination step determines the stage of the dialogue based on one or more of the following: matching with a predetermined keyword, semantic similarity with an expected answer, or the output of a trained machine learning model. The method according to claim 6.

9. A program that is executed in the interaction between a user and a large-scale language model, A group of first score calculation commands that calculate a first score based on the user's facial expression acquired by the camera, A set of prompt generation instructions that generate a prompt based on the first score, A set of prompt-adding commands that add the prompt to the user's input to the large-scale language model, A program that is executed by a computer.

10. The first score is a score relating to one or more of the user's level of understanding, level of interest, and level of emotional positivity. The program according to claim 9.

11. A program that is executed in the interaction between a user and a large-scale language model, A group of second score calculation instructions that calculate a second score based on the user's input to the large-scale language model or the output from the large-scale language model to the user, A set of prompt generation instructions that generate a prompt based on the second score, A set of prompt-adding commands that add the prompt to the user's input to the large-scale language model, A program that is executed by a computer.

12. The second score is a score relating to one or more of the user's level of understanding, level of interest, and level of emotional positivity. The program according to claim 11.

13. The second score calculation instruction set includes an instruction set that calculates a second score based on one or more of the following: matching with a predetermined keyword, semantic similarity with an expected answer, or the output of a trained machine learning model. The program according to claim 11.

14. A program that is executed in the interaction between a user and a large-scale language model, A set of stage determination commands that determine the stage of the aforementioned dialogue based on the user's input to the large-scale language model, or the output from the large-scale language model to the user, A set of prompt generation instructions that generate a prompt based on the aforementioned stage, A set of prompt-adding commands that add the prompt to the user's input to the large-scale language model, A program that is executed by a computer.

15. The aforementioned dialogue stages include one or more of the following: opening, needs assessment, presentation, addressing customer concerns and questions, closing, and after-sales follow-up. The program according to claim 14.

16. The aforementioned set of stage determination commands includes a set of commands that determine the stage of the dialogue based on one or more of the following: keyword matching, semantic similarity with a reference document, or the output of a trained machine learning model. The program according to claim 14.

17. A computer system configured to perform the method described in any one of claims 1 to 8.

Images