Information provision method

Abstract

To provide an information providing method, an information providing system and a program that enable improved precision in answers to questions in a chat bot.SOLUTION: An information providing system, which receives questions from a user terminal (information processing device) that the user operates, and automatically provides answers for such questions by a chat bot composed of a web server connected to a network, comprises: a question answer unit that, in accordance with a question sentence input from the user, outputs a question candidate sentence corresponding to the question sentence to the user; a detection unit that detects a behavior of the user with respect to the question candidate sentence; and an evaluation unit that performs an evaluation of the question candidate sentence with respect to the question sentence in accordance with the behavior thereof.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to an information providing method, an information providing system, and a program for providing information in response to a question from a user. [Background technology]

[0002] A system known as a chatbot uses a web server on the Internet or a computer installed in a store to automatically answer questions entered by a user. For example, a chatbot stores Q&A data consisting of combinations of candidate question sentences and answer sentences in advance, analyzes a question entered by a user, extracts candidate question sentences corresponding to the question, and presents one or more candidate question sentences to the user. The chatbot then asks the user to select the candidate question sentence that most closely matches the question the user wants to ask from the candidate question sentences, and displays the answer sentence associated with the selected candidate question sentence. For example, an example of a chatbot is described in Patent Document 1.

[0003] In order to improve the accuracy of responses to user questions, chatbots have been equipped with a function for obtaining feedback from the user as to whether the final answer presented is correct. For example, after displaying an answer, the chatbot requests the user to input an evaluation of the answer as "correct," "incorrect," or "unresolved." This allows the chatbot to analyze and learn from the user's evaluation of the answer, thereby improving the accuracy of responses to subsequent user questions. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-185614 Summary of the Invention [Problem to be solved by the invention]

[0005] However, as mentioned above, when a user is requested to rate the answers provided by the chatbot, the user may not be able to rate the answers. As a result, the chatbot is unable to obtain feedback from the user, which makes it difficult to improve the accuracy of responses to questions.

[0006] Therefore, the object of the present invention is to provide an information provision method that can solve the above-mentioned problem of being unable to improve the accuracy of responses to questions in a chatbot. [Means for solving the problem]

[0007] An information providing method according to one aspect of the present invention includes: In response to a question sentence input by a user, outputting a question candidate sentence corresponding to the question sentence to the user; Detecting a user's behavior with respect to the candidate question sentence; evaluating the candidate question sentence for the question sentence according to the behavior; The structure is as follows.

[0008] Furthermore, an information providing system according to one aspect of the present invention includes: a question answering unit that, in response to a question sentence input by a user, outputs to the user a question candidate sentence corresponding to the question sentence; a detection unit that detects a user's behavior in response to the question candidate sentence; an evaluation unit that evaluates the candidate question sentence for the question sentence according to the behavior; Equipped with The structure is as follows.

[0009] Furthermore, a program according to one aspect of the present invention includes: In the information processing device, a question answering unit that, in response to a question sentence input by a user, outputs to the user a question candidate sentence corresponding to the question sentence; a detection unit that detects a user's behavior in response to the question candidate sentence; an evaluation unit that evaluates the candidate question sentence for the question sentence according to the behavior; To realize The structure is as follows. [Effects of the Invention]

[0010] By configuring the present invention as described above, it is possible to improve the accuracy of responses to questions in a chatbot. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing the configuration of a chatbot according to the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of data stored in the chatbot disclosed in FIG. 1. [Figure 3] FIG. 2 is a diagram illustrating an example of data stored in the chatbot disclosed in FIG. 1. [Figure 4A] FIG. 2 is a diagram illustrating an example of data stored in the chatbot disclosed in FIG. 1. [Figure 4B] FIG. 2 is a diagram illustrating an example of data stored in the chatbot disclosed in FIG. 1. [Figure 5A] FIG. 2 is a diagram showing a state in which the chatbot disclosed in FIG. 1 and the user terminal perform a question-and-answer operation. [Figure 5B] FIG. 2 is a diagram showing a state in which the chatbot disclosed in FIG. 1 and the user terminal perform a question-and-answer operation. [Figure 5C] FIG. 2 is a diagram showing a state in which the chatbot disclosed in FIG. 1 and the user terminal perform a question-and-answer operation. [Figure 5D] FIG. 2 is a diagram showing a state in which the chatbot disclosed in FIG. 1 and the user terminal perform a question-and-answer operation. [Figure 5E] FIG. 2 is a diagram showing a state in which the chatbot disclosed in FIG. 1 and the user terminal perform a question-and-answer operation. [Figure 5F]FIG. 2 is a diagram showing a state in which the chatbot disclosed in FIG. 1 and the user terminal perform a question-and-answer operation. [Figure 5G] FIG. 2 is a diagram showing a state in which the chatbot disclosed in FIG. 1 and the user terminal perform a question-and-answer operation. [Figure 6] 2 is a flowchart showing the operation of the chatbot disclosed in FIG. 1. [Figure 7] FIG. 10 is a block diagram showing a hardware configuration of an information providing system according to a second embodiment of the present invention. [Figure 8] FIG. 10 is a block diagram showing the configuration of an information providing system according to a second embodiment of the present invention. [Figure 9] 10 is a flowchart showing the operation of the information providing system according to the third embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0012] <Embodiment 1> A first embodiment of the present invention will be described with reference to Figures 1 to 6. Figures 1 to 4 are diagrams for explaining the configuration of a chatbot, and Figures 5 and 6 are diagrams for explaining the processing operation of the chatbot.

[0013] [composition] The chatbot 10 of the present invention is configured as a web server connected to a network, and functions as an information provision system that accepts questions from a user terminal 20 (information processing device) operated by a user U and automatically provides answers to those questions. For example, the chatbot 10 may be managed by a specific company and provide answers to questions from employees (users) of that company, or may be managed by a business that provides specific products or services and automatically provide answers to questions about products or services from users who access it via a network.

[0014] However, the chatbot 10 of the present invention may be used in any situation and may provide any information. Furthermore, the chatbot 10 is not necessarily limited to being configured as an information processing system that accepts questions from a user terminal 20 via a network and provides answers. For example, the chatbot 10 may be configured as an installed terminal installed in a store or the like, and may be an information processing system configured to accept questions directly from users and provide answers via text information or voice information.

[0015] The chatbot 10 in this embodiment is composed of one or more information processing devices each equipped with a calculation device and a storage device. As shown in FIG. 1, the chatbot 10 comprises a question and answer unit 11, a detection unit 12, and an evaluation unit 13, which are constructed by the calculation device executing a program. The chatbot 10 also comprises a Q&A data storage unit 14, a correct answer rate data storage unit 15, and an evaluation data storage unit 16, which are formed in the storage device. Each component will be described in detail below.

[0016] The question and answer unit 11 first outputs a chat screen A1, a message input field A2, and a send button A3 to the display screen of the accessed user terminal 20, as shown in Fig. 5A. The chat screen A1 is a screen that displays messages exchanged between the user U and the operator P. The message input field A2 is an input field where the user U inputs a question via the user terminal 20, and by pressing the send button A3, the question is sent to the chatbot 10 and accepted by the chatbot 10.

[0017] Then, when question and answering unit 11 receives a question sentence from user terminal 20, it outputs to display the question sentence from user U in question field U1 of chat screen A1, as shown in Fig. 5B, and in response, outputs to display a candidate question sentence in response field P1 of operator P on chat screen A1. At this time, question and answering unit 11 searches and extracts candidate question sentences corresponding to the question sentence from user U from within Q&A data storage unit 14, and displays the extracted candidate question sentences in response field P1 of chat screen A1.

[0018] Here, the Q&A data storage unit 14 stores Q&A data consisting of combinations of prepared question candidate sentences and answer sentences, as shown in Fig. 2. For example, as an example of Q&A data, the Q&A data for QAID "QA1" stores a combination of a question candidate sentence "Please tell me the procedures for maternity leave" and an answer sentence "The procedures for maternity leave can be found on the following website... (URL)". Note that the Q&A data may be a combination of any content of question candidate sentences and answer sentences.

[0019] Then, question answering unit 11 extracts a candidate question sentence corresponding to the question sentence from user U from the candidate question sentences stored in Q&A data storage unit 14. For example, question answering unit 11 stores a model in which candidate question sentences (Q&A data) corresponding to question sentences have been machine-learned in advance, and by inputting the question sentence from user U into such a model, one or more candidate question sentences are output, and question answering unit 11 extracts the output candidate question sentence. Note that the extraction of candidate question sentences corresponding to the question sentence by question answering unit 11 may be performed by any method, including other known methods.

[0020] The question answering unit 11 displays a list of the question and answer sentences extracted as described above in a response column P1 for the operator P, as shown in FIG. 5B. In this example, the question answering unit 11 extracts multiple candidate question sentences (1, 2, 3), and displays all of them in the response column P1. At this time, the question answering unit 11 displays and outputs each candidate question sentence so that the user U can select it. However, the number of candidate question sentences that the question answering unit 11 displays in the response column P1 is arbitrary. Note that if no candidate question sentence is extracted for a question sentence, the question answering unit 11 can determine that the question sentence itself is inappropriate, and may therefore assign an evaluation to the question sentence itself that it is inappropriate.

[0021] Additionally, the question and answering unit 11 displays in the response field P1, along with the list of the above-mentioned question candidate sentences (1, 2, 3), an option such as "See more questions" that indicates whether the user U wishes to output other question candidate sentences. At this time, the question and answering unit 11 displays and outputs the option "See more questions" so that it can be selected by the user U. Note that the phrase "See more questions" may be replaced with other phrases as long as the user U wishes to output other question candidate sentences in addition to the displayed question candidate sentences.

[0022] Furthermore, the question answering unit 11 displays an option such as "Not among these" in the response field P1 in addition to the options for each of the above-mentioned question candidate sentences (1, 2, 3) and an option such as "View more questions." The option "Not among these" is an option for conveying that the question candidate sentence desired by the user U is not among the already displayed question candidate sentences, and the wording may be other words. Then, the question answering unit 11 displays and outputs the option "Not among these" so that the user U can select it.

[0023] The question and answer unit 11 further displays various information on the chat screen A1 to the user U in accordance with the behavior of the user U detected by the detection unit 12, which will be described later in detail.

[0024] As shown in Fig. 5 described above, the detection unit 12 detects the behavior of the user U after a question candidate sentence or the like is displayed in the response field P1 of the chat screen A1. For example, the detection unit 12 detects, as the behavior of the user U, the selection status by the user U for each option including the question candidate sentence displayed in the response field P1. As an example, the detection unit 12 detects which of the question candidate sentences (1, 2, 3) displayed in the response field P1, "View more questions," and "Not in these" is selected by the user U on the chat screen A1 displayed on the display screen of the user terminal 20 by operating the user terminal 20.

[0025] Furthermore, the detection unit 12 detects, as a behavior of the user U, that after the question candidate sentence or the like is displayed in the response field P1, the user U further inputs another question sentence in the message input field A2 and presses the send button A3. Note that the other question sentence input at this time is accepted by the question answering unit 11 described above, and similarly as described above, a question candidate sentence or the like corresponding to the other question sentence is output to the user terminal 20. Furthermore, the detection unit 12 detects, as a behavior of the user U, an action of terminating the question, such as closing the chat screen A1 without selecting a question candidate sentence or the like in the response field P1 or inputting another question sentence.

[0026] In addition, when the user U selects one of the question candidate sentences as described below and the question answering unit 11 outputs an answer sentence corresponding to the selected question candidate sentence, the detection unit 12 also detects the behavior of the user U with respect to the answer sentence (second behavior). For example, when the answer sentence contains a link (address information) to another web page where a more detailed answer is posted, the detection unit 12 detects whether the link was selected by the user U as the behavior of the user U. In addition, when the user U inputs an evaluation of the answer following the answer sentence, the detection unit 12 detects the degree of evaluation. For example, when a button indicating that the answer was useful to the user U and a button indicating that the answer was not useful are displayed, the detection unit 12 detects which button was selected.

[0027] Now, let us return to the explanation of the question and answering unit 11. After displaying candidate question sentences in response to a question from the user U as described above, the question and answering unit 11 takes various measures, such as providing information to the user U, in accordance with the behavior of the user U detected by the detection unit 12. In the following explanation, the behavior of the user U will be assigned a "pattern number."

[0028] First, if the user U ends the question by closing the chat screen A1 without making any selection from the candidate question sentences displayed in the response field P1 shown in FIG. 5B (pattern 1), the question answering unit 11 ends the answering process without continuing to answer the question.

[0029] Furthermore, when the user U selects "View more questions" for a question candidate sentence or the like displayed in the response column P1 shown in FIG. 5B (pattern 2), the question answering unit 11 further extracts other question candidate sentences corresponding to the initial question sentence from the Q&A data storage unit 14, as described above. Then, as shown in FIG. 5C, the question answering unit 11 displays "View more questions" in the question column U1 of the user U on the chat screen A1, and also displays the further extracted question candidate sentences (4, 5, 6) and other options in the response column P1 of the operator P. Note that, when the user U selects "Not in these" for a question candidate sentence or the like displayed in the response column P1 shown in FIG. 5B (pattern 6), the question answering unit 11 either further extracts other question candidate sentences corresponding to the initial question sentence as described above, or terminates the answering process without continuing to answer the question.

[0030] Furthermore, when the user U selects one question candidate sentence (e.g., question candidate 1) from the question candidate sentences, etc., displayed in the response field P1 shown in FIG. 5B, the question answering unit 11 identifies the Q&A data including the selected question candidate sentence from the Q&A data storage unit 14, and reads out the answer sentence associated with the selected question candidate sentence from the identified Q&A data. Then, as shown in FIG. 5D, the question answering unit 11 displays "question candidate 1" in the question field U1 of the user U on the chat screen A1, and displays an "answer sentence" corresponding to the selected question candidate sentence in the response field P1 of the operator P. Then, when the user U subsequently ends the question by, for example, closing the chat screen A1 (pattern 3), the question answering unit 11 determines that the answer to the question has been completed and ends the answering process.

[0031] At this time, if the answer text includes a link (address information (e.g., URL)) to another web page where a more detailed answer is posted, the question and answering unit 11 also displays the link to the other web page in the response field P1 of the chat screen A1, as shown in FIG. 5D. Then, when the user U selects a link included in the answer text, the question and answering unit 11 displays and provides the linked web page on the user terminal 20 (pattern 8). Furthermore, as shown in FIG. 5E, the question and answering unit 11 displays an input means for inputting the user's level of evaluation of the answer following the answer text. As such input means, for example, a button indicating that the answer was helpful to the user U and a button indicating that the answer was not helpful are displayed. Then, when either button is selected by the user U (pattern 9), the question and answering unit 11 ends the response to the user's question. Note that the input means for inputting the user's level of evaluation of the answer is not limited to the above-mentioned two buttons, and may display an input means for inputting an evaluation on three or more levels.

[0032] 5D, if an answer sentence corresponding to a question candidate sentence is already displayed, but the user U selects another question candidate sentence (for example, question candidate 2) for the question candidate sentence displayed in the response column P1 (pattern 4), the question answering unit 11 displays the selected other question candidate sentence in the question column U1 of the user U, as shown in FIG. 5F. Then, although not shown, the question answering unit 11 displays an answer sentence corresponding to the other question candidate sentence in the response column P1 in the same manner as described above.

[0033] 5G, when an answer sentence corresponding to the question candidate sentence is already displayed, if the user U inputs a new question sentence in the message input field A2 and presses the send button A3 (pattern 5), the question and answering unit 11 accepts the new question sentence and displays it in the question field U1 of the user U on the chat screen A1 in the same manner as described above. Also, when the user U does not select the question candidate sentence displayed in the response field P1 and inputs a new question sentence in the message input field A2 (pattern 7), as shown in FIG. 5B, the question and answering unit 11 accepts the new question sentence in the same manner as described above and displays it in the question field U1 of the user U on the chat screen A1 in the same manner as described above. Then, the question and answering unit 11 extracts a new question candidate sentence corresponding to the new question sentence and displays it in the response field P1 in the same manner as described above.

[0034] The evaluation unit 13 evaluates the candidate question sentences for the question sentence based on the behavior of the user U detected by the detection unit 12 as described above. In this embodiment, a correct answer rate is set for each pattern number corresponding to the behavior of the user U described above, and the evaluation unit 13 associates the correct answer rate with evaluation data containing pairs of the actually input question sentence and the candidate question sentence extracted corresponding to the question sentence. Then, as shown in FIG. 4A , the evaluation unit 13 associates the user ID of the user U who input the question, the input question sentence, the QAID of the Q&A data including the extracted candidate question sentence, and the correct answer rate, and stores the associated data as evaluation data in the evaluation data storage unit 16. The correct answer rate is a value representing the degree to which the candidate question sentence output in response to the question sentence input by the user U is considered to be correct. It is assumed that the correct answer rate is set in advance for each pattern number corresponding to the behavior of the user U, as shown in the correct answer rate table of FIG. 3 .

[0035] Specifically, the evaluation unit 13 calculates the accuracy rate of the candidate question sentences extracted corresponding to the question sentence according to the behavior of the user U detected by the detection unit 12 as follows. First, consider the case of pattern 1, that is, the case where the user U ends the question by closing the chat screen A1 without making any selection in response to the displayed candidate question sentences, etc. In this case, it can be determined that all the candidate question sentences extracted corresponding to the question sentence are highly likely to be "incorrect answers," so the accuracy rate is set to "-0.5," and this value is calculated as the accuracy rate of all the candidate question sentences for the question sentence. Note that in pattern 2, where the user U selects "See more questions" in response to the displayed candidate question sentences, etc., and in pattern 6, where the user U selects "Not among these," it can also be determined that all the candidate question sentences extracted corresponding to the question sentence are highly likely to be "incorrect answers," so the accuracy rate is set to "-0.5," and this value is calculated as the accuracy rate.

[0036] Next, consider the case of pattern 3, that is, when user U selects one candidate question sentence (e.g., question candidate 1) from the displayed candidate question sentences, an answer sentence corresponding to that candidate question sentence is displayed, and then user U ends the question by closing chat screen A1, etc. In this case, since it can be determined that the candidate question sentence selected for the question sentence has a high probability of being the "correct answer," the correct answer rate is set to "0.5," and this value is calculated as the correct answer rate of the candidate question sentence selected for the question sentence.

[0037] Next, consider the case of pattern 4, that is, as described above, when user U selects one candidate question sentence (e.g., candidate question sentence 1) for a candidate question sentence or the like, and an answer sentence corresponding to that candidate question sentence is displayed, but then user U selects another candidate question sentence (e.g., candidate question 2). In this case, the candidate question sentence initially selected by user U (e.g., candidate question sentence 1) is likely to be an "incorrect answer," but because it has been selected once, the correct answer rate is set to "-0.4," and this value is calculated as the correct answer rate for the candidate question sentence initially selected for the question sentence.

[0038] Next, consider the case of pattern 5, that is, first, candidate question sentences etc. are displayed in response to an initial question sentence from user U, user U selects one candidate question sentence from the candidate question sentences etc., an answer sentence corresponding to the selected candidate question sentence is displayed, and then user U inputs a new question sentence. In this case, the evaluation unit 13 analyzes the similarity between the initial question sentence and the new question sentence. For example, the evaluation unit 13 determines whether the initial question sentence and the new question sentence are similar using a known method. As an example, the evaluation unit 13 performs morphological analysis on each of the initial question sentence and the new question sentence to convert them into vector numerical values, calculates the similarity between them, and determines that they are similar if the similarity is equal to or greater than a predetermined value. However, the analysis of the similarity between the initial question sentence and the new question sentence may be performed using any method. If the evaluation unit 13 determines that the initial question sentence and the new question sentence are similar, it can be determined that the candidate question sentence selected for the initial question sentence is highly likely to be an "incorrect answer," and a negative value is set as the correct answer rate, which is calculated as the correct answer rate of the candidate question sentence selected for the initial question sentence.On the other hand, if the evaluation unit 13 determines that the initial question sentence and the new question sentence are not similar, it can be determined that the candidate question sentence selected for the initial question sentence is highly likely to be a "correct answer," and a positive value is set as the correct answer rate, which is calculated as the correct answer rate of the candidate question sentence selected for the initial question sentence.

[0039] Next, consider the case of pattern 7, that is, a case where candidate question sentences etc. are displayed in response to an initial question sentence from user U, but user U does not select any of the candidate question sentences etc. and instead enters a new question sentence. In this case, the evaluation unit 13 analyzes the similarity between the initial question sentence and the new question sentence in the same manner as described above. If the evaluation unit 13 determines that the initial question sentence and the new question sentence are similar, it determines that all candidate question sentences displayed in response to the initial question sentence are highly likely to be "incorrect answers," and a negative value is set to the correct answer rate, which is calculated as the correct answer rate of the candidate question sentences displayed in response to the initial question sentence. On the other hand, if the evaluation unit 13 determines that the initial question sentence and the new question sentence are not similar, it cannot determine that all candidate question sentences displayed in response to the initial question sentence are "correct" or "incorrect," and so the correct answer rate is set to 0, and this value is calculated as the correct answer rate of the candidate question sentences displayed in response to the initial question sentence.

[0040] Next, consider the case of pattern 8, in which candidate question sentences, etc. are displayed in response to an initial question sentence from user U, user U selects one candidate question sentence for the candidate question sentence, an answer sentence corresponding to the selected candidate question sentence is displayed, and the answer sentence includes a link to another webpage containing a detailed answer. In this case, if user U selects the link, evaluation unit 13 determines that the selected candidate question sentence for the question sentence is highly likely to be the "correct answer," sets the accuracy rate to "1," and calculates this value as the accuracy rate of the candidate question sentence selected for the question sentence. On the other hand, if user U does not select the link, evaluation unit 13 determines that the selected candidate question sentence for the question sentence is highly likely to be the "incorrect answer," sets the accuracy rate to "-0.5," and calculates this value as the accuracy rate of the candidate question sentence selected for the question sentence.

[0041] Next, consider the case of pattern 9, in which candidate question sentences etc. are displayed in response to the first question sentence from user U, user U selects one candidate question sentence for the candidate question sentence etc., an answer sentence corresponding to the selected candidate question sentence is displayed, and a selection button (a button for selecting "whether the answer was helpful or not") indicating the user's level of evaluation of the answer is further displayed. In this case, if user U selects the button indicating "the answer was helpful," evaluation unit 13 determines that the selected candidate question sentence for the question sentence is highly likely to be a "correct answer," sets the correct answer rate to "1," and calculates this value as the correct answer rate of the candidate question sentence selected for the question sentence. On the other hand, if user U selects the button indicating "the answer was not helpful," evaluation unit 13 determines that the selected candidate question sentence for the question sentence is highly likely to be an "incorrect answer," sets the correct answer rate to "-1," and calculates this value as the correct answer rate of the candidate question sentence selected for the question sentence.

[0042] The evaluation unit 13 also has a function of correcting the accuracy rate associated with the question candidate sentence for the question sentence. At this time, the evaluation unit 13 calculates a similarity indicating the degree of similarity between the evaluation data and corrects the accuracy rate included in each evaluation data based on the similarity. Specifically, the evaluation unit 13 first calculates the similarity between two evaluation data based on the question sentence and the question candidate sentence included in each evaluation data. For example, the evaluation unit 13 determines the similarity of the question candidate sentence based on whether the QAIDs of the Q&A data including the question candidate sentence are the same, and calculates the similarity between the question sentences by performing morphological analysis. The evaluation unit 13 then comprehensively determines the similarity between the question candidate sentence and the question sentence. If the evaluation unit 13 determines that the two evaluation data are similar, it corrects the accuracy rate by adding the accuracy rate of the other to its own accuracy rate. For example, as shown in FIG. 4B, if it is determined that the evaluation data corresponding to the question sentences by user IDs A and E are similar to each other, the correct answer rates are added together and corrected.

[0043] The calculation of the correct answer rate by the evaluation unit 13 described above is an example, and the correct answer rate may be calculated according to the behavior of the user U using other criteria or methods, and the correct answer rate of the candidate question sentence for the question sentence may be set.

[0044] The above-mentioned evaluation data is stored in the evaluation data storage unit 16 and is later used as learning data for machine learning to generate a model to be used when extracting candidate question sentences from question sentences. The accuracy rate included in the evaluation data is used as a weight when training the model. Note that the above-mentioned evaluation data is not necessarily limited to being used as learning data for machine learning, and may be used in any situation.

[0045] [Operation] Next, the operation of the chatbot 10 described above will be explained mainly with reference to the display screen of the user terminal 20 shown in Fig. 5 and the flowchart in Fig. 6. When the chatbot 10 is accessed by the user terminal 20, it displays a chat screen A1 as shown in Fig. 5A on the user terminal 20 and accepts a question text entered in the message input field A2 (step S1 in Fig. 6).

[0046] Next, the chatbot 10 searches for and extracts candidate question sentences corresponding to the question sentence from the user U from the Q&A data storage unit 14 (step S2 in FIG. 6), and displays the question sentence from the user U in the question field U1 as shown on the chat screen A1 in FIG. 5B, and also displays a list of the extracted candidate question sentences in the response field P1 of the chat screen A1 (step S3 in FIG. 6). At this time, the chatbot 10 displays, in the response field P1, the list of candidate question sentences (candidate question sentences 1, 2, and 3), as well as options such as "See more questions" and "Not included," as shown in FIG. 5B.

[0047] Next, the chatbot 10 detects the behavior of the user U after the candidate question sentences and the like are displayed in the response field P1 of the chat screen A1 (step S4 in FIG. 6). For example, the chatbot 10 detects, as the behavior of the user U, the selection status by the user U for the candidate question sentences and each option displayed in the response field P1, the operation by the user U after the answer sentences are displayed, and the input of another question sentence by the user U into the message input field A2.

[0048] Then, the chatbot 10 outputs various displays in the question field U1 and the response field P1 of the chat screen A1 according to the behavior of the user U, as shown in Figures 5C to 5G, and calculates the correct answer rate of the candidate question sentences for the question sentence (step S5 in Figure 6). At this time, the chatbot 10 refers to a preset correct answer rate table shown in Figure 3, determines the correct answer rate of the candidate question sentences for the question sentence according to the behavior of the user U, and stores the correct answer rate as evaluation data in association with the question sentence and the candidate question sentence, as shown in Figure 4A (step S6 in Figure 6).

[0049] The chatbot 10 later corrects the correct answer rate included in the evaluation data at any timing. For example, it calculates the similarity between the evaluation data, that is, the similarity between the question sentences and the candidate question sentences included in each evaluation data, and if it determines that the evaluation data are similar, it corrects it by adding the correct answer rate of the other to its own correct answer rate.

[0050] The chatbot 10 can then use the evaluation data as learning data for generating a model to be used when extracting candidate question sentences from question sentences. At this time, the chatbot 10 uses the correct answer rate included in the evaluation data as a weight when learning the model.

[0051] As described above, according to the present invention, the chatbot 10 outputs to the user candidate question sentences corresponding to question sentences input by the user, and evaluates the candidate question sentences for the question sentences according to the user's behavior with respect to the candidate question sentences. Therefore, the chatbot 10 can detect the behavior of the user from the time the user asks a question until the time the user receives an answer, and can obtain an evaluation of the candidate question sentence for the question sentence according to the behavior. As a result, the chatbot 10 can obtain an evaluation of the response it made to the question from the user U, and can improve the accuracy of responses to questions.

[0052] Although the chatbot 10 in the above-described embodiment is described as being configured to communicate with the user U using text, the chatbot of the present invention may also communicate with the user via voice. That is, the chatbot may receive a question text from the user via voice, output candidate question texts to the user, and detect the user's behavior.

[0053] <Embodiment 2> Next, a second embodiment of the present invention will be described with reference to Fig. 7 to Fig. 9. Fig. 7 to Fig. 9 are block diagrams showing the configuration of an information provision system in the second embodiment, and Fig. 7 to Fig. 8 are flowcharts showing the operation of the information provision system. Note that this embodiment shows an outline of the configuration of the chatbot and information provision method described in the first embodiment.

[0054] First, the hardware configuration of the information providing system 100 in this embodiment will be described with reference to Fig. 7. The information providing system 100 is configured with a general information processing device, and is equipped with the following hardware configuration, as an example. ·CPU(Central Processing Unit)101(Arithmetic unit) ROM (Read Only Memory) 102 (storage device) RAM (Random Access Memory) 103 (storage device) ·Programs 104 loaded into RAM 303 A storage device 105 for storing a program group 304 A drive device 106 that reads and writes from a storage medium 110 external to the information processing device A communication interface 107 that connects to a communication network 111 outside the information processing device Input / output interface 108 for inputting and outputting data Bus 109 connecting each component

[0055] The information providing system 100 can be equipped with a question and answer unit 121, a detection unit 122, and an evaluation unit 123 shown in Fig. 8 by having the CPU 101 acquire and execute the group of programs 104. The group of programs 104 may be stored in advance in the storage device 105 or the ROM 102, for example, and loaded into the RAM 103 by the CPU 101 as needed for execution. The group of programs 104 may be supplied to the CPU 101 via the communication network 111, or may be stored in advance in the storage medium 110, with the drive device 106 reading out the programs and supplying them to the CPU 101. However, the above-mentioned question and answer unit 121, detection unit 122, and evaluation unit 123 may be constructed using electronic circuits.

[0056] 7 shows an example of the hardware configuration of the information processing device constituting the information providing system 100, and the hardware configuration is not limited to the above-described case. For example, the information processing device may be configured with only a part of the above-described configuration, such as excluding the drive device 106.

[0057] The information providing system 100 then executes the information providing method shown in the flowchart of FIG. 9 using the functions of the question and answering unit 121, the detection unit 122, and the evaluation unit 123, which are constructed by the program as described above.

[0058] As shown in FIG. 9, the information providing system 100 includes: In response to a question sentence input by a user, a candidate question sentence corresponding to the question sentence is output to the user (step S101); Detecting user behavior in response to the question candidate sentence (step S102); The question candidate sentence is evaluated for the question sentence according to the behavior (step S103).

[0059] In this embodiment, as configured above, the information provision system 100 outputs, to the user, candidate question sentences corresponding to question sentences input by the user, and evaluates the candidate question sentences based on the user's behavior with respect to the candidate question sentences. Therefore, the information provision system 100 can detect the user's behavior from the time the user asks a question to the time the user receives an answer, and can obtain an evaluation of the candidate question sentences based on the behavior. As a result, the information provision system 100 can obtain an evaluation of the responses made to questions from the user U, thereby improving the accuracy of responses to questions.

[0060] <Additional Notes> A part or all of the above-described embodiments can be described as follows: The following provides an overview of the configurations of the information providing method, information providing system, and program of the present invention. However, the present invention is not limited to the following configurations.

[0061] (Appendix 1) In response to a question sentence input by a user, outputting a question candidate sentence corresponding to the question sentence to the user; Detecting a user's behavior with respect to the candidate question sentence; evaluating the candidate question sentence for the question sentence according to the behavior; Information provision method.

[0062] (Appendix 2) 10. The information providing method according to claim 1, the question and answer unit outputs the question candidate sentence to a display screen of an information processing device operated by a user; the detection unit detects, as the behavior, an operation by a user on a display screen displaying the question candidate sentence; the evaluation unit evaluates the question candidate sentence for the question sentence in response to the operation. Information provision method.

[0063] (Appendix 3) 1. The information providing method according to claim 1 or 2, the detection unit detects, as the behavior, a selection status by the user of the question candidate sentence output to the user; the evaluation unit evaluates the question candidate sentence according to a selection status of the question candidate sentence by a user; Information provision method.

[0064] (Appendix 4) 1. The information providing method according to claim 3, the question and answering unit outputs the question candidate sentence to the user, and also outputs to the user an option for selecting other question candidate sentences to be output; the detection unit detects, as the behavior, a selection status by a user for the question candidate sentence and the option; the evaluation unit evaluates the question candidate sentence with respect to the question sentence in accordance with a selection status by a user for the question candidate sentence and the options. Information provision method.

[0065] (Appendix 5) 10. The information providing method according to claim 1 or 4, the detection unit detects that another question sentence has been input by the user after the question candidate sentence has been output to the user; the evaluation unit evaluates the candidate question sentence with respect to the question sentence based on the question sentence and the other question sentences; Information provision method.

[0066] (Appendix 6) 10. The information providing method according to claim 5, the evaluation unit analyzes a similarity between the question sentence and the other question sentences, and evaluates the candidate question sentences with respect to the question sentence based on an analysis result. Information provision method.

[0067] (Appendix 7) 7. The information providing method according to any one of Supplementary Notes 1 to 6, the question answering unit outputs an answer sentence corresponding to the question candidate sentence to the user in accordance with the user's behavior with respect to the question candidate sentence; the detection unit detects a second behavior of the user in response to the answer sentence; the evaluation unit evaluates the candidate question sentence for the question sentence in accordance with the second behavior. Information provision method.

[0068] (Appendix 8) 8. The information providing method according to any one of Supplementary Notes 1 to 7, the evaluation unit calculates a correct answer rate representing a degree to which the question candidate sentence output in response to the question sentence is deemed correct in accordance with the behavior as an evaluation of the question candidate sentence with respect to the question sentence, and stores the correct answer rate in association with data paired with the question sentence and the question candidate sentence. Information provision method.

[0069] (Appendix 9) 10. The information providing method according to claim 8, comprising: the evaluation unit calculates a similarity indicating a degree of similarity between the data paired with the question sentence and the question candidate sentence, and modifies the correct answer rate associated with each of the data in accordance with the similarity. Information provision method.

[0070] (Appendix 10) a question answering unit that, in response to a question sentence input by a user, outputs to the user a question candidate sentence corresponding to the question sentence; a detection unit that detects a user's behavior in response to the question candidate sentence; an evaluation unit that evaluates the candidate question sentence for the question sentence according to the behavior; An information provision system equipped with:

[0071] (Appendix 11) 11. The information provision system according to claim 10, the question and answer unit outputs the question candidate sentence to a display screen of an information processing device operated by a user; the detection unit detects, as the behavior, an operation by a user on a display screen displaying the question candidate sentence; the evaluation unit evaluates the question candidate sentence for the question sentence in response to the operation. Information provision system.

[0072] (Appendix 12) 12. The information providing system according to claim 10 or 11, the detection unit detects, as the behavior, a selection status by the user of the question candidate sentence output to the user; the evaluation unit evaluates the question candidate sentence according to a selection status of the question candidate sentence by a user; Information provision system.

[0073] (Appendix 13) 13. The information provision system according to claim 12, the question and answering unit outputs the question candidate sentence to the user, and also outputs to the user an option for selecting other question candidate sentences to be output; the detection unit detects, as the behavior, a selection status by a user for the question candidate sentence and the option; the evaluation unit evaluates the question candidate sentence with respect to the question sentence in accordance with a selection status by a user for the question candidate sentence and the options. Information provision system.

[0074] (Appendix 14) 14. The information providing system according to claim 10, the detection unit detects that another question sentence has been input by the user after the question candidate sentence has been output to the user; the evaluation unit evaluates the candidate question sentence with respect to the question sentence based on the question sentence and the other question sentences; Information provision system.

[0075] (Appendix 15) 15. The information provision system according to claim 14, the evaluation unit analyzes a similarity between the question sentence and the other question sentences, and evaluates the candidate question sentences with respect to the question sentence based on an analysis result. Information provision system.

[0076] (Appendix 16) 16. The information providing system according to any one of Supplementary Notes 10 to 15, the question answering unit outputs an answer sentence corresponding to the question candidate sentence to the user in accordance with the user's behavior with respect to the question candidate sentence; the detection unit detects a second behavior of the user in response to the answer sentence; the evaluation unit evaluates the candidate question sentence for the question sentence in accordance with the second behavior. Information provision system.

[0077] (Appendix 17) 17. The information providing system according to any one of Supplementary Notes 10 to 16, the evaluation unit calculates a correct answer rate representing a degree to which the question candidate sentence output in response to the question sentence is deemed correct in accordance with the behavior as an evaluation of the question candidate sentence with respect to the question sentence, and stores the correct answer rate in association with data paired with the question sentence and the question candidate sentence. Information provision system.

[0078] (Appendix 18) 18. The information provision system according to claim 17, the evaluation unit calculates a similarity indicating a degree of similarity between the data paired with the question sentence and the question candidate sentence, and modifies the correct answer rate associated with each of the data in accordance with the similarity. Information provision system.

[0079] (Appendix 19) In the information processing device, a question answering unit that, in response to a question sentence input by a user, outputs to the user a question candidate sentence corresponding to the question sentence; a detection unit that detects a user's behavior in response to the question candidate sentence; an evaluation unit that evaluates the candidate question sentence for the question sentence according to the behavior; A program to achieve this.

[0080] The above-described program can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can supply the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0081] Although the present invention has been described above with reference to the above-mentioned embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Explanation of symbols]

[0082] 10. Chatbots 11 Question and answer section 12 Detector 13 Evaluation Section 14 Q&A Data Storage Section 15 Correct answer rate data storage section 16 Evaluation data storage unit 20 User terminal 100 Information Provision System 101 CPU 102 ROM 103 RAM 104 Programs 105 Storage device 106 Drive device 107 Communication Interface 108 Input / Output Interface 109 Bus 110 Storage medium 111 Communication Network 121 Question and answer section 122 Detector 123 Evaluation Department

Claims

1. The information processing device outputting a plurality of candidate question sentences output by the machine learning model in response to a question sentence input by a user to a terminal of the user; receiving a request from the user to output other candidate questions; In response to the request, outputting a question candidate sentence different from the plurality of output question candidate sentences to the user's terminal using the machine-learned model; In response to the user inputting one of the output candidate question sentences, an answer corresponding to the candidate question sentence is output to the user's terminal, and at the same time, two buttons are output as input means, each indicating that the answer was helpful to the user and another indicating that the answer was not helpful, which indicate the user's evaluation of the answer; learning the machine-learned model based on the user's evaluation input to the input means; Information provision method.

2. an answering unit that outputs a plurality of candidate question sentences output by the machine learning model in response to a question sentence input by a user to the user's terminal; an evaluation unit that performs learning based on the user's evaluation, the answering unit, in response to receiving a request from the user to output another question candidate, outputs a question candidate sentence different from the plurality of question candidate sentences being output to the user's terminal using the machine-learned model, and, in response to the user inputting one of the question candidate sentences being output by the user, outputs an answer corresponding to the question candidate sentence to the user's terminal, and at the same time outputs, as input means, a button indicating that the answer was helpful to the user and a button indicating that the answer was not helpful, which indicate the user's evaluation of the answer; The evaluation unit performs learning of the machine-learned model based on the user's evaluation input to the input means. Information provision system.

3. In the information processing device, a process of outputting a plurality of candidate question sentences output by the machine learning model in response to a question sentence input by a user to a terminal of the user; receiving a request from the user to output other question candidates; In response to the request, a process of outputting a question candidate sentence different from the plurality of output question candidate sentences to the user's terminal using the machine-learned model; a process of outputting an answer corresponding to the question candidate sentence to the user's terminal in response to the user inputting one of the output question candidate sentences, and simultaneously outputting, as input means, a button indicating that the answer was helpful to the user and a button indicating that the answer was not helpful, which indicate the user's evaluation of the answer; a process of learning the machine-learned model based on the user's evaluation input to the input means; A program to make this happen.

Images