Answer support device, answer support method, and program

Abstract

Users may need extra effort to determine whether the responses generated by the AI ​​are appropriate. [Solution] The answer support device according to this disclosure includes an answer request means that transmits a user question to multiple generating AIs and receives generated answers generated by the multiple generating AIs in response to the user question, a matching rate calculation means that calculates the matching rate of the generated answers by the multiple generating AIs, an answer creation means and an output means. The answer creation means creates an output answer based on the generated answers by the multiple generating AIs when the matching rate is equal to or greater than a matching rate threshold. The answer creation means creates an output answer based on the generated answers with a confidence level equal to or greater than a confidence level threshold when there are generated answers with a confidence level equal to or greater than a confidence level threshold, and creates an output answer that requests the user to change the user question when there are no generated answers with a confidence level equal to or greater than a confidence level threshold. The output means then outputs the output answer.

Description

Technical Field

[0001] The present disclosure relates to an answer support device, an answer support method, and a program.

Background Art

[0002] When a user sends a question to a generative AI (Artificial Intelligence), the generative AI generates an answer to the question.

[0003] Patent Document 1 describes a program that obtains a judgment result on whether a question sentence is appropriate as input data for a question-answering system capable of outputting an answer sentence by inputting the question sentence received from a user as input data into a large language model.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The answer generated by the generative AI may not be appropriate. Therefore, the user may sometimes spend time determining whether the answer by the generative AI is appropriate.

[0006] One object of the present disclosure is to provide an answer support device or the like that can easily determine the appropriateness of an answer generated by a generative AI.

Means for Solving the Problems

[0007] An answer support device in one aspect of this disclosure includes: a question receiving means for receiving a user question from a user; an answer request means for transmitting the user question to a plurality of generating AIs and receiving generated answers generated by the plurality of generating AIs in response to the user question; a matching rate calculation means for calculating the matching rate of the generated answers from the plurality of generating AIs; an answer creation means for creating an output answer based on the generated answers from the plurality of generating AIs when the matching rate is equal to or greater than a matching rate threshold; an output means for outputting the output answer; and a mutual evaluation means for requesting evaluation of each of the plurality of generated answers from a generating AI other than the generating AI that generated the generated answer, and calculating a reliability based on the evaluation result when the matching rate is lower than a matching rate threshold. The answer creation means creates an output answer based on a generated answer with a reliability of equal to or greater than a reliability threshold when there is a generated answer with a reliability of equal to or greater than a reliability threshold, and creates an output answer requesting the user to change the user question when there is no generated answer with a reliability of equal to or greater than a reliability threshold.

[0008] An answer support method in one aspect of this disclosure receives a user question from a user, sends the user question to multiple generating AIs, receives generated answers from the multiple generating AIs in response to the user question, calculates the agreement rate of the generated answers from the multiple generating AIs, requests evaluation of each generated answer from the multiple generating AIs by other generating AIs if the agreement rate is lower than the agreement rate threshold, calculates the confidence level based on the evaluation results, creates an output answer based on the generated answers with a confidence level of or higher than the confidence level threshold if there are generated answers with a confidence level of or higher than the confidence level threshold, creates an output answer requesting the user to change the user question if there are no generated answers with a confidence level of or higher than the confidence level threshold, and creates an output answer based on the generated answers from the multiple generating AIs if the agreement rate is higher than the agreement rate threshold, and outputs the output answer.

[0009] A program in one aspect of this disclosure receives a user question from a user, sends the user question to multiple generating AIs, receives generated answers from the multiple generating AIs in response to the user question, calculates the agreement rate of the generated answers from the multiple generating AIs, requests evaluation of each generated answer from each of the multiple generating AIs by other generating AIs if the agreement rate is lower than the agreement rate threshold, calculates the confidence level based on the evaluation results, creates an output answer based on the generated answers with a confidence level of or higher than the confidence level threshold if there are generated answers with a confidence level of or higher than the confidence level threshold, creates an output answer requesting the user to change the user question if there are no generated answers with a confidence level of or higher than the confidence level threshold, creates an output answer based on the generated answers from the multiple generating AIs if the agreement rate is higher than the agreement rate threshold, and causes the computer to execute the process of outputting the output answer.

[0010] Each program may be stored on a non-temporary storage medium that is readable by the computer. [Effects of the Invention]

[0011] One example of the effects of this disclosure is that it becomes easier to determine the appropriateness of responses generated by the AI. [Brief explanation of the drawing]

[0012] [Figure 1] This figure shows an example of the configuration of a system including an answer support device. [Figure 2] This block diagram shows an example of the configuration of an answer support device. [Figure 3] This figure shows an example of a screen where a user enters a question. [Figure 4] This is the first explanatory diagram showing an example of interaction between the answer support device and the generation AI. [Figure 5] This is the second explanatory diagram, showing an example of interaction between the answer support device and the generation AI. [Figure 6] This is the third explanatory diagram, showing an example of interaction between the answer support device and the generation AI. [Figure 7]It is a fourth explanatory diagram showing an example of the interaction between the answer support device and the generative AI. [Figure 8] It is an explanatory diagram showing an example of a method for calculating the reliability of each generated answer. [Figure 9] It is a first diagram showing an example of a screen on which an output answer is displayed. [Figure 10] It is a second diagram showing an example of a screen on which an output answer is displayed. [Figure 11] It is a third diagram showing an example of a screen on which an output answer is displayed. [Figure 12A] It is a first flowchart showing the operation of the answer support device. [Figure 12B] It is a second flowchart showing the operation of the answer support device. [Figure 13] It is a block diagram showing an example of the configuration of the answer support device. [Figure 14] It is a first diagram showing an example of a screen on which an output answer including alternative questions is displayed. [Figure 15] It is a second diagram showing an example of a screen on which an output answer including alternative questions is displayed. [Figure 16A] It is a third flowchart showing the operation of the answer support device. [Figure 16B] It is a fourth flowchart showing the operation of the answer support device. [Figure 17] It is a diagram showing an example of the hardware configuration of the answer support device. [Embodiments for Carrying Out the Invention]

[0013] Embodiments of the present disclosure will be described in detail with reference to the drawings.

[0014] [First Embodiment] Referring to Figure 1, an example configuration of a response support system including a response support device 10 will be described. Figure 1 is a diagram showing an example configuration of a response support system including a response support device. The response support system includes a response support device 10, a user terminal 90, and a generation server 91. The response support device 10 is connected to the user terminal 90 by a wired or wireless network. The response support device 10 is connected to each generation server 91 by a wired or wireless network.

[0015] The user terminal 90 is a terminal used by the user. First, a user is a person who asks questions to the generating AI. The user asks questions to the generating AI in order to obtain answers from the generating AI. The user terminal 90 is the terminal used by the user when sending questions to the generating AI. The user can input questions into the user terminal 90. Examples of user terminals 90 include smartphones, tablet devices, and personal computers. However, the user terminal 90 is not limited to these. In the example shown in Figure 1, the answer support device 10 is connected to one user terminal 90. The number of user terminals 90 that the answer support device 10 can connect to is not limited to this. The answer support device 10 can connect to multiple user terminals 90. In this case, the answer support device 10 may connect to user terminals 90 of multiple users. Also, the answer support device 10 may connect to multiple user terminals 90 used by a single user.

[0016] The generation server 91 is a server equipped with a generation AI. The generation AI is a text generation AI capable of generating text. Examples of generation AIs include ChatGPT®, Google Gemini, and Microsoft Copilot. However, the generation AI is not limited to these; any text generation AI used for text generation is acceptable. Each generation server 91 is equipped with a different type of generation AI. The answer support device 10 connects to multiple generation servers 91. In the example shown in Figure 1, the answer support device 10 is connected to three generation servers 91. Each of the three generation servers 91 is equipped with a different generation AI. The number of generation servers 91 that the answer support device 10 connects to is not limited to this. The answer support device 10 connects to multiple generation servers 91. Here, one generation server 91 may be equipped with multiple generation AIs. In this case, the answer support device 10 may connect to only one generation server 91. In other words, the answer support device 10 only needs to be able to use multiple generation AIs by connecting to the generation servers 91.

[0017] The following explanation will describe the case where the answer support device 10 is connected to three generation servers 91. Let the generation servers 91 be referred to as generation server 91a, generation server 91b, and generation server 91c. Each generation server 91 is equipped with a different generation AI. Let the generation AIs be referred to as generation AIa, generation AIb, and generation AIc.

[0018] Referring to Figure 2, the configuration of the answer support device 10 will be described. Figure 2 is a block diagram showing an example of the configuration of the answer support device. The answer support device 10 includes a question receiving unit 101, an answer request unit 102, an agreement rate calculation unit 103, an answer creation unit 104, a peer evaluation unit 105, and an output unit 106.

[0019] The question receiving unit 101 is one form of a question receiving means for receiving user questions from a user. The user asks a question to the generating AI. At this time, the user inputs the question to, for example, the user terminal 90. In the following description, the question that the user asks to the generating AI will be called a user question. The user terminal 90 transmits the input question to the question receiving unit 101. The question receiving unit 101 can then receive the user question from the user terminal 90. The question receiving unit 101 can accept a user question by receiving the user question from the user terminal 90.

[0020] Figure 3 illustrates an example of the screen on the user terminal 90 where a user question has been entered. Figure 3 is a diagram showing an example of a screen for entering a user question. As shown in the example in Figure 3, the user can enter a user question on the user terminal 90. In Figure 3, the user question is displayed in a chat format. First, the user enters the question in the text box. Then, the user can send the question entered in the text box by pressing the "Send" button. The sent question is represented by a speech bubble on the screen of the user terminal 90. The question reception unit 101 can then receive the question sent by the user as a user question. The display method on the user terminal 90 is not limited to this. Also, the method of operation by the user using the user terminal 90 is not limited to this.

[0021] The answer request unit 102 is one embodiment of an answer request means that sends a user question to multiple generating AIs and receives generated answers generated by the multiple generating AIs in response to the user question. The answer request unit 102 sends the user question received by the question reception unit 101 to multiple generating servers 91. The answer request unit 102 can send the user question to multiple generating AIs. Each generating AI receives the user question from the answer request unit 102. Each generating AI then generates an answer to the user question. The answer to the user question generated by the generating AI is called a generated answer. Each generating AI outputs its generated answer to the answer request unit 102. The answer request unit 102 receives generated answers from multiple generating AIs.

[0022] The interaction between the answer support device 10 and each of the generating AIs will be explained using Figure 4. Figure 4 is a first explanatory diagram showing an example of interaction between the answer support device and the generating AIs. For example, the answer request unit 102 sends a user question to each of the generating AIs a, b, and c. Each of the generating AIs a, b, and c generates an answer to the user question. Each of the generating AIs outputs a generated answer to the answer request unit 102. The answer request unit 102 then receives the answers to the user question from each of the generating AIs a, b, and c. Here, the answers generated by the generating AIs a, b, and c are called generated answer a, b, and c, respectively. The answer request unit 102 can receive generated answer a, b, and c.

[0023] The matching rate calculation unit 103 is one embodiment of a matching rate calculation means for calculating the matching rate of generated responses by multiple generating AIs. The matching rate calculation unit 103 calculates the matching rate of generated responses generated by each of the multiple generating AIs received by the response request unit 102. The matching rate is the percentage of generated responses whose content is common. Matching of generated responses means that the content of the generated responses is the same; the wording of the generated responses themselves does not need to be the same.

[0024] The matching rate calculation unit 103 calculates the matching rate of the generated responses generated by multiple generating AIs. For example, if generating AI a, generating AI b, and generating AI c generate generated responses a, b, and c respectively, the matching rate calculation unit 103 calculates the matching rate of generated responses a, b, and c.

[0025] The agreement rate calculation unit 103 calculates the agreement rate using a known method for comparing distributed text representations. For example, the agreement rate calculation unit 103 calculates the agreement rate of the generated responses using the TF-IDF method, the SCDV method, or WRD. The method for calculating the agreement rate is not limited to these.

[0026] The matching rate calculation unit 103 compares the calculated matching rate with the matching rate threshold. The matching rate threshold is a threshold used to determine the level of matching rate of the generated response. The matching rate threshold can be set to any value. For example, the matching rate threshold can be set by the user. The matching rate threshold is a value used to determine the correctness of the generated response. The matching rate threshold is a value at which there is a high probability that the content of the generated response is correct when the matching rate exceeds the threshold. The response creation unit 104, which will be described next, creates an output response according to the comparison result between the calculated matching rate and the matching rate threshold.

[0027] The answer creation unit 104 is one embodiment of an answer creation means that creates an output answer based on the generated answers from multiple generation AIs when the agreement rate is equal to or greater than the agreement rate threshold. In the agreement rate calculation unit 103, the agreement rate and the agreement rate threshold are compared. If the agreement rate is equal to or greater than the agreement rate threshold, the answer creation unit 104 creates an output answer based on the generated answers from multiple generation AIs. Here, the agreement rate is the percentage of the content of the generated answers that are common, as described above. If the percentage of the content of the generated answers from all generation AIs is high, there is a high probability that the content of those generated answers is correct. Furthermore, if the percentage of the content of the generated answers from all generation AIs is high, there is a high probability that the content of the generated answer is appropriate. An example of an appropriate answer is an answer with correct content. Therefore, if the agreement rate is equal to or greater than the agreement rate threshold, the answer creation unit 104 creates an output answer using multiple generated answers.

[0028] The response creation unit 104 creates an output response using all the generated responses produced by multiple generation AIs. A high agreement rate means that a large proportion of the content of the generated responses from multiple generation AIs is common. In other words, there is a high probability that the content of the generated responses from multiple generation AIs is an appropriate response. Therefore, the response creation unit 104 creates, for example, a summary of all the generated responses as the output response. The output response is the response that is output to the user who asked the question to the generation AI. The output of the output response will be described later.

[0029] The response generation unit 104 generates output responses using known natural language generation techniques. The matching rate calculation unit 103 calculates the matching rate for each phrase, for example, using a comparison method for distributed text representations. The response generation unit 104 can then create a summary of all generated responses as an output response, using phrases with high matching rates as a reference. The method of generating output responses by the response generation unit 104 is not limited to this.

[0030] The response generation unit 104 generates an output response even if the agreement rate is lower than the agreement rate threshold. The content and method of generating the output response created by the response generation unit 104 will be described later.

[0031] The mutual evaluation unit 105 is one form of mutual evaluation means that, when the agreement rate is lower than the agreement rate threshold, requests evaluation of each of the multiple generated responses from a generating AI other than the generating AI that generated the generated response, and calculates the reliability based on the evaluation results. The agreement rate calculation unit 103 compares the agreement rate with the agreement rate threshold. The agreement rate being lower than the agreement rate threshold means that each generating AI has generated a generated response with different content. When each generating AI has generated a generated response with different content, it may not be possible to output an appropriate response to the user.

[0032] When different generating AIs produce different answers, this includes cases where some generating AIs do not produce correct answers, and cases where there may be multiple correct answers to a user's question. If a generating AI does not produce a correct answer to a user's question, that is, if the generating AI produces an incorrect answer, that answer can be said to be inappropriate. Also, if there may be multiple correct answers to a user's question, some of those answers may differ from the answer the user is looking for. For example, each generated answer may not be an incorrect answer to the user's question, but it may be a deviation from the answer the user is looking for. In this case, it may be difficult to output an appropriate answer to the user. The answer the user is looking for is not limited to an answer that the user clearly recognizes as being what they are looking for. For example, if a user wanted to know the meaning of a certain word in a specific field, but the generating AI provides a meaning in a different field, that answer can be said to be inappropriate. On the other hand, if the generating AI provides the meaning of the word in the specific field that the user wanted to know, that answer can be said to be appropriate. Thus, the answer the user is looking for is, for example, an answer that aligns with the intent of the user's question.

[0033] Therefore, the mutual evaluation unit 105 requests evaluation of each generated response from other generating AIs. The mutual evaluation unit 105 requests evaluation from other generating AIs regarding whether each generated response is correct or not. For example, the mutual evaluation unit 105 confirms with other generating AIs whether they agree with each generated response. The mutual evaluation unit 105 may also ask other generating AIs whether each generated response is correct. The specific content of the questions asked to other generating AIs is not limited to these. Here, "other generating AIs" refers to generating AIs different from the generating AI that generated the generated response being evaluated.

[0034] The mutual evaluation unit 105 requests evaluation from other generating AIs for each of the generated responses from multiple generating AIs if the agreement rate is lower than the agreement rate threshold. The mutual evaluation unit 105 requests other generating AIs to evaluate whether each generated response is a correct response. Requesting an evaluation means, for example, sending a question to other generating AIs asking whether they agree or disagree with a generated response from one generating AI. The other generating AIs receive a question asking whether they agree or disagree with the generated response that is the subject of the mutual evaluation. The other generating AIs then output a response indicating whether they agree or disagree with the generated response that is the subject of the mutual evaluation. The mutual evaluation unit 105 then receives the responses from the other generating AIs indicating whether they agree or disagree with the generated response that is the subject of the mutual evaluation.

[0035] The mutual evaluation unit 105 may ask other generating AIs whether their respective generated responses are correct. In this case, the mutual evaluation unit 105 sends a question to the other generating AI asking whether the generated response of one generating AI is correct or not. The other generating AI receives the question asking whether the generated response to be mutually evaluated is correct or not. The other generating AI then outputs a response indicating whether the generated response to be mutually evaluated is correct or not. The mutual evaluation unit 105 then receives the response from the other generating AI indicating whether the generated response to be mutually evaluated is correct or not.

[0036] In the following explanation, the mutual evaluation unit 105 will be described as a unit that confirms whether or not it agrees with a given generated response to other generating AIs.

[0037] The mutual evaluation by the mutual evaluation unit 105 will be explained with reference to Figures 5 to 7. Figures 5 to 7 illustrate the case where generation AIa, generation AIb, and generation AIc are used. Figure 5 is a second explanatory diagram showing an example of interaction between the answer support device and the generation AI. Figure 6 is a third explanatory diagram showing an example of interaction between the answer support device and the generation AI. Figure 7 is a fourth explanatory diagram showing an example of interaction between the answer support device and the generation AI.

[0038] As shown in Figure 5 as an example, the mutual evaluation unit 105 asks the generating AIb and generating AIc whether they agree or disagree with the generated response a by generating AIa. Then, generating AIb and generating AIc output their answers to the mutual evaluation unit 105, indicating whether they agree or disagree with generated response a. The mutual evaluation unit 105 can receive the answers from generating AIb and generating AIc indicating whether they agree or disagree with generated response a.

[0039] As shown in Figure 6 as an example, the mutual evaluation unit 105 similarly asks the generating AI c and generating AI a question about the generated response b generated by generating AI b, asking whether they agree or disagree. Then, generating AI c and generating AI a output their answers to the mutual evaluation unit 105, indicating whether they agree or disagree with generated response b. The mutual evaluation unit 105 can receive the answers from generating AI c and generating AI a indicating whether they agree or disagree with generated response b.

[0040] As shown in Figure 7 as an example, the mutual evaluation unit 105 similarly asks the generating AI a and generating AI b whether they agree or disagree with the generated response c generated by the generating AI c. Then, the generating AI a and generating AI b output their answers to the mutual evaluation unit 105, indicating whether they agree or disagree with generated response c. The mutual evaluation unit 105 can receive the answers from the generating AI a and generating AI b indicating whether they agree or disagree with generated response c.

[0041] In this way, the mutual evaluation unit 105 asks a different generating AI than the one that generated the generated response whether it agrees or disagrees with all the generated responses received by the response request unit 102. A single generating AI receives a question from another generating AI asking whether it agrees or disagrees with the generated response generated by that AI.

[0042] The mutual evaluation unit 105 calculates the confidence level based on the evaluation results. The confidence level is calculated for each generated response. The confidence level is the percentage of other generating AIs that evaluated each generated response as correct. In other words, the confidence level of each generated response is the percentage of other generating AIs that evaluated that generated response as correct. For example, the confidence level is the percentage of generating AIs other than the generating AI that generated that response that agreed with each generated response. Alternatively, the confidence level may be the percentage of generating AIs other than the generating AI that generated that response that answered that it was correct. The confidence level of a given generated response is the number of generating AIs that evaluated that generated response as correct divided by the number of generating AIs that were asked to evaluate that generated response. The confidence level is expressed as a percentage, for example.

[0043] Refer to Figure 8 to explain the calculation of confidence levels. Figure 8 is an explanatory diagram showing an example of how to calculate the confidence level of each generated response. In Figure 8, the example is given when generated AIa, generated AIb, and generated AIc are used. In the example shown in Figure 8, the subjects of mutual evaluation are generated responses a, b, and c. The verification results for each generated response are shown for generated AIa, b, and c. For generated response a, both generated AIb and c agree. For generated response b, both generated AIc and agree. For generated response c, both generated AIa and agree. For combinations of generated responses and generated AIs where mutual evaluation is not performed, a "-" is indicated in the table box.

[0044] The peer evaluation unit 105 calculates the confidence level for each generated response based on the evaluation results. In the example shown in Figure 8, the confidence level of generated response a is calculated to be 100%. The confidence level of generated response b is calculated to be 50%, and the confidence level of generated response c is calculated to be 0%.

[0045] The peer evaluation unit 105 compares the calculated confidence level with the confidence threshold. The confidence threshold is a threshold used to determine whether a generated response is a correct response. As mentioned above, confidence is the percentage of other generating AIs that evaluate a generated response as correct. A high confidence level for a generated response means that, compared to a low confidence level, the generated response has been evaluated as correct by more generating AIs. A generated response that is evaluated as correct by more generating AIs is likely to be a correct response. Therefore, the peer evaluation unit 105 can determine whether each generated response is a correct response by comparing the confidence level calculated for each generated response with the confidence threshold. The confidence threshold can be set to any value. The agreement rate threshold can be set, for example, by the user.

[0046] The mutual evaluation unit 105 allows the generating AIs to mutually evaluate each generated response by having other generating AIs evaluate each other's generated responses. For example, a generated response whose calculated confidence level is above the confidence threshold may have a high probability of being the correct response. On the other hand, a generated response whose calculated confidence level is below the confidence threshold may have a low probability of being the correct response. In this way, the mutual evaluation unit 105 makes it possible to mutually evaluate whether each generated response is the correct response.

[0047] The response generation unit 104 described above creates output responses according to the confidence level of each generated response. Let's explain again the output responses created by the response generation unit 104.

[0048] First, we will explain the case where, among the multiple generated responses, there is a generated response whose confidence level is above the confidence threshold. As shown in Figure 8 as an example, if, among the multiple generated responses, there is a generated response whose confidence level is above the confidence threshold, the response creation unit 104 creates an output response based on the generated response with the higher confidence level above the confidence threshold.

[0049] As mentioned above, generated responses with a calculated confidence level above the confidence threshold are likely to be correct. Therefore, the response creation unit 104 creates, for example, a summary of high-confidence generated responses with a confidence level above the confidence threshold as the output response. On the other hand, generated responses with a calculated confidence level below the confidence threshold are unlikely to be correct. Therefore, generated responses with a calculated confidence level below the confidence threshold are not used to create the output response.

[0050] The response generation unit 104 generates an output response using a known natural language generation technique. The response generation unit 104 may also request the generation AI with the highest confidence level to generate the output response. For example, the response generation unit 104 may request the generation AI that generated the highest confidence level to create a summary of the generated responses whose confidence level is above a confidence threshold.

[0051] For example, in the example in Figure 8, the confidence level of the generated response by the generating AIa is the highest. Therefore, the response creation unit 104 may request the generating AIa to create a summary of the generated responses whose confidence level is above the confidence threshold. At this time, the response creation unit 104 sends instructions to the generating AIa to create a generated response whose confidence level is above the confidence threshold, and a summary of the generated responses whose confidence level is above the confidence threshold. The generating AIa generates a summary based on the generated responses received from the response creation unit 104. Then, the generating AIa outputs the generated summary to the response creation unit 104. The response creation unit 104 can receive the summary from the generating AIa.

[0052] Furthermore, if there are multiple generated answers with a confidence level equal to or higher than the confidence threshold, the answer generation unit 104 may generate an output answer indicating that there may be multiple answers to the user question.

[0053] As mentioned above, a lower agreement rate than the agreement rate threshold occurs when each generating AI produces different generated answers. This includes cases where some generating AIs produce incorrect answers, and cases where there are multiple correct answers to a user question. If there are multiple generated answers with a confidence level above the confidence threshold, all of those generated answers may be correct. In other words, a user question may have multiple correct answers.

[0054] Therefore, if there are multiple generated answers with a confidence level equal to or greater than the confidence threshold, the answer generation unit 104 may create an output answer that includes, for example, an expression indicating the possibility that there are multiple answers to the user question. For example, the answer generation unit 104 may create an output answer that includes the sentence, "This is a question for which multiple answers are possible," as an expression indicating the possibility that there are multiple answers to the user question. This is not an example of an expression indicating the possibility that there are multiple answers to the user question. In this case, the output answer includes an expression indicating the possibility that there are multiple answers to the user question, and a summary of the generated answers with a confidence level equal to or greater than the confidence threshold.

[0055] Next, we will explain the case where there are no generated answers with a confidence level above the confidence threshold. When there are no generated answers with a confidence level above the confidence threshold, the answer creation unit 104 creates an output answer requesting the user to change the user question. If there are no generated answers with a confidence level above the confidence threshold, it is possible that none of the generating AIs have generated the correct answer. One reason why the generating AIs may not be generating the correct answer is that the user question is inappropriate. Therefore, the answer creation unit 104 can request the user to change the user question in the output answer. For example, the user may be able to obtain the correct answer from the generating AI by changing the user question to an appropriate one.

[0056] The answer generation unit 104 creates an output response that includes a request to the user to change the user question. An example of an output response that requests the user to change the user question is an output response that includes the phrase, "Please ask the question again." The output response may also include the phrase, "Please revise the question." The answer generation unit 104 may also create an output response that indicates that it was not possible to obtain a correct answer from the generating AI. For example, the answer generation unit 104 may create an output response that includes the phrase, "It is possible that you will not be able to obtain a correct answer."

[0057] The output unit 106 is one form of output means for outputting an output response. The output unit 106 outputs the output response created by the response creation unit 104. As described above, the output response is the response output to the user who asked the question. The output unit 106 outputs the output response to, for example, the user terminal 90. The user terminal 90 receives the output response from the output unit 106. The user terminal 90 can then display the output response on its screen. For example, if a user sends a user question in chat format, the user terminal 90 can display the output response in chat format as an answer to the user question. The display format on the user terminal 90 screen is not limited to this. Furthermore, the user terminal 90, having received the output response from the output unit 106, may output the output response as audio.

[0058] The output unit 106 outputs different output responses depending on the agreement rate of the generated responses. If the agreement rate is equal to or greater than the agreement rate threshold, the output unit 106 outputs an output response created based on the generated responses from multiple AI generators. As a specific example, the output unit 106 can output a summary of all generated responses as an output response.

[0059] Figure 9 illustrates an example of the screen display on the user terminal 90 when the matching rate is equal to or greater than the matching rate threshold. Figure 9 is the first figure showing an example of a screen displaying an output response. In the example shown in Figure 9, the user question and the output response are displayed in a chat format. When the matching rate is equal to or greater than the matching rate threshold, the output unit 106 outputs the output response created based on the generated responses from multiple generation AIs to the user terminal 90. The user terminal 90 then displays the output response created based on the generated responses from multiple generation AIs on its screen. In the example in Figure 9, "Tokyo." is displayed as the output response. The display format of the user question and output response is not limited to this. The user question and output response only need to be displayed in a way that the user can see.

[0060] Furthermore, if the agreement rate is lower than the agreement rate threshold, the output unit 106 outputs a different output response depending on the confidence level of the generated response. If there is a generated response with a confidence level of or higher than the confidence level threshold, the output unit 106 can output an output response created based on the generated response with a confidence level of or higher than the confidence level threshold. For example, the output unit 106 may output a summary of the highly confident generated response with a confidence level of or higher than the confidence level threshold as an output response.

[0061] Furthermore, if there are multiple generated answers with a confidence level above the confidence threshold, the output unit 106 may output an output response indicating that there may be multiple answers to the user question. As described above, if there are multiple generated answers with a confidence level above the confidence threshold, the answer creation unit 104 may, for example, create an output response that includes an expression indicating that there may be multiple answers to the user question. Therefore, the output unit 106 can output an output response that includes an expression indicating that there may be multiple answers to the user question.

[0062] If there are multiple generated responses with a confidence level above the confidence threshold, the output unit 106 may also output a request to the user to choose whether or not to ask the generating AI the question again. When there are multiple generated responses with a confidence level above the confidence threshold, there is a possibility that there are multiple correct answers to the user's question. In this case, for example, there may be multiple correct answers to the user's question if the user's question is inappropriate. Therefore, the user may be able to obtain an appropriate answer by improving the user's question. Therefore, the output unit 106 may output a request to the user to choose whether or not to ask the generating AI the question again. The user can then decide whether or not to ask the question again.

[0063] Using Figure 10, an example of the screen display on the user terminal 90 is explained when the agreement rate is lower than the agreement rate threshold, and there are multiple generated answers with a confidence level equal to or greater than the confidence threshold. Figure 10 is the second figure showing an example of a screen displaying output answers. In Figure 10, the user question and output answers are displayed in a chat format. When there are multiple generated answers with a confidence level equal to or greater than the confidence threshold, the output answer includes a summary of the generated answers with a confidence level equal to or greater than the confidence threshold, and an expression indicating the possibility that there are multiple answers to the user question. In Figure 10, the generated answers are listed in bullet points as a summary of the generated answers with a confidence level equal to or greater than the confidence threshold. In addition, as an output answer that includes an expression indicating the possibility that there are multiple answers to the user question, the sentence "This is a question that can have multiple answers." is displayed in a speech bubble on the screen. The specific sentences indicating the possibility that there are multiple answers to the user question are not limited to these. Furthermore, the display method on the user terminal 90 is not limited to these.

[0064] The output unit 106 outputs a screen that allows the user to select whether to ask additional questions. For example, the output unit 106 outputs an output response to the user terminal 90 that includes a request to select whether to ask additional questions. The output unit 106 then displays a screen on the user terminal 90 that allows the user to select whether to ask additional questions. If there are multiple generated responses with a confidence level above the confidence threshold, there may be multiple correct answers to the user question. In such cases, the user may be able to obtain the appropriate answer by changing the user question, for example. In other words, the user may be able to obtain the appropriate answer they are looking for from among multiple correct answers. Therefore, the answer creation unit 104 may create an output response that includes a request to select whether to ask additional questions. An example of an output response that includes a request to select whether to ask additional questions is the sentence "Do you want to ask additional questions?" shown in Figure 10.

[0065] Figure 10 shows an example of a screen on the user terminal 90 where the user can choose whether to ask an additional question. The screen displays a speech bubble with the text "Do you want to ask an additional question?" along with "Yes" and "No" buttons. If the user wants to ask an additional question, they press the "Yes" button. If the user presses the "Yes" button, the screen may switch to a screen where the user can enter a question. If the user presses the "No" button, the chat may be ended.

[0066] The output response, which includes a request to choose whether to ask additional questions, may also include a sentence requesting the user to enter additional questions if they wish to do so. For example, the output response may include the sentence, "If you wish to ask additional questions, please enter them." In this case, if the user does not enter additional questions, it may be considered that the user has chosen not to ask any additional questions.

[0067] The output unit 106 outputs an output response requesting the user to change the user question if there are no generated answers with a confidence level above the confidence threshold. If there are no generated answers with a confidence level above the confidence threshold, it is possible that not all generating AIs have generated the correct answer. For example, the user may be able to obtain the correct answer from the generating AI by changing the user question to an appropriate one. Therefore, the answer creation unit 104 can create an output response requesting the user to change the user question if there are no generated answers with a confidence level above the confidence threshold. The output unit 106 outputs the output response created by the answer creation unit 104, requesting the user to change the user question.

[0068] Using Figure 11, we will explain an example of the screen display on the user terminal 90 when the agreement rate is lower than the agreement rate threshold and there are no generated answers with a confidence level equal to or greater than the confidence level threshold. Figure 11 is the third figure showing an example of a screen where an output answer is displayed. In Figure 11, the user question and the output answer are displayed in a chat format. When there are no generated answers with a confidence level equal to or greater than the confidence level threshold, the output answer includes, for example, a request to the user to change the user question. In Figure 11, as an output answer that includes a request to the user to change the user question, the text "Please ask the question again." is displayed in a speech bubble on the screen. By seeing this screen, the user can understand that they need to change their question. The user can then, for example, enter a new user question in the text box.

[0069] Furthermore, the user terminal 90 may display a message indicating that it was not possible to obtain a correct answer from the generating AI. In Figure 11, the message "It is possible that you will not be able to obtain a correct answer." is displayed. By displaying a message requesting the user to change their question, along with another message requesting the user to change their question, the user can understand why they need to change their question. Here, the specific wording of the output answer is not limited to these. Also, the display method on the user terminal 90 is not limited to these.

[0070] Referring to Figures 12A and 12B, the operation of the answer support device 10, which includes a question receiving unit 101, an answer request unit 102, an agreement rate calculation unit 103, an answer creation unit 104, a peer evaluation unit 105, and an output unit 106, will be described. Figure 12A is a first flowchart showing an example of the operation of the answer support device. Figure 12B is a second flowchart showing an example of the operation of the answer support device.

[0071] In step S101, the question receiving unit 101 receives a user question. In step S102, the answer request unit 102 sends the user question to multiple generating AIs. In step S103, the answer request unit 102 receives the generated answers generated by the multiple generating AIs in response to the user question. In step S104, the agreement rate calculation unit 103 calculates the agreement rate of the generated answers from the multiple generating AIs. In step S105, the agreement rate calculation unit 103 compares the calculated agreement rate with the agreement rate threshold. The agreement rate calculation unit 103 determines whether the calculated agreement rate is equal to or greater than the agreement rate threshold.

[0072] If the answer in step S105 is Yes, then in step S106, the answer creation unit 104 creates an output answer based on the answers generated by multiple generation AIs. The answer in step S105 is Yes if the agreement rate is equal to or greater than the agreement rate threshold. In step S107, the output unit 106 outputs the output answer. Then, the answer support device 10 terminates its operation.

[0073] If the answer is No in step S105, in step S108, the mutual evaluation unit 105 requests evaluation of each of the generated responses by the multiple generating AIs by other generating AIs. The answer No in step S105 is when the agreement rate is lower than the agreement rate threshold. In step S109, the mutual evaluation unit 105 calculates the confidence level. Then, in step S110, the mutual evaluation unit 105 compares the calculated confidence level with the confidence level threshold. The mutual evaluation unit 105 determines whether there are any generated responses whose calculated confidence level is equal to or higher than the confidence level threshold.

[0074] If the answer is Yes in step S110, in step S111, the answer generation unit 104 creates an output answer based on the generated answer whose confidence level is equal to or greater than the confidence threshold. The answer is Yes in step S110 if there is a generated answer whose confidence level is equal to or greater than the confidence threshold. In step S112, the output unit 106 outputs the output answer. Then, the answer support device 10 terminates its operation.

[0075] If the answer in step S110 is No, then in step S113, the answer creation unit 104 creates an output response requesting the user to change the user question. The answer in step S110 is No if there are no generated responses with a confidence level equal to or higher than the confidence threshold. In step S114, the output unit 106 outputs the output response. Then, the process returns to step S101. The user who has confirmed the output response output in step S114 inputs the changed user question. Then, in step S101, the question receiving unit 101 accepts the user question that has been changed by the user.

[0076] In this embodiment, the answer support device 10 includes: an answer request unit 102 that transmits a user question to multiple generating AIs and receives generated answers generated by the multiple generating AIs in response to the user question; a matching rate calculation unit 103 that calculates the matching rate of the generated answers by the multiple generating AIs; an answer creation unit 104 that creates an output answer based on the generated answers by the multiple generating AIs when the matching rate is equal to or greater than a matching rate threshold; and a mutual evaluation unit 105 that, when the matching rate is lower than the matching rate threshold, requests evaluation of each of the multiple generated answers from a generating AI other than the generating AI that generated the generated answer, and calculates a reliability level based on the evaluation result. The answer creation unit 104 then creates an output answer based on a generated answer with a reliability level equal to or greater than the reliability threshold when there is a generated answer with a reliability level equal to or greater than the reliability threshold. The answer creation unit 104 also creates an output answer that requests the user to change the user question when there is no generated answer with a reliability level equal to or greater than the reliability threshold. However, the answers generated by the generating AIs may not be appropriate. Therefore, it may take time for the user to determine whether the answers from the generating AIs are appropriate. For example, users may need to judge for themselves whether an answer is appropriate. Alternatively, users may ask additional questions to the generating AI or to other generating AIs regarding the appropriateness of the answer. However, the configuration of the answer support device 10 makes it possible for users to easily judge the appropriateness of the answer generated by the generating AI.

[0077] The answers generated by AI generators may not always be appropriate. For example, the AI ​​generator's answer may be incorrect, meaning the answer is wrong. In this case, the user may need to judge for themselves whether the answer is correct. However, it can be difficult for the user to determine whether the AI ​​generator's answer is correct. Furthermore, the user may mistakenly believe that an incorrect answer is correct. To determine whether the answer is correct, the user may ask the AI ​​generator additional questions or ask other AI generators questions. In this way, it can be time-consuming for the user to determine whether the AI ​​generator's answer is correct.

[0078] However, in the answer support device 10, the agreement rate calculation unit 103 calculates the agreement rate and compares it with the agreement rate threshold. The relationship between the agreement rate and the agreement rate threshold makes it possible to determine the correctness of the generated answers. If the agreement rate is greater than or equal to the agreement rate threshold, there is a high probability that the content of all generated answers by the generation AI is correct. In this case, the answer creation unit 104 creates an output answer based on the generated answers by multiple generation AIs, and the output unit 106 outputs the output answer. As a result, the user can obtain an answer that is likely to be correct. In other words, the user can avoid the trouble of judging the correctness of the answer themselves or asking additional questions to the generation AI. Therefore, the user can easily judge the appropriateness of the answers generated by the generation AI.

[0079] Furthermore, even if the agreement rate is lower than the agreement rate threshold, the peer evaluation unit 105 calculates a confidence level for each generated response. The peer evaluation unit 105 then compares the confidence level with the confidence level threshold to determine whether each generated response is likely to be correct. Therefore, users can easily judge the appropriateness of the responses generated by the AI.

[0080] In the answer support device 10 of this embodiment, the answer request unit 102 transmits user questions to multiple generating AIs. Here, the learning content differs depending on the generating AI. Therefore, even if the same question is asked to multiple generating AIs, different answers may be generated. In other words, the user may be able to obtain multiple answers for a single user question. If the user is using only one generating AI, it may be difficult to obtain multiple answers for a single user question. Therefore, obtaining multiple answers can be beneficial to the user.

[0081] Incidentally, when multiple AI generators produce different answers, users may need to determine whether each answer is correct. It can be difficult for users to judge whether each generated answer is correct. However, in the answer support device 10, the mutual evaluation unit 105 requests evaluation of each generated answer from multiple AI generators by other AI generators and calculates a confidence level based on the evaluation results. In other words, even if multiple AI generators produce different answers, it is possible to determine whether a generated answer is correct by having other AI generators evaluate the generated answer. Therefore, users can easily judge the appropriateness of the answers generated by the AI ​​generators.

[0082] In this embodiment, the answer generation unit 104 creates an output answer indicating that there may be multiple answers to a user question when there are multiple generated answers with a confidence level equal to or higher than the confidence threshold. There may be multiple correct answers to a user question. In this case, multiple generating AIs may each generate answers with different content. For example, even if each generated answer is correct, it may contain answers that deviate from the answer the user is looking for. In this case as well, the generated answers can be said to be inappropriate. Therefore, the answer generation unit 104 creates an output answer indicating that there may be multiple answers to the user question, and the output unit 106 outputs the output answer, allowing the user to recognize that there may be multiple correct answers to the user question they entered. The user can then, if necessary, ask the generating AI additional questions. Thus, the likelihood of the user obtaining an appropriate answer can be improved.

[0083] In this embodiment, the answer creation unit 104 requests the generation AI with the highest reliability to create the output answer. By requesting the generation AI with the highest reliability to create the output answer, the likelihood of creating a good output answer is increased. A good output answer is, for example, an output answer that is easy for the user to understand. Examples of good output answers are not limited to these. In addition, since the answer creation unit 104 requests the generation AI installed on the generation server 91 connected to the answer support device 10 to create the output answer, it is possible to reduce the processing in the answer support device 10.

[0084] [Second Embodiment] This embodiment will be described in detail with reference to the drawings. To the extent that the description of this embodiment remains clear, any information that overlaps with the previous description will be omitted.

[0085] The configuration of the response support device 20 will be described with reference to Figure 13. Figure 13 is a block diagram showing an example of the configuration of the response support device.

[0086] The answer support device 20 in this embodiment includes a question creation unit 207, compared to the answer support device 10 in the above-described embodiment. The question receiving unit 201, answer request unit 202, agreement rate calculation unit 203, answer creation unit 204, mutual evaluation unit 205, and output unit 206 in the answer support device 20 correspond to the question receiving unit 101, answer request unit 102, agreement rate calculation unit 103, answer creation unit 104, mutual evaluation unit 105, and output unit 106, respectively.

[0087] The question creation unit 207 is one embodiment of a question creation means that, if the matching rate is lower than the matching rate threshold, creates an alternative question that has the potential to provide an appropriate answer to the user question, based on past user questions and output answers. As described above, the matching rate calculation unit 203 calculates the matching rate. The matching rate calculation unit 203 then compares the calculated matching rate with the matching rate threshold. If the matching rate is lower than the matching rate threshold, the question creation unit 207 creates an alternative question that has the potential to provide an appropriate answer to the user question.

[0088] A matching rate lower than the matching rate threshold occurs when each generating AI produces different generated answers. This could happen, for example, when some generating AIs fail to produce the correct answer, or when there are multiple correct answers to a user question. For instance, if there are multiple correct answers to a user question, some of these answers may differ from what the user is seeking. Specifically, each generated answer may not be incorrect, but they may deviate from the user's desired response. One possible reason for this is that the user question itself is inappropriate. Therefore, by submitting a new user question to multiple generating AIs, the user may be able to obtain a more appropriate answer.

[0089] The question generation unit 207 creates alternative questions as candidates for new user questions. Alternative questions are questions that have the potential to provide an appropriate answer to a user question. Questions that have the potential to provide an appropriate answer to a user question are, for example, questions whose answer content is uniquely determined. In other words, questions that have the potential to provide an appropriate answer to a user question are questions in which, when asked to multiple generative AIs, the content of the answers generated by multiple generative AIs is likely to match. For example, if there are multiple correct answers to a user question, some of those answers may be different from the answer the user is looking for. Therefore, by using questions that have the potential to provide an appropriate answer to a user question, the user may be able to obtain the answer they are looking for.

[0090] Questions that have the potential to provide an appropriate answer to a user's question may also include, for example, questions that a generative AI can generate a correct answer for. If some or all of the generative AIs are unable to generate a correct answer, the correct answer may be obtained through questions that have the potential to provide an appropriate answer to a user's question.

[0091] Examples of questions that may yield an appropriate answer to a user question are not limited to these. Alternative questions may also be questions that can clarify the intent of the user's question. For example, a user may want to know the meaning of a word in a specific field, but the generating AI may provide a meaning in a different field. In this case, the user's intent to know the meaning of the word in a specific field may not be reflected in the user question. Therefore, the question creation unit 207 may create an alternative question that can clarify the intent of the user's question. An example of an alternative question is a question that can identify the genre to which the already submitted user question relates. A genre is the field, industry, or situation to which the user question relates. Genres are not limited to these. A question that can identify a genre is a question that narrows down the genre to which the user question relates. In a user question, if the genre to which the user question relates is not narrowed down, there may be multiple correct answers to the user question. Therefore, an alternative question is a question that can narrow down the genre to which the user question relates.

[0092] This section explains how the question creation unit 207 creates alternative questions. The question creation unit 207 creates alternative questions based on past user questions and output answers. Past user questions and output answers refer to user questions previously sent to the generation AI and the output answers to those user questions. The combination of user questions previously sent to the generation AI and the output answers to those user questions is called the question history.

[0093] The question creation unit 207 creates an alternative question from the question history that is similar to the user question. The alternative question may be a user question included in the question history. The alternative question may be a modified version of a user question included in the question history. Alternatively, the alternative question may be a combination of a user question included in the question history and a user question already submitted. The question creation unit 207 may also create an alternative question using a user question included in the question history. The alternative question may be an improved version of a user question already submitted to the generation AI.

[0094] To generate questions similar to user questions from the question history, known natural language processing techniques and known methods for comparing distributed text representations are used. The method for creating alternative questions is not limited to these. The question creation unit 207 may create one alternative question or multiple alternative questions. The number of alternative questions to be generated may be set to any number by the user, for example.

[0095] Here, the question history used to create alternative questions includes question histories where the agreement rate was above the agreement rate threshold, and question histories where the confidence level was above the confidence level threshold. An agreement rate above the agreement rate threshold means, for example, when the content of the generated answers from multiple generating AIs is the same. In other words, a user question with an agreement rate above the agreement rate threshold can be considered an example of a question from which an appropriate answer can be obtained. Therefore, the question history with an agreement rate above the agreement rate threshold is used by the question creation unit 207 to create alternative questions. Furthermore, a confidence level above the confidence level threshold means that a certain generated answer is evaluated as correct by a high percentage of other generating AIs. A generated answer with a confidence level above the confidence level threshold is likely to be a correct answer. A user question with a confidence level above the confidence level threshold can be considered an example of a question from which an appropriate answer can be obtained. Therefore, the question history with an agreement rate above the agreement rate threshold is used by the question creation unit 207 to create alternative questions.

[0096] The question history is stored, for example, in a database (not shown). The database may be stored in the answer support device 20. The database may also be stored in an information processing device (not shown) different from the answer support device 20. In this case, the answer support device 20 and the information processing device are connected by a wired or wireless network. The question history is saved to the database by, for example, the answer creation unit 204.

[0097] The response generation unit 204 generates an output response that includes alternative questions if the question generation unit 207 has created alternative questions. The number of alternative questions included in the output response may be one or multiple. The number of alternative questions included in the output response may be the same as the number of alternative questions created by the question generation unit 207.

[0098] Examples of output responses, including alternative questions, will be explained by considering different cases.

[0099] If the agreement rate is lower than the agreement rate threshold, and there are multiple generated answers with a confidence level above the confidence threshold, the output response may include a statement indicating the possibility of multiple answers to the user question, a summary of the generated answers with a confidence level above the confidence threshold, and alternative questions. An example of a statement indicating the possibility of multiple answers to a user question is the phrase, "This is a question with multiple possible answers." In addition to alternative questions, the output response may also include a statement asking the user to select a question from the multiple alternative questions to send to the generation AI. For example, the output response may include a statement such as, "To obtain a more appropriate answer, please select a new question from the following questions."

[0100] If the agreement rate is lower than the agreement rate threshold and there are no generated answers with a confidence level equal to or greater than the confidence level threshold, the output response includes an expression requesting the user to select a question to send to the generation AI from among several alternative questions, and alternative questions. An output response including alternative questions is an example of an output response that requests the user to change the user question. The response creation unit 204 may also create an output response indicating that it was not able to obtain a correct answer from the generation AI.

[0101] The output unit 206 outputs the output response created by the response creation unit 204. The output unit 206 outputs the output response including alternative questions. The output unit 206 may also output a screen in which the user can select from multiple alternative questions to send to multiple generating AIs. For example, the output unit 206 outputs the output response including multiple alternative questions to the user terminal 90. Then, the output unit 206 displays a screen on the user terminal 90 in which the user can select from multiple alternative questions to send to multiple generating AIs.

[0102] The output destination of the output response from the output unit 206 is as described above. An example of the display on the user terminal 90 screen when the output response is output to the user terminal 90 by the output unit 206 will be explained below.

[0103] Using Figure 14, an example of the screen display on the user terminal 90 in a case where the agreement rate is lower than the agreement rate threshold, and there are multiple generated responses with a confidence level equal to or greater than the confidence threshold, will be explained. Figure 14 is the first figure showing an example of a screen displaying output responses including alternative questions. In Figure 14, the user question and output responses are displayed in a chat format. When there are multiple generated responses with a confidence level equal to or greater than the confidence threshold, the output responses include an expression indicating the possibility of multiple answers to the user question, a summary of the generated responses with a confidence level equal to or greater than the confidence threshold, and alternative questions. The expression indicating the possibility of multiple answers to the user question and the summary of the generated responses with a confidence level equal to or greater than the confidence threshold are the same as those described in Figure 10.

[0104] In Figure 14, after the summary of the generated answer, a message is displayed requesting the user to select a question to send to the generating AI from several alternative questions: "To obtain a more appropriate answer, please select a new question from the following questions." Three alternative questions are then displayed. In the example shown in Figure 14, the user question asks for the meaning of "buffer." The word "buffer" can have different meanings depending on the field in which it is used. Therefore, the alternative questions are displayed to ask for the meaning of "buffer" in a specific field. Each alternative question may also be displayed in a way that allows the user to select it. For example, each alternative question could be a button that the user can press. When the user presses any of the alternative question buttons, that alternative question is sent to the chat. In other words, the question reception unit 201 receives the alternative question as a user question. Then, the answer request unit 202 sends the alternative question to the generating AI.

[0105] The display of alternative questions is not limited to these examples. For instance, each alternative question may have a checkbox, allowing the user to select which alternative question to send by checking the corresponding box. Alternatively, after selecting an alternative question, the user may press a "Send" button to send the question to the chat.

[0106] Using Figure 15, an example of the screen display on the user terminal 90 when the agreement rate is lower than the agreement rate threshold and there are no generated answers with a confidence level equal to or higher than the confidence level threshold will be explained. Figure 15 is the second figure showing an example of a screen displaying output answers including alternative questions. In Figure 15, the user question and output answer are displayed in a chat format. In the example output answer shown in Figure 15, the message "It is possible that you will not receive a correct answer." is displayed to indicate that the correct answer could not be obtained from the generating AI. Then, a message requesting the user to select a question to send to the generating AI from multiple alternative questions, and the alternative questions themselves are displayed. Each alternative question may be a button. When the user presses any of the alternative question buttons, that alternative question is sent to the chat. The display format of the alternative questions is not limited to these. As mentioned above, for example, there may be a checkbox for each alternative question, and the user may select the alternative question to send by checking the checkbox. Alternatively, after selecting an alternative question, the user may press the "Send" button to send that alternative question to the chat.

[0107] Furthermore, the output unit 206 may output a screen that allows the user to edit the alternative question selected by the user. If the output unit 206 displays a screen that allows the user to select an alternative question to send to multiple generating AIs from among multiple alternative questions, the user will select an alternative question to send. At this time, the user may want to edit the alternative question. Therefore, the output unit 206 may display a screen on the user terminal 90 that allows the user to edit the selected alternative question.

[0108] Referring to Figures 16A and 16B, the operation of the answer support device 20, which comprises a question receiving unit 201, an answer request unit 202, an agreement rate calculation unit 203, an answer creation unit 204, a peer evaluation unit 205, an output unit 206, and a question creation unit 207, will be described. Figure 16A is a third flowchart showing an example of the operation of the answer support device. Figure 16B is a fourth flowchart showing an example of the operation of the answer support device.

[0109] Steps S201 to S210 are the same as steps S101 to S110 described in Figures 12A and 12B.

[0110] If the answer in step S210 is Yes, then in step S211, the question creation unit 207 creates an alternative question that has the potential to provide an appropriate answer to the user question, based on past user questions and output answers. The answer in step S210 is Yes if there is a generated answer with a confidence level equal to or greater than the confidence threshold. In step S212, the answer creation unit 204 creates an output answer based on the generated answer with a confidence level equal to or greater than the confidence threshold. In step S213, the output unit 206 outputs the output answer. In step S214, the question reception unit 201 determines whether it has accepted the selection of an alternative question to be included in the output answer. If the answer in step S214 is Yes, that is, if the question reception unit 201 has accepted the selection of an alternative question, the process returns to step S202. If the answer in step S214 is No, that is, if the selection of an alternative question has not been accepted, the answer support device 20 terminates its operation.

[0111] If the answer in step S210 is No, then in step S215, the question creation unit 207 creates an alternative question that has the potential to provide an appropriate answer to the user question, based on past user questions and output answers. If the answer in step S110 is No, then there are no generated answers with a confidence level equal to or higher than the confidence threshold. In step S216, the answer creation unit 204 creates an output answer requesting the user to change the user question. In step S217, the output unit 206 outputs the output answer. Then, in step S218, the question reception unit 201 accepts the selection of an alternative question. Then, the process returns to step S202.

[0112] The answer support device 20 in this embodiment further includes a question creation unit 207 that, when the match rate is lower than a match rate threshold, creates an alternative question that has the potential to provide an appropriate answer to the user question based on past user questions and output answers. The output unit 206 then creates an output answer that includes the alternative question. The case where the match rate is lower than a match rate threshold is when each generating AI generates a generated answer with different content. When each generating AI generates a generated answer with different content, this includes cases where some generating AIs do not generate the correct answer, and cases where there may be multiple correct answers to the user question. In this case, the user question may be inappropriate. Therefore, the user may ask additional questions. Therefore, by having the question creation unit 207 create an alternative question and the output unit 206 output an output answer that includes the alternative question, the user can be saved the trouble of thinking of additional questions themselves. For example, even if the user does not fully understand the content of the user question or is unable to articulate it well, an output answer that includes an alternative question is output, so the user can easily ask questions.

[0113] Furthermore, the question creation unit 207 creates alternative questions that have the potential to provide appropriate answers to user questions. These alternative questions are, for example, questions that can identify the genre from the user question. This improves the likelihood that the user will receive appropriate answers.

[0114] In this embodiment, the output unit 206 outputs a screen that allows the user to select an alternative question to send to multiple generating AIs from among multiple alternative questions. For example, the user can select a question from among the multiple alternative questions displayed on the user terminal 90. This eliminates the need for the user to think of additional questions themselves. Furthermore, since multiple alternative questions are output, the user can choose one that seems closest to their intention. In other words, it is possible to improve the likelihood that the user will receive an appropriate answer.

[0115] In the answer support device 20 according to this embodiment, past user questions and output answers include user questions and output answers where the agreement rate was equal to or greater than the agreement rate threshold, and user questions and output answers where the confidence level was equal to or greater than the confidence level threshold. An agreement rate equal to or greater than the agreement rate threshold means, for example, that the content of the generated answers from multiple generating AIs is the same. In other words, a user question where the agreement rate was equal to or greater than the agreement rate threshold can be said to be an example of a question from which an appropriate answer can be obtained. Furthermore, a confidence level equal to or greater than the confidence level threshold means that a certain generated answer was evaluated as correct by a high percentage of other generating AIs. A generated answer with a confidence level equal to or greater than the confidence level threshold is likely to be a correct answer. A user question with a confidence level equal to or greater than the confidence level threshold can be said to be an example of a question from which an appropriate answer can be obtained. By using this question history, the question creation unit 207 can create alternative questions that have a higher probability of obtaining an appropriate answer to the user question.

[0116] In this embodiment, the output unit 206 outputs a screen on which the selected alternative question can be edited. When the output unit 206 displays a screen on which the user can select an alternative question to send to multiple generating AIs from among multiple alternative questions, the user selects an alternative question to send. At this time, the user may want to edit the alternative question. For example, the user may come up with a better question while looking at the alternative questions. At this time, the user may want to edit the alternative question. Therefore, the output unit 206 outputs a screen on which the selected alternative question can be edited, allowing the user to edit the alternative question. By enabling editing by the user, the user can ask the generating AI a better question than the alternative question. As a result, it is possible to improve the likelihood that the user will receive an appropriate answer.

[0117] [Example Hardware Configuration] Figure 17 shows an example of the hardware configuration of the answer support device. The answer support device 30 is implemented using a computer. The answer support device 30 is an example of a computer implementation of the answer support device 10 or the answer support device 20.

[0118] The answer support device 30 includes a processor 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, a storage device 304 such as a hard disk for storing programs, an input / output interface 305 for data input and output, and a communication interface 306 for network connection. Each component is connected via a bus 307.

[0119] The processor 301 controls the entire computer by running the operating system. Examples of the processor 301 include a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). The processor 301 loads programs stored in, for example, a ROM 302 or a storage device 304. Then, the processor 301 executes each process coded in the program. The processor 301 may execute processes or instructions in the illustrated flowchart based on the program.

[0120] ROM 302 stores application programs, programs related to each embodiment, etc. RAM 303 is used as the work area of ​​processor 301.

[0121] The storage device 304 may be, for example, a semiconductor memory such as flash memory, or a hard disk drive (HDD). The storage device 304 stores, for example, OS (Operating System) programs, application programs, programs according to each embodiment, and so on.

[0122] The input / output interface 305 is connected to peripheral devices (not shown). The connection method may be a wired network or a wireless network.

[0123] The communication interface 306 is connected to a communication network (not shown), such as a LAN (Local Network) or WAN (Wide Area Network), via a wireless or wired network. The communication network may consist of multiple communication networks. This allows the computer to connect to external devices via the communication network. The answer support device 30 may have components other than those shown in Figure 17. For example, the answer support device 30 may include a drive device. For instance, the processor 301 may be mounted on a drive device or the like and read programs and data stored on a non-temporary tangible recording medium into the RAM 303.

[0124] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the configuration and details of the present disclosure are possible, as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, the configurations in each embodiment can be combined with one another, as long as they do not depart from the scope of the present disclosure.

[0125] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0126] (Note 1) A question submission method that accepts user questions from other users, A response request means that sends the user question to multiple generating AIs and receives generated answers generated by the multiple generating AIs in response to the user question, A means for calculating the agreement rate of the generated responses by multiple AI generation systems, If the agreement rate is equal to or greater than the agreement rate threshold, the response creation means creates an output response based on the responses generated by multiple AI generation systems, Output means for outputting the aforementioned output response, The system includes a mutual evaluation means that, when the agreement rate is lower than the agreement rate threshold, requests evaluation for each of the multiple generated responses from a generating AI other than the generating AI that generated the response, and calculates a reliability based on the evaluation results. The aforementioned means for generating the answer is: If there is a generated response whose confidence level is equal to or greater than the confidence threshold, the output response is created based on the generated response whose confidence level is equal to or greater than the confidence threshold. If there are no generated responses whose confidence level is equal to or greater than the confidence threshold, an output response is created that requests the user to change the user question. Answer support device.

[0127] (Note 2) The response generation means generates an output response that indicates the possibility of multiple answers to the user question when there are multiple generated responses whose confidence level is equal to or greater than the confidence threshold. The answer support device described in Appendix 1.

[0128] (Note 3) The response generation means requests the generation AI with the highest confidence level to create the output response. The answer support device described in Appendix 1 or 2.

[0129] (Note 4) If the matching rate is lower than the matching rate threshold, the system further provides a question creation means that creates an alternative question that may provide an appropriate answer to the user question based on past user questions and output answers. The response generation means generates the output response, which includes the alternative question. A response support device as described in any one of the items 1 to 3 in the appendix.

[0130] (Note 5) The output means outputs a screen in which the user can select from a plurality of alternative questions to send to a plurality of generating AIs. The answer support device described in Appendix 4.

[0131] (Note 6) The aforementioned alternative question is a question that allows the genre to be identified from the aforementioned user question. The answer support device described in Appendix 4 or 5.

[0132] (Note 7) Past user questions and output answers include user questions and output answers where the agreement rate was equal to or greater than the agreement rate threshold, and user questions and output answers where the confidence level was equal to or greater than the confidence level threshold. A response support device as described in any one of the appendices 4 to 6.

[0133] (Note 8) The output means outputs an editable screen for the selected alternative question. The answer support device described in Appendix 5.

[0134] (Note 9) The output means outputs a screen in which the user can select whether to ask additional questions. The answer support devices described in Appendix 1 to 8.

[0135] (Note 10) We accept user questions from users. The user question is sent to multiple generation AIs, and the generated answers for the user question are received from the multiple generation AIs. The agreement rate of the generated responses by multiple AI generators is calculated, If the agreement rate is lower than the agreement rate threshold, for each of the multiple generated responses, an evaluation is requested from a different generating AI among the multiple generating AIs that generated the response, and a confidence level is calculated based on the evaluation results. If there is a generated response whose confidence level is equal to or greater than the confidence threshold, an output response is created based on the generated response whose confidence level is equal to or greater than the confidence threshold. If there are no generated responses whose confidence level is equal to or greater than the confidence threshold, an output response is created that requests the user to change the user question. If the agreement rate is equal to or greater than the agreement rate threshold, the output response is created based on the generated responses from the multiple generation AIs. Output the aforementioned output response. Answer support method.

[0136] (Note 11) We accept user questions from users. The user question is sent to multiple generation AIs, and the generated answers for the user question are received from the multiple generation AIs. The agreement rate of the generated responses by multiple AI generators is calculated, If the agreement rate is lower than the agreement rate threshold, for each of the multiple generated responses, an evaluation is requested from a different generating AI among the multiple generating AIs that generated the response, and a confidence level is calculated based on the evaluation results. If there is a generated response whose confidence level is equal to or greater than the confidence threshold, an output response is created based on the generated response whose confidence level is equal to or greater than the confidence threshold. If there are no generated responses whose confidence level is equal to or greater than the confidence threshold, an output response is created that requests the user to change the user question. If the agreement rate is equal to or greater than the agreement rate threshold, the output response is created based on the generated responses from the multiple generation AIs. Output the aforementioned output response. A program that instructs a computer to perform a process.

[0137] (Note 12) We accept user questions from users. The user question is sent to multiple generation AIs, and the generated answers generated by the multiple generation AIs for the user question are received. The agreement rate of the generated responses by multiple AI generators is calculated, If the agreement rate is lower than the agreement rate threshold, for each of the multiple generated responses, an evaluation is requested from a different generating AI among the multiple generating AIs that generated the response, and a confidence level is calculated based on the evaluation results. If there is a generated response whose confidence level is equal to or greater than the confidence threshold, an output response is created based on the generated response whose confidence level is equal to or greater than the confidence threshold. If there are no generated responses whose confidence level is equal to or greater than the confidence threshold, an output response is created that requests the user to change the user question. If the agreement rate is equal to or greater than the agreement rate threshold, the output response is created based on the generated responses from the multiple generation AIs. Output the aforementioned output response. A recording medium that stores programs that cause a computer to perform a process.

[0138] Some or all of the configurations described in Appendices 2-9, which are dependent on Appendice 1 above, may also be dependent on Appendices 10-12 in the same manner as with Appendices 2-9. Not limited to Appendices 1 and 10-12, some or all of the configurations described as appendices may also be dependent on various hardware, software, various recording devices or systems for recording software, without departing from the embodiments described above. [Explanation of Symbols]

[0139] 10, 20 Answer support device 101, 201 Question Reception Department 102, 202 Response request section 103, 203 Match rate calculation section 104, 204 Answer Creation Department 105, 205 Mutual Evaluation Department 106, 206 Output section 207 Question Creation Department 208 Databases 301 Processor 302 ROM 303 RAM 304 Storage device 305 Input / Output Interface 306 Communication Interface 307 Bus 90 User terminals 91 Generation Server

Claims

1. A question submission method that accepts user questions from other users, A response request means that transmits the user question to multiple generating AIs and receives generated answers generated by the multiple generating AIs in response to the user question, A means for calculating the agreement rate of the generated responses by multiple AI generators, If the agreement rate is equal to or greater than the agreement rate threshold, the response creation means creates an output response based on the responses generated by multiple AI generators, Output means for outputting the aforementioned output response, The system includes a mutual evaluation means that, when the agreement rate is lower than the agreement rate threshold, requests an evaluation for each of the multiple generated responses from a generation AI other than the generation AI that generated the response, and calculates a reliability based on the evaluation result, The aforementioned means for generating the answer is, If there is a generated response whose confidence level is equal to or greater than the confidence threshold, the output response is created based on the generated response whose confidence level is equal to or greater than the confidence threshold. If there are no generated responses whose confidence level is equal to or greater than the confidence threshold, an output response is created that requests the user to change the user question. Answer support device.

2. The response generation means generates an output response that indicates the possibility of multiple answers to the user question when there are multiple generated responses whose confidence level is equal to or greater than the confidence threshold. The answer support device according to claim 1.

3. The response generation means requests the generation AI with the highest confidence level to generate the output response. The answer support device according to claim 1.

4. If the matching rate is lower than the matching rate threshold, the system further provides a question creation means that creates an alternative question that may provide an appropriate answer to the user question based on past user questions and output answers. The response generation means generates the output response including the alternative question. The answer support device according to any one of claims 1 to 3.

5. The output means outputs a screen in which the user can select from a plurality of alternative questions to send to a plurality of generating AIs. The answer support device according to claim 4.

6. The aforementioned alternative question is a question that allows the genre to be identified from the aforementioned user question. The answer support device according to claim 4.

7. Past user questions and output answers include user questions and output answers where the agreement rate was equal to or greater than the agreement rate threshold, and user questions and output answers where the confidence level was equal to or greater than the confidence level threshold. The answer support device according to claim 4.

8. The output means outputs an editable screen for the selected alternative question. The answer support device according to claim 5.

9. We accept user questions from users. The user question is sent to multiple generation AIs, and the generated answers for the user question are received from the multiple generation AIs. The agreement rate of the generated responses by multiple AI generators is calculated, If the agreement rate is lower than the agreement rate threshold, for each of the multiple generated responses, an evaluation is requested from a generation AI among the multiple generation AIs that is different from the generation AI that generated the generated response, and the confidence level is calculated based on the evaluation result. If there is a generated response whose confidence level is equal to or greater than the confidence threshold, an output response is created based on the generated response whose confidence level is equal to or greater than the confidence threshold. If there are no generated responses whose confidence level is equal to or greater than the confidence threshold, an output response is created that requests the user to change the user question. If the agreement rate is equal to or greater than the agreement rate threshold, the output response is created based on the generated responses from the multiple AI generators. Output the aforementioned output response. Answer support method.

10. We accept user questions from users. The user question is sent to multiple generation AIs, and the generated answers for the user question are received from the multiple generation AIs. The agreement rate of the generated responses by multiple AI generators is calculated, If the agreement rate is lower than the agreement rate threshold, for each of the multiple generated responses, an evaluation is requested from a generation AI among the multiple generation AIs that is different from the generation AI that generated the generated response, and the confidence level is calculated based on the evaluation result. If there is a generated response whose confidence level is equal to or greater than the confidence threshold, an output response is created based on the generated response whose confidence level is equal to or greater than the confidence threshold. If there are no generated responses whose confidence level is equal to or greater than the confidence threshold, an output response is created that requests the user to change the user question. If the agreement rate is equal to or greater than the agreement rate threshold, the output response is created based on the generated responses from the multiple AI generators. Output the aforementioned output response. A program that instructs a computer to perform a process.

Images