Information processing device, information processing method, and program

Abstract

To make it easier to verify whether the answer is appropriate. [Solution] An information processing apparatus comprising: an acquisition unit that acquires target information including an inquiry and an answer to the inquiry; and a processing unit that inputs the target information and a request to evaluate the answer to the inquiry to a model, acquires an evaluation result obtained by the model evaluating the answer to the inquiry, and stores evaluation information in a storage unit in which the evaluation result and the target information are associated.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Conventionally, a customer support system is known that includes a public solution database publicly available to users by a customer support entity and an internal solution database that is normally used within the customer support entity (see Patent Document 1). When the user makes an inquiry to the customer support entity, this customer support system transmits the search results of the public solution database by the user to the customer support entity together with the inquiry content, and displays the answer obtained by searching the internal solution database and processing by an agent on the customer's terminal.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Conventionally, sufficient verification has not been performed on whether the answer is appropriate. When a person visually verifies, it is costly.

[0005] The present invention has been made in consideration of such circumstances, and one of the objects is to provide an information processing apparatus, an information processing method, and a program that can more easily confirm whether an answer is appropriate.

Means for Solving the Problems

[0006] One aspect of the present invention is an information processing device comprising: an acquisition unit that acquires target information including an inquiry and an answer to the inquiry; and a processing unit that inputs the target information and a request to evaluate the answer to the inquiry to a model, acquires an evaluation result obtained by the model evaluating the answer to the inquiry, and stores evaluation information in a storage unit in which the evaluation result and the target information are associated. [Effects of the Invention]

[0007] According to one aspect of the present invention, it is possible to more easily verify whether the answer is appropriate. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram showing an example of the configuration of an information processing system. [Figure 2] This is a configuration diagram of service server 100. [Figure 3] This figure shows an example of an interface screen IF1 that the service application 20 displays on the display unit of the user terminal device 10. [Figure 4] This figure shows an example of the content of target information 174. [Figure 5] This diagram illustrates the process of evaluating responses. [Figure 6] This figure shows an example of the content of evaluation information 176. [Figure 7] This figure shows an example of a prompt. [Figure 8] This figure shows another example of a prompt. [Figure 9] This figure shows another example of a prompt. [Figure 10] This figure shows another example of a prompt. [Figure 11] This flowchart shows an example of the processing flow performed by the service server 100. [Figure 12] This figure shows an example of responses and inquiries below a threshold displayed on the administrator's terminal device. [Figure 13] This flowchart shows an example of the processing flow performed by the service server 100. [Figure 14] This diagram shows an example of the target information for each service. [Figure 15] This figure shows an example of information associated with prompts for each service. [Modes for carrying out the invention]

[0009] The following describes embodiments of the information processing apparatus, information processing method, and program of the present invention with reference to the drawings. Various devices such as "servers" that appear below, which provide services to users or perform internal analysis, may be implemented by a distributed group of devices, and the operators of each device may be different. Furthermore, the owner of the hardware of the devices (the provider of the cloud server) and the operator that actually operates them may also be different.

[0010] Figure 1 shows an example of the configuration of an information processing system. The information processing system comprises a user terminal device 10, a service server 100, a chatbot 200, and a model 300. These devices communicate with each other, for example, via a network NW. The network NW includes, for example, the internet, a LAN (Local Area Network), a wireless base station, and provider equipment.

[0011] Some or all of the functional components included in the information processing system may be distributed across multiple devices in any form, or integrated into any device. For example, some or all of the chatbot 200, or model 300, may be included in the service server 100.

[0012] [User terminal device] The user terminal device 10 is, for example, a portable terminal device such as a smartphone or a tablet terminal. The user terminal device 10 is a computer device having at least an optical reading function, a communication function, a display function, an input reception function, and a program execution function. In the following description, the configurations for realizing these functions are respectively referred to as a camera, a communication device, a touch panel, a CPU (Central Processing Unit), etc. In the user terminal device 10, the service application 20 is executed by a processor such as a CPU, and operates to provide a service to the user in cooperation with the service server 100. The service application 20 is installed in the user terminal device 10 from, for example, an application store, and controls a camera, a communication device, a touch panel, etc.

[0013] The service is, for example, any service such as a service related to settlement or a service related to finance. The service is, for example, a service related to a credit card. The service is, for example, a service that executes a settlement using a credit card owned by the user or provides information related to the settlement. The service may include, for example, a service related to cashing using a credit card.

[0014] [Service Server] FIG. 2 is a configuration diagram of the service server 100. The service server 100 provides the above-described services to the service application 20. The service server 100 may provide the services to the user terminal device 10 without going through the service application 20. The service server 100 includes, for example, a communication unit 110, a content providing unit 120, an information management unit 130, an information processing unit 140, an inquiry processing unit 150, an evaluation unit 160, and a storage unit 170. Components other than the communication unit 110 and the storage unit 170 are realized, for example, when a hardware processor such as a CPU executes a program (software). Some or all of these components may be realized by hardware (including a circuit unit; circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), and SOC (System On Chip), or may be realized by the cooperation of software and hardware. The program may be stored in advance in a storage device (a storage device having a non-transitory storage medium) such as an HDD (Hard Disk Drive) or a flash memory, or may be stored in a removable storage medium (a non-transitory storage medium) such as a DVD or a CD-ROM, and may be installed in the storage device when the storage medium is mounted on a drive device. The information management unit 130 is an example of an "acquisition unit", and the evaluation unit 160 is an example of a "processing unit".

[0015] The storage unit 170 is an HDD, a flash memory, a RAM (Random Access Memory), or the like. The storage unit 170 may be a NAS (Network Attached Storage) device that can be accessed by the service server 100 via a network. Information such as management information 172, target information 174, and evaluation information 176 is stored in the storage unit 170 (details of these will be described later).

[0016] The communication unit 110 is a communication interface for connecting to a network NW. The communication unit 110 is, for example, a network interface card.

[0017] The content provider unit 120, for example, has the functionality of a web server and provides information (content) for displaying various screens of the service to the user terminal device 10. The content provider unit 120 reads the necessary content as appropriate from the content information (not shown) stored in the storage unit 170 and provides it to the user terminal device 10. The user terminal device 10 receives various inputs from the user while the content is being played by the service application 20, and transmits the received information to the service server 100. The above content may be generated by the service application 20. In this case, the content provider unit 120 provides the service application 20 with the information necessary for generating the content.

[0018] The information management unit 130 acquires information provided by other devices such as the user terminal device 10, the chatbot 200, and models. The information management unit 130 manages management information 172, target information 174, and evaluation information 176. Management information 172 is, for example, information that associates user identification information with information about the user of the user identification information. Information about the user is, for example, information about the user related to the services of the service server 100, such as information about outstanding payments, late payment penalties, and bank accounts. In addition to the above, information about the user includes various information about the user related to the services provided by the service server 100.

[0019] The information processing unit 140 performs various processes in response to requests from the user terminal device 10. For example, the information processing unit 140 receives various services transmitted by the user via the user terminal device 10 and provides the received services to the user terminal device 10.

[0020] The inquiry processing unit 150 provides the user terminal device 10 with a response to the inquiry sent by the user terminal device 10. The inquiry processing unit 150 may also provide the user with information about the user associated with the user identification information of the user in the management information 172, in response to the user's request.

[0021] The evaluation unit 160 evaluates the response to the inquiry provided to the user by the inquiry processing unit 150 (details will be described later).

[0022] [Chatbot] Chatbot 200 is a model trained to output answers to inquiries in response to requests from the inquiry processing unit 150. Chatbot 200 is, for example, a Large Language Model (LLM) or a Generative AI (Artificial Intelligence). Chatbot 200 is a model trained to learn from a large amount of data and output answers in response to requests.

[0023] [Model] Model 300 is a model trained to output information indicating whether the response output by chatbot 200 is appropriate for the inquiry. For example, Model 300 outputs information indicating whether the response is appropriate or inappropriate. Model 300 could be, for example, a Large-Scale Language Model (LLM) or a Generative AI. Model 300 is a model trained to learn from a large amount of data and output responses that correspond to requests. Model 300 may be the same model as chatbot 200 described above.

[0024] [Interface screen] Figure 3 shows an example of an interface screen IF1 that the service application 20 displays on the display unit of the user terminal device 10. Interface screen IF1 includes, for example, information about credit card services and an inquiry button B. When the user operates the inquiry button B, the service application 20 displays an interface screen for inputting the inquiry on the display unit of the user terminal device 10. When the user enters the inquiry and performs the prescribed operation, the answer to the inquiry is displayed on the display unit. The answer is displayed through the following process.

[0025] The inquiry processing unit 150 of the service server 100 acquires the inquiry (first inquiry information) sent from the user terminal device 10. Next, the service server 100 generates a prompt that includes the user's inquiry and a request to output an answer to the user's inquiry, and provides the generated prompt to the chatbot 200. The service server 100 acquires the answer output by the chatbot 200. The service server 100 provides the acquired answer to the user terminal device 10 so that it is displayed on the display unit of the user terminal device 10.

[0026] [Regarding support information] The information management unit 130 stores the target information 174, which associates inquiries with answers, in the storage unit 170. Figure 4 shows an example of the contents of the target information 174. The target information 174 is the subject of evaluation by the evaluation unit 160.

[0027] [Summary of response evaluation] This section describes the process of evaluating whether the response is appropriate to the query. The information management unit 130 of the service server 100 acquires target information, including the query and the response to the query. The evaluation unit 160 of the service server 100 inputs the target information and a request to the model 300 to evaluate the response to the query, and acquires the evaluation result obtained by the model 300. The evaluation unit 160 of the service server 100 stores the evaluation information 176, which associates the evaluation result with the target information, in the storage unit 170A.

[0028] Model 300 is a model trained to output information indicating whether the response to the target information is appropriate for the query, given target information and a request as input. Model 300 may be the same model as Model 300 described above, or it may be a different model. For example, Model 300 may output information indicating that the response is appropriate, information indicating that the response is inappropriate, or information indicating that the response is neither good nor bad.

[0029] The above request may also be a request to output a score indicating whether the response to the target information is appropriate for the query about the target information. In this case, model 300 is a model that, when the target information and the request are input, outputs a score indicating whether the response to the target information is appropriate for the query about the target information.

[0030] [Process for evaluating responses] Figure 5 is a diagram illustrating the process of evaluating responses. (1) The service server 100 generates a prompt that includes the target information (inquiries and answers to those inquiries) and the request. (2) The service server 100 inputs the generated prompt to the model 300. (3) The service server 100 obtains the evaluation results output by the model 300. (4) The service server 100 stores the evaluation information 182, which associates the evaluation results with the target information, in the storage unit 170.

[0031] Figure 6 shows an example of the content of evaluation information 182. Evaluation information 182 is information to which a score is associated with each piece of target information, such as an inquiry and an answer. For example, a higher score indicates a higher evaluation.

[0032] [prompt] Figures 7 to 10 show examples of prompts. Prompts are stored, for example, in the memory unit 170. Some of the information in Figures 7 to 10 may be omitted in the prompts. For example, the information in Figures 9 and 10 may be omitted.

[0033] Figure 7 shows an example of a prompt. The prompt includes, for example, the objective, the subject to be evaluated, and the service overview. The objective is the purpose of the evaluation. In this embodiment, the objective is "to determine whether the answer is relevant to the question in terms of content."

[0034] The evaluation target is Model 300's (chatbot's) response to user inquiries (questions). The service overview is an overview of the service for which the inquiry is made. In this embodiment, it is a service related to credit cards, and for example, inquiries regarding unpaid balances are assumed. The service overview includes, for example, suggestions and solutions depending on the situation in the service, and an explanation of what the situation is. The service overview also includes the meaning, interpretation, and definition of terms.

[0035] Figure 8 shows another example of a prompt. A prompt may include, for example, evaluation rules. Evaluation rules are guidelines or criteria for conducting an evaluation. Evaluation rules specify criteria for each score. For example, scores may be assigned on a 10-point scale, but are not limited to this. Evaluation rules may specify evaluation criteria for a score of 10, for a score of 9, for a score of 1, and for a score of 0 (not evaluated).

[0036] A rating of 10 indicates an answer that is nearly perfect, for example, deeply understanding the user's situation and intentions, perfectly matching their intentions, being specific and practical, and directly solving the problem. It demonstrates a deep understanding of the situation, background, and intentions. It perfectly addresses the user's fundamental questions and feelings, leaving no lingering anxieties or doubts. A rating of 9 indicates an answer that is very good, for example, almost perfectly capturing the main intent of the question and the underlying needs, and exhibiting very high quality, accuracy, and naturalness.

[0037] A rating of 1 indicates a response that is completely irrelevant, such as an answer that is completely unrelated to the question, a mechanically generated canned response, or an incomprehensible response that does not function as a response. A rating of 0 indicates a response that is incomplete, incomprehensible, or a one-sided report from the user ("I did...", "I did...") that does not require a response from the chatbot.

[0038] Figure 9 shows another example of a prompt. The prompt includes, for example, evaluation points that are particularly important in the evaluation. These evaluation points are, for example, guidelines for when the evaluation is high or low. The prompt includes, for example, evaluation points and detailed information about those evaluation points. For example, an evaluation point is that it is important to acknowledge the user's "claim." Specifically, if the user claims a particular situation such as "paid," the chatbot will first acknowledge that claim and then guide the user through possible situations (e.g., delayed payment reflection, other unpaid) and how to check, which will lead to a high evaluation.

[0039] Figure 10 shows another example of a prompt. A prompt may include, for example, an evaluation example. An evaluation example includes a question, an answer to the question, and a score for the answer to the question. An evaluation example may also include, for example, a question, an answer, an evaluation of the answer to the question (expected score), and the reason for the evaluation. For example, the question is "How long does it take for the payment to be reflected after payment?" and the answer is "The timing of when the payment is reflected varies depending on the payment method and payment date. If you transfer the payment to our designated account, ..., ..., our customer center will contact you." The expected score for the answer to this question is 9 points. The reason for the 9 points is that it directly and specifically presents the timing of reflection to the user's question, and providing detailed information for each payment method greatly contributes to alleviating the user's anxiety. An evaluation example like the one above is included in a prompt.

[0040] The above evaluation makes it easier to verify whether the answers are appropriate. For example, since evaluation can be performed automatically without requiring human visual confirmation, costs can be reduced. Model 300 performs evaluations based on predefined criteria, eliminating subjectivity by evaluators and ensuring objective and uniform evaluations. Automation allows for rapid acquisition of evaluation results. This enables quick and efficient identification of areas for improvement in answers, and rapid and efficient improvement of answer quality. As described above, the service server 100 can improve convenience for administrators and users through automated evaluation. For example, by omitting human evaluation for answers with an evaluation above a certain level and requiring human evaluation for those below a certain level, costs can be reduced while providing more appropriate evaluations.

[0041] In this embodiment, the service server 100 acquires target information including the inquiry entered by the user and the answer to the inquiry output by the chatbot 200, and acquires an evaluation result for the chatbot 200's answer. However, instead of this (or in addition), the response of another device or a human operator may also be evaluated. In this way, the service server 100 can appropriately evaluate the answer to the inquiry in the same manner as described above.

[0042] [Specific application of evaluation results (1)] The evaluation unit 160 of the service server 100 identifies combinations of responses with scores below a threshold and inquiries related to those responses, and displays the identified combinations on the display unit of the administrator's terminal device (not shown) so that the administrator can confirm the inquiries and the responses to inquiries with low scores.

[0043] Figure 11 is a flowchart illustrating an example of the processing flow performed by the service server 100. First, the service server 100 acquires evaluation information 176 (S200). Next, the service server 100 refers to the evaluation information 176 and identifies responses and inquiries with scores below the threshold (S202). Then, the service server 100 sends the responses and inquiries below the threshold to the administrator's terminal device and displays them on the administrator's terminal device's display (S204). This completes one routine in this flowchart.

[0044] Figure 12 shows an example of responses and inquiries below a threshold displayed on the administrator's terminal device. As shown in Figure 12, for example, responses with scores below a threshold and inquiries regarding those responses are displayed on the display. By referring to the above information, the administrator can recognize information that the chatbot 200 cannot answer appropriately and improve the chatbot 200 to output appropriate responses.

[0045] [Specific application of evaluation results (2)] The service server 100 may identify combinations of answers with scores below a threshold and inquiries related to those answers, input the identified inquiries and requests for appropriate answers to those inquiries into the model 300, and display the answers output by the model 300 on the display unit of the administrator's terminal device. If the administrator approves the answers output by the model 300, the service server 100 may train the chatbot 200 that provided the answers to learn the answers output by the model 300 as answers to inquiries. Training means inputting into the chatbot 200 that the approved answers are appropriate answers (correct answers) as inquiries related to those answers, or registering them in the database as inquiries or answers similar to inquiries.

[0046] Figure 13 is a flowchart illustrating an example of the processing flow performed by the service server 100. First, the service server 100 identifies queries with low ratings and the corresponding answers (S200). Next, the service server 100 inputs a request to the model 300 to query and generate a high-rated answer (S202). For example, the query, the above request, and specific information are input to the model 300. The specific information may be one or more of the following: service overview, evaluation rules, evaluation points that are particularly emphasized in the evaluation, and evaluation examples. For example, the service server 100 asks the model 300 to generate an answer that will result in a high rating for the query, using the specific information as a reference.

[0047] When the service server 100 obtains a response from the model 300, it displays the inquiry and the response to the inquiry on the display unit of the administrator's terminal device (S204). If the administrator approves the inquiry and the response, the service server 100 trains the chatbot 200 with the response as a candidate answer to the inquiry (S208). If the administrator does not approve the inquiry and the response, the service server 100 trains the chatbot 200 with the response entered by the administrator as a candidate answer to the inquiry (S210). For example, on the approval screen, an approval button and an area for entering a response are provided, allowing the administrator to approve the response to the inquiry or enter a more appropriate response.

[0048] As described above, the service server 100 can provide users with appropriate answers to inquiries by having the model 300 generate an appropriate response, and if the generated response is appropriate, having the chatbot 200 learn from it.

[0049] [Specific application of evaluation results (3)] The service server 100 may identify combinations of answers with scores below a threshold and inquiries related to those answers, and train the chatbot 200 to recognize that the answers to the identified inquiries are inappropriate, or it may notify the administrator's terminal device of the inappropriate answers and inquiries related to those answers. For example, the service server 100 may notify the chatbot 200 of answers with scores below a threshold, inquiries related to those answers, and information indicating that the above answers are inappropriate. In response to the notification, the chatbot 200 learns that the answers to those inquiries are inappropriate and suppresses (or excludes) outputting the same answers for those inquiries or similar inquiries, or lowers the output priority. In addition, by notifying the administrator's terminal device of inappropriate answers and inquiries, the administrator can tune the chatbot 200 to prevent inappropriate answers from being output to inquiries. For example, by evaluating answers at predetermined intervals or each time an answer is output and performing the above processing, the output of inappropriate answers can be suppressed.

[0050] Through the above process, the service server 100 can prevent inappropriate responses from being output again.

[0051] [Identifying the reasons for the evaluation results] The service server 100 may input to the model 300 a request that evaluate the target information and the response to the query, as well as a request that outputs the reason why the response was evaluated as inappropriate if it is not appropriate (a request that outputs the reason for outputting the evaluation result). The model 300 outputs the reason for outputting the evaluation result. For example, the service server 100 may input a request that outputs the reason if the evaluation result is inappropriate, causing the model 300 to respond with the reason. The above reason may be requested at a different time than when the evaluation result is obtained. In this case, for example, the service server 100 may request the query, the response to the query, and information indicating the reason for the evaluation at a different time than when the response is obtained. Even if an appropriate response is obtained, the service server 100 may input a request to the model 300 to output why it was evaluated as appropriate, causing the model 300 to output the reason for the evaluation.

[0052] As described above, the service server 100 can easily analyze the responses output by the chatbot 200 by having the model 300 output the reasons for the evaluation results.

[0053] [Use of service-specific prompts] The service server 100 acquires target information for multiple different services, refers to the information that associates each of the multiple services with a prompt, and identifies a prompt according to the service of the target information to be evaluated. The service server 100 inputs the identified prompt and target information into the model 300, obtains the evaluation result from the model 300, and stores the evaluation information in the storage unit 170.

[0054] The memory unit 170 stores target information 174 for each service. Figure 14 shows an example of target information for each service. A service may be a type such as service A, B, or C as shown in Figure 14, or it may be the service used for the inquiry (a channel such as an interface screen). The type of service may be a classification within a certain service. For example, a service related to credit card usage, credit card payment repayment, or cash advance. For example, an inquiry made by a user from a credit card usage screen (service) may be classified as an inquiry for a different service than an inquiry made from a cash advance screen (service).

[0055] As shown in Figure 15, the memory unit 170 stores information to which prompts are associated with each service. Service A has prompts to better evaluate the answers to service A queries, and service B has prompts to better evaluate the answers to service B queries. The same applies to service C. A service-specific prompt is a prompt in which one or more pieces of information, such as the service overview, evaluation rules, evaluation points emphasized in the evaluation, and evaluation examples, are tailored to that service. For example, the service overview may be tailored to that service, or the evaluation points may be specific to that service. By inputting the service-specific prompt into the model 300, the service server 100 can obtain the results of an accurate evaluation of the appropriateness of the answers to service queries.

[0056] According to the embodiment described above, the service server 100 inputs the target information 174 and a request to the model 300 to evaluate the response to the query, obtains the evaluation result obtained by the model 300's evaluation of the response to the query, and stores the evaluation information 176, which associates the evaluation result with the target information 174, in the storage unit 170, thereby making it easier to confirm whether the response is appropriate.

[0057] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of symbols]

[0058] 10. User terminal device 20 Service Apps 100 service servers 120 Content Provision Department 130 Information Management Department 140 Information Processing Section 150 Inquiry Processing Unit 160 Evaluation Department 170 Storage section 172 Management information 174 Target Information 176 Review Information 300 Models

Claims

1. An acquisition unit that acquires target information including the inquiry and the response to the said inquiry, The aforementioned target information and a request to evaluate the response to the aforementioned inquiry are input into the model. The model obtains an evaluation result that evaluates the response to the query, The system includes a processing unit that stores evaluation information in which the evaluation results and the target information are associated, The aforementioned processing unit, The model identifies combinations of responses whose scores are below a threshold and queries corresponding to those responses, and inputs the identified queries and requests that output appropriate responses to those queries into the model. The response output by the aforementioned model is displayed on the display unit of the administrator's terminal device. When the administrator approves the response output by the model, the chatbot that provided the response is trained to use the response output by the model as the answer to the inquiry. If the administrator does not approve the answer output by the aforementioned model, the answer entered by the administrator will be used as the answer to the aforementioned inquiry for training purposes. Information processing device.

2. An acquisition unit that acquires target information including the inquiry and the response to the said inquiry, A storage unit that stores prompts associated with each of several services, including an overview of the service including the meaning, definition, or interpretation of terms in the service, and evaluation rules including evaluation criteria, From the prompts stored in the memory unit, a prompt corresponding to the service related to the target information to be evaluated is identified, and the target information, the identified prompt, and a request to evaluate the response to the query are input to the model. The model obtains an evaluation result that evaluates the response to the query, A processing unit that stores evaluation information in which the evaluation results and the target information are associated in the storage unit, An information processing device equipped with the following features.

3. The storage unit stores, for each of the multiple services, a prompt containing an overview of the service, evaluation rules, evaluation points to be emphasized in the evaluation, and / or an evaluation example, The evaluation points emphasized in the aforementioned evaluation include guidelines for when the evaluation is high or when the evaluation is low. The aforementioned evaluation example includes an inquiry, an answer to the inquiry, and a score for the answer to the inquiry. The information processing apparatus according to claim 2.

4. The aforementioned model is a model that, upon receiving the aforementioned target information and the aforementioned request, outputs information indicating whether the response regarding the target information is appropriate for the query regarding the target information. The information processing apparatus according to any one of claims 1 to 3.

5. The aforementioned request is a request to output a score indicating whether the response to the aforementioned information is appropriate for the query regarding the aforementioned information. The aforementioned model is a model that, upon receiving the aforementioned target information and the aforementioned request, outputs a score indicating whether the response regarding the target information is appropriate to the query regarding the target information. The information processing apparatus according to any one of claims 1 to 3.

6. The processing unit identifies combinations of responses whose scores are below a threshold and inquiries corresponding to those responses, and displays the identified combinations on the display unit of the administrator's terminal device so that the administrator can confirm the inquiries and the responses to inquiries with low scores. The information processing apparatus according to claim 2 or 3.

7. The acquisition unit acquires the target information, which includes the inquiry entered by the user and the response to the inquiry output by the chatbot. The processing unit obtains the evaluation result for the chatbot's response. The information processing apparatus according to any one of claims 1 to 3.

8. The aforementioned processing unit, Identify the combination of the score, which is the evaluation result, of a response below a threshold, and the inquiry corresponding to that response. The chatbot that outputs answers to inquiries is trained to recognize that the answer to the identified inquiry is inappropriate, or the administrator's terminal device is notified of the inappropriate answer and the inquiry related to that answer. The information processing apparatus according to claim 2 or 3.

9. The aforementioned processing unit, In addition to the aforementioned target information and a request to evaluate the response to the aforementioned inquiry, the model is also input a request to output the reason why the response was evaluated as inappropriate if it is deemed inappropriate. The model is instructed to output the reason why the answer was deemed inappropriate if it was deemed inappropriate. The information processing apparatus according to any one of claims 1 to 3.

10. Computers We obtain the target information, including the inquiry and the response to the said inquiry. The aforementioned target information and a request to evaluate the response to the aforementioned inquiry are input into the model. The model obtains an evaluation result that evaluates the response to the query, The evaluation results and the target information are associated with each other and stored in the storage unit. The model identifies combinations of responses whose scores are below a threshold and queries corresponding to those responses, and inputs the identified queries and requests that output appropriate responses to those queries into the model. The response output by the aforementioned model is displayed on the display unit of the administrator's terminal device. When the administrator approves the response output by the model, the chatbot that provided the response is trained to use the response output by the model as the answer to the inquiry. If the administrator does not approve the answer output by the aforementioned model, the answer entered by the administrator will be used as the answer to the aforementioned inquiry for training purposes. Information processing methods.

11. On the computer, The system will obtain the target information, including the inquiry and the response to the said inquiry. The model is given the aforementioned target information and a request to evaluate the response to the aforementioned inquiry. The model obtains an evaluation result that evaluates the response to the query. The evaluation results and the target information are associated with each other and stored in the storage unit. The model identifies combinations of responses whose scores are below a threshold and queries corresponding to those responses, and inputs the identified queries and requests that output appropriate responses to those queries into the model. The response output by the aforementioned model is displayed on the display unit of the administrator's terminal device. When the administrator approves the response output by the model, the chatbot that provided the response is trained to use the response output by the model as the answer to the inquiry. If the administrator does not approve the answer output by the aforementioned model, the answer entered by the administrator will be used as the answer to the aforementioned inquiry for training purposes. program.

12. Computers We obtain the target information, including the inquiry and the response to the said inquiry. For each of the multiple services, a storage unit stores prompts associated with a service overview, including the meaning, definition, or interpretation of terms in the service, and evaluation rules, including evaluation criteria. From these prompts, the system identifies a prompt corresponding to the service related to the target information being evaluated. The target information, the identified prompt, and a request to evaluate the response to the query are then input into the model. The model obtains an evaluation result that evaluates the response to the query, The evaluation information, which associates the evaluation result with the target information, is stored in the storage unit. Information processing methods.

13. On the computer, The system will obtain the target information, including the inquiry and the response to the said inquiry. For each of the multiple services, a storage unit stores prompts associated with a service overview, including the meaning, definition, or interpretation of terms in the service, and evaluation rules, including evaluation criteria. From these prompts, the system identifies a prompt corresponding to the service related to the target information being evaluated, and the model is input with the target information, the identified prompt, and a request to evaluate the response to the query. The model obtains an evaluation result that evaluates the response to the query. The evaluation information, which associates the evaluation result with the target information, is stored in the storage unit. program.

Images