Information processing apparatus, method, program, and information processing system

Abstract

To improve the quality of an answer to a prompt from a user.SOLUTION: An information processing apparatus according to an embodiment of the present invention includes a reception unit configured to receive a prompt from a user terminal, a document search unit configured to search for a document based on the prompt, an answer generation unit configured to cause a large scale language model to generate an answer to the prompt based on the prompt and the document, a transmission unit configured to transmit the answer to the user terminal, an evaluation unit configured to evaluate the answer based on an action of a user who operates the user terminal, and an answer management unit configured to store the answer and a result of the evaluation in association with each other.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a method, a program, and an information processing system. [Background technology]

[0002] A conventional technique known as Retrieval-Augmented Generation (RAG) is to generate an answer to a prompt (question, etc.) from a large-scale language model by searching for a document based on the prompt from within a company or other documents, and then generating an answer based on the prompt and the document. This technique makes it possible to generate an appropriate answer.

[0003] In Non-Patent Document 1, information on whether the user was satisfied with the answer is stored in a cache database as feedback, and for inquiries similar to questions with highly rated answers, the cache is returned directly without using a large-scale language model, thereby improving answer quality and cost efficiency (Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] ExaWizards Inc., "ExaWizards launches 'RAGOps', which allows for operational improvements to generative AI and business data integration. If the quality of answers is not satisfactory, humans can intervene to make additions and corrections, and the service is provided through exaBase Studio." May 13, 2024, Internet<URL: https: / / exawizards.com / archives / 27382 / > Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional method, the answer to the prompt may not be appropriate for the user. Therefore, an object of the present invention is to improve the quality of the answer to the prompt from the user. [Means for solving the problem]

[0006] An information processing device according to one embodiment of the present invention includes a receiving unit that receives a prompt from a user terminal, a document search unit that searches for a document based on the prompt, an answer generation unit that causes a large-scale language model to generate an answer to the prompt based on the prompt and the document, a transmitting unit that transmits the answer to the user terminal, an evaluation unit that evaluates the answer based on the behavior of a user operating the user terminal, and an answer management unit that associates and stores the answer with the result of the evaluation. [Effects of the Invention]

[0007] The present invention can improve the quality of responses to prompts from users. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing an overall configuration according to an embodiment of the present invention; [Figure 2] 1 is a functional configuration diagram of an information processing device according to an embodiment of the present invention; [Figure 3] 4 is an example of data stored in a response storage unit according to one embodiment of the present invention. [Figure 4] 4 is an example of data stored in a user information storage unit according to an embodiment of the present invention. [Figure 5] FIG. 10 is a diagram illustrating evaluation of an answer based on a user's behavior according to an embodiment of the present invention. [Figure 6] FIG. 10 is a diagram illustrating evaluation of an answer based on a user's behavior according to an embodiment of the present invention. [Figure 7]FIG. 10 is a sequence diagram of a process for evaluating an answer based on a user's behavior according to an embodiment of the present invention. [Figure 8] FIG. 10 is a sequence diagram of a process for extracting answers based on user attributes according to an embodiment of the present invention. [Figure 9] 1 is a hardware configuration diagram of an information processing apparatus according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] <Overall structure> Fig. 1 is a diagram showing the overall configuration according to one embodiment of the present invention. The information processing system 1 includes an information processing device (server) 10 and a user terminal 20. The information processing device 10 and the user terminal 20 can transmit and receive data via any network. Each of these will be described below.

[0011] <<Information processing equipment>> The information processing device 10 is a server that receives a prompt input by a user 21 to the user terminal 20 from the user terminal 20 and transmits an answer to the prompt to the user terminal 20. The information processing device 10 searches for a document from among internal company documents, etc., based on the prompt, and causes a large-scale language model (LLM) to generate an answer based on the prompt and the document (also known as Retrieval-Augmented Generation (RAG)). The information processing device 10 is composed of one or more computers.

[0012] [Evaluation of answers based on user behavior] The information processing device 10 can evaluate an answer based on the behavior of the user 21. Specifically, the information processing device 10 receives a prompt from the user terminal 20. The information processing device 10 searches for a document based on the prompt and causes a large-scale language model to generate an answer to the prompt based on the prompt and the document. The information processing device 10 transmits the answer to the user terminal 20. The information processing device 10 evaluates the answer based on the behavior of the user 21 who operates the user terminal 20, and associates the answer with the evaluation result and stores it. Note that the information processing device 10 can also further associate the answer and the evaluation result with the attributes of the evaluator and store them.

[0013] [Extract answers based on user attributes] The information processing device 10 can extract answers based on the attributes of the user 21. Specifically, the information processing device 10 stores answers to prompts generated by a large-scale language model based on prompts and documents searched based on the prompts. The information processing device 10 receives a prompt from the user terminal 20, and extracts from the stored answers an answer that is an answer to a prompt that is the same as or similar to the prompt received from the user terminal 20, has a high rating, and has a high degree of similarity between the attributes of the evaluator and the attributes of the user 21 operating the user terminal 20, and transmits the extracted answer to the user terminal 20. Note that the rating is, for example, based on the behavior of the user 21 operating the user terminal 20, but may also be an rating input by the user 21 to the user terminal 20.

[0014] <<User terminal>> The user terminal 20 is a terminal operated by the user 21. For example, the user terminal 20 is a smartphone, a tablet, a personal computer, etc. The user terminal 20 transmits a prompt input by the user 21 to the user terminal 20 to the information processing device 10, receives an answer to the prompt from the information processing device 10, and displays the answer on the display of the user terminal 20. Note that although FIG. 1 illustrates a case in which there are three user terminals 20, the number of user terminals 20 is not limited to this.

[0015] <Functional configuration> 2 is a functional configuration diagram of an information processing device 10 according to one embodiment of the present invention. The information processing device 10 can include a receiving unit 101, a document searching unit 102, an answer generating unit 103, a transmitting unit 104, an evaluation unit 105, an answer managing unit 106, an answer extracting unit 107, a document storage unit 108, a large-scale language model storage unit 109, an answer storing unit 110, and a user information storage unit 111. By executing a program, the information processing device 10 can function as the receiving unit 101, the document searching unit 102, the answer generating unit 103, the transmitting unit 104, the evaluation unit 105, the answer managing unit 106, and the answer extracting unit 107. Each of these units will be described below.

[0016] The receiving unit 101 receives a prompt (for example, a question, which may be a command, instruction, command, etc.) from the user terminal 20. The receiving unit 101 also receives a request from the user terminal 20 to regenerate an answer to the prompt.

[0017] Based on the prompt received by the receiving unit 101, the document searching unit 102 searches for one or more documents corresponding to the prompt from among the documents stored in the document storage unit 108.

[0018] The answer generation unit 103 causes the large-scale language model stored in the large-scale language model storage unit 109 to generate one or more answers to the prompt based on the prompt received by the receiving unit 101 and the document searched by the document search unit 102.

[0019] [Generating an answer when there is no answer in the answer memory] If the answer storage unit 110 does not contain an answer to a prompt that is the same as or similar to the prompt received by the receiving unit 101, the answer generating unit 103 can cause the large-scale language model to generate an answer to the prompt.

[0020] [Regenerate Answer] When the receiving unit 101 receives a request from the user terminal 20 to regenerate an answer to a prompt, the answer generating unit 103 causes the large-scale language model to generate an answer to the prompt based on the prompt received by the receiving unit 101 and the document searched by the document search unit 102.

[0021] The transmission unit 104 transmits one or more answers to the prompt to the user terminal 20. Specifically, the transmission unit 104 transmits to the user terminal 20 the answers that the answer generation unit 103 has generated using the large-scale language model and the answers that the answer extraction unit 107 has extracted from the answers stored in the answer storage unit 110.

[0022] The evaluation unit 105 evaluates the answer to the prompt based on the behavior of the user 21 operating the user terminal 20 (i.e., the behavior in response to the answer). When there are multiple answers to one prompt, the evaluation unit 105 evaluates each answer.

[0023] For example, if the information processing device 10 receives an additional prompt from the user terminal 20 within a predetermined time, or if the information processing device 10 receives a request from the user terminal 20 to regenerate an answer to the prompt, the evaluation unit 105 will rate the answer low.

[0024] For example, the evaluation unit 105 evaluates the answer highly if the source of the answer is viewed, if at least a part of the answer is copied, or if the link destination of the answer is clicked.

[0025] The answer management unit 106 associates an answer to a prompt with the result of evaluation of the answer by the evaluation unit 105, and stores them in the answer storage unit 110. The answer management unit 106 can also further associate the answer to the prompt and the result of evaluation of the answer by the evaluation unit 105 with the attributes of the evaluator (i.e., the user 21 who performed the behavior that formed the basis of the evaluation), and store them in the answer storage unit 110. The answer management unit 106 references the attributes of the user 21 stored in the user information storage unit 111 based on the user ID etc. received from the user terminal 20.

[0026] The answer extraction unit 107 extracts one or more answers from the answers stored in the answer storage unit 110 that are answers to a prompt that is the same as or similar to the prompt received from the user terminal 20, that have a high evaluation, and that have a high degree of similarity between the attributes of the evaluator and the attributes of the user 21 operating the user terminal 20. The answer extraction unit 107 refers to the attributes of the user 21 stored in the user information storage unit 111 based on the user ID etc. received from the user terminal 20.

[0027] Various documents are stored in the document storage unit 108. The documents are, for example, documents describing internal company regulations (e.g., documents describing various internal procedures), but may also be documents describing procedures to be carried out by customers of financial institutions. Note that the documents are assigned information for identifying the documents (e.g., document IDs).

[0028] A large-scale language model is stored in the large-scale language model storage unit 109. Note that the large-scale language model may be stored in a storage unit of a device other than the information processing device 10.

[0029] The answer storage unit 110 stores answers transmitted by the information processing device 10 to the user terminal 20. Specifically, the answer storage unit 110 stores, in association with each other, information for identifying a document used when generating an answer (for example, a document ID), attributes of the evaluator (i.e., the user 21 who performed the behavior that is the basis of the evaluation), a prompt, an answer generated by the answer generation unit 103 using a large-scale language model, and a result of the evaluation of the answer by the evaluation unit 105.

[0030] The attributes of each user are stored in the user information storage unit 111. The attributes may be, for example, the attributes of a company employee (for example, department, position, job type; it may also be years of service, age, etc.), but may also be the attributes of a financial institution's customer.

[0031] <Database> 3 shows an example of data stored in the answer storage unit 110 according to one embodiment of the present invention. For example, the answer storage unit 110 stores the "answer ID," "document ID," "evaluator attributes," "prompt," "answer," and "evaluation result" stored by the answer management unit 106. Note that the data stored in the answer storage unit 110 may be deleted at any time (for example, when it has not been used in receiving the past N prompts).

[0032] The “answer ID” is assigned to each answer that the information processing device 10 transmits to the user terminal 20.

[0033] The "document ID" is information for identifying the document used when generating the answer.

[0034] "Attributes of evaluator" are attributes (for example, department, position, job type) of the evaluator (that is, the user 21 who performed the behavior that is the basis of the evaluation).

[0035] “Prompt” is the prompt received from the user terminal 20 .

[0036] The "answer" is an answer that the answer generation unit 103 has caused the large-scale language model to generate.

[0037] The "evaluation result" is the result of evaluating the answer by the evaluation unit 105. For example, the "evaluation result" is a value obtained by quantifying the behavior (the sum of the scores of each behavior).

[0038] In this way, the answer storage unit 110 stores the evaluation results of evaluators with different attributes for the same prompt and answer (answer IDs: 001, 002, 003 in FIG. 3). In some cases, the evaluation results of evaluators with different attributes are different, as in answer IDs: 001 and 002 in FIG. 3, and in other cases, the evaluation results of evaluators with different attributes are the same, as in answer IDs: 002 and 003 in FIG. 3. In addition, evaluation results of evaluators with the same attribute for different answers to the same prompt are stored (answer IDs: 001 and 004 in FIG. 3).

[0039] 4 shows an example of data stored in the user information storage unit 111 according to one embodiment of the present invention. For example, the user information storage unit 111 stores a "user ID" and a "user attribute."

[0040] The “user ID” is information for identifying the user 21 .

[0041] "User attributes" are attributes of the user 21 (for example, department, position, job type).

[0042] Each process will be described in detail below.

[0043] [Search for documents] The document search unit 102 can use vector search or keyword search to search for one or more documents to be used for generating an answer from among the various documents stored in the document storage unit 108. For example, the document search unit 102 vectorizes (embedding) the prompt using a large-scale language model, and uses the vector to search for a document from among the various documents stored in the document storage unit 108. For example, the document search unit 102 extracts keywords from the prompt using a large-scale language model, and uses the keywords to search for a document from among the various documents stored in the document storage unit 108. Note that the document search unit 102 may analyze vectors and keywords to be used for document search from the prompt without using a large-scale language model.

[0044] [Generate Answers] The answer generation unit 103 inputs the prompt received by the receiving unit 101 and one or more documents searched by the document search unit 102 into the large-scale language model stored in the large-scale language model storage unit 109, and causes the large-scale language model to output an answer to the prompt. Note that the answer generation unit 103 may cause the large-scale language model to generate one answer or multiple answers for one prompt.

[0045] [Evaluate your answers] The evaluation unit 105 calculates the actions of the user 21 operating the user terminal 20 as a numerical value (the sum of the scores of each action). Details will be described with reference to FIGS.

[0046] Figure 5 shows an example of a user's behavior after viewing answers to a prompt.

[0047] In step 1 (S1), the user 21 inputs a prompt into the user terminal 20.

[0048] In step 2 (S2), the user 21 views the answers to the prompts entered in S1, which are displayed on the user terminal 20.

[0049] In step 3 (S3), the user 21 determines whether the answer viewed in S2 is the answer the user 21 wants. If it is the answer the user 21 wants, the process proceeds to step 4, and if it is not the answer the user 21 wants, the process proceeds to step 9 or step 10.

[0050] In step 4 (S4), the user 21 checks the facts. Specifically, the user 21 views the source of the answer viewed in S2. Note that there are cases where the user 21 proceeds to step 5 without taking the action of S4.

[0051] In step 5 (S5), the user 21 determines whether the answer viewed in S2 is the answer that the user 21 wants. If it is the answer that the user 21 wants, the process proceeds to one of steps 6 to 8, and if it is not the answer that the user 21 wants, the process proceeds to step 10.

[0052] In step 6 (S6), the user 21 copies at least a portion of the answer viewed in S2.

[0053] In step 7 (S7), the user 21 clicks on the link of the answer viewed in S2.

[0054] Note that both actions S6 and S7 may be performed.

[0055] In step 8 (S8), the user 21 views the answers.

[0056] In step 9 (S9), the user 21 inputs an additional prompt into the user terminal 20 within a predetermined time, after which the process returns to S2.

[0057] In step 10 (S10), the user 21 inputs into the user terminal 20 a request to regenerate the answer to the prompt input in S1.

[0058] 6 shows an example of scores when the user performs each behavior (TRUE) and when the user does not perform each behavior (FALSE). The evaluation unit 105 calculates the sum of the scores of one or more behaviors performed by the user 21 and the scores of one or more behaviors not performed by the user 21.

[0059] For example, if the user's action is "Entered an additional prompt within N seconds," and the user performs this action (S9 in Figure 5), the score is "-10," and if the user does not perform this action (Yes in S3 in Figure 5), the score is "+1."

[0060] For example, if the user's behavior is "viewed the source (regardless of the number of times)" and the user performs this behavior (S4 in FIG. 5), the score is "+2".

[0061] For example, if the user's behavior is "copied part of the answer (regardless of the number of times)" and the user performs this behavior (S6 in FIG. 5), the score is "+3".

[0062] For example, if the user's behavior is "clicked on a link (regardless of the number of times)" and the user performs this behavior (S7 in FIG. 5), the score is "+7".

[0063] For example, if the user's action is "regenerated an answer" and the user performs this action (S10 in FIG. 5), the score is "-10."

[0064] [Answer Extraction] The answer extraction unit 107 extracts answers that are answers to prompts that are identical or similar (i.e., have a similarity equal to or greater than a threshold) to the prompt received from the user terminal 20 and that use the same document from the answers stored in the answer storage unit 110. The answer extraction unit 107 reranks (sorts) the answers extracted from the answer storage unit 110 based on the attributes of the evaluator, the attributes of the prompt sender (e.g., the questioner), and the evaluation results.

[0065] [[Extracting answers stored in the answer memory]] The answer extraction unit 107 extracts answers that are answers to prompts that are the same as or similar to the prompt received from the user terminal 20 and that search for the same document based on the prompt from the answers stored in the answer storage unit 110. For example, the answer extraction unit 107 can cause a large-scale language model to determine whether the prompt received from the user terminal 20 is similar to the prompt of the answer stored in the answer storage unit 110.

[0066] [[Rerank]] The answer extraction unit 107 may rerank the answers using a user collaborative filtering technique described below, or may have the large-scale language model rerank them (specifically, by inputting a prompt to the large-scale language model that instructs it to rerank the answers using answer data extracted from the answer storage unit 110 (including the results of the answer evaluations and the attributes of the evaluators) and the attributes of the sender of the prompt (e.g., the questioner), and then outputting the reranking results). One or more top-ranked answers from the reranked answers are transmitted to the user terminal 20.

[0067] [[[User Collaborative Filtering]]] The answer extraction unit 107 calculates the value of the following formula (1) for each answer extracted from the answer storage unit 110, and can rerank answers with larger values ​​as higher ranking answers.

[0068] (similarity between the prompt sender and the evaluator × evaluation result) ÷ sum of the similarities between the prompt sender and each evaluator... Equation (1)

[0069] For example, the attributes of the prompt sender (e.g., the questioner) and the attributes of the evaluator are vectorized, and the similarity between the prompt sender and the evaluator is calculated based on the cosine similarity between the two vectors. Note that the similarity between the prompt sender and the evaluator may be calculated based on the Euclidean distance, or a large-scale language model may be used to determine the similarity.

[0070] The vector of attributes of prompt sender A is

[0071]

number

[0072] and the vector of attributes of rater B is

[0073]

number

[0074] and the vector of attributes of rater C is

[0075]

number

[0076] It should be noted that any method such as TF-IDF, BoW, Ada, etc. may be used for vectorization.

[0077] The cosine similarity between the vector of attributes of prompt sender A and the vector of attributes of rater B is expressed as follows:

[0078]

number

[0079] The cosine similarity between the vector of attributes of prompt sender A and the vector of attributes of rater C is expressed as follows:

[0080]

number

[0081] For example, suppose the cosine similarity between the attribute vector of prompt sender A and the attribute vector of evaluator B is 0.8, and the cosine similarity between the attribute vector of prompt sender A and the attribute vector of evaluator C is 0.3. Also, suppose the result of evaluator B's evaluation (total score of each behavior) is 4, and the result of evaluator C's evaluation (total score of each behavior) is 5.

[0082] The value of the above formula (1) for the answer given by rater B is: (0.8×4)÷1.1=2.91 is.

[0083] The value of the above formula (1) for the answer given by rater C is: (0.3×5)÷1.1=1.36 is.

[0084] In this case, the answer extraction unit 107 reranks the answer whose evaluator is evaluator B as an answer higher than the answer whose evaluator is evaluator C.

[0085] <Example> In one embodiment of the present invention, a large-scale language model can be used to generate answers to questions that company employees have about various procedures within the company.

[0086] For example, suppose the information processing device 10 receives a prompt "What are the procedures for business trips?" from the user terminal 20. The information processing device 10 extracts the keyword "business trip procedures" from the prompt. Using the keyword, the information processing device 10 searches for the documents "ID: 01 Document name: Domestic business trip procedures, ID: 02 Document name: Overseas business trip procedures, ID: 03 Document name: Bill payment procedures, ID: 04 Document name: Travel agency arrangement procedures."

[0087] [Evaluation of responses based on employee behavior] The information processing device 10 generates answers using the large-scale model and transmits the answers (for example, Answer 1 "On a domestic business trip...", Answer 2 "On an overseas business trip...", Answer 3 "The invoice is...") to the user terminal 20. The information processing device 10 evaluates the answers based on the actions of the employees who viewed the answers, and stores the answers and the evaluation results (total scores for each action) in association with the attributes of the evaluator (i.e., the employee who performed the action that is the basis of the evaluation), such as department, position, and job type.

[0088] [Extracting answers based on the attributes of the evaluator (employee) and the questioner (other employees)] The information processing device 10 extracts answers that are answers to prompts that are the same as or similar to the prompt received from the user terminal 20 and that use the same document (ID: 01 to 04) from the answers stored in the answer storage unit 110. The information processing device 10 reranks the answers extracted from the answer storage unit 110 based on the attributes of the evaluator (i.e., the employee who performed the behavior that is the basis of the evaluation), the attributes of the questioner (other employees), and the evaluation results (the sum of the scores of each behavior). The information processing device 10 transmits the top answers of the reranked answers (for example, Answer 1 "On a domestic business trip...", Answer 2 "On an overseas business trip...", Answer 3 "The invoice is...") to the user terminal 20.

[0089] <Processing method> FIG. 7 is a sequence diagram of a process for evaluating an answer based on a user's behavior according to an embodiment of the present invention.

[0090] In step 101 (S101), the user terminal 20 transmits information for identifying the user 21 (for example, a user ID) to the information processing device 10. For example, S101 is executed when logging in to the service according to the present invention.

[0091] In step 102 (S102), the user 21 inputs a prompt into the user terminal 20.

[0092] In step 103 (S103), the user terminal 20 transmits the prompt input in S102 to the information processing device 10. The information processing device 10 receives the prompt from the user terminal 20.

[0093] In step 104 (S104), the information processing device 10 searches for a document corresponding to the prompt from among the documents stored in the document storage unit 108 based on the prompt received in S103.

[0094] In step 105 (S105), the information processing device 10 causes the large-scale language model stored in the large-scale language model storage unit 109 to generate an answer to the prompt based on the prompt received in S103 and the document searched in S104.

[0095] In step 106 (S106), the information processing device 10 transmits the answer generated by the large-scale language model in S105 to the user terminal 20. The user terminal 20 receives the answer from the information processing device 10.

[0096] In step 107 (S107), the user terminal 20 displays the answer received in S106 on the display of the user terminal 20.

[0097] In step 108 (S108), the user 21 views the answer displayed in S107 and then takes action in response to the answer.

[0098] In step 109 (S109), the user terminal 20 transmits information indicating the behavior performed by the user 21 in S108 to the information processing device 10. The information processing device 10 receives, from the user terminal 20, information indicating the behavior performed by the user 21.

[0099] In step 110 (S110), the information processing device 10 evaluates the answer transmitted to the user terminal 20 in S106 based on the behavior indicated by the information received in S110.

[0100] In step 111 (S111), the information processing device 10 associates the answer transmitted to the user terminal 20 in S106 with the result of evaluating the answer in S110 and stores them in the answer storage unit 110. Note that the information processing device 10 can also further associate the answer transmitted to the user terminal 20 in S106 and the result of evaluating the answer in S110 with the attributes of the evaluator (i.e., the user 21 who performed the behavior that is the basis of the evaluation) and store them in the answer storage unit 110.

[0101] FIG. 8 is a sequence diagram of a process for extracting an answer based on a user attribute according to an embodiment of the present invention.

[0102] In step 201 (S201), the user terminal 20 transmits information for identifying the user 21 (for example, a user ID) to the information processing device 10. For example, S201 is executed when logging in to the service according to the present invention.

[0103] In step 202 (S202), the user 21 inputs a prompt into the user terminal 20.

[0104] In step 203 (S203), the user terminal 20 transmits the prompt input in S202 to the information processing device 10. The information processing device 10 receives the prompt from the user terminal 20.

[0105] In step 204 (S204), the information processing device 10 extracts, from the answers stored in the answer storage unit 110, answers that are answers to a prompt that is the same as or similar to the prompt received in S203, have a high rating, and have a high degree of similarity between the attributes of the evaluator and the attributes of the user 21 operating the user terminal 20. If the answer storage unit 110 does not have an answer to a prompt that is the same as or similar to the prompt received in S203, the information processing device 10 causes the large-scale language model to generate an answer to the prompt.

[0106] In step 205 (S205), the information processing device 10 transmits the response of S204 to the user terminal 20. The user terminal 20 receives the response from the information processing device 10.

[0107] In step 206 (S206), the user terminal 20 displays the response received in S205 on the display of the user terminal 20.

[0108] In step 207 (S207), the user 21 inputs into the user terminal 20 a request to regenerate the answer to the prompt input in S202.

[0109] In step 208 (S208), the user terminal 20 transmits a request to regenerate an answer to the prompt input in S207 to the information processing device 10. The information processing device 10 receives the request to regenerate an answer to the prompt from the user terminal 20.

[0110] In step 209 (S209), the information processing device 10 searches for a document corresponding to the prompt from among the documents stored in the document storage unit 108 based on the prompt received in S203.

[0111] In step 210 (S210), the information processing device 10 causes the large scale language model stored in the large scale language model storage unit 109 to generate an answer to the prompt based on the prompt received in S203 and the document searched in S209.

[0112] In step 211 (S211), the information processing device 10 transmits the answer generated by the large-scale language model in S211 to the user terminal 20. The user terminal 20 receives the answer from the information processing device 10.

[0113] In step 212 (S212), the user terminal 20 displays the answer received in S211 on the display of the user terminal 20.

[0114] In step 213 (S213), the user 21 views the answer displayed in S212 and then takes action in response to the answer.

[0115] In step 214 (S214), the user terminal 20 transmits information indicating the behavior performed by the user 21 in S213 to the information processing device 10. The information processing device 10 receives, from the user terminal 20, information indicating the behavior performed by the user 21.

[0116] In step 215 (S215), the information processing device 10 evaluates the answer transmitted to the user terminal 20 in S211 based on the behavior indicated by the information received in S214.

[0117] In step 216 (S216), the information processing device 10 associates the answer transmitted to the user terminal 20 in S211 with the result of evaluating the answer in S215, and stores them in the answer storage unit 110. Note that the information processing device 10 can also further associate the answer transmitted to the user terminal 20 in S211 and the result of evaluating the answer in S215 with the attributes of the evaluator (i.e., the user 21 who performed the behavior that is the basis of the evaluation), and store them in the answer storage unit 110.

[0118] <Effects> In one embodiment of the present invention, responses to prompts are evaluated based on the behavior of the user 21 operating the user terminal 20, so there are no drawbacks such as the user 21 not being able to enter an evaluation into the user terminal 20, or the user 21 not being able to enter an accurate evaluation. In one embodiment of the present invention, answers that are highly rated by evaluators with attributes similar to the attributes of the user 21 are used, so the user 21 can obtain the answer that he or she is looking for.

[0119] <Hardware configuration> 9 is a hardware configuration diagram of the information processing device 10 according to one embodiment of the present invention. The same applies to the user terminal 20.

[0120] The information processing device 10 can include a control unit 1001, a main memory unit 1002, an auxiliary memory unit 1003, an input unit 1004, an output unit 1005, and an interface unit 1006. Each of these will be described below.

[0121] The control unit 1001 is a processor (for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc.) that executes various programs installed in the auxiliary storage unit 1003.

[0122] The main memory unit 1002 includes a non-volatile memory (Read Only Memory (ROM)) and a volatile memory (Random Access Memory (RAM)). The ROM stores various programs, data, etc. required for the control unit 1001 to execute various programs installed in the auxiliary memory unit 1003. The RAM provides a working area into which the various programs installed in the auxiliary memory unit 1003 are expanded when executed by the control unit 1001.

[0123] The auxiliary storage unit 1003 is an auxiliary storage device that stores various programs and information used when the various programs are executed.

[0124] The input unit 1004 is an input device through which the operator of the information processing device 10 inputs various instructions to the information processing device 10 .

[0125] The output unit 1005 is an output device that outputs the internal state of the information processing device 10 and the like.

[0126] The interface unit 1006 is a communication device for connecting to a network and communicating with other devices.

[0127] Although the embodiments have been described above, changes in form and details are possible without departing from the spirit and scope of the claims. [Explanation of symbols]

[0128] 1. Information Processing Systems 10. Information processing equipment 20 User terminal 21 users 101 Receiving unit 102 Document Search Unit 103 Answer generation part 104 Transmitter 105 Evaluation Department 106 Response Management Department 107 Answer extraction part 108 Document Storage Unit 109 Large-scale language model memory unit 110 Answer storage section 111 User information storage unit 1001 control section 1002 Main memory 1003 Auxiliary storage unit 1004 Input section 1005 Output section 1006 Interface section

Claims

1. a receiving unit for receiving a prompt from a user terminal; a document retrieval unit that retrieves documents based on the prompt; an answer generator that causes a large-scale language model to generate an answer to the prompt based on the prompt and the document; a transmitting unit that transmits the response to the user terminal; an evaluation unit that evaluates the answer based on the behavior of the user operating the user terminal; a response management unit that associates the responses with the evaluation results and stores them; An information processing device comprising:

2. The information processing device according to claim 1 , wherein the answer management unit further associates the answers and the evaluation results with attributes of the user and stores them.

3. 3. The information processing device according to claim 1, wherein the evaluation unit evaluates the answer lower if an additional prompt is received from the user terminal within a predetermined time, or if a request to regenerate an answer to the prompt is received from the user terminal.

4. The information processing device according to claim 1 or 2, wherein the evaluation unit highly evaluates the answer when the source of the answer is viewed, when at least a portion of the answer is copied, or when a link to the answer is clicked.

5. a receiving unit for receiving a prompt from a user terminal; an answer storage unit storing answers to the prompts generated by a large-scale language model based on the prompts and documents retrieved based on the prompts; an answer extraction unit that extracts, from the answers stored in the answer storage unit, answers that are answers to prompts that are identical or similar to the prompt received from the user terminal, have a high evaluation, and have a high degree of similarity between the attributes of the evaluator and the attributes of the user operating the user terminal; a transmitting unit that transmits the extracted answer to the user terminal; An information processing device comprising:

6. The information processing device according to claim 5 , wherein the evaluation is based on an action of a user who operates the user terminal.

7. When the receiving unit receives a request from the user terminal to regenerate an answer to the prompt, The information processing apparatus according to claim 5 , further comprising an answer generation unit that causes a large-scale language model to generate an answer to the prompt based on the prompt and the document.

8. A method executed by an information processing device, receiving a prompt from a user terminal; retrieving documents based on the prompt; causing a large-scale language model to generate an answer to the prompt based on the prompt and the document; transmitting the response to the user terminal; Evaluating the answer based on the behavior of the user operating the user terminal; storing the answer and the result of the evaluation in association with each other; A method comprising:

9. A method executed by an information processing device, receiving a prompt from a user terminal; extracting, from among answers stored in an answer storage unit that stores answers to the prompt generated by a large-scale language model based on the prompt and the documents searched based on the prompt, answers that are answers to a prompt that is identical to or similar to the prompt received from the user terminal, have a high evaluation, and have a high degree of similarity between the attributes of the evaluator and the attributes of the user operating the user terminal; transmitting the extracted answer to the user terminal; A method comprising:

10. In the information processing device, receiving a prompt from a user terminal; retrieving documents based on the prompt; causing a large-scale language model to generate an answer to the prompt based on the prompt and the document; transmitting the response to the user terminal; Evaluating the answer based on the behavior of the user operating the user terminal; storing the answer and the result of the evaluation in association with each other; A program that executes the following.

11. In the information processing device, receiving a prompt from a user terminal; extracting, from among answers stored in an answer storage unit that stores answers to the prompt generated by a large-scale language model based on the prompt and the documents searched based on the prompt, answers that are answers to a prompt that is identical to or similar to the prompt received from the user terminal, have a high evaluation, and have a high degree of similarity between the attributes of the evaluator and the attributes of the user operating the user terminal; transmitting the extracted answer to the user terminal; A program that executes the following.

12. An information processing system including an information processing device and a user terminal, The information processing device includes: a receiving unit for receiving a prompt from the user terminal; a document retrieval unit that retrieves documents based on the prompt; an answer generator that causes a large-scale language model to generate an answer to the prompt based on the prompt and the document; a transmitting unit that transmits the response to the user terminal; an evaluation unit that evaluates the answer based on the behavior of the user operating the user terminal; a response management unit that associates the responses with the evaluation results and stores them; An information processing system comprising:

13. An information processing system including an information processing device and a user terminal, The information processing device includes: a receiving unit for receiving a prompt from the user terminal; an answer storage unit storing answers to the prompts generated by a large-scale language model based on the prompts and documents retrieved based on the prompts; an answer extraction unit that extracts, from the answers stored in the answer storage unit, answers that are answers to prompts that are identical or similar to the prompt received from the user terminal, have a high evaluation, and have a high degree of similarity between the attributes of the evaluator and the attributes of the user operating the user terminal; a transmitting unit that transmits the extracted answer to the user terminal; An information processing system comprising:

Images