Information processing device, information processing method, and information processing program

Abstract

To resolve such a problem that the imperfection of an answer of a generative model cannot be easily compensated for.SOLUTION: An information processing device includes: a reception section 131 for receiving a question of a user and an answer generated for the question by a generative model; a notification section 133 for notifying a terminal of an evaluator of a demand for evaluation of the answer received by the reception section 131; and an output section 135 for outputting the evaluation by the evaluator, which has been demanded through the notification of the notification section 133, to a terminal of the user.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] In recent years, information sharing services between users via networks such as the Internet have become popular. In this type of service, for example, a user (questioner) posts a question and another user (answerer) posts an answer, thereby sharing knowledge and wisdom between users. There are also services in which users ask questions to a model that generates sentences, rather than to each other, and the generative model then generates sentences as answers to the questions.

[0003] Furthermore, in recent years, the use of crowdsourcing has been increasing as a service that allows users to request work ("tasks") from an unspecified number of people via the Internet. In crowdsourcing, people who want to request a task ("owners") request the task on the service and pay a usage fee. Meanwhile, people who want to accept a task perform the task on the service and receive a reward. In particular, with the use of AI (Artificial Intelligence) technology, an increasing number of owners are using crowdsourcing to collect the learning data necessary for creating machine learning models. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-125146 [Patent Document 2] JP 2004-538572 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the above-mentioned conventional technology, it was not easy to compensate for incomplete answers from generative models. For example, answers to questions from generative models were sometimes not up-to-date, not fact-checked, or contrary to laws, regulations, or public order and morals. This made it difficult for users of generative models to determine how much they could trust the answers. Furthermore, when the answers from generative models to questions were incorrect, humans could not provide supplementary information, making it difficult to compensate for incomplete answers from generative models.

[0006] The present application has been made in view of the above, and aims to provide an information processing device, an information processing method, and an information processing program that can compensate for incompleteness in answers from generative models. [Means for solving the problem]

[0007] The information processing device according to the present application includes: a reception unit that receives a question from a user and an answer generated by a generative model in response to the question; a confirmation unit that confirms that the user has paid the usage fee after the response is received by the reception unit; a setting unit that sets the number of evaluators according to the usage fee of the user after it is confirmed that the user has paid the usage fee; a presentation unit that presents to the user a button for requesting the set number of evaluators; and a display unit that displays to the user a button for requesting the set number of evaluators when the user presses the button. Regarding the response received by the reception unit, The aforementioned Request for evaluation of the response , the set number of people a notification unit that notifies the terminal of the evaluator; The aforementioned Notified by the Notification Department The aforementioned In response to requests and the set number of people and an output unit that outputs the evaluation made by the evaluator to the terminal of the user. 。 [Effects of the Invention]

[0008] According to one aspect of the embodiment, it is possible to compensate for incompleteness in answers from a generative model. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an overview of an information processing method according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of the information processing device according to the embodiment. [Figure 3]FIG. 3 is a diagram showing an example of a screen for requesting an evaluator according to the embodiment. [Figure 4] FIG. 4 is a diagram showing an example of a screen on which an evaluator makes an evaluation according to the embodiment. [Figure 5] FIG. 5 is a diagram showing an example of a screen on which an evaluation according to the embodiment is output. [Figure 6] FIG. 6 is a flowchart illustrating an example of a processing procedure according to the embodiment. [Figure 7] FIG. 7 is a flowchart illustrating an example of a procedure for providing an evaluation by an evaluator according to the embodiment. [Figure 8] FIG. 8 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an information processing device, an information processing method, and an information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail with reference to the drawings. Note that the information processing device, the information processing method, and the information processing program according to the present application are not limited to these embodiments. Furthermore, the embodiments can be appropriately combined as long as the processing content is not contradictory. Furthermore, the same components in the following embodiments will be assigned the same reference numerals, and redundant explanations will be omitted.

[0011] [1. Overview of Information Processing] First, an example of information processing according to the embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram illustrating an example of an information processing method according to the embodiment, which is realized by an information processing device 100. In the example of FIG. 1, an overall system 1 includes a user terminal 10, an evaluator terminal 20, the information processing device 100, and an external server 200. The user terminal 10, the evaluator terminal 20, the information processing device 100, and the external server 200 are connected to each other via a network N so as to be able to communicate with each other via wired or wireless communication. Note that FIG. 1 is an example, and the number of user terminals 10 and evaluator terminals 20 is not limited to the number shown in FIG. 1. Note that a user is a person who asks a question to a generative model, and an evaluator is a person who evaluates whether the answer of the generative model is appropriate.

[0012] First, in the information processing device 100, a user of a generative model uses a user terminal 10 to ask a question to the generative model and requests an evaluator to evaluate whether the answer generated in response to the question is correct (step S1). For example, the generative model here is a sentence generation model that has learned from data published on the Web. For example, the generative model is a GPT (Generative Pre-trained Transformer) model that generates an answer to a question. Furthermore, for example, the information processing device 100 displays an answer from the generative model to a question, and the user can request the evaluator by pressing the button. Furthermore, for example, the information processing device 100 can display the button after confirming that the user has paid the usage fee. In this way, the information processing device 100 can set the number of evaluators according to the user's usage fee and make the request. Furthermore, the information processing device 100 can make the evaluation more accurate by setting multiple evaluators.

[0013] Next, the information processing device 100 receives a question from the user and an answer from the generative model to the question (step S2). For example, in addition to the question and answer, the information processing device 100 also receives information on the number of evaluators set by the user.

[0014] Next, the information processing device 100 selects an evaluator who is suitable for the request (step S3). For example, the information processing device 100 can select an evaluator by estimating the evaluator from pre-registered crowdsourcing data. Furthermore, for example, the information processing device 100 makes an estimation based on the attributes of the user or evaluator and the attributes of the question content. This allows the information processing device 100 to select an evaluator whose attributes are the same as or related to the attributes of the question. Next, the information processing device 100 transmits information about the request to the evaluator terminal 20 (step S4). For example, the information processing device 100 transmits information together with a message that the evaluator can choose whether or not to accept the request. Next, the information processing device 100 receives a request from the evaluator terminal 20. For example, the information processing device 100 can receive a message indicating that the evaluator has decided to accept the request.

[0015] Next, the information processing device 100 transmits the contents of the request to the evaluator terminal 20 and requests evaluations from the evaluators (step S6). Next, the information processing device 100 receives evaluation information from the evaluators (step S7). For example, the information processing device 100 receives evaluation information from all evaluators to whom the contents of the request have been transmitted.

[0016] Next, the information processing device 100 outputs the evaluation obtained from the evaluator to the user terminal 10 (step S8). For example, after the user confirms the evaluator's evaluation, the information processing device 100 can provide feedback to the evaluator who provided the evaluation and store the result. Based on this feedback, the information processing device 100 can select an evaluator for the next or subsequent request.

[0017] Next, the information processing device 100 can retrain the generative model using the evaluations obtained from the evaluators (step S9). For example, the information processing device 100 can provide the evaluations obtained from the evaluators to both the user terminal 10 and the developer of the generative model. Next, the information processing device 100 determines a reward to be awarded to the evaluator (step S10). For example, the information processing device 100 determines the reward based on the time until the evaluation of the question, the number of evaluations, the evaluation content, and feedback from users to the evaluator based on the evaluations of past evaluators. Next, the information processing device 100 notifies each party that made the request of information regarding the reward (step S11). For example, the information processing device 100 can notify the evaluator, the operator of the information processing system, and the developer of the generative model that a reward will be awarded.

[0018] Next, the information processing device 100 associates the reward amount with the evaluator's identification information and notifies the external server 200 that provides the reward (step S12). For example, the external server 200 that controls payments using electronic money can be notified. As a result, the information processing device 100 provides a reward to the evaluator (step S13). For example, the information processing device 100 can set a reward such as electronic money, points, or coupons. Furthermore, for example, if there are multiple evaluators, the information processing device 100 can set a value obtained by subtracting the usage fee for the information processing system from the usage fee paid by the user for operating the information processing system, and dividing the result by the number of evaluators.

[0019] Finally, the information processing device 100 collects the usage fee for the information processing system from the user (step S14). For example, the information processing device 100 can notify the external server 200 of the usage fee and the user's identification information in association with each other. Furthermore, for example, the information processing device 100 can vary the usage fee for the information processing system depending on the number and feedback from users to the evaluators.

[0020] This allows the information processing device 100 to compensate for incompleteness in the answer of the generative model. Therefore, the information processing device 100 is configured to request an evaluation from an evaluator, thereby enabling the user to check whether the answer of the generative model is up-to-date, whether the facts have been confirmed, and whether it violates laws, regulations, or public order and morals. Furthermore, the information processing device 100 allows the user to determine whether the answer of the generative model is appropriate after requesting an evaluation from an evaluator. This allows the information processing device 100 to allow the user to make a decision after checking both the answer of the generative model and the evaluation of the evaluator. Furthermore, the information processing device 100 also provides the evaluator's evaluation to the developer of the generative model, allowing the developer to retrain the generative model and improve the accuracy of the answers of the generative model to questions from users.

[0021] 2. Configuration of Information Processing Device Next, the configuration of the information processing device 100 will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. As shown in Fig. 2, the information processing device 100 according to the embodiment includes a user terminal 10, an evaluator terminal 20, a network N, a communication unit 110, a storage unit 120, and a control unit 130.

[0022] The information processing device 100 may be, for example, a personal computer (PC), a workstation (WS), a computer with server functions, etc. The information processing device 100 performs processing based on information transmitted from a user terminal 10 via a network N, for example.

[0023] The user terminal 10 or the evaluator terminal 20 is an information processing device used by a user or an evaluator of a generative model. For example, the information processing device may be a smartphone, a tablet terminal, a desktop PC, a notebook PC, a mobile phone, a PDA (Personal Digital Assistant), or the like. Note that the example shown in FIG. 1 illustrates a case where the user terminal 10 and the evaluator terminal 20 are smartphones.

[0024] The external server 200 is a server device for collecting usage fees from users and providing rewards to evaluators. For example, the external server 200 links the usage fees with the user's identification information and notifies them. Furthermore, for example, the external server 200 can link the amount of reward with the evaluator's identification information and provide the reward.

[0025] The network N connects the information processing device 100, the user terminal 10, and the evaluator terminal 20 to each other in a wired or wireless manner so that they can communicate with each other. If the network N is wired, it may be realized by Ethernet (registered trademark) defined in IEEE802.3. If the network N is wireless, it may be realized by a wireless LAN (Local Area Network) defined in IEEE802.11.

[0026] (Regarding the communication unit 110) The communication unit 110 is realized by, for example, a network interface card (NIC), etc. The communication unit 110 is connected to the network N by wire or wirelessly, and transmits and receives information between the user terminal 10 and the evaluator terminal 20, etc.

[0027] (Regarding the storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in Fig. 2, the storage unit 120 has a user information storage unit 121, an evaluator information storage unit 122, and an output result storage unit 123. The configuration of these units will be explained below in order.

[0028] (Regarding the user information storage unit 121) The user information storage unit 121 stores information about users who use generative models. For example, the user information storage unit 121 stores the user's attributes and the fact that the user has previously used a generative model to ask a question. Specifically, the user information storage unit 121 stores the content or attributes of the question by categorizing them. This makes it easy to select an evaluator with the same or related attributes as the content of the question for the user's question.

[0029] (Regarding the evaluator information storage unit 122) The evaluator information storage unit 122 stores information about evaluators. For example, the evaluator information storage unit 122 stores information about evaluators registered in crowdsourcing. This facilitates the selection of evaluators with attributes that are the same as or related to the attributes of the user's question. For example, the evaluator information storage unit 122 stores feedback from users to evaluators based on their past evaluations. For example, the evaluator information storage unit 122 can store information about whether the user's question and the generative model's answer to the question are correct, and if incorrect, whether the evaluator adds an answer and provides feedback about the extent to which the evaluator contributed to the user. For example, the evaluator information storage unit 122 stores information about the evaluator's rating being lowered if the evaluator provides an incorrect evaluation. For example, the evaluator information storage unit 122 stores information about the transmission of information along with a message that the evaluator can choose whether to accept the request.

[0030] (Regarding the output result storage unit 123) The output result storage unit 123 stores that the evaluation result has been output to the user terminal 10 by the evaluator. For example, the output result storage unit 123 stores that the result has been sent to the user terminal 10 or the developer of the generative model. This allows the generative model to be retrained, thereby improving the accuracy of the generative model's answers to questions from users.

[0031] (Regarding the control unit 130) Next, the control unit 130 in Fig. 2 will be described. The control unit 130 is a controller that controls the information processing device 100, and is stored in the storage unit 120 of the information processing device 100 by, for example, a CPU (Central Processing Unit) or the like. The control unit 130 is realized by reading out various programs and executing them using RAM as a working area. The control unit 130 is also a controller, and may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0032] As shown in Fig. 2, the control unit 130 has a reception unit 131, an estimation unit 132, a notification unit 133, a reward setting unit 134, and an output unit 135. The control unit 130 realizes these functions by reading out a program from the storage unit 120 and executing it using the RAM as a work area, thereby realizing or executing the functions and actions of the information processing described below. These processes will be explained in order below.

[0033] (Regarding the reception unit 131) The receiving unit 131 receives a question from a user and an answer generated by a generative model in response to the question. For example, the receiving unit 131 receives the question and the answer when they are both available, and does not receive the question if either one is not available. Furthermore, for example, the receiving unit 131 displays a button for making a request to the evaluator when the answer is returned from the generative model, and the user can receive the question and the answer by pressing the displayed button. Furthermore, for example, the receiving unit 131 displays a button for making a request to the evaluator after the answer from the generative model to the question is displayed, and the request to the evaluator is received by pressing the button.

[0034] Here, an example of a screen received by the reception unit 131 will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of a screen received by the reception unit 131 of the information processing device 100 according to the embodiment. Fig. 3 illustrates a screen on which a question is asked to a generative model on the user terminal 10. In the example of Fig. 3, the screen of the user terminal 10 has items such as a question 501, an answer 502, and a button 503.

[0035] Question 501 indicates the content of a question that a user asks the generative model using the user terminal 10. Answer 502 indicates the content of the answer of the generative model to the question entered by the user. Button 503 is a button that the user presses to request an evaluator to evaluate whether the answer of the generative model is appropriate. For example, button 503 may be implemented by being called by an API (Application Programming Interface). Also, for example, button 503 may display a button for requesting an evaluator after the answer of the generative model to the question is displayed, and the user can press the button to request an evaluator. Also, for example, button 503 may display a button to determine the number of evaluators after it is confirmed that the user has paid the usage fee.

[0036] (Regarding the estimation unit 132) The estimation unit 132 estimates the attributes of the user or evaluator and the attributes of the question content, respectively. For example, the estimation unit 132 estimates the attributes of the user, the evaluator, and the question content, and thereby can request an evaluation from an evaluator whose attributes are the same as or related to the attributes of the question content. Furthermore, for example, the estimation unit 132 can estimate the attributes of the evaluator from data registered in advance in crowdsourcing. Furthermore, for example, the estimation unit 132 estimates feedback from users to the evaluator based on the time until the evaluation on the request, the number of evaluations, the evaluation content, and past evaluations by the evaluator. This allows users to be ranked.

[0037] (Regarding the notification unit 133) The notification unit 133 notifies the terminal of the evaluator of a request to evaluate the answer received by the reception unit 131. For example, the notification unit 133 notifies the terminal of an evaluator who has attributes that are the same as or related to the attributes of the question content, based on the attributes of the user or evaluator estimated by the estimation unit 132 and the attributes of the question content. Also, for example, the notification unit 133 notifies, together with the request, predetermined guidelines that serve as standards for evaluating the answer. Also, for example, the notification unit 133 notifies the terminal of the evaluator who performed the evaluation of information regarding rewards.

[0038] Furthermore, for example, the notification unit 133 can change the content of predetermined guidelines that serve as the basis for evaluating answers for each question and notify the request at the same time. Specifically, for example, the notification unit 133 can estimate the guidelines desired by the user from the content of the question, attributes, the answer of the generative model, etc., and notify the corresponding guidelines to the evaluator terminal 20. Furthermore, for example, when the user makes a request to an evaluator, the notification unit 133 can allow the user to set guidelines and notify the evaluator terminal 20.

[0039] An example of a screen that the notification unit 133 requests from the evaluator will now be described with reference to Fig. 4. Fig. 4 shows an example of a screen that the notification unit 133 of the information processing device 100 according to the embodiment requests the evaluator to make an evaluation. Fig. 4 illustrates an example of a screen that is displayed on the evaluator terminal 20. In the example of Fig. 4, the screen of the evaluator terminal 20 has items such as a check box 505 and a text box 506.

[0040] The checkbox 505 is a location where the evaluator can select whether the answer of the generative model is valid or not. For example, the checkbox 505 displays multiple options. The textbox 506 is a location where the evaluator can enter the reason for the answer in the checkbox 505 or a suggested correction. For example, if the evaluator has additional information that he or she wants to enter regarding the answer of the generative model, the textbox 506 can be used to enter the reason for the answer in the checkbox 505 or a suggested correction, regardless of which option is selected in the checkbox 505.

[0041] (Regarding the remuneration setting unit 134) The reward setting unit 134 sets a reward for the evaluator for making a request. For example, the reward setting unit 134 can set a reward based on the time until the evaluation of the request, the number of evaluations, the evaluation content, and the evaluator's past evaluations, as well as feedback from the user to the evaluator based on the evaluation of the user by the user or the operator of the information processing system. Furthermore, for example, the reward setting unit 134 can set a reward such as electronic money, points, or coupons. Furthermore, for example, if there are multiple evaluators, the reward setting unit 134 can set a value obtained by subtracting the usage fee for the information processing system from the usage fee paid by the user to the operator of the information processing system, and dividing the result by the number of evaluators.

[0042] (Regarding the output unit 135) The output unit 135 outputs the evaluation made by the evaluator for the request notified by the notification unit 133 to the user's terminal. For example, the output unit 135 outputs the evaluation made by the evaluator for the request notified by the notification unit 133 to the user's terminal or the developer of the generative model. Furthermore, for example, the output unit 135 can output feedback to the evaluator to the evaluator terminal 20 after the user confirms the evaluation from the evaluator. This allows the information processing device 100 to refer to the feedback and select an evaluator for subsequent requests. Furthermore, the output unit 135 can output the evaluator's evaluation to both the user's terminal 10 and the developer of the generative model, for example. This allows the generative model to be retrained based on the evaluation, thereby improving the accuracy of the generative model's answers to user questions.

[0043] Here, an example of a screen output to a user by the output unit 135 will be described with reference to Fig. 5. Fig. 5 is an example of a screen on which an evaluation from an evaluator is output to a user by the output unit 135 of the information processing device 100 according to the embodiment. Fig. 5 illustrates a screen output to the user terminal 10. In the example of Fig. 5, the screen of the user terminal 10 has an item called evaluation 507.

[0044] The evaluation 507 is a screen showing examples of evaluations from evaluators. For example, the evaluation 507 displays the evaluations of all evaluators on the screen, and also displays suggested corrections from the evaluators. The evaluation 507 in this case may be a numerical value or a sentence.

[0045] This allows the information processing device 100 to compensate for incompleteness in the answer of the generative model. Therefore, the information processing device 100 is configured to request an evaluation from an evaluator, thereby enabling the user to check whether the answer of the generative model is up-to-date, whether the facts have been confirmed, and whether it violates laws, regulations, or public order and morals. Furthermore, the information processing device 100 allows the user to determine whether the answer of the generative model is appropriate after requesting an evaluation from an evaluator. This allows the information processing device 100 to allow the user to make a decision after checking both the answer of the generative model and the evaluation of the evaluator. Furthermore, the information processing device 100 also provides the evaluator's evaluation to the developer of the generative model, allowing the developer to retrain the generative model and improve the accuracy of the answers of the generative model to questions from users.

[0046] [3. Information processing flow] Next, the procedure of information processing according to the embodiment will be described with reference to Fig. 6 and Fig. 7. Fig. 6 and Fig. 7 are flowcharts showing an example of information processing according to the embodiment. Below, the procedure of information processing according to the embodiment will be described with reference to the flowcharts shown in Fig. 6 and Fig. 7.

[0047] First, the procedure for processing to request an evaluation from the evaluator will be described with reference to Fig. 6. When the receiving unit 131 of the information processing device 100 receives a user's question and a generative model's answer to the question (step S101, Yes), it requests an evaluation from the evaluator (step S102). On the other hand, when the receiving unit 131 does not receive a user's question and a generative model's answer to the question (step S101, No), it waits until it receives both the question and the answer.

[0048] Next, the notification unit 133 of the information processing device 100 requests evaluations from the evaluators (step S102). The evaluation procedure will be described in detail with reference to FIG. 7. Next, the notification unit 133 of the information processing device 100 notifies whether replies have been received from all evaluators (step S103). If replies have been received from all evaluators (step S103, Yes), the notification unit 133 of the information processing device 100 outputs the evaluators' evaluations to the user and the developer of the generative model. On the other hand, if replies have not been received from all evaluators (step S103, No), the notification unit 133 waits until replies are received from all evaluators. Finally, the output unit 135 of the information processing device 100 outputs the evaluations obtained from the evaluators to the user terminal 10 and the developer of the generative model.

[0049] Next, the procedure of the process when an evaluator performs an evaluation will be described with reference to Fig. 7. First, the information processing device 100 notifies the evaluator terminal 20 of a request for evaluation based on the user's question received by the receiving unit 131 and the generative model's answer to the question (step S201). If a request notification has been sent to the evaluator terminal 20 (step S201, Yes), the notification unit 133 of the information processing device 100 checks one of the check boxes displayed on the evaluation screen (step S202). On the other hand, if a request notification has not been sent to the evaluator terminal 20 (step S201, No), the notification unit 133 of the information processing device 100 waits until a request notification arrives at the evaluator terminal 20.

[0050] Next, if any one of the check boxes is checked on the request screen notified by the notification unit 133 of the information processing device 100 (Yes in step S202), the evaluator enters the reason for the evaluation or a correction proposal in a text box (step S203). On the other hand, if any one of the check boxes is not checked on the request screen notified by the notification unit 133 of the information processing device 100 (No in step S202), the process waits until a check box is checked.

[0051] This allows the information processing device 100 to compensate for incompleteness in the answers of the generative model. Therefore, the information processing device 100 is configured to request an evaluation from an evaluator, thereby making it possible to confirm whether the answers of the generative model are the latest information, whether the facts have been confirmed, and whether they violate laws and regulations or public order and morals. Furthermore, the information processing device 100 allows a user of a generative model to request an evaluation from an evaluator and then determine whether the answers of the generative model are appropriate. Furthermore, the information processing device 100 also provides the developer of the generative model with the evaluator's evaluation, thereby enabling the developer to retrain the generative model and improve the accuracy of the answers of the generative model to questions from users.

[0052] [4. Hardware Configuration] The information processing device 100 according to the above-described embodiment is realized, for example, by a computer 1000 configured as shown in Fig. 8. Fig. 8 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 100. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which a calculation device 1030, a primary storage device 1040, a secondary storage device 1050, an output IF (Interface) 1060, an input IF 1070, and a network IF 1080 are connected via a bus 1090.

[0053] The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The primary storage device 1040 is a memory device, such as a RAM, that temporarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device that stores data used by the arithmetic device 1030 for various calculations and various databases, and is realized by a ROM (Read Only Memory), an HDD (Hard Disk Drive), a flash memory, or the like.

[0054] The output IF 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various types of information, such as a monitor or a printer, and is realized by a connector conforming to a standard such as USB (Universal Serial Bus), DVI (Digital Visual Interface), or HDMI (High Definition Multimedia Interface), etc. The input IF 1070 is an interface for receiving information from various input devices 1020, such as a mouse, keyboard, scanner, etc., and is realized by a USB, etc.

[0055] The input device 1020 may be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disk), a DVD (Digital Versatile Disk), or a PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magnet-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. The input device 1020 may also be an external storage medium such as a USB memory.

[0056] The network IF 1080 receives data from other devices via the network N and sends the data to the arithmetic device 1030, and also transmits data generated by the arithmetic device 1030 to other devices via the network N.

[0057] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0058] For example, when the computer 1000 functions as the information processing device 100, the arithmetic unit 1030 of the computer 1000 realizes the functions of the control unit 130 of the information processing device 100 by executing a program loaded onto the primary storage device 1040.

[0059] [5. Composition and Effects] The information processing device 100 according to the present disclosure includes a receiving unit 131 that receives a user's question and an answer generated by a generative model for the question, and a notification unit 133 that notifies an evaluator's terminal of a request to evaluate the answer received by the receiving unit 131; The system further includes an output unit 135 that outputs the evaluation made by the evaluator for the request notified by the notification unit 133 to the user's terminal.

[0060] This allows the information processing device 100 to compensate for incompleteness in the answer of the generative model. Therefore, the information processing device 100 is configured to request an evaluation from an evaluator, thereby enabling the user to check whether the answer of the generative model is up-to-date, whether the facts have been confirmed, and whether it violates laws, regulations, or public order and morals. Furthermore, the information processing device 100 allows the user to determine whether the answer of the generative model is appropriate after requesting an evaluation from an evaluator. This allows the information processing device 100 to allow the user to make a decision after checking both the answer of the generative model and the evaluation of the evaluator. Furthermore, the information processing device 100 also provides the evaluator's evaluation to the developer of the generative model, allowing the developer to retrain the generative model and improve the accuracy of the answers of the generative model to questions from users.

[0061] The estimation unit 132 of the information processing device 100 according to the present disclosure further includes an estimation unit 132 that estimates the attributes of the user or evaluator and the attributes of the question content, respectively, and the notification unit 133 notifies the request to the terminal of the evaluator who has attributes that are the same as or related to the attributes of the question content, based on the attributes of the user or evaluator and the attributes of the question content estimated by the estimation unit 132.

[0062] According to this configuration, the information processing device 100 is configured so that the user can determine whether the answer of the generative model is appropriate after requesting an evaluation from the evaluator. This allows the information processing device 100 to allow the user to make a decision after checking both the answer of the generative model and the evaluation by the evaluator.

[0063] The notification unit 133 of the information processing device 100 according to the present disclosure notifies, together with the request, predetermined guidelines that serve as criteria for evaluating the response.

[0064] According to this configuration, the information processing device 100 can ask the evaluator to evaluate the answer of the generative model, thereby making it possible to confirm whether the answer is the latest information, whether the facts have been confirmed, and whether it violates laws and regulations or public order and morals.

[0065] The above describes the embodiments of the present application in detail based on the drawings, but this is merely an example, and the present invention can be implemented in other forms that include the embodiments described in the Disclosure of the Invention section and that have been modified and improved in various ways based on the knowledge of those skilled in the art.

[0066] Furthermore, the above-mentioned "section, module, unit" can be read as "means" or "circuit," etc. For example, the reception unit 131 can be read as reception means or reception circuit. [Explanation of symbols]

[0067] 1. Overall System 10 User terminal 20 Evaluator terminal 100 Information processing device 110 Communications Department 120 Storage section 121 User information storage unit 122 Evaluator information storage unit 123 Output result storage unit 130 Control Unit 131 Reception 132 Estimation Department 133 Notification Department 134 Remuneration Setting Department 135 Output section N Network N

Claims

1. a reception unit that receives a question from a user and an answer generated by the generative model in response to the question; a confirmation unit that confirms that the user has paid the usage fee after the response is accepted by the acceptance unit; a setting unit that sets the number of evaluators according to the user's usage fee after it is confirmed that the user has paid the usage fee; a presentation unit that presents to the user a button for requesting the set number of evaluators; a notification unit that, when the user presses the button, notifies terminals of a set number of evaluators of a request to evaluate the answer received by the reception unit; and an output unit that outputs evaluations made by a set number of the evaluators in response to the request notified by the notification unit to a terminal of the user; An information processing device comprising:

2. an estimation unit that estimates the attributes of the user or the evaluator and the attributes of the content of the question, The information processing device according to claim 1, characterized in that the notification unit notifies the request to a terminal of an evaluator who has attributes that are the same as or related to the attributes of the content of the question, based on the attributes of the user or the evaluator estimated by the estimation unit and the attributes of the content of the question.

3. a reward setting unit for setting a reward for the evaluator for making the request; The information processing apparatus according to claim 1 , further comprising:

4. The information processing apparatus according to claim 1 , wherein the notification unit notifies, together with the request, a predetermined guideline that is a standard for evaluation of the answer.

5. The information processing device according to claim 1 , wherein the output unit outputs the evaluation made by the evaluator for the request notified by the notification unit to a terminal of the user or a developer of the generative model.

6. The information processing device according to claim 1 , wherein the notification unit notifies the terminal of the evaluator who made the evaluation of information related to remuneration.

7. An information processing method executed by an information processing device, a receiving step of receiving a question from a user and an answer generated by the generative model for the question; a confirmation step of confirming that the user has paid the usage fee after the response is accepted by the accepting step; a setting step of setting the number of evaluators according to the user's usage fee after it is confirmed that the user has paid the usage fee; a presentation step of presenting to the user a button for requesting the set number of evaluators; a notification step of, when the user presses the button, notifying terminals of a set number of evaluators of a request to evaluate the answer accepted by the accepting step; an output step of outputting evaluations made by a set number of the evaluators in response to the request notified by the notification step to a terminal of the user; An information processing method comprising:

8. a receiving step of receiving a question from a user and an answer generated by the generative model for the question; a confirmation step of confirming that the user has paid the usage fee after the response is accepted by the acceptance step; a setting step of setting the number of evaluators according to the user's fee after it is confirmed that the user has paid the fee; a presentation step of presenting to the user a button for requesting the set number of evaluators; a notification step of notifying terminals of a set number of evaluators of a request to evaluate the answer accepted in the acceptance step when the user presses the button; an output step of outputting evaluations made by a set number of the evaluators for the request notified by the notification step to a terminal of the user; An information processing program that causes a computer to execute the above.

Images