Information processing device, information processing method, and information processing program

Abstract

To resolve such a problem that the imperfection of an answer of a generative model cannot be easily compensated for.SOLUTION: An information processing device includes: a reception section 131 for receiving a question of a user and an answer generated for the question by a generative model; a notification section 133 for notifying a terminal of an evaluator of a demand for evaluation of the answer received by the reception section 131; and an output section 135 for outputting the evaluation by the evaluator, which has been demanded through the notification of the notification section 133, to a terminal of the user.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] In recent years, information sharing services between users via networks such as the Internet have become popular. In this type of service, for example, a user (questioner) posts a question and another user (answerer) posts an answer, thereby sharing knowledge and wisdom between users. There is also a service in which a user asks a question to a model that generates sentences, rather than to other users, and the generative model generates a sentence as an answer to the question.

[0003] Furthermore, in recent years, the use of crowdsourcing has been increasing as a service that allows users to request work ("tasks" as appropriate) from an unspecified number of people via the Internet. In crowdsourcing, people who want to request a task ("owners" as appropriate) request the task on the service and pay a usage fee. Meanwhile, people who want to receive a task carry out the task on the service and receive a reward. In particular, with the use of AI (Artificial Intelligence) technology, an increasing number of owners are using crowdsourcing to collect learning data necessary for creating machine learning models. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-125146 A [Patent Document 2] JP 2004-538572 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the above-mentioned conventional technology, it was not easy to compensate for incompleteness of the answer of the generative model. For example, the answer to the question of the generative model may not be the latest information, may not have been fact-checked, or may violate laws, regulations, or public order and morals. Therefore, it was difficult for the user of the generative model to judge how much the answer can be trusted, and when the answer to the question of the generative model was incorrect, a human could not supplement it, so it was not easy to compensate for the incompleteness of the answer of the generative model.

[0006] The present application has been made in consideration of the above, and aims to provide an information processing device, an information processing method, and an information processing program that can compensate for incompleteness in answers of a generative model. [Means for solving the problem]

[0007] The information processing device according to the present application is characterized in that it comprises: a reception unit that receives a user's question and an answer generated by a generative model to the question; a notification unit that notifies an evaluator's terminal of a request to evaluate the answer received by the reception unit; and an output unit that outputs, to the user's terminal, the evaluation made by the evaluator in response to the request notified by the notification unit. Effect of the Invention

[0008] According to one aspect of the embodiment, it is possible to compensate for incompleteness in answers of a generative model. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an overview of an information processing method according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of the information processing device according to the embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of a screen for making a request to an evaluator according to the embodiment. [Figure 4] FIG. 4 is a diagram showing an example of a screen on which an evaluator makes an evaluation according to the embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of a screen on which an evaluation is output according to the embodiment. [Figure 6] FIG. 6 is a flowchart illustrating an example of a processing procedure according to the embodiment. [Figure 7] FIG. 7 is a flowchart illustrating an example of a procedure for providing an evaluation by an evaluator according to the embodiment. [Figure 8] FIG. 8 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the information processing device, the information processing method, and the information processing program according to the present application (hereinafter, referred to as "embodiments") will be described in detail with reference to the drawings. Note that the information processing device, the information processing method, and the information processing program according to the present application are not limited to these embodiments. In addition, each embodiment can be appropriately combined as long as the processing contents are not contradictory. In addition, the same parts in each embodiment below are given the same reference numerals, and duplicated explanations will be omitted.

[0011] [1. Overview of Information Processing] First, an example of information processing according to the embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram showing an example of an information processing method according to the embodiment, which is realized by an information processing device 100. In the example of FIG. 1, an overall system 1 includes a user terminal 10, an evaluator terminal 20, an information processing device 100, and an external server 200. In addition, the user terminal 10, the evaluator terminal 20, the information processing device 100, and the external server 200 are each connected to be able to communicate with each other via a network N by wire or wirelessly. Note that FIG. 1 is an example, and the number of user terminals 10 and evaluator terminals 20 is not limited to the number shown in FIG. 1. Note that a user is a person who asks a question to a generative model, and an evaluator is a person who evaluates whether an answer of the generative model is appropriate or not.

[0012] First, the information processing device 100 allows a user of the generative model to use the user terminal 10 to ask a question to the generative model, and to request an evaluator to evaluate whether or not the answer generated to the question is correct (step S1). For example, the generative model here is a sentence generation model that has learned data published on the Web. For example, the generative model is a GPT (Generative Pre-trained Transformer) model that generates an answer to a question. For example, the information processing device 100 displays a button for making a request to an evaluator after the answer of the generative model to the question is displayed, and the user can make a request to the evaluator by pressing the button. For example, the information processing device 100 can display the button after it is confirmed that the user has paid the usage fee. This allows the information processing device 100 to set the number of evaluators according to the usage fee of the user and make a request. Furthermore, the information processing device 100 can make the evaluation more accurate by setting multiple evaluators.

[0013] Next, the information processing device 100 accepts a question from the user and an answer from the generative model to the question (step S2). For example, in addition to the question and the answer, the information processing device 100 also accepts information on the number of evaluators set by the user.

[0014] Next, the information processing device 100 selects an evaluator who is suitable for the request (step S3). For example, the information processing device 100 can estimate the evaluator from preregistered crowdsourcing data and select the evaluator. Also, for example, the information processing device 100 performs estimation based on the attributes of the user or evaluator and the attributes of the question content. In this way, the information processing device 100 can select an evaluator who has the same or related attributes as the question. Next, the information processing device 100 transmits information of the request to the evaluator terminal 20 (step S4). For example, the information processing device 100 transmits information together with a message that the evaluator can choose whether or not to accept the request. Next, the information processing device 100 accepts a request from the evaluator terminal 20. For example, the information processing device 100 can accept a message in which the evaluator has decided to accept the request.

[0015] Next, the information processing device 100 transmits the contents of the request to the evaluator terminal 20 and requests the evaluators to evaluate the items (step S6). Next, the information processing device 100 receives information on the evaluations from the evaluators (step S7). For example, the information processing device 100 receives information on the evaluations from all the evaluators to which the request contents have been transmitted.

[0016] Next, the information processing device 100 outputs the evaluation obtained from the evaluator to the user terminal 10 (step S8). For example, after the user confirms the evaluator's evaluation, the information processing device 100 can provide feedback to the evaluator who provided the evaluation and store the result. Based on this feedback, the information processing device 100 can select an evaluator for the next or subsequent request.

[0017] Next, the information processing device 100 can re-learn the generative model using the evaluation obtained from the evaluator (step S9). For example, the information processing device 100 can provide the evaluation obtained from the evaluator to both the user terminal 10 and the developer of the generative model. Next, the information processing device 100 determines the reward to be given to the evaluator (step S10). For example, the information processing device 100 determines the reward based on the period until the evaluation of the question, the number of evaluations, the evaluation content, and feedback from users to the evaluator based on the evaluations of the past evaluators. Next, the information processing device 100 notifies each of the places that have made the request of information regarding the reward (step S11). For example, the information processing device 100 can notify the evaluator, the operation of the information processing system, and the developer of the generative model that a reward will be given.

[0018] Next, the information processing device 100 links the remuneration amount with the identification information of the evaluator and notifies the external server 200 that provides the remuneration (step S12). For example, the external server 200 that controls payments by electronic money can be notified. As a result, the information processing device 100 provides a remuneration to the evaluator (step S13). For example, the information processing device 100 can set a remuneration such as electronic money, points, or coupons. Also, for example, when there are multiple evaluators, the information processing device 100 can set a value obtained by subtracting the usage fee for the information processing system from the usage fee paid by the user for the operation of the information processing system, and dividing the result by the number of evaluators.

[0019] Finally, the information processing device 100 collects the usage fee for the information processing system from the user (step S14). For example, the information processing device 100 can notify the external server 200 of the usage fee and the user's identification information in association with each other. Also, for example, the information processing device 100 can vary the usage fee for the information processing system depending on the number of users' feedback to the evaluators.

[0020] This allows the information processing device 100 to compensate for incompleteness of the answer of the generative model. Therefore, the information processing device 100 makes it possible to check whether the answer of the generative model is the latest information, whether the facts have been confirmed, and whether it is in violation of laws and regulations or public order and morals by requesting an evaluation from the evaluator. Furthermore, the information processing device 100 makes it possible for a user to determine whether the answer of the generative model is appropriate or not after requesting an evaluation from the evaluator. This allows the information processing device 100 to make a judgment after the user has confirmed both the answer of the generative model and the evaluation of the evaluator. Furthermore, the information processing device 100 also provides the evaluation of the evaluator to the developer of the generative model, allowing the developer to re-learn the generative model and to improve the accuracy of the answers of the generative model to questions from users.

[0021] 2. Configuration of Information Processing Device Next, the configuration of the information processing device 100 will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. As shown in Fig. 2, the information processing device 100 according to the embodiment includes a user terminal 10, an evaluator terminal 20, a network N, a communication unit 110, a storage unit 120, and a control unit 130.

[0022] The information processing device 100 may be, for example, an information processing device such as a PC (Personal Computer), a WS (Work Station), a computer having a server function, etc. The information processing device 100 performs processing based on information transmitted from a user terminal 10 via a network N, for example.

[0023] The user terminal 10 or the evaluator terminal 20 is an information processing device used by a user or an evaluator of a generative model. For example, the information processing device may be a smartphone, a tablet terminal, a desktop PC, a notebook PC, a mobile phone, a PDA (Personal Digital Assistant), etc. In the example shown in FIG. 1, the user terminal 10 and the evaluator terminal 20 are smartphones.

[0024] The external server 200 is a server device for collecting usage fees from users and providing rewards to evaluators. For example, the external server 200 links the usage fees with the identification information of the users and notifies them. Also, for example, the external server 200 can link the amount of reward with the identification information of the evaluators and provide rewards.

[0025] The network N connects the information processing device 100, the user terminal 10, and the evaluator terminal 20 in a wired or wireless manner so that they can communicate with each other. If the network N is wired, it may be realized by Ethernet (registered trademark) (ETHERNET (registered trademark)) defined in IEEE802.3. If the network N is wireless, it may be realized by a wireless LAN (Local Area Network) defined in IEEE802.11.

[0026] (Regarding communication unit 110) The communication unit 110 is realized by, for example, a network interface card (NIC) etc. The communication unit 110 is connected to the network N by wire or wirelessly, and transmits and receives information between the user terminal 10 and the evaluator terminal 20 etc.

[0027] (Regarding the storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in Fig. 2, the storage unit 120 has a user information storage unit 121, an evaluator information storage unit 122, and an output result storage unit 123. The configuration of these units will be described below in order.

[0028] (Regarding the user information storage unit 121) The user information storage unit 121 stores information about users who use generative models. For example, the user information storage unit 121 stores the attributes of the users and the fact that the users have used generative models in the past to ask questions. Specifically, the user information storage unit 121 stores the content or attributes of the questions by categorizing them. This makes it easy to select an evaluator who has the same or related attributes as the content of the question in response to the user's question.

[0029] (Regarding the evaluator information storage unit 122) The evaluator information storage unit 122 stores information on the evaluator. For example, the evaluator information storage unit 122 stores the period until the evaluation of the question, the number of evaluations, the evaluation content, the attributes of the evaluator, and the like. Also, for example, the evaluator information storage unit 122 stores information on the evaluator registered in crowdsourcing. This makes it easy to select an evaluator that is the same as or related to the attributes of the content of the user's question. Also, for example, the evaluator information storage unit 122 stores feedback from the user to the evaluator based on the past evaluation of the evaluator. This means, for example, that the evaluator information storage unit 122 can store whether the user's question and the answer of the generative model to the question are correct or not, and if not, that the evaluator adds an answer and provides feedback on the extent to which the evaluator was able to contribute to the user. Also, for example, the evaluator information storage unit 122 stores that if the evaluator provides an incorrect evaluation, the evaluator's evaluation will be lowered. Also, for example, the evaluator information storage unit 122 stores that information has been transmitted together with a message that the evaluator can choose whether or not to accept the request.

[0030] (Regarding the output result storage unit 123) The output result storage unit 123 stores that the result of the evaluation has been output to the user terminal 10 by the evaluator. For example, the output result storage unit 123 stores that the result has been transmitted to the user terminal 10 or the developer of the generative model. This allows the generative model to be re-trained, thereby improving the accuracy of the generative model's answers to questions from users.

[0031] (Regarding the control unit 130) Next, the control unit 130 in Fig. 2 will be described. The control unit 130 is a controller that controls the information processing device 100, and is stored in the storage unit 120 of the information processing device 100 by, for example, a CPU (Central Processing Unit) or the like. Also, it is realized by reading out various programs and executing them using a RAM as a working area. Also, the control unit 130 is a controller, and may be realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0032] 2, the control unit 130 has a reception unit 131, an estimation unit 132, a notification unit 133, a reward setting unit 134, and an output unit 135. The control unit 130 realizes these functions by reading out a program from the storage unit 120 and executing it using the RAM as a working area, and realizes or executes the functions and actions of the information processing described below. Below, these processes will be explained in order.

[0033] (Regarding the reception unit 131) The reception unit 131 receives a question from a user and an answer generated by the generative model to the question. For example, the reception unit 131 receives the question and the answer when they are ready, and does not receive the question if either one is not ready. For example, the reception unit 131 displays a button for making a request to the evaluator when the answer is returned from the generative model, and the user can receive the question and the answer by pressing the displayed button. For example, the reception unit 131 displays a button for making a request to the evaluator after the answer from the generative model to the question is displayed, and receives the request to the evaluator by pressing the button.

[0034] Here, an example of a screen received by the receiving unit 131 will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of a screen received by the receiving unit 131 of the information processing device 100 according to the embodiment. Fig. 3 illustrates a screen on which a question is asked to a generative model by the user terminal 10. In the example of Fig. 3, the screen of the user terminal 10 has items such as a question 501, an answer 502, and a button 503.

[0035] Question 501 indicates the content of a question that a user asks the generative model using user terminal 10. Answer 502 indicates the content of the answer of the generative model to the question input by the user. Button 503 is a button for the user to request an evaluator to evaluate whether the answer of the generative model is appropriate or not. For example, button 503 may be realized by calling it up with an API (Application Programming Interface). Also, for example, button 503 may display a button for making a request to an evaluator after the answer of the generative model to the question is displayed, and the user can make a request to the evaluator by pressing the button. Also, for example, button 503 may display a button to determine the number of evaluators after it is confirmed that the user has paid the usage fee.

[0036] (Regarding the estimation unit 132) The estimation unit 132 estimates the attributes of the user or evaluator and the attributes of the question content, respectively. For example, the estimation unit 132 can request an evaluation from an evaluator whose attributes are the same as or related to the attributes of the question content by estimating the attributes of the user, the evaluator, and the question content. Also, for example, the estimation unit 132 can estimate the attributes of the evaluator from data registered in advance in crowdsourcing. Also, for example, the estimation unit 132 estimates feedback from users to the evaluator based on the period until the evaluation on the request, the number of evaluations, the evaluation content, and the evaluations of the evaluators in the past. This makes it possible to rank the users.

[0037] (Regarding the notification unit 133) The notification unit 133 notifies the terminal of the evaluator of a request to evaluate the answer accepted by the acceptance unit 131. For example, the notification unit 133 notifies the terminal of an evaluator who has attributes that are the same as or related to the attributes of the content of the question, based on the attributes of the user or evaluator estimated by the estimation unit 132 and the attributes of the content of the question. Also, for example, the notification unit 133 notifies, together with the request, a predetermined guideline that serves as a standard for evaluating the answer. Also, for example, the notification unit 133 notifies the terminal of the evaluator who performed the evaluation of information regarding the reward.

[0038] Furthermore, for example, the notification unit 133 can change the content of a predetermined guideline that is the standard for evaluating an answer for each question and notify the request at the same time. Specifically, for example, the notification unit 133 can estimate a guideline desired by a user from the content of the question, attributes, the answer of the generative model, and the like, and notify the corresponding guideline to the evaluator terminal 20. Furthermore, for example, when a user makes a request to an evaluator, the notification unit 133 can allow the user to set a guideline and notify the evaluator terminal 20.

[0039] Here, an example of a screen on which the notification unit 133 requests the evaluator to make an evaluation will be described with reference to Fig. 4. Fig. 4 is an example of a screen on which the evaluator is requested to make an evaluation by the notification unit 133 of the information processing device 100 according to the embodiment. Fig. 4 illustrates an example of a screen displayed on the evaluator terminal 20. In the example of Fig. 4, the screen of the evaluator terminal 20 has items such as a check box 505 and a text box 506.

[0040] The checkbox 505 is a place where the evaluator selects whether the answer of the generative model is valid or not. For example, the checkbox 505 displays multiple options. The text box 506 is a place where the evaluator enters the reason for the answer in the checkbox 505 or a correction suggestion. For example, if the evaluator has additional information to enter for the answer of the generative model, the evaluator can enter the reason for the answer in the checkbox 505 or a correction suggestion in the text box 506, regardless of which option is selected in the checkbox 505.

[0041] (Regarding the remuneration setting unit 134) The reward setting unit 134 sets a reward for making a request to the evaluator. For example, the reward setting unit 134 can set a reward based on the period until the evaluation of the request, the number of evaluations, the evaluation content, and the past evaluations of the evaluator, and based on feedback from the user to the evaluator based on the evaluation of the evaluator by the user or the operation of the information processing system. Also, for example, the reward setting unit 134 can set a reward such as electronic money, points, coupons, etc. Also, for example, when there are multiple evaluators, the reward setting unit 134 can set a value obtained by subtracting the usage fee of the information processing system from the usage fee paid by the user to the operation of the information processing system, and dividing the value by the number of evaluators.

[0042] (Regarding output unit 135) The output unit 135 outputs the evaluation made by the evaluator for the request notified by the notification unit 133 to the user's terminal. For example, the output unit 135 outputs the evaluation made by the evaluator for the request notified by the notification unit 133 to the user's terminal or the developer of the generative model. In addition, for example, the output unit 135 can output feedback to the evaluator to the evaluator terminal 20 after the user confirms the evaluation from the evaluator. This allows the information processing device 100 to select an evaluator for the next or subsequent request by referring to the feedback. Furthermore, the output unit 135 can output the evaluation of the evaluator to both the user's terminal 10 and the developer of the generative model, for example. This allows the generative model to be re-learned based on the evaluation, and the accuracy of the answers of the generative model to the user's questions can be improved.

[0043] Here, an example of a screen output by the output unit 135 to the user will be described with reference to Fig. 5. Fig. 5 is an example of a screen on which an evaluation from an evaluator is output to the user by the output unit 135 of the information processing device 100 according to the embodiment. Fig. 5 illustrates an example of a screen output to the user terminal 10. In the example of Fig. 5, the screen of the user terminal 10 has an item called evaluation 507.

[0044] Evaluation 507 is a screen showing an example of evaluations from evaluators. For example, evaluations from all evaluators are output on the screen in evaluation 507, and correction suggestions from the evaluators are also output. The evaluations 507 in this case may be numerical values ​​or text.

[0045] This allows the information processing device 100 to compensate for incompleteness of the answer of the generative model. Therefore, the information processing device 100 makes it possible to check whether the answer of the generative model is the latest information, whether the facts have been confirmed, and whether it is in violation of laws and regulations or public order and morals by requesting an evaluation from the evaluator. Furthermore, the information processing device 100 makes it possible for a user to determine whether the answer of the generative model is appropriate or not after requesting an evaluation from the evaluator. This allows the information processing device 100 to make a judgment after the user has confirmed both the answer of the generative model and the evaluation of the evaluator. Furthermore, the information processing device 100 also provides the evaluation of the evaluator to the developer of the generative model, allowing the developer to re-learn the generative model and to improve the accuracy of the answers of the generative model to questions from users.

[0046] [3. Information processing flow] Next, the procedure of information processing according to the embodiment will be described with reference to Fig. 6 and Fig. 7. Fig. 6 and Fig. 7 are flowcharts showing an example of information processing according to the embodiment. Below, the procedure of information processing according to the embodiment will be described with reference to the flowcharts shown in Fig. 6 and Fig. 7.

[0047] First, the procedure of the process for requesting an evaluation from an evaluator will be described with reference to Fig. 6. When the reception unit 131 of the information processing device 100 receives a question from a user and an answer from a generative model to the question (step S101, Yes), it requests an evaluation from the evaluator (step S102). On the other hand, when the reception unit 131 does not receive a question from a user and an answer from a generative model to the question (step S101, No), it waits until it receives both the question and the answer.

[0048] Next, the notification unit 133 of the information processing device 100 requests the evaluators to make an evaluation (step S102). The evaluation procedure will be described in detail with reference to FIG. 7. Next, the notification unit 133 of the information processing device 100 notifies whether or not replies have been received from all the evaluators (step S103). If replies have been received from all the evaluators (step S103, Yes), the notification unit 133 of the information processing device 100 outputs the evaluators' evaluations to the user and the developer of the generative model. On the other hand, if replies have not been received from all the evaluators (step S103, No), the notification unit 133 waits until replies are received from all the evaluators. Finally, the output unit 135 of the information processing device 100 outputs the evaluations obtained from the evaluators to the user terminal 10 and the developer of the generative model.

[0049] Next, the procedure of the process when the evaluator performs the evaluation will be described with reference to FIG. 7. First, the information processing device 100 notifies the evaluator terminal 20 of a request for evaluation based on the user's question received by the receiving unit 131 and the answer of the generative model to the question (step S201). If a request notification has been sent to the evaluator terminal 20 (step S201, Yes), the notification unit 133 of the information processing device 100 checks any one of the check boxes displayed on the evaluation screen (step S202). On the other hand, if a request notification has not been sent to the evaluator terminal 20 (step S201, No), the notification unit 133 of the information processing device 100 waits until a request notification arrives at the evaluator terminal 20.

[0050] Next, if any one of the check boxes is checked on the request screen notified by the notification unit 133 of the information processing device 100 (step S202, Yes), the evaluator enters the reason for the evaluation or a proposed correction in a text box (step S203). On the other hand, if any one of the check boxes is not checked on the request screen notified by the notification unit 133 of the information processing device 100 (step S202, No), the process waits until a check box is checked.

[0051] This allows the information processing device 100 to compensate for incompleteness of the answer of the generative model. Therefore, the information processing device 100 makes it possible to check whether the answer of the generative model is the latest information, whether the facts have been confirmed, and whether it violates laws, regulations, or public order and morals by requesting an evaluation from the evaluator. Furthermore, the information processing device 100 makes it possible for a user of the generative model to determine whether the answer of the generative model is appropriate after requesting an evaluation from the evaluator. Furthermore, the information processing device 100 also provides the developer of the generative model with the evaluation by the evaluator, thereby enabling the developer to re-learn the generative model and improve the accuracy of the answers of the generative model to questions from users.

[0052] [4. Hardware Configuration] Moreover, the information processing device 100 according to the embodiment described above is realized, for example, by a computer 1000 having a configuration as shown in Fig. 8. Fig. 8 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 100. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which a calculation device 1030, a primary storage device 1040, a secondary storage device 1050, an output IF (Interface) 1060, an input IF 1070, and a network IF 1080 are connected via a bus 1090.

[0053] The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The primary storage device 1040 is a memory device, such as a RAM, that temporarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device that stores data used by the arithmetic device 1030 for various calculations and various databases, and is realized by a ROM (Read Only Memory), an HDD (Hard Disk Drive), a flash memory, or the like.

[0054] The output IF 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various types of information, such as a monitor or a printer, and is realized by, for example, a connector conforming to a standard such as USB (Universal Serial Bus), DVI (Digital Visual Interface), or HDMI (High Definition Multimedia Interface), etc. The input IF 1070 is an interface for receiving information from various input devices 1020, such as a mouse, keyboard, scanner, etc., and is realized by, for example, a USB, etc.

[0055] The input device 1020 may be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disk), a DVD (Digital Versatile Disk), or a PD (Phase change rewriteable Disk), a magneto-optical recording medium such as an MO (Magnet-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. The input device 1020 may also be an external storage medium such as a USB memory.

[0056] The network IF 1080 receives data from other devices via the network N and sends the data to the arithmetic device 1030, and also transmits data generated by the arithmetic device 1030 to other devices via the network N.

[0057] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0058] For example, when the computer 1000 functions as the information processing device 100, the arithmetic unit 1030 of the computer 1000 realizes the functions of the control unit 130 of the information processing device 100 by executing a program loaded onto the primary storage device 1040.

[0059] 5. Composition and Effects The information processing device 100 according to the present disclosure includes a reception unit 131 that receives a question from a user and an answer generated by a generative model to the question, and a notification unit 133 that notifies an evaluator's terminal of a request to evaluate the answer received by the reception unit 131; and an output unit 135 that outputs the evaluation made by the evaluator for the request notified by the notification unit 133 to the user's terminal.

[0060] This allows the information processing device 100 to compensate for incompleteness of the answer of the generative model. Therefore, the information processing device 100 makes it possible to check whether the answer of the generative model is the latest information, whether the facts have been confirmed, and whether it is in violation of laws and regulations or public order and morals by requesting an evaluation from the evaluator. Furthermore, the information processing device 100 makes it possible for a user to determine whether the answer of the generative model is appropriate or not after requesting an evaluation from the evaluator. This allows the information processing device 100 to make a judgment after the user has confirmed both the answer of the generative model and the evaluation of the evaluator. Furthermore, the information processing device 100 also provides the evaluation of the evaluator to the developer of the generative model, allowing the developer to re-learn the generative model and to improve the accuracy of the answers of the generative model to questions from users.

[0061] The estimation unit 132 of the information processing device 100 according to the present disclosure further includes an estimation unit 132 that estimates the attributes of the user or evaluator and the attributes of the question content, respectively, and the notification unit 133 notifies the request to the terminal of an evaluator who has attributes that are the same as or related to the attributes of the question content, based on the attributes of the user or evaluator and the attributes of the question content estimated by the estimation unit 132.

[0062] According to this configuration, the information processing device 100 is configured to determine whether the answer of the generative model is appropriate after the user requests an evaluation from the evaluator. This allows the information processing device 100 to determine whether the answer of the generative model is appropriate after the user has confirmed both the answer of the generative model and the evaluation by the evaluator.

[0063] The notification unit 133 of the information processing device 100 according to the present disclosure notifies, together with the request, a predetermined guideline that is the standard for evaluating the answer.

[0064] According to this configuration, the information processing device 100 can check whether the answer of the generative model is the latest information, whether the facts have been confirmed, and whether it violates laws, regulations, or public order and morals by requesting an evaluation from the evaluator.

[0065] The above describes the embodiment of the present application in detail based on the drawings, but this is merely an example, and the present invention can be implemented in other forms that include various modifications and improvements based on the knowledge of those skilled in the art, including the forms described in the Disclosure of the Invention section.

[0066] Moreover, the above-mentioned "section, module, unit" can be read as "means" or "circuit", etc. For example, the reception unit 131 can be read as reception means or reception circuit. [Explanation of symbols]

[0067] 1. Overall System 10 User terminal 20 Evaluator terminal 100 Information processing device 110 Communications Department 120 Storage section 121 User information storage unit 122 Evaluator information storage unit 123 Output result storage unit 130 Control section 131 Reception 132 Estimation Department 133 Notification Department 134 Remuneration Setting Department 135 Output section N Network N

Claims

1. a reception unit that receives a question from a user and an answer generated by the generative model with respect to the question; a notification unit that notifies a terminal of an evaluator of a request to evaluate the answer received by the reception unit; an output unit that outputs the evaluation made by the evaluator for the request notified by the notification unit to a terminal of the user; An information processing device comprising:

2. An estimation unit that estimates an attribute of the user or the evaluator and an attribute of the content of the question, The information processing device according to claim 1, characterized in that the notification unit notifies the request to a terminal of an evaluator who has attributes that are the same as or related to the attributes of the content of the question, based on the attributes of the user or the evaluator estimated by the estimation unit and the attributes of the content of the question.

3. a reward setting unit for setting a reward for the evaluator for making the request; The information processing apparatus according to claim 1 , further comprising:

4. The information processing apparatus according to claim 1 , wherein the notification unit notifies, together with the request, a predetermined guideline that is to be a criterion for evaluation of the answer.

5. The information processing apparatus according to claim 1 , wherein the output unit outputs the evaluation performed by the evaluator for the request notified by the notification unit to a terminal of the user or a developer of a generative model.

6. The information processing apparatus according to claim 1 , wherein the notification unit notifies a terminal of the evaluator who performed the evaluation of information related to a reward.

7. An information processing method executed by an information processing device, a receiving step of receiving a question from a user and an answer generated by the generative model for the question; a notification step of notifying a terminal of an evaluator that an evaluation will be made on the answer received by the receiving step; an output step of outputting the evaluation made by the evaluator notified in the notifying step to a terminal of the user; 2. An information processing method comprising:

8. a receiving step of receiving a question from a user and an answer generated by the generative model for the question; a notification step of notifying a terminal of an evaluator that an evaluation will be made on the answer received in the receiving step; an output step of outputting the evaluation made by the evaluator notified in the notification step to a terminal of the user; An information processing program for causing a computer to execute the above.

Images