Information processing system, program, and method

Abstract

To be able to effectively manage information. [Solution] An information processing system comprising a known information storage unit that stores known information, an input data accepting unit that accepts input data from a first user, a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained using machine learning to output output data, a correction result accepting unit that accepts correction results from a second user that have been corrected on the output data, and an update unit that determines whether the known information that matches or is similar to the correction result is stored in the known information storage unit.

Description

[Technical field]

[0001] The present invention relates to an information processing system, a program, and a method. [Background technology]

[0002] Technologies that allow machines to automatically respond to questions and requests from users have been developed, and are now being provided as automatic response services such as bots (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-3533 A Summary of the Invention [Problem to be solved by the invention]

[0004] It is difficult to control whether the responses provided by an automated response service are favorable or not, and there is a demand for more effective information management.

[0005] The present invention has been made in view of the above background, and has an object to provide a technique capable of effectively managing information. [Means for solving the problem]

[0006] The main invention of the present invention for solving the above problem is an information processing system comprising: a known information storage unit that stores known information; an input data accepting unit that accepts input data from a first user; a processing unit that provides the known information related to the input data and the input data to a trained model using machine learning to output output data; a correction result accepting unit that accepts a correction result of the output data from a second user different from the first user; and a known determination unit that determines whether the known information that matches or is similar to the correction result is stored in the known information storage unit. Effect of the Invention

[0007] According to the present invention, information can be effectively managed. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing an overview of an information processing system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing an example of the configuration of a question management table 311. [Diagram 3] 13 is a diagram showing an example of the configuration of a response management table 312. FIG. [Figure 4] FIG. 13 is a diagram showing an example of the configuration of a correction result management table 313. [Diagram 5] 13 is a diagram showing an example of the configuration of a question evaluation management table 314. FIG. [Figure 6] FIG. 13 is a diagram showing an example of the configuration of a correction content evaluation management table 315. [Figure 7] FIG. 13 is a diagram showing an example of the configuration of a questioner management table 316. [Figure 8] FIG. 13 is a diagram showing an example of the configuration of a corrector management table 317. [Figure 9] FIG. 13 is a diagram showing a flow of processing by a learning unit 224. [Figure 10] FIG. 13 is a diagram showing a process flow by the question quality judging unit 225. [Figure 11] FIG. 13 is a diagram showing a process flow by the correction quality determination unit 226. [Figure 12] FIG. 13 is a diagram showing the flow of a process for calculating an incentive for a user who has asked a question. [Figure 13] FIG. 13 is a diagram showing the flow of a process for calculating an incentive for a user who corrects a response. [Figure 14] FIG. 2 illustrates an example of a hardware configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] <Summary of the Invention> The present invention will be described below with reference to the preferred embodiments thereof. [Item 1] a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; A processing unit that provides the known information related to the input data and the input data to a trained model by machine learning and outputs output data; a correction result receiving unit that receives a correction result of correcting the output data from a second user different from the first user; a known information determination unit that determines whether the known information that matches or is similar to the correction result is stored in the known information storage unit; An information processing system comprising: [Item 2] The information processing system according to item 1, an expert storage unit that stores whether or not a user is an expert; the output unit refers to the expert storage unit, identifies the expert as the second user, and outputs the output data only to the identified second user; outputting the correction result corrected by the second user to the first user so that the correction result can be viewed by the first user; An information processing system comprising: [Item 3] The information processing system according to item 1, A correction evaluation receiving unit is further provided for receiving an evaluation from a third user on the correction result, the output unit outputs the input data and the correction result in a viewable manner to the third user; the correction result receiving unit receives the correction results from a plurality of the second users; The update unit selects the correction result in response to the evaluation, and updates the trained model using the selected correction result; An information processing system comprising: [Item 4] Item 3. An information processing system according to item 3, Further, a correction quality determination unit is provided which determines the quality of the correction result by tallying up values ​​according to at least the evaluations, the updating unit selects at least a portion of the correction results according to the quality; An information processing system comprising: [Item 5] Item 4. An information processing system according to the present invention, the correction quality determination unit determines the quality according to at least the aggregated value and the number of the third users who have viewed the correction result; An information processing system comprising: [Item 6] The information processing system according to item 1, an input evaluation receiving unit that receives an evaluation from a third user on the input data; the output unit outputs the input data so as to be viewable by the second user and the third user; The update unit updates the trained model using the input data whose evaluation satisfies a predetermined criterion and the correction result of the output data corresponding to the input data; An information processing system comprising: [Item 7] Item 6. An information processing system according to item 6, the input evaluation receiving unit receives the evaluations from a plurality of the third users; an input quality determination unit that determines a quality of the input data in accordance with at least the evaluation; The updating unit updates the trained model using the input data whose quality satisfies a predetermined standard and the correction result; An information processing system comprising: [Item 8] Item 7. An information processing system according to item 7, the input quality determination unit determines the quality in accordance with at least one of the evaluation and the number of the third users who have viewed the input data; An information processing system comprising: [Item 9] The information processing system according to item 1, a correction evaluation receiving unit that receives an evaluation from a third user on the correction result; a correction quality determination unit that determines the quality of the correction result according to at least a total value of the evaluation values ​​according to the evaluations and the number of the third users who viewed the correction result; an input evaluation receiving unit that receives an evaluation from a third user regarding the input data; an input quality determination unit that determines an input quality, which is the quality of the input data, according to at least a total value of evaluation values ​​according to the evaluations and the number of the third users who have viewed the input data; Further equipped with the output unit outputs the input data and the correction result so as to be viewable by the second user and the third user; The update unit updates the trained model using the input data whose input quality is equal to or greater than a first predetermined value and the correction result whose correction quality is equal to or greater than a second predetermined value; An information processing system comprising: [Item 10] The information processing system according to item 1, an incentive providing unit that provides an incentive to the second user who has provided the correction result; An information processing system comprising: [Item 11] The information processing system according to item 1, an incentive providing unit that provides an incentive to the first user who provides the input data; An information processing system comprising: [Item 12] Item 3. An information processing system according to item 3, an incentive providing unit that provides an incentive to the third user; An information processing system comprising: [Item 13] The information processing system according to item 1, the update unit determines whether the correction result is public information, and if the correction result is not public information, updates the trained model using the correction result; An information processing system comprising: [Item 14] storing known information; accepting input data from a first user; A step of providing the known information related to the input data and the input data to a trained model by machine learning to output output data; receiving a correction result of correcting the output data from a second user different from the first user; A step of determining whether the known information that matches or is similar to the correction result is stored in the known information storage unit; A program for causing a computer to execute the following. [Item 15] storing known information; accepting input data from a first user; A step of providing the known information related to the input data and the input data to a trained model by machine learning to output output data; receiving a correction result of correcting the output data from a second user different from the first user; A step of determining whether the known information that matches or is similar to the correction result is stored in the known information storage unit; How a computer executes. [Item 16] The information processing system according to item 1, an update unit that adds the correction result to the known information storage unit when the known information that matches or is similar to the correction result is not stored in the known information storage unit; An information processing system comprising: [Item 17] <Known check: Is it registered in the DB?> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result of correcting the output data from a second user; a known-information determination unit that determines whether the correction result is known based on whether the known information that matches or is similar to the correction result is stored in the known-information storage unit; An information processing system comprising: [Item 18] <Known judgement: Are the corrections similar to the search results for the input data?> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result of correcting the output data from a second user; a known-data determination unit that provides the input data or a keyword included in the input data to a search engine to obtain search results, and determines whether the correction result is known based on whether the obtained search results include content similar to the correction result; An information processing system comprising: [Item 19] <Known result: Search the correction results to see if there are similar results> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result of correcting the output data from a second user; a known-state determination unit that provides the correction result to a search engine to obtain search results, and determines whether the correction result is known or not based on whether the obtained search results contain content similar to the correction result; An information processing system comprising: [Item 20] <Question known status: Is it registered in the database?> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a known information determination unit that determines whether the known information that matches or is similar to the input data is stored in the known information storage unit; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; An information processing system comprising: [Item 21] Item 21. An information processing system according to item 20, a correction result receiving unit that receives a correction result of correcting the output data from a second user; An information processing system comprising: [Item 22] <Question known status determination: Search for a question and see if there are similar search results> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result of correcting the output data from a second user; Equipped with providing the input data or keywords contained in the input data to a search engine to obtain search results, and determining whether the correction results are public information based on whether the obtained search results contain content similar to the correction results; An information processing system comprising: [Item 23] <Incentive: Corrector, quantity only> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result of correcting the output data from a second user; an incentive providing unit that provides an incentive to the second user who has provided the correction result; Equipped with the incentive granting unit determines an amount of the incentive according to a correction enthusiasm which is a value according to the number of the correction results provided by the second user; An information processing system comprising: [Item 24] Item 24. The information processing system according to item 23, a correction evaluation receiving unit that receives an evaluation value from a third user on the correction result; the incentive granting unit determines an amount of the incentive according to a correction quality, which is a value corresponding to an aggregate value of the evaluation values ​​from the third user for the correction result provided by the second user, and the proactiveness of the correction; An information processing system comprising: [Item 25] Item 25. An information processing system according to item 24, the incentive granting unit determines the amount of the incentive according to an evaluator quality, which is a value according to the number of the third users who have provided the same evaluation value as the predetermined evaluation value provided by the second user, the correction quality, and the correction proactiveness for the same correction result; An information processing system comprising: [Item 26] <Incentive: Questioner, quantity only> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result of correcting the output data from a second user; an incentive granting unit that grants an incentive to the first user who provides the input data; Equipped with the incentive granting unit determines an amount of the incentive according to a question willingness, the question being a value according to the number of the input data provided by the first user; An information processing system comprising: [Item 27] Item 27. An information processing system according to item 26, an input evaluation receiving unit that receives an evaluation value from a third user for the input data; the incentive granting unit determines an amount of the incentive according to a question quality, which is a value according to an aggregate value of the evaluation values ​​from the third user for the input data provided by the first user, and the question proactiveness; An information processing system comprising: [Item 28] Item 28. An information processing system according to item 27, the incentive granting unit determines the amount of the incentive in accordance with a questioner quality, the question quality, and the question proactiveness, the question quality being a value corresponding to the number of the third users who have provided the same evaluation value as the predetermined evaluation value provided by the first user for the same question; An information processing system comprising: [Item 29] <Incentive:Evaluator> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result of correcting the output data from a second user; an incentive providing unit that provides an incentive to the second user who has provided the correction result; a correction evaluation receiving unit that receives an evaluation value from a third user on the correction result; Equipped with the incentive granting unit determines an amount of the incentive according to an evaluation proactiveness, the value being a value according to the number of the evaluation values ​​provided by the third user; An information processing system comprising: [Item 30] Item 29. An information processing system according to item 29, the incentive granting unit determines the amount of the incentive in accordance with an evaluator quality, which is a value corresponding to the number of the third users who have provided the same evaluation value as the predetermined evaluation value provided by the third user, for the same correction result, and the evaluation proactiveness; An information processing system comprising: [Item 31] <Incentives: corrector, speed of correction> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result of correcting the output data from a second user; an incentive providing unit that provides an incentive to the second user who has provided the correction result; Equipped with the incentive granting unit determines an amount of the incentive depending on how quickly the output data is output and the correction result for the output data is received; An information processing system comprising: [Item 32] <Incentives: Evaluator, speed of evaluation of corrections> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result of correcting the output data from a second user; a correction evaluation receiving unit that receives an evaluation from a third user on the correction result; an incentive providing unit that provides an incentive to the third user who has evaluated the correction result; Equipped with the incentive granting unit determines the amount of the incentive depending on how quickly the correction result is received after the correction result is received and the evaluation value for the correction result is received; An information processing system comprising: [Item 33] <Incentive:Evaluator, Question Evaluation> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result of correcting the output data from a second user; an evaluation receiving unit that receives an evaluation value from a third user for the input data; an incentive granting unit that grants an incentive to the third user who provided the evaluation value; Equipped with the incentive granting unit determines an amount of the incentive according to an evaluation proactiveness, the value being a value according to the number of the evaluation values ​​provided by the third user; An information processing system comprising: [Item 34] Item 34. The information processing system according to item 33, the incentive granting unit determines the amount of the incentive in accordance with an evaluator quality, the evaluator quality being a value corresponding to the number of the third users who have provided the same evaluation value as the predetermined evaluation value provided by the third user, for the same input data, and the evaluation aggressiveness; An information processing system comprising: [Item 35] <Incentive: Evaluator, Response Evaluation> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result of correcting the output data from a second user; an evaluation receiving unit that receives an evaluation value from a third user regarding the output data; an incentive granting unit that grants an incentive to the third user who provided the evaluation value; Equipped with the incentive granting unit determines an amount of the incentive according to an evaluation proactiveness, the value being a value according to the number of the evaluation values ​​provided by the third user; An information processing system comprising: [Item 36] Item 36. An information processing system according to item 35, the incentive granting unit determines an amount of the incentive in accordance with an evaluator quality, the evaluator quality being a value corresponding to the number of the third users who have provided the same evaluation value as the predetermined evaluation value provided by the third user, for the same output data, and the evaluation aggressiveness; An information processing system comprising: [Item 37] <Incentives: questioner, number of question views> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result of correcting the output data from a second user; an incentive granting unit that grants an incentive to the first user who provides the input data; Equipped with the incentive granting unit determines an amount of the incentive according to the number of views of the input data provided by the first user; An information processing system comprising: [Item 38] Item 37. An information processing system according to item 37, the incentive granting unit determines an amount of the incentive according to a question enthusiasm, which is a value according to the number of input data provided by the first user, and the number of views; An information processing system comprising: [Item 39] Item 38. An information processing system according to item 38, an input evaluation receiving unit that receives an evaluation value from a third user for the input data; the incentive granting unit determines an amount of the incentive according to a question quality, which is a value corresponding to an aggregate value of the evaluation values ​​from the third users for the input data provided by the first user, the question proactiveness, and the number of views; An information processing system comprising: [Item 40] Item 39. An information processing system according to item 39, the incentive granting unit determines the amount of the incentive according to a questioner quality, the question quality, the question proactiveness, and the number of views, the question being a value according to the number of third users who have provided the same evaluation value as the predetermined evaluation value provided by the first user for the same question; An information processing system comprising: [Item 41] <Incentives: correctors, number of views of corrections> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result of correcting the output data from a second user; an incentive providing unit that provides an incentive to the second user who has provided the correction result; Equipped with the incentive granting unit determines an amount of the incentive according to the number of views of the correction result provided by the second user; An information processing system comprising: [Item 42] Item 41. An information processing system according to item 41, the incentive granting unit determines an amount of the incentive according to a correction enthusiasm, which is a value according to the number of the correction results provided by the second user, and the number of views; An information processing system comprising: [Item 43] Item 43. An information processing system according to item 42, a correction evaluation receiving unit that receives an evaluation value from a third user on the correction result; the incentive granting unit determines an amount of the incentive according to a correction quality, which is a value corresponding to an aggregate value of the evaluation values ​​from the third user for the correction result provided by the second user, the proactiveness of correction, and the number of views; An information processing system comprising: [Item 44] Item 44. An information processing system according to item 43, the incentive granting unit determines the amount of the incentive according to an evaluator quality, which is a value according to the number of the third users who have provided the same evaluation value as the predetermined evaluation value provided by the second user, the correction quality, the proactiveness in correction, and the number of views for the same correction result; An information processing system comprising: [Item 45] <Check if the answer is known: Is it registered in the database?> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a response data receiving unit that receives response data to the input data from a second user; a known information determination unit that determines whether the response data is known based on whether the known information that matches or is similar to the response data is stored in the known information storage unit; An information processing system comprising: [Item 46] <Known judgement of human answers: Are the human answers similar to the search results of the input data?> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a response data receiving unit that receives response data to the input data from a second user; a known-data determination unit that provides the input data or a keyword included in the input data to a search engine to obtain search results, and determines whether the response data is known based on whether the obtained search results include content similar to the response data; An information processing system comprising: [Item 47] <Determine if human answers are known: Search for human answers to see if there are similar search results> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a response data receiving unit that receives response data to the input data from a second user; a known determination unit that provides the response data to a search engine to obtain search results, and determines whether the response data is known based on whether the obtained search results contain content similar to the response data; An information processing system comprising:

[0010] <System configuration> 1 is a diagram showing an overview of an information processing system according to an embodiment of the present invention. The information processing system of this embodiment is configured to include a response device 2. The response device 2 is communicably connected to a user terminal 1 via a communication network. The communication network is, for example, the Internet, and is constructed by a public telephone line network, a mobile phone line network, a wireless communication path, Ethernet (registered trademark), etc.

[0011] The user terminal 1 is a computer operated by a user. The user terminal 1 can be, for example, a smartphone, a tablet computer, a personal computer, etc. A web browser or an application runs on the user terminal 1, and the user can access the response device 2 via the web browser or the application.

[0012] The response device 2 may be a general-purpose computer, such as a workstation or a personal computer, or may be logically realized by cloud computing.

[0013] In the information processing system of this embodiment, the response device 2 accumulates knowledge (known information) that is the basis for generating an answer to a question, and generates an answer to a question from a user (questioner) using a machine learning model (trained model) that has been trained by machine learning based on this known information, and accepts corrections to this answer from a user (corrector) other than the user who asked the question. In this embodiment, the machine learning model is assumed to be a large-scale language model (LLM), and data that serves as an answer can be generated by using this machine learning model. Note that this embodiment assumes a so-called QA task of generating an answer (response) to a question, but it is not limited to the form of the QA task, and may be in any form in which some input is given to the machine learning model and output. That is, the "question" in this embodiment can be understood as the "input" to the learning model, and the "answer" can be understood as the "output" from the learning model.

[0014] In this embodiment, it is assumed that the corrector is a user different from the questioner, the evaluator who evaluates the question is a user different from the questioner, and the evaluator who evaluates the correction result is a user different from the corrector. The corrector may be the same as the questioner, or the questioner may be included as part of the correctors. The evaluator, questioner, and corrector may be the same person, or the questioner and the corrector may be included as part of the evaluators. The questioner may be one or more people. The corrector may be one or more people. The evaluator may be one or more people.

[0015] <Software configuration> As shown in FIG. 1, the user terminal 1 includes a question input unit 111, a response content display unit 112, a question evaluation input unit 113, a response correction input unit 114, a correction result display unit 115, a correction evaluation input unit 116, a question quality judgment result display unit 117, a correction quality judgment result display unit 118, and an incentive grant number display unit 119.

[0016] 1, the response device 2 includes a WEB server 21 that transmits and receives data to and from the user terminal 1, an AP server 22 that functions as middleware, and a DB server 23 that manages a database. Note that this configuration is merely an example, and the response device 2 may be configured with one or two computers. Also, the response device 2 may be configured with four or more computers.

[0017] The DB server 23 includes a data storage unit 231, a trained model storage unit 232, a viewing log storage unit 233, and a known information storage unit 234.

[0018] The WEB server 21 includes a question receiving unit 211, a response content display unit 212, a question evaluation receiving unit 213, a correction result receiving unit 214, a correction result display unit 215, a correction evaluation receiving unit 216, a question quality judgment result display unit 217, a correction quality judgment result display unit 218, a number of incentives granted display unit 219, and a viewing log acquisition unit 220.

[0019] The AP server 22 includes a response unit 221 , a response content creation unit 222 , a correction result content creation unit 223 , a learning unit 224 , a question quality determination unit 225 , a correction quality determination unit 226 , and an incentive given number calculation unit 227 .

[0020] The data storage unit 231 of the DB server 23 includes a question management table 311 , a response management table 312 , a correction result management table 313 , a question evaluation management table 314 , a correction content evaluation management table 315 , a questioner management table 316 , and a corrector management table 317 .

[0021] The question management table 311 is a table for managing information (hereinafter referred to as question information) relating to questions received from users (questioners) who ask questions. Fig. 2 is a diagram showing an example of the configuration of the question management table 311. Records (question information) managed in the question management table 311 include information for identifying a question (question ID), information for identifying a user who has asked a question (questioner ID), and the content of the question.

[0022] The response management table 312 is a table for managing information (hereinafter referred to as response information) related to the content of an automatic response to a question from a user (questioner). Fig. 3 is a diagram showing an example of the configuration of the response management table 312. A record (response information) registered in the response management table 312 includes a question ID for identifying the question, information for identifying the response (response ID), and the response content.

[0023] The correction result management table 313 is a table for managing information on the correction results (hereinafter referred to as correction result information) received from a user (corrector) in response to the content of an automatic response (response content). Fig. 4 is a diagram showing an example of the configuration of the correction result management table 313. Records (correction result information) managed in the correction result management table 313 include a response ID that identifies the response, information that identifies the user who performed the correction (corrector ID), information for identifying the correction result (correction ID), and correction content.

[0024] The question evaluation management table 314 is a table for managing information on evaluations from users (evaluators) on the quality of questions (hereinafter, referred to as question evaluation information). FIG. 5 is a diagram showing an example of the configuration of the question evaluation management table 314. The records (question evaluation information) managed in the question evaluation management table 314 include a question ID for identifying a question, evaluations (in this embodiment, the number of users who gave "Like" to the question content is set as the number of evaluations) received from at least users other than the user who asked the question (even if the user who asked the question is included, it is acceptable), the number of views of the question, and a score (quality evaluation score) determined in consideration of the number of evaluations and the number of views. That is, in this embodiment, both the evaluations received from users and the score (final evaluation) determined based on the evaluations are managed in the question evaluation information. Note that the evaluation may be a score value set by the evaluator, such as 1 to 5 stars. Also, instead of an evaluation value from a user (evaluator), a value may be automatically calculated based on a given rule or function. For example, the number of times that the machine learning model has cited the question or the correction result may be set as the evaluation value. In addition, text data such as comments, audio data, image data, etc. can also be given as evaluations. In this case, an evaluation value for the content of the evaluation can be estimated by classifying the text data, audio data, or image data using a classifier.

[0025] The correction content evaluation management table 315 is a table for managing information on the evaluation of the quality of the correction content (hereinafter, referred to as correction content evaluation information). FIG. 6 is a diagram showing an example of the configuration of the correction content evaluation management table 315. The records (correction content evaluation information) managed in the correction content evaluation management table 315 include information for identifying the correction content (correction ID), evaluations received from at least users other than the user who corrected the correction content (in this embodiment, the number of users who gave "Like" to the correction content is the number of evaluations), the number of views of the correction content, and a score (quality evaluation score) determined in consideration of the number of evaluations and the number of views. That is, in this embodiment, both the evaluation received from the user and the score (final evaluation) determined based on the evaluation are managed in the correction content evaluation information. Note that the evaluation is not limited to the number of "Likes", but may be the number of times the "Useful" button was pressed, or may be a score value set by the evaluator, such as 1 to 5 stars. Furthermore, instead of an evaluation value from a user (evaluator), it may be a value that is automatically calculated based on a given rule or function. For example, the number of times that the machine learning model has cited the question or the correction result can be the evaluation value. Also, text data such as comments, voice data, image data, etc. can be given as the evaluation. In this case, the evaluation value for the content of the evaluation can be estimated by classifying the text data, voice data, or image data with a classifier.

[0026] The questioner management table 316 is a table for managing information related to users who have asked questions (hereinafter referred to as questioner information). Fig. 7 is a diagram showing an example of the configuration of the questioner management table 316. A record (questioner information) managed in the questioner management table 316 includes information (questioner ID) that identifies the user who asked the question, a question ID that identifies the question, a quality evaluation score acquired for the question by the user, and a question ID that identifies the question of another person that was evaluated by the user.

[0027] The corrector management table 317 is a table for managing information related to the user who made the correction (hereinafter referred to as corrector information). Fig. 8 is a diagram showing an example of the configuration of the corrector management table 317. The records (corrector information) managed in the corrector management table 317 include information (corrector ID) that identifies the user who made the correction, a correction ID that identifies the correction result, a quality evaluation score acquired for the correction by that user, and a correction ID that identifies the correction result of another person that was evaluated by that user.

[0028] The trained model storage unit 232 included in the DB server 23 stores a trained machine learning model (parameters constituting the model). For example, a trained model trained externally can be used as the machine learning model. Note that the DB server 23 may not be provided with a trained machine learning model, and an API or the like provided by another server may be used. In this embodiment, it is assumed that the trained model is a large-scale language model (LLM) and is a generator that generates an answer to an input prompt. Note that a prompt may be sent to an external server that provides a generator using the large-scale language model via an API or the like, and the external server may receive a result generated in response to the large-scale language model and the prompt.

[0029] The view log storage unit 233 included in the DB server 23 stores a log of the user viewing questions and correction contents on the user terminal 1. The view log can be a general access log. The view log has necessary and sufficient information to count how many times a question ID or correction ID has been viewed.

[0030] The known information storage unit 234 included in the DB server 23 can store any information (hereinafter, referred to as known information). In this embodiment, the known information is assumed to be text data, but it can also be image data, audio data, or the like. The known information storage unit 234 can include a file storage unit that stores a file storing the known information, and a vector store that stores vector data in which the known information is embedded. The file storage unit can be, for example, a file system or an object database. The file storage unit may be configured as a functional unit that stores a file of the known information in another file server and performs processing to read the known information from the other file server. The vector store can store information that identifies the known information (hereinafter, referred to as known information ID. It is sufficient that the information is unique, and it may be a numerical value or a character string issued as an ID, or it may be, for example, a path to a file or a URL) in association with the vector data. One piece of known information can be embedded for each chunk into which it is divided, and multiple pieces of vector data can be stored in association with one known information ID. The known information storage unit 234 can store a user ID indicating a creator and / or provider of the known information in association with the known information. For example, the vector store may store a user ID indicating a creator and / or a user ID indicating a provider, a known information ID, and vector data in association with each other, or may include a provider-known information correspondence table that associates a user ID indicating a creator and / or a user ID indicating a provider with a known information ID. Furthermore, information specifying a question may be associated with the known information. The known information storage unit 234 may include a question-known information correspondence table that associates a known information ID specifying the known information with a question ID specifying a question, for example.

[0031] ==Responses to Questions== The question input unit 111 of the user terminal 1 accepts input of a question from a user. In this embodiment, the question is assumed to be text data, but the question can also be image data, audio data, or the like. The question input unit 111 transmits the question accepted from the user to the response device 2. The question input unit 111 can transmit the question by attaching it to an HTTP request, for example.

[0032] The question receiving unit 211 of the WEB server 21 can receive a question transmitted from the user terminal 1. The question receiving unit 211 can, for example, decode a question encoded in an HTTP request and transmit the decoded question to the AP server 22.

[0033] When the response unit 221 of the AP server 22 receives a question from the WEB server 21, the response unit 221 generates a response to the question by providing a prompt including the question to the machine learning model stored in the trained model storage unit 232 of the DB server 23, and transmits the generated response to the response content creation unit 222. The response unit 221 can also set the decoded question as the question content, set a questioner ID indicating the user of the user terminal 1 who asked the question, generate a new question ID to create question information, and register the created question information in the question management table 311 provided in the data storage unit 231 of the DB server 23. In this embodiment, the response unit 221 generates a response by so-called RAG (Retrieval-Augmented Generation). That is, the response unit 221 can search for known information similar to the question from the known information storage unit 234, and can provide a prompt including the searched known information and the question to the trained model to generate a response. The prompt can include an instruction to generate an answer to the question based on the known information.

[0034] The response content creation unit 222 of the AP server 22 creates content (hereinafter referred to as response content; for example, it may be screen data written in HTML) for displaying the response received from the response unit 221 on the user terminal 1. The response content creation unit 222 transmits the created response content to the WEB server 21. Note that the response content may include not only the response but also a question. The response content may also include the number of ratings for the question (the number of "likes" for the question). The response content may also include information that identifies the known information included in the prompt.

[0035] The response content display unit 212 of the WEB server 21 receives response content for displaying the response from the AP server 22, and transmits the received response content to the user terminal 1. The response content display unit 212 can transmit content for displaying the question and the response not only to the user terminal 1 that is the sender of the question, but also to the user terminals 1 of other users.

[0036] In response to a request from the user terminal 1, the response content display unit 212 of the WEB server 21 transmits a message to the response content creation unit 222 of the AP server 22, and the response content creation unit 222 reads one or more pieces of question information from the question management table 311 provided in the data storage unit 231 of the DB server 23, reads response information corresponding to the question ID from the response management table 312 for each of the read question information, creates response content for displaying the question content and the response content, and transmits the response content to the WEB server 21, and the response content display unit 212 transmits the response content to the sender of the request. The response content creation unit 222 can also obtain the number of ratings corresponding to the question ID from the question rating management table 314 and include it in the response content.

[0037] The response content display unit 112 of the user terminal 1 receives the response content for displaying the response transferred from the WEB server 21, and is able to display the response content to the user based on the response content.

[0038] ==Public / Private setting function== The questions and / or correction contents may be set to be public or private. In this case, information indicating public or private (public setting information) is set in the question information stored in the question management table 311 and / or the correction result information stored in the correction result management table 313, and only the question information and / or the correction result information whose public setting information indicates "public" can be made public to general users.

[0039] It should be noted that even if the correction contents are not public, private corrections may be accepted. In this case, if the disclosure setting information indicates that the correction contents are not public, the correction contents may be made public only to users of the company. This makes it possible to accumulate knowledge exclusive to the company while preventing the company's know-how from being leaked.

[0040] ==Mask setting function== Also, a part of the question and / or the correction content may be masked. In this case, for example, a character to be masked or a condition for masking may be set for each question and / or correction content in the question information and / or the correction result information, and the set character may be replaced with a replacement character, or a part of the question and / or the correction content that satisfies the set condition may be replaced with a replacement character, and then the question and / or the correction content may be made public and used for re-learning.

[0041] ==Question Rating== The question evaluation input unit 113 of the user terminal 1 accepts input of an evaluation of a question from a user. The question evaluation input unit 113 may accept a "like" for a question posted by a user other than the user using the user terminal 1, if the question is a good question. The question evaluation input unit 113 transmits the evaluation of the accepted question (e.g., "like") to the response device 2 together with a question ID that identifies the question.

[0042] The question evaluation receiving unit 213 of the WEB server 21 receives evaluations (e.g., "Like") on a question from the user terminal 1. The question evaluation receiving unit 213 can increment the number of evaluations for question information in the question evaluation management table 314 corresponding to the question ID that identifies the question.

[0043] ==Response Evaluation== The response may be evaluated from the trained model. In this case, the user terminal 1 includes a response evaluation input unit, the WEB server 21 includes a response evaluation receiving unit, and the data storage unit 231 of the DB server 23 includes a response evaluation management table. In the response evaluation management table, the evaluation (for example, the number of likes), the number of views of the response, and the quality evaluation score for the response can be stored in association with a response ID that identifies the response. The evaluation of the response may be a score value set by the evaluator, for example, 1 to 5 stars. Also, instead of the evaluation value from the user (evaluator), it may be a value that is automatically calculated based on a given rule or function. For example, the number of times the machine learning model cites the question or the correction result may be the evaluation value. Also, text data such as comments, voice data, image data, etc. can be given as the evaluation, and in this case, the evaluation value for the content of the evaluation can be estimated by classifying the text data, voice data, and image data with a classifier.

[0044] The response evaluation input unit of the user terminal 1 accepts input of an evaluation of a response from a user. The response evaluation input unit can accept an evaluation of the content of a response viewed by a user. For example, the response evaluation input unit may accept a "like" when the content of the response is good. The response evaluation input unit transmits the evaluation of the accepted response (for example, "like") to the response device 2 together with a response ID that identifies the response (or a question ID that identifies the question).

[0045] The response evaluation receiving unit of the WEB server 21 can receive an evaluation (for example, "Like") on a response from the user terminal 1, and increment the number of evaluations in the response evaluation management table corresponding to the response ID that identifies the response.

[0046] ==Response correction== The response correction input unit 114 of the user terminal 1 can receive correction contents from the user for the response that the user has viewed. The response correction input unit 114 transmits the received correction contents to the WEB server 21 together with a response ID that identifies the response and a corrector ID that identifies the user.

[0047] The correction result receiving unit 214 of the Web server 21 receives the correction content from the user in response to the response. The correction result receiving unit 214 transmits the received correction content to the AP server 22 together with the response ID and the correction ID.

[0048] When the correction result content creation unit 223 of the AP server 22 receives the correction content transmitted from the WEB server 21, it can register the correction result information including the response ID, the corrector ID, the newly assigned correction ID (which may be assigned by the DB server 23), and the correction content in the correction result management table 313 provided in the data storage unit 231. The correction result content creation unit 223 can create content indicating that the correction result has been accepted (hereinafter, referred to as correction result content). The correction result content can include the correction result received from the WEB server 21. The correction result content can also include, for example, a list of past correction results registered in the correction result management table 313. The correction result content creation unit 223 transmits the correction result content to the WEB server 21.

[0049] The correction result display unit 215 of the Web server 21 transmits the correction result content received from the AP server 22 to the user terminal 1 .

[0050] The correction result display unit 115 of the user terminal 1 can display the correction contents to the user based on the correction result content received from the Web server 21.

[0051] In response to a request from the user terminal 1, the correction result display unit 215 of the WEB server 21 transmits a message to the correction result content creation unit 223 of the AP server 22, and the correction result content creation unit 223 reads one or more pieces of response information from the response management table 312 included in the data storage unit 231 of the DB server 23, reads the question content corresponding to the question ID from the question management table 311 for each of the read response information, reads the correction result information corresponding to the response ID from the correction result management table 313, creates correction result content for displaying the question content, the response content, and the correction content (multiple correction contents are possible), and transmits it to the WEB server 21, and the correction result display unit 215 transmits this correction result content to the sender of the request. The correction result content creation unit 223 can also obtain the number of evaluations corresponding to the correction ID from the correction content evaluation management table 315 and include it in the correction result content.

[0052] ==Correction Evaluation== The correction evaluation input unit 116 of the user terminal 1 accepts input of an evaluation of the correction result from the user. The correction evaluation input unit 116 may accept a "like" for a correction result made by a user other than the user using the user terminal 1, when the correction result is good. The correction evaluation input unit 116 transmits the evaluation of the accepted correction result (for example, a "like") to the response device 2 together with a correction ID that identifies the correction result.

[0053] The correction evaluation receiving unit 216 of the WEB server 21 receives an evaluation (e.g., “Like”) on the correction result from the user terminal 1. The correction evaluation receiving unit 216 can increment the number of evaluations in the correction content evaluation information in the correction content evaluation management table 315 corresponding to the correction ID that identifies the correction result.

[0054] ==Machine Learning== The learning unit 224 (corresponding to the update unit of the present invention) of the AP server 22 can learn the correction result in which the user corrects the response generated by the trained model in response to the question. In this embodiment, the learning unit 224 performs learning by adding the correction result to the known information storage unit 234. The learning unit 234 adds the correction result (which may be only the corrected part or the entire response reflecting the correction result) to the known information storage unit 234. For each correction content registered in the correction result management table 313, the learning unit 224 can register the correction content in the known information storage unit 234 in association with a user ID (corrector ID) indicating the user who provided the correction result. The learning unit 224 can also obtain a user ID indicating the creator of the known information used to generate the response (known information similar to the question and identified by information identifying the known information included in the response content) from the known information storage unit 234, associate the user ID as the creator, and also associate a user ID (corrector ID) indicating the user who provided the correction content as the provider, and register the correction content in the known information storage unit 234.

[0055] For example, the learning unit 224 may read out correction result information corresponding to the question ID for each piece of question information registered in the question management table 311 from the correction result management table 313, and add the correction content included in the read correction result information to the known information storage unit 234. For example, the learning unit 224 may register the question in the known information storage unit 234 in association with the correction content.

[0056] ==Known Verdicts== The learning unit 224 (known determination unit) may determine whether the correction result is known information. The learning unit 224 may not add known information to the known information storage unit 234. If the correction result is known, it may be determined that the correction result is public information (similarly, if the correction result is not known, the correction result is private information).

[0057] Regarding whether the information is known, the learning unit 224 can determine the similarity between the known information stored in the known information storage unit 234 and the correction result, and determine that the correction result is known if the similarity is a predetermined value or more. The learning unit 224 can also determine the similarity between the result output by giving a question to the trained model and the correction result, and determine that the correction result is known if the similarity is a predetermined value or more.

[0058] Regarding whether or not the information is known, the learning unit 224 can, for example, provide the question or a keyword or phrase contained in the question to a publicly available search engine, determine whether or not the search results contain content similar to the correction result, and if it is determined that the correction result contains content similar to the correction result, determine that the correction result is known information. The learning unit 224 can also, for example, provide the correction result or a keyword or phrase contained in the correction result to a publicly available search engine, determine whether or not the search results contain content similar to the correction result, and if it is determined that the correction result contains content similar to the correction result, determine that the correction result is known information.

[0059] Also, it may be determined whether the question is known. In this case, the learning unit 224 calculates the similarity between the question and the known information stored in the known information storage unit 234, and can determine that the question is known if the similarity is equal to or greater than a predetermined value. Also, if a question is stored in the known information storage unit 234 in association with the question, the learning unit 224 calculates the similarity between the question stored in the known information storage unit 234 and the question content managed in the question management table 311, and can determine that the question is known if the similarity is equal to or greater than a predetermined value. If the learning unit 224 determines that the question is not known, or if the similarity is equal to or less than a predetermined threshold (which may be a value lower than the above-mentioned predetermined value), it can add the question to the known information storage unit 234 in association with the correction result.

[0060] == Additional Conditions == The learning unit 224 may, for example, accumulate only the correction results whose number of ratings and / or quality evaluation score are each equal to or greater than a given threshold. For example, for each piece of question information registered in the question management table 311, the learning unit 224 can add to the known information storage unit 234, among the correction result information corresponding to the question ID, the number of ratings of which is equal to or greater than a given threshold and / or the quality evaluation score of which is equal to or greater than a given threshold (which may be the same as or different from the threshold for the number of ratings).

[0061] The learning unit 224 may also add only correction results related to responses to questions for which the number of ratings and / or the quality evaluation score are each equal to or greater than a given threshold. For example, the learning unit 224 may select, from among the question information registered in the question management table 311, those for which the number of ratings is equal to or greater than a given threshold and / or those for which the quality evaluation score is equal to or greater than a given threshold (which may be the same as or different from the threshold for the number of ratings), and add the correction contents of the correction result information corresponding to the question ID of the selected question information to the known information storage unit 234 in association with the question ID indicating the selected question information. At this time, it may also be configured to add only those correction result information corresponding to the question ID for which the number of ratings is equal to or greater than a given threshold for the correction result and / or those for which the quality evaluation score is equal to or greater than a given threshold for the correction result (which may be the same as or different from the threshold for the number of ratings).

[0062] ==Subject to additional data== The learning unit 224 may use not only the correction results but also explanatory articles on papers and web pages as data to be added to the known information storage unit 234. In this case, too, the incentive amount calculation unit 227, which will be described later, can award incentives to the authors of papers and web pages.

[0063] ==Judgment of Question Quality== The question quality judgment unit 225 of the AP server 22 judges the quality of the question. For each piece of question evaluation information registered in the question evaluation management table 314, the question quality judgment unit 225 can determine a quality evaluation score according to, for example, the number of evaluations. For example, the question quality judgment unit 225 can calculate the quality evaluation score by dividing the number of evaluations of the question evaluation information by the number of views. Note that instead of the number of views, the number of users who viewed the question may be used. The question quality judgment unit 225 creates content for displaying the quality evaluation score for the question (hereinafter, referred to as question quality evaluation content). The question quality evaluation content can be, for example, content for allowing the user to view a good question. The question quality judgment unit 225 can create screen data written in HTML for displaying the quality evaluation scores for one or more pieces of question evaluation information. The question quality judgment unit 225 transmits the question quality evaluation content to the WEB server 21.

[0064] The question quality judgment result display unit 217 of the WEB server 21 can receive question quality evaluation content received from the AP server 22, and provide the received question quality evaluation content to the user terminal 1. In response to a request from the user terminal 1, the question quality judgment result display unit 217 may read question evaluation information from the question evaluation management table 314, and create question quality evaluation content to respond.

[0065] The question quality judgment result display unit 117 of the user terminal 1 receives the question quality evaluation content transmitted from the Web server 21, and can display to the user a screen showing the quality evaluation score of the question based on the question quality evaluation content.

[0066] ==Judgment of the quality of the corrections== The correction quality judgment unit 226 of the AP server 22 judges the quality of the correction contents. The correction quality judgment unit 226 can determine a quality evaluation score for each piece of correction contents evaluation information registered in the correction contents evaluation management table 315, for example, according to the number of evaluations. The correction quality judgment unit 226 can calculate a quality evaluation score by dividing the number of evaluations of the correction contents evaluation information by the number of views, for example. Note that instead of the number of views, the number of users who viewed the correction contents may be used. The correction quality judgment unit 226 creates content for displaying a quality evaluation score for the correction contents (hereinafter, referred to as correction quality evaluation content). The correction quality evaluation content can be, for example, content for providing a best answer to a user. The correction quality judgment unit 226 can create screen data described in HTML for displaying a quality evaluation score for one or more pieces of correction contents evaluation information. The correction quality judgment unit 226 transmits the correction quality evaluation content to the WEB server 21.

[0067] The correction quality judgment result display unit 218 of the WEB server 21 can receive the correction quality evaluation content received from the AP server 22, and provide the received correction quality evaluation content to the user terminal 1. In response to a request from the user terminal 1, the correction quality judgment result display unit 218 may read out correction content evaluation information from the correction content evaluation management table 315, and create correction quality evaluation content to respond.

[0068] The correction quality judgment result display unit 118 of the user terminal 1 can receive the correction quality evaluation content transmitted from the WEB server 21, and display to the user a screen showing the quality evaluation score of the correction content based on the correction quality evaluation content.

[0069] ==Providing incentives for corrections== The incentive amount calculation unit 227 of the AP server 22 determines the amount of incentive to be given to at least one of the user who asked the question, the user who evaluated the question, the user who corrected the response, and the user who evaluated the correction content.

[0070] The incentive may be, for example, points circulating in the market, virtual currency, or a token using blockchain technology. The incentive may also be a coupon or the like. Digital content may also be given as the incentive. For example, the incentive may be a digital content or a real lottery ticket, and the number of attempts to draw the lottery may be the amount of the incentive, or the probability of winning the lottery may be the amount of the incentive.

[0071] In this embodiment, incentives are given to the user who asked the question and the user who corrected the response. The amount of incentive given to the user who asked the question and the user who corrected the response may be different. For example, a larger incentive may be given to the user who corrected the response than to the user who asked the question.

[0072] The incentive grant number calculation unit 227 can determine the amount of incentive to be granted to each user, for example. The incentive grant number calculation unit 227 can calculate, for example, an incentive for a user who asked a question (questioner) and an incentive for a user who corrected the response (corrector). The incentive grant number calculation unit 227 can determine the amount of incentive so that a larger incentive is given to a user who provided a question and / or correction content that has a larger number of evaluations and / or quality evaluation scores.

[0073] ==Incentives for Questioners== The incentive grant amount calculation unit 227 can determine the amount of incentive to be granted to the user who has asked a question, depending on at least one of the popularity of the question, the proactiveness of the question, the quality of the question, and the quality of the questioner. The popularity of the question can be, for example, the number of views of the question. The proactiveness of the question is the degree to which the user is proactively asking questions, and can be evaluated, for example, by the number of questions asked by the user, or the statistical amount thereof. The quality of the question is the degree to which the question is considered to be good by many users, and can be evaluated, for example, by the number of "likes" given by other users to a question from the user, or the statistical amount thereof. The quality of the questioner is whether the user who has asked the question has discerning eyes, that is, the degree to which other users also consider a question that the user considers good to be good, and can be evaluated, for example, by the number of "likes" given by other users to a question that the user has "liked."

[0074] The incentive grant amount calculation unit 227 can calculate the amount of incentive for each of the evaluation value of the question proactiveness (question proactiveness), the evaluation value of the question quality (question quality), and the evaluation value of the questioner quality (questioner quality) as described above, using the following formula. Incentive amount = coefficient a x question activity + coefficient b x question quality + coefficient c x questioner quality

[0075] The degree to which the proactiveness of the question, the quality of the question, and the quality of the questioner are emphasized may be adjusted by coefficients, which may be set arbitrarily. The above formula is not limited to a linear sum, and any formula may be adopted in which at least one of the proactiveness of the question, the quality of the question, and the quality of the questioner is used as a variable, and the larger the value of each evaluation value, the larger the amount of the incentive is calculated.

[0076] The incentive grant amount calculation unit 227 can calculate the amount of incentive only from the question enthusiasm, for example, as in the following formula. Incentive amount = coefficient x question enthusiasm

[0077] The incentive award amount calculation unit 227 can calculate the amount of incentive only from the quality of the question, for example, as in the following formula. Incentive amount = coefficient x question quality

[0078] The incentive award amount calculation unit 227 can calculate the amount of incentive based only on the quality of the questioner, for example, as in the following formula. Incentive amount = coefficient x questioner quality

[0079] The amount of the incentive may be calculated according to a combination of two or more values ​​obtained by multiplying each of the question proactiveness, the question quality, and the questioner quality by a coefficient.

[0080] The incentive-granted number calculation unit 227 can also grant an incentive based on the popularity (number of views) of a question. The incentive-granted number calculation unit 227 can calculate the amount of the incentive, for example, by the following formula. Incentive amount = coefficient x question popularity

[0081] The incentive given number calculation unit 227 may determine the amount of the incentive according to at least two of the above-mentioned question enthusiasm, the quality of the question, the quality of the questioner, and the popularity of the question. The incentive given number calculation unit 227 can calculate the amount of the incentive, for example, by the following formula. Incentive amount = coefficient x question popularity

[0082] Furthermore, the incentive award amount calculation unit 227 may award an incentive depending on whether the question is known or not.

[0083] Furthermore, the incentive grant number calculation unit 227 may correct the amount of incentive depending on whether the question is known or not. The incentive grant number calculation unit 227 may increase the amount of incentive when the question is not known, or decrease the amount of incentive when the question is known. The amount of increase or decrease may be linear, may be stepped, or may be determined by some function.

[0084] ==Incentives for correctors== The incentive grant number calculation unit 227 can determine the type and / or amount of incentive to be granted to a user who has corrected a response, depending on at least one of, for example, the promptness of the correction, the popularity of the correction, the proactiveness of the correction, the quality of the correction, and the quality of the corrector. The promptness of the correction can be the speed at which a correction is provided for a response to a question. In this case, the response management table 312 is configured to add the date and time at which the response corresponding to the question is provided, and the correction result management table 313 is configured to add the date and time at which the correction is provided, and the length of time from the date and time at which the response corresponding to the question is provided to the date and time at which the correction is provided (hereinafter, referred to as the correction time) can be used. The popularity of the correction can be, for example, the number of views of the correction result. The proactiveness of the correction is the degree to which a user actively performs corrections, and can be evaluated, for example, by the number of corrections made by the user, or the statistics thereof. The quality of the correction is the degree to which a correction is considered to be good by many users, and can be evaluated, for example, by the number of "likes" given by other users to the corrections made by the user. The quality of a corrector is determined by whether or not the user who made the correction has an eye for quality, i.e., the degree to which other users also consider corrections that the user thinks are good to be good. For example, the quality can be evaluated based on the number of "likes" given by other users to corrections that the user has "liked," or on statistics about such numbers.

[0085] The incentive grant amount calculation unit 227 can calculate the amount of incentive for each of the evaluation value of the proactiveness of correction (proactiveness of correction), the evaluation value of the quality of correction (quality of correction), and the evaluation value of the quality of the corrector (quality of corrector) as described above, using the following formula. Incentive amount = coefficient d x willingness to correct + coefficient e x quality of correction + coefficient f x quality of corrector

[0086] The incentive grant amount calculation unit 227 can also determine the amount of incentive based only on the proactiveness of correction, for example, as in the following formula. Incentive amount = coefficient x proactiveness in correction

[0087] The incentive amount calculation unit 227 can also calculate the amount of incentive based only on the quality of correction, for example, as in the following formula. Incentive amount = coefficient x quality of correction

[0088] The incentive amount calculation unit 227 can also calculate the amount of incentive based only on the quality of the corrector, for example, as in the following formula. Incentive amount = coefficient x quality of corrector

[0089] The incentive-to-be-given calculation unit 227 can also grant an incentive for the speed of correction. The incentive-to-be-given calculation unit 227 can calculate the amount of the incentive, for example, by the following formula. Incentive amount = coefficient x (threshold - correction time)

[0090] In addition, the incentive given number calculation unit 227 can also give an incentive based on the popularity (number of views) of the correction. The incentive given number calculation unit 227 can calculate the amount of the incentive, for example, by the following formula. Incentive amount = coefficient x popularity of corrections

[0091] The incentive given number calculation unit 227 can also give an incentive for a combination of the proactiveness in correction and the popularity of the correction (number of views). The incentive given number calculation unit 227 can calculate the amount of the incentive, for example, by the following formula. Incentive amount = Coefficient 1 x Proactiveness in correction + Coefficient 2 x Popularity of correction

[0092] The incentive-to-be-given calculation unit 227 can grant an incentive for the speed of the correction. The incentive-to-be-given calculation unit 227 can calculate the amount of the incentive, for example, by the following formula. Incentive amount = coefficient x (threshold - correction time)

[0093] The incentive grant number calculation unit 227 may determine the type of incentive depending on the speed of correction. For example, an incentive setting unit may be provided that presets the type of incentive to be given for each range of the value of threshold minus correction time, and the incentive grant number calculation unit 227 may determine the type of incentive corresponding to the value of threshold minus correction time.

[0094] In addition, the amount of the incentive can be calculated according to a combination of two or more values ​​obtained by multiplying a coefficient for each of the proactiveness in correction, the quality of the correction, the quality of the corrector, and the speed of correction (threshold value-correction time).

[0095] The degree to which the willingness to correct, the quality of the correction, and the quality of the corrector are emphasized may be adjusted by coefficients, which can be set arbitrarily. The above formula is not limited to a linear sum, and any formula can be adopted in which at least one of the willingness to correct, the quality of the correction, and the quality of the corrector is used as a variable, and the larger the value of each evaluation value, the larger the amount of incentive is calculated.

[0096] Furthermore, the incentive award amount calculation unit 227 may award an incentive depending on whether the correction result is known or not.

[0097] Furthermore, the incentive grant number calculation unit 227 may correct the amount of incentive depending on whether the correction result is known or not. The incentive grant number calculation unit 227 may increase the amount of incentive when the correction result is not known, or decrease the amount of incentive when the correction result is known. The amount of increase or decrease may be linear, may be stepped, or may be determined by some function.

[0098] ==Incentives for reviewers== In addition, an incentive may be given to a user who has evaluated a question, a response, or a correction result. In this case, the incentive award number calculation unit 227 can determine the type and / or amount of incentive to be given to a user who has evaluated a question, a response, or a correction result, depending on at least one of, for example, the promptness of the evaluation, the positivity of the evaluation, the quality of the evaluation, and the quality of the evaluator. The promptness of the evaluation can be the speed of the time until the question, the response, or the correction result is evaluated. In this case, the data storage unit 231 is provided with an evaluation management table, and a question ID indicating the question to be evaluated or a correction ID indicating the correction result to be evaluated, a user ID (evaluator ID) indicating the evaluator, and an evaluation date and time can be stored in association with each other. In addition, the question management table 311, the response evaluation management table (not shown), and / or the correction result management table 313 can be added with the date and time of the question, the response, or the correction, and the length of time from the evaluation date and time to the date and time when the question, the response, or the correction is evaluated (hereinafter, referred to as evaluation time) can be used. The proactiveness of evaluation is the degree to which a user is proactive in evaluating, and can be evaluated, for example, by the number of times the user has evaluated something such as "liked," or by the statistical amount thereof. The quality of evaluation is the degree to which many users give good evaluations to things that they think deserve good evaluations, and can be evaluated, for example, by the number of "likes" from other users to questions or corrections that the user has "liked," or by the statistical amount thereof.

[0099] The incentive grant amount calculation unit 227 can calculate the amount of incentive for each of the evaluation value of the evaluation proactiveness (evaluation proactiveness), the evaluation value of the evaluation quality (evaluation quality), and the evaluation value of the evaluator quality (evaluator quality) using the following formula. Incentive amount = coefficient g × evaluation aggressiveness + coefficient h × evaluation quality + coefficient i × evaluator quality

[0100] The incentive given amount calculation unit 227 can also determine the amount of incentive based only on the evaluation aggressiveness, for example, as in the following formula. Incentive amount = coefficient x evaluation aggressiveness

[0101] The incentive given amount calculation unit 227 can also calculate the amount of incentive from only the quality of the evaluation, for example, as in the following formula. Incentive amount = coefficient x evaluation quality

[0102] The incentive award amount calculation unit 227 can also calculate the amount of incentive from only the quality of the evaluator, for example, as in the following formula. Incentive amount = coefficient x evaluator quality

[0103] The incentive-to-be-given calculation unit 227 can also grant an incentive for the speed of evaluation. The incentive-to-be-given calculation unit 227 can calculate the amount of the incentive, for example, by the following formula. Incentive amount = coefficient x (threshold - evaluation time)

[0104] The amount of incentive may be calculated according to a combination of two or more values ​​obtained by multiplying each of the evaluation aggressiveness, the quality of the evaluation, the quality of the evaluator, and the speed of the evaluation (threshold value-evaluation time) by a coefficient.

[0105] The incentive given number calculation unit 227 may calculate the incentive for the evaluation of the question, the incentive for the evaluation of the response, and the incentive for the correction result separately. In this case, the incentive given number calculation unit 227 can calculate the promptness, proactiveness, quality of evaluation, or quality of the evaluator for the question, the promptness, proactiveness, quality of evaluation, or quality of the evaluator for the response, and the promptness, proactiveness, quality of evaluation, or quality of the evaluator for the correction result, respectively, and then calculate the incentives as described above.

[0106] When receiving an evaluation for a response, a response evaluation management table may be provided. The response evaluation management table may be a table for managing information on the evaluation of a response (hereinafter, referred to as response evaluation information). The record (response evaluation information) managed in the response evaluation management table may include information for identifying a response (which may be a response ID or a question ID), an evaluation received from a user (evaluator) for the response (in this embodiment, the number of users who gave "Like" to the content of the response is the number of evaluations), the number of views of the response, the number of evaluations, and a score (quality evaluation score) determined in consideration of the number of views. Both the evaluation received from the user and the score (final evaluation) determined based on the evaluation can be managed in the response evaluation information. Note that the evaluation is not limited to the number of "Likes", but may be the number of times the "Useful" button was pressed, or may be a score value set by the evaluator, such as a score of 1 to 5 stars. In addition, instead of an evaluation value from a user (evaluator), it may be a value automatically calculated based on a given rule or function.

[0107] Furthermore, the incentive grant number calculation unit 227 may correct the amount of incentive depending on whether the evaluated question or correction result is known or not. For example, the incentive grant number calculation unit 227 may increase the amount of incentive when the question or correction result is not known, or decrease the amount of incentive when the question or correction result is known. The amount of increase or decrease may be linear, may be stepped, or may be determined by some function.

[0108] ==Calculation of incentives per question / correction== The incentive grant number calculation unit 227 can also grant incentives on a per question and / or per correction basis. For example, the incentive grant number calculation unit 227 can grant an amount of incentive to the questioner or the corrector according to the number of views of the question or the correction. The incentive grant number calculation unit 227 may continue to grant an incentive to the questioner or the corrector every time the question or the correction is viewed.

[0109] The incentive grant number calculation unit 227 can issue a non-fungible token (NFT) indicating the right to receive an incentive. The incentive grant number calculation unit 227 can issue an NFT linked to a question and / or a correction, for example. The issuance of an NFT is omitted here as it uses general blockchain technology. The incentive grant number calculation unit 227 can grant an incentive for a question and / or a correction to the owner of the NFT. This NFT may be tradable.

[0110] The incentive award number calculation unit 227 may award an incentive when the learning unit 224 uses the correction result for re-learning.

[0111] The incentive grant number calculation unit 227 can grant an incentive when the correction result is non-public information (closed data). The incentive grant number calculation unit 227 can grant a larger incentive for the correction result of non-public information than for the correction result that is public information. The incentive grant number calculation unit 227 may not grant an incentive for public information.

[0112] The incentive award amount calculation unit 227 transmits to the Web server 21 content (hereinafter referred to as incentive content) for displaying the determined incentive details and amount (amount awarded).

[0113] The incentive given number display unit 219 of the WEB server 21 transmits the incentive content received from the AP server 22 to the user terminal 1. An incentive management table that stores the content and amount of the incentive calculated by the incentive given number calculation unit 227 in association with the user ID indicating the user to be given the incentive may be provided in the data storage unit 231, and the incentive given number display unit 219 may, for example, in response to a request from the user terminal 1, read out the content and amount of the incentive corresponding to the user from the incentive management table, create incentive content, and respond to the user terminal 1 with the created incentive content.

[0114] The incentive amount display unit 119 of the user terminal 1 can display on the screen the type and amount of the incentive given to the user based on the incentive content transmitted from the Web server 21.

[0115] == Viewing log == The viewing log acquisition unit 220 of the WEB server 21 can acquire an access log of access to the WEB server 21 from the user terminal 1, for example. The viewing log acquisition unit 220 can acquire an access log of a general WEB server. The viewing log acquisition unit 220 can register the acquired viewing log in the viewing log storage unit 233 managed by the DB server 23.

[0116] <Operation> 9 is a diagram showing a flow of processing by the learning unit 224. First, the learning unit 224 reads data from the response management table 312, the question evaluation management table 311, and the correction content evaluation management table 315 in the data storage unit 231 (S401). Next, the learning unit 224 refers to the value of the quality evaluation score in the question evaluation management table 311 and selects a question with a certain score or higher. The learning unit 224 refers to the response management table 312 for the correction ID corresponding to the selected question, refers to the value of the quality evaluation score in the correction content evaluation management table 315 corresponding to the response ID, and selects a correction result with a certain score or higher (S402). The learning unit 224 adds the pair of the high-quality question and the correction result extracted in step S402 to the known information storage unit 234 (S403).

[0117] 10 is a diagram showing a process flow by the question quality judgment unit 225. The question quality judgment unit 225 reads data from the question evaluation management table 311 in the data storage unit 231 (S421). The question quality judgment unit 225 reads data of the browsing log from the browsing log storage unit 233 (S422). The question quality judgment unit 225 counts the number of accesses (number of views) to the relevant WEB page corresponding to the question ID from the browsing log, and sets the number of views of the question evaluation information managed in the question evaluation management table 311 (S423). The question quality judgment unit 225 calculates a quality evaluation score from the number of evaluations (number of likes) and the number of views in the question evaluation management table 311 (S424). The question quality judgment unit 225 sets the calculated quality evaluation score as the quality evaluation score of the question evaluation information managed in the question evaluation management table 314 (S425). The question quality judgment unit 225 stores the question evaluation management table 311 in the data storage unit 231 (S426).

[0118] FIG. 11 is a diagram showing a flow of processing by the correction quality judgment unit 226. The correction quality judgment unit 226 reads data from the correction content evaluation management table 315 in the data storage unit 231 (S441). The correction quality judgment unit 226 reads the data of the viewing log from the viewing log storage unit 233 (S442). The correction quality judgment unit 226 counts the number of accesses (number of views) to the relevant WEB page corresponding to the correction ID from the viewing log, and sets the number of views of the correction content evaluation information managed in the correction content evaluation management table 315 (S443). The correction quality judgment unit 226 calculates a quality evaluation score from the number of evaluations (number of likes) and the number of views in the correction content evaluation management table 315 (S444). The correction quality judgment unit 226 sets the calculated quality evaluation score to the quality evaluation score of the correction content evaluation information managed in the correction content evaluation management table 315 (S445). The correction quality judgment unit 226 stores the correction content evaluation management table 315 in the data storage unit 231 (S446).

[0119] FIG. 12 is a diagram showing a flow of calculation processing of incentives for users who have asked questions. The incentive-granted number calculation unit 227 reads data from the questioner management table 316 in the data storage unit 231 (S461). The incentive-granted number calculation unit 227 counts the number of question IDs of questioner information managed in the questioner management table 316 for each questioner ID to calculate the number of questions (S462). The incentive-granted number calculation unit 227 calculates the statistics (which may be the sum, average, deviation, etc.) of the acquired quality scores of the questioner information managed in the questioner management table 316 for each questioner ID (S463). The incentive-granted number calculation unit 227 calculates the statistics (sum, average, deviation) of the "acquired quality evaluation scores" linked to the "question IDs of other people who have evaluated" in the questioner management table 316 for each questioner ID (S464). The incentive-granted number calculation unit 227 determines the number of incentives for each questioner ID in consideration of the calculated values ​​(S465). The incentive award amount calculation unit 227 can award an incentive to the user who asked the question based on the calculated incentive amount (S466).

[0120] FIG. 13 is a diagram showing the flow of calculation processing of incentives for users who have corrected responses. The incentive grant number calculation unit 227 reads data from the corrector management table 317 in the data storage unit 231 (S481). The incentive grant number calculation unit 227 counts the number of correction IDs of the corrector information managed in the corrector management table 317 for each corrector ID to calculate the number of corrections (S482). The incentive grant number calculation unit 227 calculates the statistics (which may be the sum, average, deviation, etc.) of the acquired quality scores of the corrector information managed in the corrector management table 317 for each corrector ID (S483). The incentive grant number calculation unit 227 calculates the statistics (the sum, average, deviation) of the "acquired quality evaluation score" linked to the "correction ID of another person who evaluated" of the corrector information managed in the corrector management table 317 for each corrector ID (S484). The incentive grant number calculation unit 227 determines the number of incentives for each corrector ID in consideration of the calculated value (S485). The incentive grant number calculation unit 227 can grant incentives to the user who performed the correction based on the calculated number of incentives (S486).

[0121] Fig. 14 is a diagram showing an example of a hardware configuration of a computer. Note that the illustrated configuration is an example, and other configurations may be used. The computer shown in Fig. 14 can implement a user terminal 1 and a response device 2 (a WEB server 21, an AP server 22, and a DB server 23).

[0122] The computer includes a CPU 201, a memory 202, a storage device 203, a communication interface 204, an input device 205, and an output device 206. The storage device 203 stores various data and programs, and is, for example, a hard disk drive, a solid state drive, or a flash memory. The communication interface 204 is an interface for connecting to a communication network, and is, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone line network, a wireless communication device for wireless communication, a USB (Universal Serial Bus) connector or an RS232C connector for serial communication, etc. The input device 205 is, for example, a keyboard, a mouse, a touch panel, a button, a microphone, etc. for inputting data. The output device 206 is, for example, a display, a printer, a speaker, etc. for outputting data. In addition, each functional unit of the above-mentioned user terminal 1 and response device 2 (WEB server 21, AP server 22, DB server 23) is realized by the CPU 201 reading out a program stored in the storage device 203 into the memory 202 and executing it, and each storage unit can be realized as part of the storage area provided by the memory 202 and the storage device 203.

[0123] Although the present embodiment has been described above, the above embodiment is intended to facilitate understanding of the present invention, and is not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit of the present invention, and equivalents thereof are also included in the present invention.

[0124] <Select from multiple candidates> For example, in the present embodiment, it is assumed that the output (response) from the trained model is one for one question, but multiple outputs may be made. For example, the response unit 221 may generate multiple responses by trying to give a question to the trained model and generate a response multiple times. When the trained model is a generator, different responses can be generated by increasing a randomness parameter (for example, the temperature parameter of GPT). In addition, multiple types of trained models may be prepared, and a prompt may be given to each of the trained models to generate multiple answers. In addition, when multiple pieces of known information are searched for, each piece of known information may be included in a prompt and given to the trained model to generate multiple answers.

[0125] Here, multiple responses (output results from the trained model) may be output, and the corrector may select the output that he or she considers to be of the highest quality, and then correct the selected response.

[0126] Alternatively, multiple output results may be output, and the viewer may select which of the multiple output results is the highest quality output, without correcting them. In this case, the learning unit 224 may register the output result selected by the viewer in the known information storage unit 234.

[0127] The response unit 221 can also output multiple responses to a question from the trained model and automatically select the highest quality output from among them as the final response. The quality can be determined based on the acceptability to the user or the validity of the answer based on a rule base or preset conditions.

[0128] <References> The response unit 221 may provide information that may be useful for correction together with the response from the trained model. Information that may be useful for correction can be specified by, for example, collecting reference information from information sources such as websites, blog information, and papers, determining the similarity between the collected data and the response from the trained model, and selecting information whose similarity is equal to or greater than a predetermined value. The response unit 221 may display a list of the response from the trained model and the selected reference information.

[0129] In addition, the information that the corrector used as a reference or basis for making the corrections may be registered in the known information storage unit 234.

[0130] <Human Answer> In this embodiment, known information corresponding to a question is extracted and given to the large-scale language model to generate an answer to the question, but the answer to the question may be provided by a user who is an answerer. For example, when known information corresponding to a question cannot be extracted (for example, when known information having a similarity equal to or greater than a predetermined value cannot be searched for), a predetermined answerer may answer the question, and the answer content may be associated with a user ID indicating the answerer and registered in the known information storage unit 234. In this case, the answer content may be regarded as a response from the large-scale language model in the above-mentioned embodiment and may be corrected by a corrector, or the answer content may be regarded as a correction result in the above-mentioned embodiment.

[0131] <Incentive Beneficiary Rights> In this embodiment, the questioner, corrector, and evaluator receive incentives, but the right to receive these incentives may be transferable to others. For example, the management server 2 can connect to a blockchain network and mint an NFT linked to a user by a smart contract implemented in the blockchain network. The NFT to be minted is linked to information identifying the user (e.g., a user ID). The user ID may be set in the metadata of the NFT. In this case, the management server 2 can pay incentives to the wallets of the owners of the NFTs linked to the user who asked the question, the user who corrected the question, and the user who evaluated the question.

[0132] Furthermore, when an NFT is resold (secondary distribution), a portion of the transfer fee may be distributed to the original user. In this case, a transfer smart contract is deployed on a blockchain network, and when an NFT is transferred, the smart contract can pay a portion of the transfer fee (for example, an arbitrary percentage such as 10%) to the user associated with the NFT to be transferred, and pay the remaining amount (which can be deducted from system fees, gas fees, etc.) to the transferor of the NFT. In this case, the payment process can be assumed to be made in virtual currency. [Explanation of symbols]

[0133] 1 User terminal 2 Answering Machine 21 Web Server 22 AP Server 23 Database Server

Claims

1. a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result of correcting the output data from a second user; a known-data determination unit that provides the input data or a keyword included in the input data to a search engine to obtain search results, and determines whether the correction result is known based on whether the obtained search results include content similar to the correction result; An information processing system comprising:

2. storing known information; accepting input data from a first user; A step of providing the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputting output data; receiving a correction result of correcting the output data from a second user; a step of providing the input data or a keyword included in the input data to a search engine to obtain search results, and judging whether the correction result is known based on whether the obtained search results include content similar to the correction result; An information processing method characterized by being executed by a computer.

3. storing known information; accepting input data from a first user; A step of providing the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputting output data; receiving a correction result of correcting the output data from a second user; a step of providing the input data or a keyword included in the input data to a search engine to obtain search results, and judging whether the correction result is known based on whether the obtained search results include content similar to the correction result; A program for causing a computer to execute the following.

Images