Information processing system, program, and method

Abstract

To effectively manage information.SOLUTION: An information processing system includes a known information storage unit configured to store known information, an input data receiving unit configured to receive input data from a first user, a processing unit configured to provide known information related to the input data and the input data to a learning model trained by machine learning and cause the learning model to output output data, a correction result receiving unit configured to receive a correction result obtained by correcting the output data from a second user, and an updating unit configured to determine whether the known information matching or similar to the correction result is stored in the known information storage unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system, a program, and a method. [Background technology]

[0002] Technologies that allow machines to automatically respond to questions and requests from users have been developed, and are now being provided as automatic response services such as bots (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-3533 Summary of the Invention [Problem to be solved by the invention]

[0004] It is difficult to control whether the responses from automated response services are favorable, and there is a demand for more effective information management.

[0005] The present invention has been made in view of the above background, and has as its object to provide a technique that enables effective management of information. [Means for solving the problem]

[0006] The main invention of the present invention for solving the above problem is an information processing system comprising: a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a trained model using machine learning to output output data; a correction result receiving unit that receives correction results of the output data from a second user different from the first user; and a known determination unit that determines whether the known information that matches or is similar to the correction result is stored in the known information storage unit. [Effects of the Invention]

[0007] According to the present invention, information can be managed effectively. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an overview of an information processing system according to an embodiment of the present invention. [Figure 2] FIG. 10 is a diagram illustrating an example of the configuration of a question management table 311. [Figure 3] FIG. 10 is a diagram illustrating an example of the configuration of a response management table 312. [Figure 4] FIG. 10 is a diagram showing an example of the configuration of a correction result management table 313. [Figure 5] FIG. 10 is a diagram showing an example of the configuration of a question evaluation management table 314. [Figure 6] FIG. 10 is a diagram showing an example of the configuration of a correction content evaluation management table 315. [Figure 7] FIG. 10 is a diagram illustrating an example of the configuration of a questioner management table 316. [Figure 8] FIG. 10 is a diagram showing an example of the configuration of a corrector management table 317. [Figure 9] FIG. 10 is a diagram showing the flow of processing by a learning unit 224. [Figure 10] FIG. 10 is a diagram showing the flow of processing by the question quality determination unit 225. [Figure 11] FIG. 10 is a diagram showing the flow of processing by the correction quality determination unit 226. [Figure 12] FIG. 10 is a diagram showing the flow of a process for calculating an incentive for a user who has asked a question. [Figure 13] FIG. 10 is a diagram showing the flow of a process for calculating an incentive for a user who corrects a response. [Figure 14] FIG. 2 illustrates an example of the hardware configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0009] <Summary of the Invention> The present invention will be described by listing the contents of the embodiments. For example, the present invention has the following configuration. [Item 1] a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a trained model by machine learning and outputs output data; a correction result receiving unit that receives correction results of the output data from a second user different from the first user; a known information determination unit that determines whether the known information that matches or is similar to the correction result is stored in the known information storage unit; An information processing system comprising: [Item 2] Item 1, an information processing system according to item 1, an expert storage unit that stores whether or not a user is an expert; the output unit refers to the expert storage unit, identifies the expert as the second user, and outputs the output data only to the identified second user; outputting the correction results corrected by the second user so that the first user can view them; An information processing system characterized by: [Item 3] Item 1, an information processing system according to item 1, further comprising a correction evaluation receiving unit that receives an evaluation from a third user on the correction result; the output unit outputs the input data and the correction result so as to be viewable by the third user; the correction result receiving unit receives the correction results from a plurality of the second users; the updating unit selects the correction result in accordance with the evaluation, and updates the trained model using the selected correction result; An information processing system characterized by: [Item 4] Item 3. The information processing system according to item 3, Further, a correction quality determination unit is provided that calculates at least the values ​​corresponding to the evaluations and determines the quality of the correction results, the updating unit selects at least a part of the correction results according to the quality; An information processing system characterized by: [Item 5] Item 4. The information processing system according to item 4, the correction quality determination unit determines the quality according to at least the aggregated value and the number of the third users who have viewed the correction results; An information processing system characterized by: [Item 6] Item 1, an information processing system according to item 1, further comprising an input evaluation receiving unit that receives an evaluation from a third user regarding the input data; the output unit outputs the input data so as to be viewable by the second user and the third user; the updating unit updates the trained model using the input data whose evaluation satisfies a predetermined criterion and the correction result of the output data corresponding to the input data; An information processing system characterized by: [Item 7] Item 6. The information processing system according to item 6, the input evaluation receiving unit receives the evaluations from a plurality of the third users; an input quality determination unit that determines the quality of the input data in accordance with at least the evaluation; the updating unit updates the trained model using the input data whose quality satisfies a predetermined standard and the correction result; An information processing system characterized by: [Item 8] Item 7. The information processing system according to item 7, the input quality determination unit determines the quality in accordance with at least one of the evaluation and the number of the third users who have viewed the input data; An information processing system characterized by: [Item 9] Item 1, an information processing system according to item 1, a correction evaluation receiving unit that receives an evaluation from a third user regarding the correction result; a correction quality determination unit that determines the quality of the correction result based on at least a total value of evaluation values ​​according to the evaluations and the number of third users who have viewed the correction result; an input evaluation receiving unit that receives an evaluation from a third user regarding the input data; an input quality determination unit that determines the input quality of the input data according to at least a total value of evaluation values ​​according to the evaluations and the number of the third users who have viewed the input data; Furthermore, the output unit outputs the input data and the correction result so as to be viewable by the second user and the third user; the updating unit updates the trained model using the input data whose input quality is equal to or greater than a first predetermined value and the correction result whose correction quality is equal to or greater than a second predetermined value; An information processing system characterized by: [Item 10] Item 1, an information processing system according to item 1, an incentive granting unit that grants an incentive to the second user who has provided the correction result; An information processing system characterized by: [Item 11] Item 1, an information processing system according to item 1, an incentive granting unit that grants an incentive to the first user who provides the input data; An information processing system characterized by: [Item 12] Item 3. The information processing system according to item 3, an incentive granting unit that grants an incentive to the third user; An information processing system characterized by: [Item 13] Item 1, an information processing system according to item 1, the update unit determines whether the correction result is public information, and if the correction result is not public information, updates the trained model using the correction result; An information processing system characterized by: [Item 14] storing known information; accepting input data from a first user; A step of providing the known information related to the input data and the input data to a trained model by machine learning and outputting output data; receiving a correction result of correcting the output data from a second user different from the first user; a step of determining whether or not the known information that matches or is similar to the correction result is stored in the known information storage unit; A program that causes a computer to execute the following. [Item 15] storing known information; accepting input data from a first user; A step of providing the known information related to the input data and the input data to a trained model by machine learning and outputting output data; receiving a correction result of correcting the output data from a second user different from the first user; a step of determining whether or not the known information that matches or is similar to the correction result is stored in the known information storage unit; How a computer runs. [Item 16] Item 1, an information processing system according to item 1, an update unit that adds the correction result to the known information storage unit when the known information that matches or is similar to the correction result is not stored in the known information storage unit; An information processing system characterized by: [Item 17] <Known: Is it registered in the database?> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result obtained by correcting the output data from a second user; a known information determination unit that determines whether the correction result is known based on whether the known information that matches or is similar to the correction result is stored in the known information storage unit; An information processing system comprising: [Item 18] <Known judgement: Are the corrections similar to the search results of the input data?> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result obtained by correcting the output data from a second user; a known-data determination unit that provides the input data or keywords included in the input data to a search engine to obtain search results, and determines whether the correction results are known based on whether the obtained search results include content similar to the correction results; An information processing system comprising: [Item 19] <Known result determination: Search the correction results to see if there are any similar results> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result obtained by correcting the output data from a second user; a known-data determination unit that provides the correction results to a search engine to obtain search results, and determines whether the correction results are known based on whether the obtained search results contain content similar to the correction results; An information processing system comprising: [Item 20] <Question known status: Is it registered in the database?> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a known information determination unit that determines whether the known information that matches or is similar to the input data is stored in the known information storage unit; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; An information processing system comprising: [Item 21] Item 20. The information processing system according to item 20, a correction result receiving unit that receives correction results of the output data from a second user; An information processing system characterized by: [Item 22] <Question known status determination: Search for a question and see if there are similar search results> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result obtained by correcting the output data from a second user; Equipped with providing the input data or keywords contained in the input data to a search engine to obtain search results, and determining whether the correction results are public information based on whether the obtained search results contain content similar to the correction results; An information processing system comprising: [Item 23] <Incentive: Corrector, quantity only> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result obtained by correcting the output data from a second user; an incentive granting unit that grants an incentive to the second user who provided the correction result; Equipped with the incentive granting unit determines the amount of the incentive according to a correction enthusiasm that is a value according to the number of the correction results provided by the second user; An information processing system characterized by: [Item 24] Item 23. The information processing system according to Item 23, a correction evaluation receiving unit that receives an evaluation value from a third user for the correction result; the incentive granting unit determines the amount of the incentive according to the correction quality, which is a value according to an aggregate value of the evaluation values ​​from the third user for the correction result provided by the second user, and the correction enthusiasm; An information processing system characterized by: [Item 25] Item 24. The information processing system according to item 24, the incentive granting unit determines the amount of the incentive according to an evaluator quality, which is a value according to the number of third users who have provided the same evaluation value as the predetermined evaluation value provided by the second user, the correction quality, and the correction proactiveness for the same correction result; An information processing system characterized by: [Item 26] <Incentive: Questioner, quantity only> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result obtained by correcting the output data from a second user; an incentive granting unit that grants an incentive to the first user who provided the input data; Equipped with the incentive granting unit determines the amount of the incentive according to a question willingness, which is a value according to the number of the input data provided by the first user; An information processing system characterized by: [Item 27] Item 27. The information processing system according to Item 26, an input evaluation receiving unit that receives an evaluation value from a third user for the input data; the incentive granting unit determines the amount of the incentive according to a question quality, which is a value according to an aggregate value of the evaluation values ​​from the third users for the input data provided by the first user, and the question proactiveness; An information processing system characterized by: [Item 28] Item 27. The information processing system according to Item 27, the incentive granting unit determines the amount of the incentive according to a questioner quality, which is a value according to the number of third users who have provided the same evaluation value as the predetermined evaluation value provided by the first user, the question quality, and the questioning proactiveness for the same question; An information processing system characterized by: [Item 29] <Incentive:Evaluator> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result obtained by correcting the output data from a second user; an incentive granting unit that grants an incentive to the second user who provided the correction result; a correction evaluation receiving unit that receives an evaluation value from a third user regarding the correction result; Equipped with the incentive granting unit determines the amount of the incentive according to an evaluation aggressiveness, which is a value according to the number of the evaluation values ​​provided by the third user; An information processing system characterized by: [Item 30] Item 29. The information processing system according to Item 29, the incentive granting unit determines the amount of the incentive in accordance with evaluator quality, which is a value corresponding to the number of third users who have provided the same evaluation value as the predetermined evaluation value provided by the third user, for the same correction result, and the evaluation aggressiveness; An information processing system characterized by: [Item 31] <Incentives: Corrector, speed of correction> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result obtained by correcting the output data from a second user; an incentive granting unit that grants an incentive to the second user who provided the correction result; Equipped with the incentive granting unit determines the amount of the incentive depending on how quickly the output data is output and the correction result for the output data is received; An information processing system characterized by: [Item 32] <Incentives: Evaluator, speed of correction evaluation> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result obtained by correcting the output data from a second user; a correction evaluation receiving unit that receives an evaluation from a third user regarding the correction result; an incentive granting unit that grants an incentive to the third user who evaluated the correction result; Equipped with the incentive granting unit determines the amount of the incentive depending on how quickly the correction result is received and the evaluation value for the correction result is received; An information processing system characterized by: [Item 33] <Incentive: Evaluator, Question Evaluation> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result obtained by correcting the output data from a second user; an evaluation receiving unit that receives an evaluation value from a third user for the input data; an incentive granting unit that grants an incentive to the third user who provided the evaluation value; Equipped with the incentive granting unit determines the amount of the incentive according to an evaluation aggressiveness, which is a value according to the number of the evaluation values ​​provided by the third user; An information processing system characterized by: [Item 34] Item 33. The information processing system according to Item 33, the incentive granting unit determines the amount of the incentive in accordance with evaluator quality, which is a value corresponding to the number of third users who have provided the same evaluation value as the predetermined evaluation value provided by the third user, for the same input data, and the evaluation aggressiveness; An information processing system characterized by: [Item 35] <Incentive: Rater, Response Evaluation> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result obtained by correcting the output data from a second user; an evaluation receiving unit that receives an evaluation value from a third user for the output data; an incentive granting unit that grants an incentive to the third user who provided the evaluation value; Equipped with the incentive granting unit determines the amount of the incentive according to an evaluation aggressiveness, which is a value according to the number of the evaluation values ​​provided by the third user; An information processing system characterized by: [Item 36] Item 36. The information processing system according to Item 35, the incentive granting unit determines the amount of the incentive in accordance with evaluator quality, which is a value corresponding to the number of third users who have provided the same evaluation value as the predetermined evaluation value provided by the third user, and the evaluation aggressiveness, for the same output data; An information processing system characterized by: [Item 37] <Incentives: Questioner, number of question views> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result obtained by correcting the output data from a second user; an incentive granting unit that grants an incentive to the first user who provided the input data; Equipped with the incentive granting unit determines the amount of the incentive according to the number of views of the input data provided by the first user; An information processing system characterized by: [Item 38] Item 37, an information processing system according to item 37, the incentive granting unit determines the amount of the incentive according to a question willingness, which is a value according to the number of input data provided by the first user, and the number of views; An information processing system characterized by: [Item 39] Item 38, an information processing system according to item 38, an input evaluation receiving unit that receives an evaluation value from a third user for the input data; the incentive granting unit determines the amount of the incentive according to a question quality, which is a value according to an aggregate value of the evaluation values ​​from the third users for the input data provided by the first user, the question proactiveness, and the number of views; An information processing system characterized by: [Item 40] Item 39. The information processing system according to Item 39, the incentive granting unit determines the amount of the incentive according to a questioner quality, which is a value according to the number of third users who have provided the same evaluation value as the predetermined evaluation value provided by the first user, the question quality, the question proactiveness, and the number of views for the same question; An information processing system characterized by: [Item 41] <Incentives: Correctors, number of corrections viewed> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a processing unit that provides the known information related to the input data and the input data to a learning model that has been trained by machine learning, and outputs output data; a correction result receiving unit that receives a correction result obtained by correcting the output data from a second user; an incentive granting unit that grants an incentive to the second user who provided the correction result; Equipped with the incentive granting unit determines the amount of the incentive according to the number of views of the correction result provided by the second user; An information processing system characterized by: [Item 42] Item 41. An information processing system according to Item 41, the incentive granting unit determines the amount of the incentive according to a correction willingness, which is a value according to the number of correction results provided by the second user, and the number of views; An information processing system characterized by: [Item 43] Item 42. An information processing system according to Item 42, a correction evaluation receiving unit that receives an evaluation value from a third user for the correction result; the incentive granting unit determines the amount of the incentive according to the correction quality, which is a value according to an aggregate value of the evaluation values ​​from the third users for the correction result provided by the second user, the proactiveness in correction, and the number of views; An information processing system characterized by: [Item 44] Item 43. The information processing system according to Item 43, the incentive granting unit determines the amount of the incentive according to an evaluator quality, which is a value according to the number of third users who have provided the same evaluation value as the predetermined evaluation value provided by the second user, the correction quality, the proactiveness in correction, and the number of views for the same correction result; An information processing system characterized by: [Item 45] <Check if the answer is known: Is it registered in the database?> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a response data receiving unit that receives response data in response to the input data from a second user; a known information determination unit that determines whether the response data is known based on whether the known information that matches or is similar to the response data is stored in the known information storage unit; An information processing system comprising: [Item 46] <Known human responses: Are the human responses similar to the search results for the input data?> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a response data receiving unit that receives response data in response to the input data from a second user; a known-data determination unit that provides the input data or a keyword included in the input data to a search engine to obtain search results, and determines whether the response data is known based on whether the obtained search results include content similar to the response data; An information processing system comprising: [Item 47] <Determining whether a human answer is known: Search for a human answer and see if there are similar search results> a known information storage unit that stores known information; an input data receiving unit that receives input data from a first user; a response data receiving unit that receives response data in response to the input data from a second user; a known-data determination unit that provides the response data to a search engine to obtain search results, and determines whether the response data is known based on whether the obtained search results contain content similar to the response data; An information processing system comprising:

[0010] <System configuration> 1 is a diagram showing an overview of an information processing system according to one embodiment of the present invention. The information processing system of this embodiment is configured to include a response device 2. The response device 2 is communicably connected to a user terminal 1 via a communication network. The communication network is, for example, the Internet, and is constructed using a public telephone network, a mobile phone network, a wireless communication path, Ethernet (registered trademark), etc.

[0011] The user terminal 1 is a computer operated by a user. The user terminal 1 can be, for example, a smartphone, a tablet computer, or a personal computer. A web browser or an application runs on the user terminal 1, and the user can access the response device 2 via the web browser or the application.

[0012] The response device 2 may be a general-purpose computer such as a workstation or a personal computer, or may be logically realized by cloud computing.

[0013] In the information processing system of this embodiment, the response device 2 accumulates knowledge (known information) that serves as the basis for generating answers to questions. Based on this known information, an answer to a question from a user (questioner) is generated using a machine learning model (trained model) that has been trained by machine learning. The answer receives corrections from users (correctors) other than the user who asked the question. In this embodiment, the machine learning model is assumed to be a large-scale language model (LLM), and answer data can be generated using this machine learning model. Note that this embodiment assumes a so-called QA task of generating an answer (response) to a question. However, the present invention is not limited to the QA task, and any form of input provided to a machine learning model for output may be used. In other words, in this embodiment, a "question" can be understood as an "input" to the learning model, and an "answer" can be understood as an "output" from the learning model.

[0014] In this embodiment, it is assumed that the corrector is a user different from the questioner, the evaluator who evaluates the question is a user different from the questioner, and the evaluator who evaluates the correction result is a user different from the corrector. Note that the corrector may be the same as the questioner, or the questioner may be included as part of the correctors. The evaluator, questioner, and corrector may be the same person, or the questioner and corrector may be included as part of the evaluators. The questioner may be one or more people. The corrector may be one or more people. The evaluator may be one or more people.

[0015] <Software configuration> As shown in Figure 1, the user terminal 1 includes a question input unit 111, a response content display unit 112, a question evaluation input unit 113, a response correction input unit 114, a correction result display unit 115, a correction evaluation input unit 116, a question quality judgment result display unit 117, a correction quality judgment result display unit 118, and an incentive grant number display unit 119.

[0016] 1, the response device 2 includes a WEB server 21 that transmits and receives data to and from the user terminal 1, an AP server 22 that functions as middleware, and a DB server 23 that manages a database. Note that this configuration is merely an example, and the response device 2 may be configured with one or two computers. The response device 2 may also be configured with four or more computers.

[0017] The DB server 23 includes a data storage unit 231, a learned model storage unit 232, a viewing log storage unit 233, and a known information storage unit 234.

[0018] The web server 21 includes a question receiving unit 211, a response content display unit 212, a question evaluation receiving unit 213, a correction result receiving unit 214, a correction result display unit 215, a correction evaluation receiving unit 216, a question quality judgment result display unit 217, a correction quality judgment result display unit 218, an incentive grant number display unit 219, and a viewing log acquisition unit 220.

[0019] The AP server 22 includes a response unit 221 , a response content creation unit 222 , a correction result content creation unit 223 , a learning unit 224 , a question quality determination unit 225 , a correction quality determination unit 226 , and an incentive grant number calculation unit 227 .

[0020] The data storage unit 231 of the DB server 23 includes a question management table 311, a response management table 312, a correction result management table 313, a question evaluation management table 314, a correction content evaluation management table 315, a questioner management table 316, and a corrector management table 317.

[0021] The question management table 311 is a table for managing information (hereinafter referred to as question information) relating to questions received from users (questioners) who ask questions. Fig. 2 is a diagram showing an example of the configuration of the question management table 311. Records (question information) managed in the question management table 311 include information for identifying the question (question ID), information for identifying the user who asked the question (questioner ID), and the question content.

[0022] The response management table 312 is a table for managing information (hereinafter referred to as response information) relating to the content of an automatic response to a question from a user (questioner). Fig. 3 is a diagram showing an example of the configuration of the response management table 312. A record (response information) registered in the response management table 312 includes a question ID that identifies the question, information (response ID) that identifies the response, and the response content.

[0023] The correction result management table 313 is a table for managing information relating to the correction results (hereinafter referred to as correction result information) received from the user (corrector) in response to the content of the automatic response (response content). Fig. 4 is a diagram showing an example of the configuration of the correction result management table 313. The records (correction result information) managed in the correction result management table 313 include a response ID that identifies the response, information that identifies the user who performed the correction (corrector ID), information that identifies the correction result (correction ID), and the correction content.

[0024] The question evaluation management table 314 is a table for managing information regarding evaluations from users (evaluators) on the quality of questions (hereinafter referred to as question evaluation information). FIG. 5 is a diagram illustrating an example of the configuration of the question evaluation management table 314. Each record (question evaluation information) managed in the question evaluation management table 314 includes a question ID that identifies a question, evaluations (in this embodiment, the number of users who "liked" the question content) received from at least users other than the user who asked the question (even if the user who asked the question is included) for the question, the number of views of the question, and a score (quality evaluation score) determined taking into account the number of evaluations and the number of views. That is, in this embodiment, the question evaluation information manages both the evaluations received from users and the score (final evaluation) determined based on the evaluations. Note that the evaluation may be a score value set by the evaluator, such as 1 to 5 stars. Furthermore, instead of an evaluation value from a user (evaluator), a value may be automatically calculated based on a given rule or function. For example, the evaluation value may be the number of times a machine learning model cites the question or the correction results. In addition, text data such as comments, audio data, image data, etc. can also be given as evaluations. In this case, the evaluation value for the content of the evaluation can be estimated by classifying the text data, audio data, or image data using a classifier.

[0025] The correction content evaluation management table 315 is a table for managing information related to evaluations of the quality of correction content (hereinafter referred to as correction content evaluation information). FIG. 6 is a diagram showing an example of the configuration of the correction content evaluation management table 315. The records (correction content evaluation information) managed in the correction content evaluation management table 315 include information for identifying the correction content (correction ID), evaluations of the correction content received from at least users other than the user who corrected it (in this embodiment, the number of users who "liked" the correction content is defined as the number of evaluations), the number of views of the correction content, and a score (quality evaluation score) determined taking into account the number of evaluations and the number of views. That is, in this embodiment, the correction content evaluation information manages both the evaluations received from users and the score (final evaluation) determined based on those evaluations. Note that the evaluation is not limited to the number of "likes," but may also be the number of times the "helpful" button was pressed, or a score value set by the evaluator, such as 1 to 5 stars. Furthermore, instead of an evaluation value from a user (evaluator), it may be a value automatically calculated based on a given rule or function. For example, the number of times the machine learning model has cited the question or the correction result can be used as the evaluation value. Also, text data such as comments, audio data, image data, etc. can be used as evaluations. In this case, the evaluation value can be estimated for the content of the evaluation by classifying the text data, audio data, or image data using a classifier.

[0026] The questioner management table 316 is a table for managing information related to users who have asked questions (hereinafter referred to as questioner information). Fig. 7 is a diagram showing an example of the configuration of the questioner management table 316. The records (questioner information) managed in the questioner management table 316 include information identifying the user who asked the question (questioner ID), a question ID identifying the question, a quality evaluation score acquired for the question by the user, and a question ID identifying a question by another person that the user evaluated.

[0027] The corrector management table 317 is a table for managing information related to users who have made corrections (hereinafter referred to as corrector information). Fig. 8 is a diagram showing an example of the configuration of the corrector management table 317. The records (corrector information) managed in the corrector management table 317 include information (corrector ID) that identifies the user who made the corrections, a correction ID that identifies the correction results, a quality evaluation score obtained for the corrections made by that user, and a correction ID that identifies the correction results of others that the user evaluated.

[0028] The trained model storage unit 232 included in the DB server 23 stores a trained machine learning model (parameters constituting the model). For example, a trained model trained externally can be used as the machine learning model. Note that the DB server 23 may not be provided with the trained machine learning model, and an API or the like provided by another server may be used. In this embodiment, it is assumed that the trained model is a large-scale language model (LLM), and is a generator that generates an answer to an input prompt. Note that a prompt may be sent to an external server that provides a generator using the large-scale language model via an API or the like, and the external server may receive a result generated in response to the large-scale language model and the prompt.

[0029] The view log storage unit 233 provided in the DB server 23 stores a log of users viewing questions and correction contents on the user terminal 1. The view log can be a general access log. The view log contains sufficient information to count the number of times a question ID or correction ID has been viewed.

[0030] The known information storage unit 234 included in the DB server 23 can store any information (hereinafter referred to as known information). In this embodiment, the known information is assumed to be text data, but it can also be image data, audio data, etc. The known information storage unit 234 can include a file storage unit that stores files containing known information and a vector store that stores vector data in which the known information has been embedded. The file storage unit can be, for example, a file system or an object database. The file storage unit can be configured as a functional unit that stores files of known information in another file server and performs processing to read the known information from the other file server. The vector store can store information that identifies the known information (hereinafter referred to as a known information ID. Any unique information may be used, such as a numeric value or a character string issued as an ID, or, for example, a path to a file or a URL) in association with the vector data. A single piece of known information can be divided into chunks and embedded, and multiple pieces of vector data can be stored in association with one known information ID. The known information storage unit 234 can store user IDs indicating creators and / or providers of known information in association with each other. For example, the vector store may store user IDs indicating creators and / or providers, known information IDs, and vector data in association with each other, or may include a provider-known information correspondence table that associates user IDs indicating creators and / or providers with known information IDs. Furthermore, information specifying a question may be associated with the known information. The known information storage unit 234 may include, for example, a question-known information correspondence table that associates known information IDs specifying known information with question IDs specifying questions.

[0031] ==Answer to Questions== The question input unit 111 of the user terminal 1 accepts input of a question from a user. In this embodiment, the question is assumed to be text data, but the question can also be image data, audio data, or the like. The question input unit 111 transmits the question accepted from the user to the response device 2. The question input unit 111 can transmit the question, for example, attached to an HTTP request.

[0032] The question receiving unit 211 of the web server 21 can receive a question sent from the user terminal 1. The question receiving unit 211 can, for example, decode a question encoded in an HTTP request and send the decoded question to the AP server 22.

[0033] When the response unit 221 of the AP server 22 receives a question from the WEB server 21, the response unit 221 generates a response to the question by providing a prompt including the question to the machine learning model stored in the trained model storage unit 232 of the DB server 23, and transmits the generated response to the response content creation unit 222. The response unit 221 can also set the decoded question as the question content, set a questioner ID indicating the user of the user terminal 1 who asked the question, generate a new question ID to create question information, and register the created question information in the question management table 311 provided in the data storage unit 231 of the DB server 23. In this embodiment, the response unit 221 generates a response using so-called Retrieval-Augmented Generation (RAG). That is, the response unit 221 searches the known information storage unit 234 for known information similar to the question, and provides a prompt including the searched known information and the question to the trained model to generate a response. The prompt can include an instruction to generate an answer to the question based on the known information.

[0034] The response content creation unit 222 of the AP server 22 creates content (hereinafter referred to as response content; for example, this may be screen data written in HTML) for displaying the response received from the response unit 221 on the user terminal 1. The response content creation unit 222 transmits the created response content to the WEB server 21. Note that the response content may include not only the response but also a question. The response content may also include the number of ratings for the question (the number of "likes" for the question). The response content may also include information that identifies the known information included in the prompt.

[0035] The response content display unit 212 of the WEB server 21 receives response content for displaying the response from the AP server 22, and transmits the received response content to the user terminal 1. The response content display unit 212 can transmit content for displaying the question and the response not only to the user terminal 1 that is the sender of the question, but also to the user terminals 1 of other users.

[0036] In response to a request from the user terminal 1, the response content display unit 212 of the WEB server 21 sends a message to the response content creation unit 222 of the AP server 22, and the response content creation unit 222 reads one or more pieces of question information from the question management table 311 provided in the data storage unit 231 of the DB server 23, reads response information corresponding to the question ID for each of the read question information from the response management table 312, creates response content for displaying the question content and the response content, and sends it to the WEB server 21. The response content display unit 212 can also transmit this response content to the sender of the request. The response content creation unit 222 can also obtain the number of ratings corresponding to the question ID from the question rating management table 314 and include it in the response content.

[0037] The response content display unit 112 of the user terminal 1 receives the response content for displaying the response transferred from the Web server 21, and can display the response content to the user based on the response content.

[0038] ==Public / Private Setting Function== The questions and / or correction contents may be set to be public or private. In this case, information indicating public or private (public setting information) is set in the question information stored in the question management table 311 and / or the correction result information stored in the correction result management table 313, and only the question information and / or correction result information whose public setting information indicates "public" can be made public to general users.

[0039] It is also possible to accept private corrections even when the correction content is not public. In this case, if the disclosure setting information indicates that the correction is not public, the correction can be made public only to users of the company. This allows the company to accumulate knowledge exclusive to the company while preventing the company's know-how from being leaked.

[0040] ==Mask setting function== Furthermore, a part of the question and / or correction content may be masked. In this case, for example, for each question and / or correction content, characters to be masked or conditions for masking are set in the question information and / or correction result information, and the set characters are replaced with replacement characters, or parts of the question and / or correction content that satisfy the set conditions are replaced with replacement characters, and then the question and / or correction content can be made public and used for re-learning.

[0041] ==Question Rating== The question evaluation input unit 113 of the user terminal 1 accepts input of an evaluation of a question from a user. The question evaluation input unit 113 may accept a "like" for a question posted by a user other than the user using the user terminal 1, if the question is a good question. The question evaluation input unit 113 transmits the evaluation of the accepted question (for example, a "like") to the response device 2 together with a question ID that identifies the question.

[0042] The question evaluation receiving unit 213 of the web server 21 receives evaluations (e.g., "like") for questions from the user terminal 1. The question evaluation receiving unit 213 can increment the number of evaluations for question information in the question evaluation management table 314 that corresponds to the question ID that identifies the question.

[0043] ==Response Evaluation== Responses may be evaluated using a trained model. In this case, the user terminal 1 includes a response evaluation input unit, the web server 21 includes a response evaluation receiving unit, and the data storage unit 231 of the DB server 23 includes a response evaluation management table. The response evaluation management table can store, in association with a response ID that identifies the response, evaluations (e.g., the number of likes), the number of views of the response, and a quality evaluation score for the response. The response evaluation may be a score value set by an evaluator, such as 1 to 5 stars. Alternatively, the response evaluation may be a value automatically calculated based on a given rule or function, rather than an evaluation value from a user (evaluator). For example, the machine learning model may use the number of times the question or the correction result is cited as the evaluation value. Text data such as comments, audio data, or image data may also be given as an evaluation. In this case, the evaluation value for the content of the evaluation can be estimated by classifying the text data, audio data, or image data using a classifier.

[0044] The response evaluation input unit of the user terminal 1 accepts input of an evaluation of a response from a user. The response evaluation input unit can accept an evaluation of the content of a response viewed by the user. For example, the response evaluation input unit may accept a "like" if the content of the response is good. The response evaluation input unit transmits the evaluation of the accepted response (for example, "like") to the response device 2 together with a response ID that identifies the response (or a question ID that identifies the question).

[0045] The response evaluation receiving unit of the Web server 21 can receive an evaluation (for example, "Like") about a response from the user terminal 1, and increment the number of evaluations in the response evaluation management table corresponding to the response ID that identifies the response.

[0046] ==Response Correction== The response correction input unit 114 of the user terminal 1 can receive correction content from the user for the response that the user has viewed. The response correction input unit 114 transmits the received correction content to the WEB server 21 together with a response ID that identifies the response and a corrector ID that identifies the user.

[0047] The correction result receiving unit 214 of the Web server 21 receives the correction content from the user in response to the response. The correction result receiving unit 214 transmits the received correction content to the AP server 22 together with the response ID and correction ID.

[0048] When the correction result content creation unit 223 of the AP server 22 receives the correction content transmitted from the WEB server 21, it can register correction result information including the response ID, the corrector ID, a newly assigned correction ID (which may be assigned by the DB server 23), and the correction content in the correction result management table 313 provided in the data storage unit 231. The correction result content creation unit 223 can create content indicating that the correction result has been accepted (hereinafter referred to as correction result content). The correction result content can include the correction result received from the WEB server 21. The correction result content can also include, for example, a list of past correction results registered in the correction result management table 313. The correction result content creation unit 223 transmits the correction result content to the WEB server 21.

[0049] The correction result display unit 215 of the web server 21 transmits the correction result content received from the AP server 22 to the user terminal 1 .

[0050] The correction result display unit 115 of the user terminal 1 can display the correction details to the user based on the correction result content received from the web server 21.

[0051] In response to a request from the user terminal 1, the correction result display unit 215 of the WEB server 21 sends a message to the correction result content creation unit 223 of the AP server 22, and the correction result content creation unit 223 reads one or more pieces of response information from the response management table 312 provided in the data storage unit 231 of the DB server 23, and for each of the read response information, reads the question content corresponding to the question ID from the question management table 311, reads the correction result information corresponding to the response ID from the correction result management table 313, creates correction result content for displaying the question content, response content, and correction content(s), and sends it to the WEB server 21, and the correction result display unit 215 sends this correction result content to the sender of the request. The correction result content creation unit 223 can also obtain the number of evaluations corresponding to the correction ID from the correction content evaluation management table 315 and include it in the correction result content.

[0052] ==Evaluation of corrections== The correction evaluation input unit 116 of the user terminal 1 accepts input of an evaluation of the correction result from the user. The correction evaluation input unit 116 may accept a "like" if the correction result made by a user other than the user using the user terminal 1 is good. The correction evaluation input unit 116 transmits the evaluation of the accepted correction result (for example, "like") to the response device 2 together with a correction ID that identifies the correction result.

[0053] The correction evaluation receiving unit 216 of the web server 21 receives an evaluation (for example, "like") on the correction result from the user terminal 1. The correction evaluation receiving unit 216 can increment the number of evaluations in the correction content evaluation information in the correction content evaluation management table 315 corresponding to the correction ID that identifies the correction result.

[0054] ==Machine Learning== The learning unit 224 (corresponding to the update unit of the present invention) of the AP server 22 can learn the correction results in which a user corrects a response generated by a trained model in response to a question. In this embodiment, the learning unit 224 performs learning by adding the correction results to the known information storage unit 234. The learning unit 234 adds the correction results (which may be only the corrected portion or the entire response reflecting the correction results) to the known information storage unit 234. For each correction content registered in the correction result management table 313, the learning unit 224 can register the correction content in the known information storage unit 234 in association with a user ID (corrector ID) indicating the user who provided the correction result. The learning unit 224 can also obtain a user ID indicating the creator of the known information used to generate the response (known information similar to the question, identified by information identifying the known information included in the response content) from the known information storage unit 234, associate the user ID as the creator, and also associate a user ID (corrector ID) indicating the user who provided the correction content as the provider, and register the correction content in the known information storage unit 234.

[0055] For example, the learning unit 224 may read, for each piece of question information registered in the question management table 311, correction result information corresponding to the question ID from the correction result management table 313, and add the correction content included in the read correction result information to the known information storage unit 234. For example, the learning unit 224 may register the question in the known information storage unit 234 in association with the correction content.

[0056] ==Known Verdicts== The learning unit 224 (known information determination unit) may determine whether the correction result is known information. The learning unit 224 may not add known information to the known information storage unit 234. If the correction result is known, it may be determined that the correction result is public information (similarly, if the correction result is not known, the correction result is private information).

[0057] Furthermore, regarding whether or not the information is known, the learning unit 224 can determine the similarity between the known information stored in the known information storage unit 234 and the correction result, and determine that the correction result is known if the similarity is equal to or greater than a predetermined value. Furthermore, the learning unit 224 can determine the similarity between the result output by giving a question to the trained model and the correction result, and determine that the correction result is known information if the similarity is equal to or greater than a predetermined value.

[0058] Regarding whether or not something is known information, the learning unit 224 can, for example, provide the question or a keyword or phrase contained in the question to a publicly available search engine, determine whether or not the search results contain content similar to the correction result, and if it is determined that the correction result contains content similar to the correction result, determine that the correction result is known information. Furthermore, the learning unit 224 can, for example, provide the correction result or a keyword or phrase contained in the correction result to a publicly available search engine, determine whether or not the search results contain content similar to the correction result, and if it is determined that the correction result contains content similar to the correction result, determine that the correction result is known information.

[0059] Furthermore, it may be configured to determine whether a question is known. In this case, the learning unit 224 calculates the similarity between the question and known information stored in the known information storage unit 234, and can determine that the question is known if the similarity is equal to or greater than a predetermined value. Furthermore, if a question is stored in the known information storage unit 234 in association with another question, the learning unit 224 can calculate the similarity between the question stored in the known information storage unit 234 and the question content managed in the question management table 311, and can determine that the question is known if the similarity is equal to or greater than a predetermined value. If the learning unit 224 determines that the question is not known, or if the similarity is equal to or less than a predetermined threshold (which may be a value lower than the predetermined value), the learning unit 224 can add the question to the known information storage unit 234 in association with the correction result.

[0060] == Additional Conditions == The learning unit 224 may, for example, accumulate only correction results whose number of ratings and / or quality evaluation scores are equal to or greater than a given threshold. For example, for each piece of question information registered in the question management table 311, the learning unit 224 can add to the known information storage unit 234 correction result information corresponding to the question ID whose number of ratings is equal to or greater than a given threshold and / or whose quality evaluation score is equal to or greater than a given threshold (which may be the same as or different from the threshold for the number of ratings).

[0061] The learning unit 224 may also add only correction results related to responses to questions whose number of ratings and / or quality evaluation scores are equal to or greater than a given threshold. For example, the learning unit 224 may select, from among the question information registered in the question management table 311, those whose number of ratings is equal to or greater than a given threshold and / or those whose quality evaluation scores are equal to or greater than a given threshold (which may be the same as or different from the threshold for the number of ratings), and add the correction contents of the correction result information corresponding to the question ID of the selected question information to the known information storage unit 234 in association with a question ID indicating the selected question information. At this time, the learning unit 224 may also add only those correction result information corresponding to the question ID whose number of ratings is equal to or greater than a given threshold for the correction results and / or those whose quality evaluation scores are equal to or greater than a given threshold for the correction results (which may be the same as or different from the threshold for the number of ratings).

[0062] ==Additional Data Subject== The learning unit 224 may use not only the correction results but also explanatory articles on papers and web pages as data to be added to the known information storage unit 234. In this case, too, the incentive grant number calculation unit 227, which will be described later, can grant incentives to the authors of papers and web pages.

[0063] ==Judgement of Question Quality== The question quality judgment unit 225 of the AP server 22 judges the quality of a question. For each piece of question evaluation information registered in the question evaluation management table 314, the question quality judgment unit 225 can determine a quality evaluation score, for example, according to the number of ratings. For example, the question quality judgment unit 225 can calculate the quality evaluation score by dividing the number of ratings for the question evaluation information by the number of views. Note that the number of users who viewed the question may be used instead of the number of views. The question quality judgment unit 225 creates content for displaying the quality evaluation score for the question (hereinafter referred to as question quality evaluation content). The question quality evaluation content can be, for example, content for allowing users to view high-quality questions. The question quality judgment unit 225 can create screen data written in HTML for displaying the quality evaluation scores for one or more pieces of question evaluation information. The question quality judgment unit 225 transmits the question quality evaluation content to the WEB server 21.

[0064] The question quality judgment result display unit 217 of the WEB server 21 can receive the question quality evaluation content received from the AP server 22 and provide the received question quality evaluation content to the user terminal 1. In response to a request from the user terminal 1, the question quality judgment result display unit 217 may read out question evaluation information from the question evaluation management table 314, create question quality evaluation content, and respond.

[0065] The question quality judgment result display unit 117 of the user terminal 1 can receive the question quality evaluation content transmitted from the Web server 21 and display to the user a screen that displays the quality evaluation score of the question based on the question quality evaluation content.

[0066] ==Judgment of the quality of the corrections== The correction quality judgment unit 226 of the AP server 22 judges the quality of the correction content. For each piece of correction content evaluation information registered in the correction content evaluation management table 315, the correction quality judgment unit 226 can determine a quality evaluation score, for example, according to the number of evaluations. For example, the correction quality judgment unit 226 can calculate the quality evaluation score by dividing the number of evaluations for the correction content evaluation information by the number of views. Note that instead of the number of views, the number of users who viewed the correction content may be used. The correction quality judgment unit 226 creates content for displaying the quality evaluation score for the correction content (hereinafter referred to as correction quality evaluation content). The correction quality evaluation content can be, for example, content for providing a best answer to the user. The correction quality judgment unit 226 can create screen data written in HTML for displaying the quality evaluation scores for one or more pieces of correction content evaluation information. The correction quality judgment unit 226 transmits the correction quality evaluation content to the WEB server 21.

[0067] The correction quality judgment result display unit 218 of the WEB server 21 can receive the correction quality evaluation content received from the AP server 22 and provide the received correction quality evaluation content to the user terminal 1. In response to a request from the user terminal 1, the correction quality judgment result display unit 218 may read out correction content evaluation information from the correction content evaluation management table 315, and create and respond with correction quality evaluation content.

[0068] The correction quality judgment result display unit 118 of the user terminal 1 can receive the correction quality evaluation content sent from the web server 21 and display to the user a screen that displays the quality evaluation score of the correction content based on the correction quality evaluation content.

[0069] ==Providing incentives for corrections== The incentive grant amount calculation unit 227 of the AP server 22 determines the amount of incentive to be granted to at least one of the user who asked the question, the user who evaluated the question, the user who corrected the response, and the user who evaluated the correction content.

[0070] The incentive may be, for example, points circulating in the market, virtual currency, or a token using blockchain technology. Alternatively, the incentive may be a coupon or the like. Alternatively, digital content may be provided as an incentive. Alternatively, the incentive may be, for example, digital content or a physical lottery ticket, and the number of lottery attempts or the probability of winning the lottery may be the amount of the incentive.

[0071] In this embodiment, incentives are given to the user who asked the question and the user who corrected the response. The amount of incentive given to the user who asked the question and the user who corrected the response may be different. For example, a larger incentive may be given to the user who corrected the response than to the user who asked the question.

[0072] The incentive grant number calculation unit 227 can determine the amount of incentive to be granted to each user, for example. The incentive grant number calculation unit 227 can calculate, for example, an incentive for a user who asked a question (questioner) and an incentive for a user who corrected the response (corrector). The incentive grant number calculation unit 227 can determine the amount of incentive so that a larger incentive is given to a user who provided a question and / or correction content that received a larger number of ratings and / or quality evaluation score.

[0073] ==Incentives for Questioners== The incentive grant amount calculation unit 227 can determine the amount of incentive to be granted to a user who has asked a question, for example, based on at least one of the popularity of the question, the proactiveness of the question, the quality of the question, and the quality of the questioner. The popularity of a question can be, for example, the number of views of the question. The proactiveness of a question refers to the degree to which a user is proactively asking questions, and can be evaluated, for example, by the number of questions asked by the user or a statistical value thereof. The quality of a question refers to the degree to which many users consider a question to be good, and can be evaluated, for example, by the number of "likes" given by other users to a question from the user or a statistical value thereof. The quality of a questioner refers to whether the user who asked the question has good judgment, i.e., the degree to which other users also consider a question that the user considers good to be good, and can be evaluated, for example, by the number of "likes" given by other users to a question that the user has "liked"

[0074] The incentive grant amount calculation unit 227 can calculate the amount of incentive for each of the evaluation value of the proactiveness of the question (question proactiveness), the evaluation value of the quality of the question (question quality), and the evaluation value of the quality of the questioner (questioner quality) as described above, using the following formula. Incentive amount = coefficient a × question activity + coefficient b × question quality + coefficient c × questioner quality

[0075] The degree of importance to be attached to the proactiveness of the question, the quality of the question, and the quality of the questioner may be adjusted by coefficients, which can be set arbitrarily. Furthermore, the above formula is not limited to a linear sum, and any formula can be adopted in which at least one of the proactiveness of the question, the quality of the question, and the quality of the questioner is used as a variable, and the larger the value of each evaluation value, the larger the amount of incentive calculated.

[0076] The incentive grant amount calculation unit 227 can calculate the amount of incentive only from the question positivity, for example, as in the following formula. Incentive amount = Coefficient x Questioning activity

[0077] The incentive award amount calculation unit 227 can calculate the amount of incentive only from the quality of the question, for example, as in the following formula. Incentive amount = Coefficient x Question quality

[0078] The incentive grant amount calculation unit 227 can calculate the amount of incentive based only on the quality of the questioner, for example, as in the following formula. Incentive amount = Coefficient x Questioner quality

[0079] The amount of incentive may be calculated according to a combination of two or more values ​​obtained by multiplying the question proactiveness, the question quality, and the questioner quality by coefficients.

[0080] The incentive-granted number calculation unit 227 can also grant an incentive based on the popularity (number of views) of a question. The incentive-granted number calculation unit 227 can calculate the amount of the incentive, for example, using the following formula. Incentive amount = coefficient x question popularity

[0081] The incentive grant number calculation unit 227 may also determine the amount of incentive depending on at least two of the above-mentioned question enthusiasm, question quality, questioner quality, and question popularity. The incentive grant number calculation unit 227 can calculate the amount of incentive, for example, using the following formula: Incentive amount = coefficient x question popularity

[0082] The incentive award number calculation unit 227 may also award an incentive depending on whether the question is known or not.

[0083] Furthermore, the incentive grant number calculation unit 227 may correct the amount of incentive depending on whether the question is known or not. The incentive grant number calculation unit 227 may increase the amount of incentive when the question is not known, or decrease the amount of incentive when the question is known. The amount of increase or decrease may be linear, may be stepwise, or may be determined by some kind of function.

[0084] ==Incentives for Correctors== The incentive grant number calculation unit 227 can determine the type and / or amount of incentive to grant to a user who corrects a response based on, for example, at least one of the following: promptness of correction, popularity of correction, proactiveness of correction, quality of correction, and quality of corrector. The promptness of correction can be measured by the speed at which corrections are provided for a response to a question. In this case, the response management table 312 is configured to store the date and time at which a response to a question is provided, and the correction result management table 313 is configured to store the date and time at which a correction is provided. The length of time from the date and time at which a response to a question is provided to the date and time at which a correction is provided (hereinafter referred to as correction time) can be used. The popularity of corrections can be measured, for example, by the number of views of the correction results. The proactiveness of corrections refers to the degree to which a user actively provides corrections, and can be evaluated, for example, by the number of corrections the user has made or statistics thereof. The quality of corrections refers to the degree to which many users consider the corrections to be good, and can be evaluated, for example, by the number of "likes" received from other users on the user's corrections. The quality of the corrector is determined by whether the user who made the correction has an eye for quality, that is, the degree to which other users also consider corrections that the user considers good to be good. For example, the quality can be evaluated by the number of "likes" given by other users to corrections that the user has given a "like" to, or a statistical value thereof.

[0085] The incentive grant amount calculation unit 227 can calculate the amount of incentive for each of the evaluation value of the proactiveness of correction (proactiveness of correction), the evaluation value of the quality of correction (quality of correction), and the evaluation value of the quality of the corrector (quality of corrector) as described above, using the following formula. Incentive amount = coefficient d × activeness in correction + coefficient e × quality of correction + coefficient f × quality of corrector

[0086] The incentive grant amount calculation unit 227 can also determine the amount of incentive based only on the willingness to correct, for example, as in the following formula. Incentive amount = coefficient x proactiveness in correction

[0087] The incentive grant amount calculation unit 227 can also calculate the amount of incentive based only on the quality of correction, for example, as in the following formula. Incentive amount = coefficient x quality of correction

[0088] The incentive grant amount calculation unit 227 can also calculate the amount of incentive based only on the quality of the corrector, for example, as in the following formula. Incentive amount = coefficient x quality of corrector

[0089] The incentive-granting number calculation unit 227 can also grant incentives for the speed of correction. The incentive-granting number calculation unit 227 can calculate the amount of incentive, for example, using the following formula. Incentive amount = coefficient x (threshold - correction time)

[0090] The incentive-granting number calculation unit 227 can also grant an incentive based on the popularity of the correction (number of views). The incentive-granting number calculation unit 227 can calculate the amount of the incentive, for example, using the following formula. Incentive amount = coefficient x popularity of correction

[0091] The incentive grant number calculation unit 227 can also grant an incentive based on a combination of the proactiveness in correction and the popularity of the correction (number of views). The incentive grant number calculation unit 227 can calculate the amount of the incentive, for example, using the following formula. Incentive amount = Coefficient 1 x Proactiveness in correction + Coefficient 2 x Popularity of correction

[0092] The incentive grant amount calculation unit 227 can grant an incentive for the speed of correction. The incentive grant amount calculation unit 227 can calculate the amount of the incentive, for example, by the following formula. Incentive amount = coefficient x (threshold - correction time)

[0093] The incentive grant number calculation unit 227 may determine the type of incentive depending on the speed of correction. For example, an incentive setting unit may be provided that sets in advance the type of incentive to be granted for each range of the threshold value-correction time value, and the incentive grant number calculation unit 227 may determine the type of incentive corresponding to the value of the threshold value-correction time value.

[0094] In addition, the amount of incentive can be calculated according to a combination of two or more values ​​obtained by multiplying a coefficient for each of the following: proactiveness in correction, quality of correction, quality of corrector, and speed of correction (threshold - correction time).

[0095] The degree to which importance is attached to the willingness to correct, the quality of the correction, and the quality of the corrector may be adjusted using coefficients, which can be set arbitrarily. Furthermore, the above formula is not limited to a linear sum, and any formula can be adopted in which at least one of the willingness to correct, the quality of the correction, and the quality of the corrector is used as a variable, and the larger the value of each evaluation value, the larger the amount of incentive calculated.

[0096] Furthermore, the incentive grant amount calculation unit 227 may grant an incentive depending on whether the correction result is known or not.

[0097] Furthermore, the incentive grant number calculation unit 227 may correct the amount of incentive depending on whether the correction result is known or not. The incentive grant number calculation unit 227 can increase the amount of incentive when the correction result is not known, or decrease the amount of incentive when the correction result is known. The amount of increase or decrease may be linear, may be stepwise, or may be determined by some kind of function.

[0098] ==Incentives for Evaluators== Incentives may be awarded to users who evaluate questions, responses, or corrections. In this case, the incentive award calculation unit 227 may determine the type and / or amount of incentives to be awarded to users who evaluate questions, responses, or corrections, based on, for example, at least one of the promptness of the evaluation, the proactiveness of the evaluation, the quality of the evaluation, and the quality of the evaluator. The promptness of the evaluation may be determined by the speed with which the question, response, or correction is evaluated. In this case, the data storage unit 231 may be provided with an evaluation management table, which may store a question ID indicating the question to be evaluated or a correction ID indicating the correction result to be evaluated, a user ID (evaluator ID) indicating the evaluator, and the date and time of evaluation in association with each other. Furthermore, the question management table 311, the response evaluation management table (not shown), and / or the correction result management table 313 may be added with the date and time of the question, response, or correction, and the length of time from the date and time of evaluation to the date and time the question, response, or correction is evaluated (hereinafter referred to as evaluation time). The proactiveness of evaluation is the degree to which a user actively evaluates, and can be evaluated, for example, by the number of times the user has evaluated something, such as "liked," or by the statistical amount thereof. The quality of evaluation is the degree to which many users give good evaluations to things they think deserve good evaluations, and can be evaluated, for example, by the number of "likes" from other users on questions or correction results that the user has "liked," or by the statistical amount thereof.

[0099] The incentive grant quantity calculation unit 227 can calculate the amount of incentive for each of the evaluation value of the evaluation aggressiveness (evaluation aggressiveness), the evaluation value of the evaluation quality (evaluation quality), and the evaluation value of the evaluator quality (evaluator quality), for example, using the following formula. Incentive amount = coefficient g × evaluation aggressiveness + coefficient h × evaluation quality + coefficient i × evaluator quality

[0100] The incentive grant amount calculation unit 227 can also determine the amount of incentive based only on the evaluation aggressiveness, for example, as in the following formula. Incentive amount = Coefficient x Evaluation aggressiveness

[0101] The incentive grant amount calculation unit 227 can also calculate the amount of incentive from only the quality of the evaluation, for example, as in the following formula. Incentive amount = Coefficient x Evaluation quality

[0102] The incentive grant amount calculation unit 227 can also calculate the amount of incentives based only on the quality of the evaluator, for example, as in the following formula. Incentive amount = Coefficient x Evaluator quality

[0103] The incentive-granting number calculation unit 227 can also grant an incentive for the speed of evaluation. The incentive-granting number calculation unit 227 can calculate the amount of incentive, for example, by the following formula. Incentive amount = coefficient × (threshold - evaluation time)

[0104] The amount of incentive can be calculated according to a combination of two or more values ​​obtained by multiplying each of the evaluation aggressiveness, the evaluation quality, the evaluator quality, and the speed of the evaluation (threshold-evaluation time) by a coefficient.

[0105] The incentive grant number calculation unit 227 may calculate the incentive for the evaluation of the question, the incentive for the evaluation of the response, and the incentive for the correction result separately. In this case, the incentive grant number calculation unit 227 can calculate the above-mentioned incentives after calculating, for example, the promptness, proactiveness, quality of the evaluation, or quality of the evaluator of the evaluation of the question, the promptness, proactiveness, quality of the evaluation, or quality of the evaluator of the evaluation of the response, and the promptness, proactiveness, quality of the evaluation, or quality of the evaluator of the correction result.

[0106] When receiving evaluations for responses, a response evaluation management table may be provided. The response evaluation management table may be a table for managing information regarding evaluations of responses (hereinafter referred to as response evaluation information). Records (response evaluation information) managed in the response evaluation management table may include information for identifying a response (which may be a response ID or a question ID), evaluations received from users (evaluators) for the response (in this embodiment, the number of users who "liked" the content of the response is defined as the number of evaluations), the number of views of the response, and a score (quality evaluation score) determined taking into account the number of evaluations and the number of views. The response evaluation information may manage both the evaluations received from users and the score (final evaluation) determined based on the evaluations. Note that the evaluation is not limited to the number of "likes," but may also be the number of times the "helpful" button was pressed, or a score value set by the evaluator, such as 1 to 5 stars. Furthermore, instead of an evaluation value from a user (evaluator), the value may be automatically calculated based on a given rule or function.

[0107] Furthermore, the incentive grant number calculation unit 227 may correct the amount of incentive depending on whether the evaluated question or correction result is known. For example, the incentive grant number calculation unit 227 may increase the amount of incentive when the question or correction result is not known, or decrease the amount of incentive when the question or correction result is known. The amount of increase or decrease may be linear, stepwise, or determined by some kind of function.

[0108] ==Calculation of incentives per question / correction== The incentive grant number calculation unit 227 can also grant incentives on a per question and / or per correction basis. The incentive grant number calculation unit 227 can grant an incentive to the asker or corrector in an amount corresponding to the number of views of the question or correction, for example. The incentive grant number calculation unit 227 may continue to provide an incentive to the asker or corrector every time the question or correction is viewed.

[0109] The incentive grant number calculation unit 227 can issue non-fungible tokens (NFTs) that represent the right to receive incentives. The incentive grant number calculation unit 227 can issue NFTs linked to questions and / or corrections, for example. The issuance of NFTs uses general blockchain technology, so a description thereof will be omitted here. The incentive grant number calculation unit 227 can grant incentives for questions and / or corrections to the owner of the NFT. This NFT may be tradable.

[0110] The incentive granting number calculation unit 227 may grant an incentive when the learning unit 224 uses the correction result for relearning.

[0111] The incentive grant number calculation unit 227 can grant an incentive when the correction result is non-public information (closed data). The incentive grant number calculation unit 227 can grant a larger incentive for the correction result of non-public information than for the correction result that is public information. The incentive grant number calculation unit 227 may not grant an incentive for public information.

[0112] The incentive award amount calculation unit 227 transmits to the web server 21 content (hereinafter referred to as incentive content) for displaying the determined details and amount (amount of award) of the incentive.

[0113] The incentive granted number display unit 219 of the Web server 21 transmits the incentive content received from the AP server 22 to the user terminal 1. An incentive management table that stores the content and amount of the incentive calculated by the incentive granted number calculation unit 227 in association with the user ID indicating the user to whom the incentive is to be granted may be provided in the data storage unit 231, and the incentive granted number display unit 219 may, for example, read the content and amount of the incentive corresponding to the user from the incentive management table in response to a request from the user terminal 1, create incentive content, and respond to the user terminal 1 with the created incentive content.

[0114] The incentive amount display unit 119 of the user terminal 1 can display on the screen the type and amount of incentives to be awarded to the user based on the incentive content transmitted from the web server 21.

[0115] == Viewing Log == The browsing log acquisition unit 220 of the WEB server 21 can acquire, for example, an access log of access to the WEB server 21 from the user terminal 1. The browsing log acquisition unit 220 can acquire the access log of a general WEB server. The browsing log acquisition unit 220 can register the acquired browsing log in the browsing log storage unit 233 managed by the DB server 23.

[0116] <Operation> 9 is a diagram showing the flow of processing by the learning unit 224. First, the learning unit 224 reads data from the response management table 312, the question evaluation management table 311, and the correction content evaluation management table 315 in the data storage unit 231 (S401). Next, the learning unit 224 references the quality evaluation score values ​​in the question evaluation management table 311 and selects questions with a certain score or higher. The learning unit 224 references the correction ID corresponding to the selected question in the response management table 312 and the quality evaluation score values ​​in the correction content evaluation management table 315 corresponding to the response ID, and selects correction results with a certain score or higher (S402). The learning unit 224 adds the pairs of high-quality questions and correction results extracted in step S402 to the known information storage unit 234 (S403).

[0117] 10 is a diagram showing the flow of processing by the question quality judgment unit 225. The question quality judgment unit 225 reads data from the question evaluation management table 311 in the data storage unit 231 (S421). The question quality judgment unit 225 reads viewing log data from the viewing log storage unit 233 (S422). The question quality judgment unit 225 counts the number of accesses (views) to the web page corresponding to the question ID from the viewing log, and sets the number of views of the question evaluation information managed in the question evaluation management table 311 (S423). The question quality judgment unit 225 calculates a quality evaluation score from the number of ratings (number of likes) and the number of views in the question evaluation management table 311 (S424). The question quality judgment unit 225 sets the calculated quality evaluation score as the quality evaluation score of the question evaluation information managed in the question evaluation management table 314 (S425). The question quality judgment unit 225 stores the question evaluation management table 311 in the data storage unit 231 (S426).

[0118] 11 is a diagram showing the flow of processing by the correction quality judgment unit 226. The correction quality judgment unit 226 reads data from the correction content evaluation management table 315 in the data storage unit 231 (S441). The correction quality judgment unit 226 reads viewing log data from the viewing log storage unit 233 (S442). The correction quality judgment unit 226 counts the number of accesses (number of views) to the web page corresponding to the correction ID from the viewing log, and sets this as the number of views of the correction content evaluation information managed in the correction content evaluation management table 315 (S443). The correction quality judgment unit 226 calculates a quality evaluation score from the number of ratings (number of likes) and the number of views in the correction content evaluation management table 315 (S444). The correction quality judgment unit 226 sets the calculated quality evaluation score as the quality evaluation score of the correction content evaluation information managed in the correction content evaluation management table 315 (S445). The correction quality determination unit 226 stores the correction content evaluation management table 315 in the data storage unit 231 (S446).

[0119] FIG. 12 is a diagram showing the flow of the process of calculating incentives for users who have asked questions. The incentive-granting number calculation unit 227 reads data from the questioner management table 316 in the data storage unit 231 (S461). The incentive-granting number calculation unit 227 counts the number of question IDs in the questioner information managed in the questioner management table 316 for each questioner ID to calculate the number of questions (S462). The incentive-granting number calculation unit 227 calculates, for each questioner ID, statistics (which may be total, average, deviation, etc.) of the acquired quality scores of the questioner information managed in the questioner management table 316 (S463). The incentive-granting number calculation unit 227 calculates, for each questioner ID, statistics (total, average, deviation) of the "acquired quality evaluation scores" linked to the "question IDs of other people who evaluated" in the questioner management table 316 (S464). The incentive-granting number calculation unit 227 determines the number of incentives for each questioner ID taking into account the calculated values ​​(S465). The incentive grant number calculation unit 227 can grant an incentive to the user who asked the question based on the calculated number of incentives (S466).

[0120] 13 is a diagram showing the flow of the process of calculating incentives for users who have corrected responses. The incentive grant number calculation unit 227 reads data from the corrector management table 317 in the data storage unit 231 (S481). The incentive grant number calculation unit 227 counts the number of correction IDs of the corrector information managed in the corrector management table 317 for each corrector ID to calculate the number of corrections (S482). The incentive grant number calculation unit 227 calculates, for each corrector ID, statistics (which may be total, average, deviation, etc.) of the acquired quality scores of the corrector information managed in the corrector management table 317 (S483). The incentive grant number calculation unit 227 calculates, for each corrector ID, statistics (total, average, deviation) of the "acquired quality evaluation scores" linked to the "correction IDs of other people who evaluated" of the corrector information managed in the corrector management table 317 (S484). The incentive grant number calculation unit 227 determines the number of incentives for each corrector ID in consideration of the calculated value (S485). The incentive grant number calculation unit 227 can grant incentives to the users who have corrected the work based on the calculated number of incentives (S486).

[0121] Fig. 14 is a diagram showing an example of the hardware configuration of a computer. Note that the configuration shown in the figure is an example, and other configurations may be used. The computer shown in Fig. 14 can implement the user terminal 1 and response device 2 (WEB server 21, AP server 22, DB server 23).

[0122] The computer includes a CPU 201, memory 202, storage device 203, communication interface 204, input device 205, and output device 206. The storage device 203 stores various data and programs, and is, for example, a hard disk drive, a solid state drive, or a flash memory. The communication interface 204 is an interface for connecting to a communication network, and is, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone network, a wireless communication device for wireless communication, or a USB (Universal Serial Bus) connector or an RS232C connector for serial communication. The input device 205 is, for example, a keyboard, a mouse, a touch panel, a button, a microphone, or the like for inputting data. The output device 206 is, for example, a display, a printer, a speaker, or the like for outputting data. Each functional unit of the above-mentioned user terminal 1 and response device 2 (WEB server 21, AP server 22, DB server 23) is realized by the CPU 201 reading a program stored in the storage device 203 into the memory 202 and executing it, and each storage unit can be realized as part of the storage area provided by the memory 202 and the storage device 203.

[0123] Although the present embodiment has been described above, the above embodiment is intended to facilitate understanding of the present invention and is not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit thereof, and equivalents thereof are also included in the present invention.

[0124] <Select from multiple candidates> For example, in this embodiment, it is assumed that the trained model outputs one response to one question, but multiple responses may be output. For example, the response unit 221 may generate multiple responses by multiple attempts to provide a question to the trained model and generate a response. When the trained model is a generator, different responses may be generated by increasing a randomness parameter (e.g., the temperature parameter of GPT). Furthermore, multiple types of trained models may be prepared, and prompts may be provided to each trained model to generate multiple responses. Furthermore, when multiple pieces of known information are searched for, each piece of known information may be included in a prompt and provided to the trained model to generate multiple answers.

[0125] Here, multiple responses (output results from the trained model) may be output, and the selected response may be corrected so that the corrector can select the output that he or she considers to be of the highest quality.

[0126] Alternatively, multiple output results may be output, and the viewer may select which of the multiple output results is the highest quality output, without correcting it. In this case, the learning unit 224 may register the output result selected by the viewer in the known information storage unit 234.

[0127] The response unit 221 can also cause the trained model to output multiple responses to a question and automatically select the output with the highest quality as the final response. The quality can be determined based on the acceptability to users or the validity of the answer based on a rule base or preset conditions.

[0128] <References> The response unit 221 may provide information that may be useful for correction along with the response from the trained model. Information that may be useful for correction can be identified by, for example, collecting reference information from information sources such as websites, blog information, and papers, determining the similarity between the collected data and the response from the trained model, and selecting information whose similarity is equal to or greater than a predetermined value. The response unit 221 may display a list of the response from the trained model and the selected reference information together.

[0129] Furthermore, the information that the corrector uses as a reference or basis for correction may be registered in the known information storage unit 234.

[0130] <Human Answer> In this embodiment, known information corresponding to a question is extracted and provided to a large-scale language model to generate an answer to the question, but the answer to the question may also be provided by a user who answers the question. For example, if known information corresponding to a question cannot be extracted (e.g., if known information with a similarity equal to or greater than a predetermined value cannot be searched for), a predetermined answerer may answer the question, and the answer content may be associated with a user ID indicating the answerer and registered in the known information storage unit 234. In this case, the answer content may be regarded as a response from the large-scale language model in the above-described embodiment and may be corrected by a corrector, or the answer content may be regarded as the correction result in the above-described embodiment.

[0131] <Incentive Beneficiary Rights> In this embodiment, the questioner, corrector, and evaluator receive incentives, but the right to receive these incentives may be transferable to others. For example, the management server 2 can connect to a blockchain network and mint NFTs linked to users using smart contracts implemented on the blockchain network. Information identifying the user (e.g., a user ID) is linked to the minted NFT. The user ID may be set in the metadata of the NFT. In this case, the management server 2 can pay incentives to the wallets of the owners of the NFTs linked to the user who asked the question, the user who corrected the question, and the user who provided the evaluation.

[0132] Furthermore, when an NFT is resold (secondary distribution), a portion of the transfer fee may be distributed to the original user. In this case, a transfer smart contract is deployed on a blockchain network, and when an NFT is transferred, the smart contract pays a portion of the transfer fee (for example, an arbitrary percentage such as 10%) to the user associated with the NFT to be transferred, and pays the remaining amount (after which system fees, gas fees, etc. can be deducted) to the transferor of the NFT. In this case, payment processing can be assumed to be in virtual currency. [Explanation of symbols]

[0133] 1. User terminal 2 Answering Machine 21 Web Server 22 AP Server 23 Database Server

Claims

[Claim 1] Information processing system.

Images