Information processing system, program and method

Abstract

To provide an information processing system allowing users to efficiently update a machine learning model.SOLUTION: An information processing system includes: a learning model storage part for storing a learned model, the learning model learned by machine learning; an input data receiving part for receiving input data from a first user; a processing part for providing the learning model with the input data and causing it to output the output data; an output part for outputting the output data to the first user and a second user different from the first user or only to the second user in such a manner as to be able to browse it; a corrected result receiving part for receiving a corrected result obtained by correcting the output data, from the second user; and an update part for updating a learning model by using the corrected result.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system, a program, and a method. [Background technology]

[0002] Technologies that allow machines to automatically respond to questions and requests from users have been developed, and are now being provided as automatic response services such as bots (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-3533 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional technologies that allow machines to respond automatically require setting rules in advance so that they can respond, and preparing huge amounts of training data to build machine learning models, which entails huge costs in terms of human labor, required computer resources, and time for training.

[0005] The present invention has been made in consideration of the above background, and aims to provide a technique that can efficiently update a machine learning model. [Means for solving the problem]

[0006] The main invention of the present invention for solving the above problem is an information processing system comprising: a learning model memory unit that stores a learning model that has been trained by machine learning; an input data receiving unit that receives input data from a first user; a processing unit that provides the input data to the learning model and outputs output data; an output unit that outputs the output data to the first user and a second user different from the first user, or so that the output data can be viewed only by the second user; a correction result receiving unit that receives correction results of the output data from the second user; and an update unit that updates the learning model using the correction results. Effect of the Invention

[0007] According to the present invention, a machine learning model can be updated efficiently. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing an overview of an information processing system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing an example of the configuration of a question management table 311. [Diagram 3] 13 is a diagram showing an example of the configuration of a response management table 312. FIG. [Figure 4] FIG. 13 is a diagram showing an example of the configuration of a correction result management table 313. [Diagram 5] 13 is a diagram showing an example of the configuration of a question evaluation management table 314. FIG. [Figure 6] FIG. 13 is a diagram showing an example of the configuration of a correction content evaluation management table 315. [Figure 7] FIG. 13 is a diagram showing an example of the configuration of a questioner management table 316. [Figure 8] FIG. 13 is a diagram showing an example of the configuration of a corrector management table 317. [Figure 9] FIG. 13 is a diagram showing a flow of processing by a learning unit 224. [Figure 10] 13 is a diagram showing a flow of processing by the question quality judgment unit 225. [Figure 11]13 is a diagram showing the flow of processing by the correction quality determination unit 226. FIG. [Figure 12] FIG. 13 is a diagram showing the flow of a process for calculating an incentive for a user who has asked a question. [Figure 13] FIG. 13 is a diagram showing the flow of a process for calculating an incentive for a user who corrects a response. [Figure 14] FIG. 2 illustrates an example of a hardware configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] <Summary of the Invention> The present invention will be described below with reference to the preferred embodiments thereof. [Item 1] A learning model storage unit that stores a learning model that has been learned by machine learning; an input data receiving unit that receives input data from a first user; A processing unit that provides the input data to the learning model and outputs output data; an output unit that outputs the output data to the first user and a second user different from the first user, or in a manner that can be viewed only by the second user; a correction result receiving unit that receives a correction result of correcting the output data from the second user; an update unit that updates the learning model using the correction result; An information processing system comprising: [Item 2] The information processing system according to item 1, an expert storage unit that stores whether or not a user is an expert; the output unit refers to the expert storage unit, identifies the expert as the second user, and outputs the output data only to the identified second user; outputting the correction result corrected by the second user to the first user so that the correction result can be viewed by the first user; An information processing system comprising: [Item 3] The information processing system according to item 1, A correction evaluation receiving unit is further provided for receiving an evaluation value from a third user for the correction result, the output unit outputs the input data and the correction result in a viewable manner to the third user; the correction result receiving unit receives the correction results from a plurality of the second users; the update unit selects the correction result in accordance with the evaluation value, and updates the learning model using the selected correction result; An information processing system comprising: [Item 4] Item 3. An information processing system according to item 3, Further, a correction quality determination unit is provided which determines the quality of the correction result by tallying up at least the evaluation values, the updating unit selects at least a portion of the correction results according to the quality; An information processing system comprising: [Item 5] Item 4. An information processing system according to the present invention, the correction quality determination unit determines the quality according to at least the aggregated value of the evaluation values ​​and the number of the third users who have viewed the correction result; An information processing system comprising: [Item 6] The information processing system according to item 1, an input evaluation receiving unit that receives an evaluation value from a third user for the input data; the output unit outputs the input data so as to be viewable by the second user and the third user; the updating unit updates the learning model using the input data having the evaluation value equal to or greater than a predetermined value and the correction result of the output data corresponding to the input data; An information processing system comprising: [Item 7] Item 6. An information processing system according to item 6, the input evaluation receiving unit receives the evaluation values ​​from a plurality of the third users; an input quality determination unit that aggregates at least the evaluation values ​​to determine the quality of the input data; the updating unit updates the learning model using the input data whose quality is equal to or greater than a predetermined value and the correction result; An information processing system comprising: [Item 8] Item 7. An information processing system according to item 7, the input quality determination unit determines the quality according to at least the aggregated value of the evaluation values ​​and the number of the third users who have viewed the input data; An information processing system comprising: [Item 9] The information processing system according to item 1, a correction evaluation receiving unit that receives an evaluation value from a third user on the correction result; a correction quality determination unit that determines the quality of the correction result, which is the quality of the correction result, in accordance with at least the aggregated value of the evaluation values ​​and the number of the third users who have viewed the correction result; an input evaluation receiving unit that receives an evaluation value from a third user for the input data; an input quality determination unit that determines an input quality, which is the quality of the input data, in accordance with at least the aggregated value of the evaluation values ​​and the number of the third users who have viewed the input data; Further equipped with the output unit outputs the input data and the correction result so as to be viewable by the second user and the third user; the updating unit updates the learning model using the input data whose input quality is equal to or greater than a first predetermined value and the correction result whose correction quality is equal to or greater than a second predetermined value; An information processing system comprising: [Item 10] The information processing system according to item 1, an incentive providing unit that provides an incentive to the second user who has provided the correction result; An information processing system comprising: [Item 11] The information processing system according to item 1, an incentive providing unit that provides an incentive to the first user who provides the input data; An information processing system comprising: [Item 12] Item 3. An information processing system according to item 3, an incentive providing unit that provides an incentive to the third user; An information processing system comprising: [Item 13] The information processing system according to item 1, the update unit determines whether the correction result is public information, and if the correction result is not public information, updates the learning model using the correction result; An information processing system comprising: [Item 14] Storing a learned model that has been learned by machine learning; accepting input data from a first user; providing the input data to the learning model to output output data; outputting the output data to the first user and a second user different from the first user, or to the second user only; receiving a correction result of correcting the output data from the second user; updating the learning model using the correction results; A program for causing a computer to execute the following. [Item 15] Storing a learned model that has been learned by machine learning; accepting input data from a first user; providing the input data to the learning model to output output data; outputting the output data to the first user and a second user different from the first user, or to the second user only; receiving a correction result of correcting the output data from the second user; updating the learning model using the correction results; How a computer executes.

[0010] <System configuration> 1 is a diagram showing an overview of an information processing system according to an embodiment of the present invention. The information processing system of this embodiment is configured to include a response device 2. The response device 2 is communicably connected to a user terminal 1 via a communication network. The communication network is, for example, the Internet, and is constructed by a public telephone line network, a mobile phone line network, a wireless communication path, Ethernet (registered trademark), etc.

[0011] The user terminal 1 is a computer operated by a user. The user terminal 1 can be, for example, a smartphone, a tablet computer, a personal computer, etc. A web browser or an application runs on the user terminal 1, and the user can access the response device 2 via the web browser or the application.

[0012] The response device 2 may be a general-purpose computer, such as a workstation or a personal computer, or may be logically realized by cloud computing.

[0013] In the information processing system of this embodiment, the response device 2 stores a machine learning model that has been trained by machine learning to generate an answer to a question, generates an answer to a question from a certain user (questioner) using the machine learning model, accepts corrections to this answer from users (correctors) other than the user who asked the question, and updates the machine learning model using the correction results. This allows the machine learning model to be autonomously updated to obtain more up-to-date information using distributed user resources and to achieve higher accuracy. In addition, the question and the correction results can be evaluated by users (viewers, evaluators) who have viewed the question and the correction results. By relearning highly rated questions and / or correction results, the machine learning model can be retrained to generate highly rated answers.

[0014] In this embodiment, it is assumed that the corrector is a user different from the questioner, the evaluator who evaluates the question is a user different from the questioner, and the evaluator who evaluates the correction result is a user different from the corrector. Note that the corrector may include the questioner. The evaluator may include the questioner and the corrector. The questioner may be one or more people. The corrector may be one or more people. The evaluator may be one or more people.

[0015] <Software configuration> As shown in FIG. 1, the user terminal 1 includes a question input unit 111, a response content display unit 112, a question evaluation input unit 113, a response correction input unit 114, a correction result display unit 115, a correction evaluation input unit 116, a question quality judgment result display unit 117, a correction quality judgment result display unit 118, and an incentive grant number display unit 119.

[0016] 1, the response device 2 includes a WEB server 21 that transmits and receives data to and from the user terminal 1, an AP server 22 that functions as middleware, and a DB server 23 that manages a database. Note that this configuration is merely an example, and the response device 2 may be configured with one or two computers. Also, the response device 2 may be configured with four or more computers.

[0017] The DB server 23 includes a data storage unit 231, a trained model storage unit 232, and a viewing log storage unit 233.

[0018] The WEB server 21 includes a question receiving unit 211, a response content display unit 212, a question evaluation receiving unit 213, a correction result receiving unit 214, a correction result display unit 215, a correction evaluation receiving unit 216, a question quality judgment result display unit 217, a correction quality judgment result display unit 218, a number of incentives granted display unit 219, and a viewing log acquisition unit 220.

[0019] The AP server 22 includes a response unit 221 , a response content creation unit 222 , a correction result content creation unit 223 , a learning unit 224 , a question quality determination unit 225 , a correction quality determination unit 226 , and an incentive given number calculation unit 227 .

[0020] The data storage unit 231 of the DB server 23 includes a question management table 311 , a response management table 312 , a correction result management table 313 , a question evaluation management table 314 , a correction content evaluation management table 315 , a questioner management table 316 , and a corrector management table 317 .

[0021] The question management table 311 is a table for managing information (hereinafter referred to as question information) relating to questions received from users (questioners) who ask questions. Fig. 2 is a diagram showing an example of the configuration of the question management table 311. Records (question information) managed in the question management table 311 include information for identifying a question (question ID), information for identifying a user who has asked a question (questioner ID), and the content of the question.

[0022] The response management table 312 is a table for managing information (hereinafter referred to as response information) related to the content of an automatic response to a question from a user (questioner). Fig. 3 is a diagram showing an example of the configuration of the response management table 312. A record (response information) registered in the response management table 312 includes a question ID for identifying the question, information for identifying the response (response ID), and the response content.

[0023] The correction result management table 313 is a table for managing information on the correction results (hereinafter referred to as correction result information) received from a user (corrector) in response to the content of an automatic response (response content). Fig. 4 is a diagram showing an example of the configuration of the correction result management table 313. Records (correction result information) managed in the correction result management table 313 include a response ID that identifies the response, information that identifies the user who performed the correction (corrector ID), information for identifying the correction result (correction ID), and correction content.

[0024] The question evaluation management table 314 is a table for managing information on evaluations from users (evaluators) on the quality of a question (hereinafter, referred to as question evaluation information). FIG. 5 is a diagram showing an example of the configuration of the question evaluation management table 314. The records (question evaluation information) managed in the question evaluation management table 314 include a question ID that identifies a question, evaluations (in this embodiment, the number of users who "liked" the question content is taken as the number of evaluations) received for the question from at least users other than the user who asked the question (it is acceptable even if the user who asked the question is included), the number of views of the question, and a score (quality evaluation score) determined in consideration of the number of evaluations and the number of views. That is, in this embodiment, both the evaluations received from users and the score (final evaluation) determined based on the evaluations are managed in the question evaluation information.

[0025] The correction content evaluation management table 315 is a table for managing information on the evaluation of the quality of the correction content (hereinafter, referred to as correction content evaluation information). FIG. 6 is a diagram showing an example of the configuration of the correction content evaluation management table 315. The records (correction content evaluation information) managed in the correction content evaluation management table 315 include information for identifying the correction content (correction ID), evaluations received from at least users other than the user who corrected the correction content (in this embodiment, the number of users who gave "Like" to the correction content is the number of evaluations), the number of views of the correction content, and a score (quality evaluation score) determined in consideration of the number of evaluations and the number of views. That is, in this embodiment, both the evaluation received from the user and the score (final evaluation) determined based on the evaluation are managed in the correction content evaluation information. Note that the evaluation is not limited to the number of "Likes", but may be the number of times the "Useful" button was pressed, or may be a score value set by the evaluator, such as 1 to 5 stars. Also, instead of an evaluation value from a user (evaluator), it may be a value that is automatically calculated based on a given rule or function, etc. For example, the number of times that the machine learning model has cited the question or the correction result may be the evaluation value.

[0026] The questioner management table 316 is a table for managing information related to users who have asked questions (hereinafter referred to as questioner information). Fig. 7 is a diagram showing an example of the configuration of the questioner management table 316. A record (questioner information) managed in the questioner management table 316 includes information (questioner ID) that identifies the user who asked the question, a question ID that identifies the question, a quality evaluation score acquired for the question by the user, and a question ID that identifies the question of another person that was evaluated by the user.

[0027] The corrector management table 317 is a table for managing information related to the user who made the correction (hereinafter referred to as corrector information). Fig. 8 is a diagram showing an example of the configuration of the corrector management table 317. The records (corrector information) managed in the corrector management table 317 include information (corrector ID) that identifies the user who made the correction, a correction ID that identifies the correction result, a quality evaluation score acquired for the correction by that user, and a correction ID that identifies the correction result of another person that was evaluated by that user.

[0028] The trained model storage unit 232 included in the DB server 23 stores a trained machine learning model (parameters constituting the model). For example, a trained model trained externally can be used as the machine learning model. Note that the DB server 23 may not include a trained machine learning model, and an API provided by another server may be used.

[0029] The view log storage unit 233 included in the DB server 23 stores a log of the user viewing questions and correction contents on the user terminal 1. The view log can be a general access log. The view log has necessary and sufficient information to count how many times a question ID or correction ID has been viewed.

[0030] ==Responses to Questions== The question input unit 111 of the user terminal 1 accepts input of a question from a user. In this embodiment, the question is assumed to be text data, but the question can also be image data, audio data, or the like. The question input unit 111 transmits the question accepted from the user to the response device 2. The question input unit 111 can transmit the question by attaching it to an HTTP request, for example.

[0031] The question receiving unit 211 of the WEB server 21 can receive a question transmitted from the user terminal 1. The question receiving unit 211 can, for example, decode a question encoded in an HTTP request and transmit the decoded question to the AP server 22.

[0032] When the response unit 221 of the AP server 22 receives a question from the WEB server 21, it issues the question to the machine learning model stored in the learned model storage unit 232 of the DB server 23, generates a response to the question, and transmits the generated response to the response content creation unit 222. The response unit 221 can also set the decoded question as the question content, set a questioner ID indicating the user of the user terminal 1 who asked the question, generate a new question ID to create question information, and register the created question information in the question management table 311 provided in the data storage unit 231 of the DB server 23.

[0033] The response content creation unit 222 of the AP server 22 creates content (hereinafter referred to as response content; for example, it may be screen data written in HTML) for displaying the response received from the response unit 221 on the user terminal 1. The response content creation unit 222 transmits the created response content to the WEB server 21. Note that the response content may include not only the response but also a question. The response content may also include the number of ratings for the question (the number of "likes" for the question).

[0034] The response content display unit 212 of the WEB server 21 receives response content for displaying the response from the AP server 22, and transmits the received response content to the user terminal 1. The response content display unit 212 can transmit content for displaying the question and the response not only to the user terminal 1 that is the sender of the question, but also to the user terminals 1 of other users.

[0035] In response to a request from the user terminal 1, the response content display unit 212 of the WEB server 21 transmits a message to the response content creation unit 222 of the AP server 22, and the response content creation unit 222 reads one or more pieces of question information from the question management table 311 provided in the data storage unit 231 of the DB server 23, reads response information corresponding to the question ID from the response management table 312 for each of the read question information, creates response content for displaying the question content and the response content, and transmits the response content to the WEB server 21, and the response content display unit 212 transmits the response content to the sender of the request. The response content creation unit 222 can also obtain the number of ratings corresponding to the question ID from the question rating management table 314 and include it in the response content.

[0036] The response content display unit 112 of the user terminal 1 receives the response content for displaying the response transferred from the WEB server 21, and is able to display the response content to the user based on the response content.

[0037] ==Public / Private setting function== The questions and / or correction contents may be set to be public or private. In this case, information indicating public or private (public setting information) is set in the question information stored in the question management table 311 and / or the correction result information stored in the correction result management table 313, and only the question information and / or the correction result information whose public setting information indicates "public" can be made public to general users.

[0038] Even if the correction contents are not disclosed, a learning model dedicated to the user who corrected the text may be prepared, and the private correction contents may be used in learning the user's dedicated learning model. In this case, for example, a learning model dedicated to the company can be developed while preventing the company's know-how from being leaked.

[0039] ==Mask setting function== Also, a part of the question and / or the correction content may be masked. In this case, for example, a character to be masked or a condition for masking may be set for each question and / or correction content in the question information and / or the correction result information, and the set character may be replaced with a replacement character, or a part of the question and / or the correction content that satisfies the set condition may be replaced with a replacement character, and then the question and / or the correction content may be made public and used for re-learning.

[0040] ==Question Rating== The question evaluation input unit 113 of the user terminal 1 accepts input of an evaluation of a question from a user. The question evaluation input unit 113 may accept a "like" for a question posted by a user other than the user using the user terminal 1, if the question is a good question. The question evaluation input unit 113 transmits the evaluation of the accepted question (e.g., "like") to the response device 2 together with a question ID that identifies the question.

[0041] The question evaluation receiving unit 213 of the WEB server 21 receives evaluations (e.g., "Like") on a question from the user terminal 1. The question evaluation receiving unit 213 can increment the number of evaluations for question information in the question evaluation management table 314 corresponding to the question ID that identifies the question.

[0042] ==Response correction== The response correction input unit 114 of the user terminal 1 can receive correction contents from the user for the response that the user has viewed. The response correction input unit 114 transmits the received correction contents to the WEB server 21 together with a response ID that identifies the response and a corrector ID that identifies the user.

[0043] The correction result receiving unit 214 of the Web server 21 receives the correction content from the user in response to the response. The correction result receiving unit 214 transmits the received correction content to the AP server 22 together with the response ID and the correction ID.

[0044] When the correction result content creation unit 223 of the AP server 22 receives the correction content transmitted from the WEB server 21, it can register the correction result information including the response ID, the corrector ID, the newly assigned correction ID (which may be assigned by the DB server 23), and the correction content in the correction result management table 313 provided in the data storage unit 231. The correction result content creation unit 223 can create content indicating that the correction result has been accepted (hereinafter, referred to as correction result content). The correction result content can include the correction result received from the WEB server 21. The correction result content can also include, for example, a list of past correction results registered in the correction result management table 313. The correction result content creation unit 223 transmits the correction result content to the WEB server 21.

[0045] The correction result display unit 215 of the Web server 21 transmits the correction result content received from the AP server 22 to the user terminal 1 .

[0046] The correction result display unit 115 of the user terminal 1 can display the correction contents to the user based on the correction result content received from the Web server 21.

[0047] In response to a request from the user terminal 1, the correction result display unit 215 of the WEB server 21 transmits a message to the correction result content creation unit 223 of the AP server 22, and the correction result content creation unit 223 reads one or more pieces of response information from the response management table 312 included in the data storage unit 231 of the DB server 23, reads the question content corresponding to the question ID from the question management table 311 for each of the read response information, reads the correction result information corresponding to the response ID from the correction result management table 313, creates correction result content for displaying the question content, the response content, and the correction content (multiple correction contents are possible), and transmits it to the WEB server 21, and the correction result display unit 215 transmits this correction result content to the sender of the request. The correction result content creation unit 223 can also obtain the number of evaluations corresponding to the correction ID from the correction content evaluation management table 315 and include it in the correction result content.

[0048] ==Correction Evaluation== The correction evaluation input unit 116 of the user terminal 1 accepts input of an evaluation of the correction result from the user. The correction evaluation input unit 116 may accept a "like" for a correction result made by a user other than the user using the user terminal 1, when the correction result is good. The correction evaluation input unit 116 transmits the evaluation of the accepted correction result (for example, a "like") to the response device 2 together with a correction ID that identifies the correction result.

[0049] The correction evaluation receiving unit 216 of the WEB server 21 receives an evaluation (e.g., “Like”) on the correction result from the user terminal 1. The correction evaluation receiving unit 216 can increment the number of evaluations in the correction content evaluation information in the correction content evaluation management table 315 corresponding to the correction ID that identifies the correction result.

[0050] ==Machine Learning== The learning unit 224 (corresponding to the update unit of the present invention) of the AP server 22 updates (re-learns) the machine learning model using the correction results obtained by the user on the response generated by the machine learning model in response to a question.

[0051] For example, the learning unit 224 can read out correction result information corresponding to the question ID for each piece of question information registered in the question management table 311 from the correction result management table 313, and update the machine learning model by learning the correction content included in the read correction result information. The learning unit 224 may, for example, learn combinations of questions and correction content.

[0052] The learning unit 224 may perform re-learning when the correction result is closed information (i.e., not public information). The learning unit 224 may not perform re-learning for public information.

[0053] Regarding whether or not the information is public, the learning unit 224, for example, provides the question or a keyword or phrase contained in the question to a publicly accessible search engine, determines whether or not the search results contain content similar to the correction result, and if it is determined that the correction result contains content similar to the correction result, it can determine that the correction result is public information. The learning unit 224, for example, provides the correction result or a keyword or phrase contained in the correction result to a publicly accessible search engine, determines whether or not the search results contain content similar to the correction result, and if it is determined that the correction result contains content similar to the correction result, it can determine that the correction result is public information.

[0054] Regarding whether or not information is public, the management server 2 may be provided with an external public information storage unit that stores external public information such as information disclosed on websites such as papers and Q&A sites, and past correction results, and the learning unit 224 may determine the similarity between the information stored in the external public information storage unit and the correction results, and determine that the correction results are public information if the similarity is a predetermined value or more. The learning unit 224 may also determine the similarity between the correction results and a result output by giving a question to the trained model stored in the trained model storage unit 232, and determine that the correction results are public information if the similarity is a predetermined value or more.

[0055] The learning unit 224 may perform machine learning using only correction results whose number of evaluations and / or quality evaluation scores are each equal to or greater than a given threshold. For example, for each piece of question information registered in the question management table 311, the learning unit 224 can perform machine learning using correction result information corresponding to the question ID whose number of evaluations is equal to or greater than a given threshold and / or whose quality evaluation score is equal to or greater than a given threshold (which may be the same as or different from the threshold for the number of evaluations).

[0056] The learning unit 224 may also perform machine learning using only questions whose number of ratings and / or quality evaluation scores are equal to or greater than a given threshold. For example, the learning unit 224 may select, from among the question information registered in the question management table 311, those whose number of ratings is equal to or greater than a given threshold and / or those whose quality evaluation scores are equal to or greater than a given threshold (which may be the same as or different from the threshold for the number of ratings), and perform machine learning using the question content of the selected question information and the correction content of the correction result information corresponding to the question ID of the selected question information. In addition, at this time, machine learning may be performed using only those correction result information corresponding to the question ID whose number of ratings is equal to or greater than a given threshold for the correction result and / or those whose quality evaluation scores are equal to or greater than a given threshold for the correction result (which may be the same as or different from the threshold for the number of ratings).

[0057] ==Target of training data== The learning unit 224 may use not only the correction results but also explanatory articles on papers and web pages as data to be used for relearning. In this case, too, the incentive award number calculation unit 227, which will be described later, can award incentives to the authors of papers and web pages.

[0058] ==Judgment of Question Quality== The question quality judgment unit 225 of the AP server 22 judges the quality of the question. For each piece of question evaluation information registered in the question evaluation management table 314, the question quality judgment unit 225 can determine a quality evaluation score according to, for example, the number of evaluations. For example, the question quality judgment unit 225 can calculate the quality evaluation score by dividing the number of evaluations of the question evaluation information by the number of views. Note that instead of the number of views, the number of users who viewed the question may be used. The question quality judgment unit 225 creates content for displaying the quality evaluation score for the question (hereinafter, referred to as question quality evaluation content). The question quality evaluation content can be, for example, content for allowing the user to view a good question. The question quality judgment unit 225 can create screen data written in HTML for displaying the quality evaluation scores for one or more pieces of question evaluation information. The question quality judgment unit 225 transmits the question quality evaluation content to the WEB server 21.

[0059] The question quality judgment result display unit 217 of the WEB server 21 can receive question quality evaluation content received from the AP server 22, and provide the received question quality evaluation content to the user terminal 1. In response to a request from the user terminal 1, the question quality judgment result display unit 217 may read question evaluation information from the question evaluation management table 314, and create question quality evaluation content to respond.

[0060] The question quality judgment result display unit 117 of the user terminal 1 receives the question quality evaluation content transmitted from the Web server 21, and can display to the user a screen showing the quality evaluation score of the question based on the question quality evaluation content.

[0061] ==Judgment of the quality of the corrections== The correction quality judgment unit 226 of the AP server 22 judges the quality of the correction contents. The correction quality judgment unit 226 can determine a quality evaluation score for each piece of correction contents evaluation information registered in the correction contents evaluation management table 315, for example, according to the number of evaluations. The correction quality judgment unit 226 can calculate a quality evaluation score by dividing the number of evaluations of the correction contents evaluation information by the number of views, for example. Note that instead of the number of views, the number of users who viewed the correction contents may be used. The correction quality judgment unit 226 creates content for displaying a quality evaluation score for the correction contents (hereinafter, referred to as correction quality evaluation content). The correction quality evaluation content can be, for example, content for providing a best answer to a user. The correction quality judgment unit 226 can create screen data described in HTML for displaying a quality evaluation score for one or more pieces of correction contents evaluation information. The correction quality judgment unit 226 transmits the correction quality evaluation content to the WEB server 21.

[0062] The correction quality judgment result display unit 218 of the WEB server 21 can receive the correction quality evaluation content received from the AP server 22, and provide the received correction quality evaluation content to the user terminal 1. In response to a request from the user terminal 1, the correction quality judgment result display unit 218 may read out correction content evaluation information from the correction content evaluation management table 315, and create correction quality evaluation content to respond.

[0063] The correction quality judgment result display unit 118 of the user terminal 1 can receive the correction quality evaluation content transmitted from the WEB server 21, and display to the user a screen showing the quality evaluation score of the correction content based on the correction quality evaluation content.

[0064] ==Giving incentives for corrections== The incentive amount calculation unit 227 of the AP server 22 determines the amount of incentive to be given to at least one of the user who asked the question, the user who evaluated the question, the user who corrected the response, and the user who evaluated the correction content.

[0065] The incentive may be, for example, points circulating in the market, virtual currency, or a token using blockchain technology. The incentive may also be a coupon or the like. Digital content may also be given as the incentive. For example, the incentive may be a digital content or a real lottery ticket, and the number of attempts to draw the lottery may be the amount of the incentive, or the probability of winning the lottery may be the amount of the incentive.

[0066] In this embodiment, incentives are given to the user who asked the question and the user who corrected the response. The amount of incentive given to the user who asked the question and the user who corrected the response may be different. For example, a larger incentive may be given to the user who corrected the response than to the user who asked the question.

[0067] The incentive grant number calculation unit 227 can determine the amount of incentive to be granted to each user, for example. The incentive grant number calculation unit 227 can calculate, for example, an incentive for a user who asked a question (questioner) and an incentive for a user who corrected the response (corrector). The incentive grant number calculation unit 227 can determine the amount of incentive so that a larger incentive is given to a user who provided a question and / or correction content that has a larger number of evaluations and / or quality evaluation scores.

[0068] ==Incentives for Questioners== The incentive grant amount calculation unit 227 can determine the amount of incentive to be granted to the user who has asked a question, depending on at least one of the proactiveness of the question, the quality of the question, and the quality of the questioner. The proactiveness of the question is the degree to which the user is proactively asking questions, and can be evaluated, for example, by the number of questions asked by the user, or the basic statistics thereof. The quality of the question is the degree to which the question is considered to be good by many users, and can be evaluated, for example, by the number of "likes" given by other users to a question from the user, or the basic statistics thereof. The quality of the questioner is whether the user who has asked the question has discerning eyes, that is, the degree to which other users also consider a question that the user considers good to be good, and can be evaluated, for example, by the number of "likes" given by other users to a question that the user has "liked" to.

[0069] The incentive grant amount calculation unit 227 can calculate the amount of incentive for each of the evaluation value of the question proactiveness (question proactiveness), the evaluation value of the question quality (question quality), and the evaluation value of the questioner quality (questioner quality) as described above, using the following formula. Incentive amount = coefficient a x question activity + coefficient b x question quality + coefficient c x questioner quality

[0070] The degree to which the proactiveness of the question, the quality of the question, and the quality of the questioner are emphasized may be adjusted by coefficients, which may be set arbitrarily. The above formula is not limited to a linear sum, and any formula may be adopted in which at least one of the proactiveness of the question, the quality of the question, and the quality of the questioner is used as a variable, and the larger the value of each evaluation value, the larger the amount of the incentive is calculated.

[0071] ==Incentives for correctors== The incentive grant amount calculation unit 227 can determine the amount of incentive to be granted to a user who has corrected a response, depending on at least one of, for example, the proactiveness of the correction, the quality of the correction, and the quality of the corrector. The proactiveness of the correction is the degree to which a user actively performs corrections, and can be evaluated, for example, by the number of corrections made by the user, or the basic statistics thereof. The quality of the correction is the degree to which the correction is considered good by many users, and can be evaluated, for example, by the number of "likes" given by other users to the corrections made by the user, or the basic statistics thereof. The quality of the corrector is whether the user who performed the correction has good judgment, that is, the degree to which other users also consider the corrections that the user considers good to be good, and can be evaluated, for example, by the number of "likes" given by other users to the corrections that the user has "liked" to.

[0072] The incentive grant amount calculation unit 227 can calculate the amount of incentive for each of the evaluation value of the proactiveness of correction (proactiveness of correction), the evaluation value of the quality of correction (quality of correction), and the evaluation value of the quality of the corrector (quality of corrector) as described above, using the following formula. Incentive amount = coefficient d x willingness to correct + coefficient e x quality of correction + coefficient f x quality of corrector

[0073] The degree to which the willingness to correct, the quality of the correction, and the quality of the corrector are emphasized may be adjusted by coefficients, which can be set arbitrarily. The above formula is not limited to a linear sum, and any formula can be adopted in which at least one of the willingness to correct, the quality of the correction, and the quality of the corrector is used as a variable, and the larger the value of each evaluation value, the larger the amount of incentive is calculated.

[0074] ==Incentives for reviewers== In addition, an incentive may be given to a user who has evaluated a question or a correction result. In this case, the incentive amount calculation unit 227 can determine the amount of incentive to be given to a user who has evaluated a question or a correction result, depending on at least one of the positivity of the evaluation, the quality of the evaluation, and the quality of the evaluator. The positivity of the evaluation is the degree to which a user actively performs evaluation, and can be evaluated, for example, by the number of times the user has performed evaluation such as "like" or a basic statistical amount thereof. The quality of the evaluation is the degree to which many users give good evaluations to things that they think are good evaluations, and can be evaluated, for example, by the number of "likes" from other users to a question or correction result that the user has "liked" or a basic statistical amount thereof.

[0075] ==Calculation of incentives per question / correction== The incentive grant number calculation unit 227 can also grant incentives on a per question and / or per correction basis. For example, the incentive grant number calculation unit 227 can grant an amount of incentive to the questioner or the corrector according to the number of views of the question or the correction. The incentive grant number calculation unit 227 may continue to grant an incentive to the questioner or the corrector every time the question or the correction is viewed.

[0076] The incentive grant number calculation unit 227 can issue a non-fungible token (NFT) indicating the right to receive an incentive. The incentive grant number calculation unit 227 can issue an NFT linked to a question and / or a correction, for example. The issuance of an NFT is omitted here as it uses general blockchain technology. The incentive grant number calculation unit 227 can grant an incentive for a question and / or a correction to the owner of the NFT. This NFT may be tradable.

[0077] The incentive award number calculation unit 227 may award an incentive when the learning unit 224 uses the correction result for re-learning.

[0078] The incentive grant number calculation unit 227 can grant a larger incentive for the correction result of non-public information (closed data) than for the correction result of public information. The incentive grant number calculation unit 227 may not grant an incentive for public information.

[0079] The incentive award amount calculation unit 227 transmits to the Web server 21 content (hereinafter referred to as incentive content) for displaying the determined incentive details and amount (amount awarded).

[0080] The incentive given number display unit 219 of the WEB server 21 transmits the incentive content received from the AP server 22 to the user terminal 1. An incentive management table that stores the content and amount of the incentive calculated by the incentive given number calculation unit 227 in association with the user ID indicating the user to be given the incentive may be provided in the data storage unit 231, and the incentive given number display unit 219 may, for example, in response to a request from the user terminal 1, read out the content and amount of the incentive corresponding to the user from the incentive management table, create incentive content, and respond to the user terminal 1 with the created incentive content.

[0081] The incentive amount display unit 119 of the user terminal 1 can display on the screen the type and amount of the incentive given to the user based on the incentive content transmitted from the Web server 21.

[0082] == Viewing log == The viewing log acquisition unit 220 of the WEB server 21 can acquire an access log of access to the WEB server 21 from the user terminal 1, for example. The viewing log acquisition unit 220 can acquire an access log of a general WEB server. The viewing log acquisition unit 220 can register the acquired viewing log in the viewing log storage unit 233 managed by the DB server 23.

[0083] <Operation> FIG. 9 is a diagram showing a flow of processing by the learning unit 224. First, the learning unit 224 reads data from the response management table 312, the question evaluation management table 311, and the correction content evaluation management table 315 in the data storage unit 231 (S401). Next, the learning unit 224 refers to the value of the quality evaluation score in the question evaluation management table 311 and selects a question with a certain score or more. The learning unit 224 refers to the response management table 312 for the correction ID corresponding to the selected question, refers to the value of the quality evaluation score in the correction content evaluation management table 315 corresponding to the response ID, and selects a correction result with a certain score or more (S402). Next, the learning unit 224 reads parameters of the trained model from the trained model storage unit 232 (S403). The learning unit 224 uses the trained model parameters read in step S403 as initial parameters, and the pairs of high-quality questions and correction results extracted in step S402 as training data, and executes a model re-learning process (S404). When the training is completed, the learning unit 224 saves the trained model parameters in the trained model storage unit 232 (S405). Note that re-learning (fine tuning) by machine learning employs general processing, and a description thereof will be omitted here.

[0084] 10 is a diagram showing a flow of processing by the question quality judgment unit 225. The question quality judgment unit 225 reads data from the question evaluation management table 311 in the data storage unit 231 (S421). The question quality judgment unit 225 reads data of the browsing log from the browsing log storage unit 233 (S422). The question quality judgment unit 225 counts the number of accesses (number of views) to the relevant WEB page corresponding to the question ID from the browsing log, and sets the number of views of the question evaluation information managed in the question evaluation management table 311 (S423). The question quality judgment unit 225 calculates a quality evaluation score from the number of evaluations (number of likes) and the number of views in the question evaluation management table 311 (S424). The question quality judgment unit 225 sets the calculated quality evaluation score as the quality evaluation score of the question evaluation information managed in the question evaluation management table 314 (S425). The question quality judgment unit 225 stores the question evaluation management table 311 in the data storage unit 231 (S426).

[0085] FIG. 11 is a diagram showing a flow of processing by the correction quality judgment unit 226. The correction quality judgment unit 226 reads data from the correction content evaluation management table 315 in the data storage unit 231 (S441). The correction quality judgment unit 226 reads the data of the viewing log from the viewing log storage unit 233 (S442). The correction quality judgment unit 226 counts the number of accesses (number of views) to the relevant WEB page corresponding to the correction ID from the viewing log, and sets the number of views of the correction content evaluation information managed in the correction content evaluation management table 315 (S443). The correction quality judgment unit 226 calculates a quality evaluation score from the number of evaluations (number of likes) and the number of views in the correction content evaluation management table 315 (S444). The correction quality judgment unit 226 sets the calculated quality evaluation score to the quality evaluation score of the correction content evaluation information managed in the correction content evaluation management table 315 (S445). The correction quality judgment unit 226 stores the correction content evaluation management table 315 in the data storage unit 231 (S446).

[0086] FIG. 12 is a diagram showing a flow of calculation processing of incentives for users who have asked questions. The incentive-granted number calculation unit 227 reads data from the questioner management table 316 in the data storage unit 231 (S461). The incentive-granted number calculation unit 227 counts the number of question IDs of questioner information managed in the questioner management table 316 for each questioner ID to calculate the number of questions (S462). The incentive-granted number calculation unit 227 calculates basic statistics (which may be total, average, deviation, etc.) of acquired quality scores of questioner information managed in the questioner management table 316 for each questioner ID (S463). The incentive-granted number calculation unit 227 calculates basic statistics (total, average, deviation) of "acquired quality evaluation scores" linked to "question IDs of other people who have evaluated" in the questioner management table 316 for each questioner ID (S464). The incentive-granted number calculation unit 227 determines the number of incentives for each questioner ID in consideration of the calculated values ​​(S465). The incentive award amount calculation unit 227 can award an incentive to the user who asked the question based on the calculated incentive amount (S466).

[0087] FIG. 13 is a diagram showing the flow of calculation processing of incentives for users who have corrected responses. The incentive grant number calculation unit 227 reads data from the corrector management table 317 in the data storage unit 231 (S481). The incentive grant number calculation unit 227 counts the number of correction IDs of the corrector information managed in the corrector management table 317 for each corrector ID to calculate the number of corrections (S482). The incentive grant number calculation unit 227 calculates the basic statistics (which may be the sum, average, deviation, etc.) of the acquired quality score of the corrector information managed in the corrector management table 317 for each corrector ID (S483). The incentive grant number calculation unit 227 calculates the basic statistics (the sum, average, deviation) of the "acquired quality evaluation score" linked to the "correction ID of another person who evaluated" of the corrector information managed in the corrector management table 317 for each corrector ID (S484). The incentive grant number calculation unit 227 determines the number of incentives for each corrector ID in consideration of the calculated value (S485). The incentive grant number calculation unit 227 can grant incentives to the user who performed the correction based on the calculated number of incentives (S486).

[0088] Fig. 14 is a diagram showing an example of a hardware configuration of a computer. Note that the illustrated configuration is an example, and other configurations may be used. The computer shown in Fig. 14 can implement a user terminal 1 and a response device 2 (a WEB server 21, an AP server 22, and a DB server 23).

[0089] The computer includes a CPU 201, a memory 202, a storage device 203, a communication interface 204, an input device 205, and an output device 206. The storage device 203 stores various data and programs, and is, for example, a hard disk drive, a solid state drive, or a flash memory. The communication interface 204 is an interface for connecting to a communication network, and is, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone line network, a wireless communication device for wireless communication, a USB (Universal Serial Bus) connector or an RS232C connector for serial communication, etc. The input device 205 is, for example, a keyboard, a mouse, a touch panel, a button, a microphone, etc. for inputting data. The output device 206 is, for example, a display, a printer, a speaker, etc. for outputting data. In addition, each functional unit of the above-mentioned user terminal 1 and response device 2 (WEB server 21, AP server 22, DB server 23) is realized by the CPU 201 reading out a program stored in the storage device 203 into the memory 202 and executing it, and each storage unit can be realized as part of the storage area provided by the memory 202 and the storage device 203.

[0090] Although the present embodiment has been described above, the above embodiment is intended to facilitate understanding of the present invention, and is not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit of the present invention, and equivalents thereof are also included in the present invention.

[0091] <Select from multiple candidates> For example, in the present embodiment, it is assumed that the output (response) from the trained model is one for one question, but multiple outputs may be made. For example, the response unit 221 can generate multiple responses by giving a question to the trained model and attempting to generate a response multiple times. When the trained model is a generator, different responses can be generated by increasing a randomness parameter (for example, the temperature parameter of GPT).

[0092] Here, multiple responses (output results from the trained model) may be output, and the corrector may select the output that he or she considers to be of the highest quality, and correct the selected response. In this case, the learning unit 224 may also accept designation of which output result was selected from the corrector, and re-learn the question and the correction result together with which output result was selected into the trained model, thereby improving the quality of the output of the trained model.

[0093] Alternatively, multiple output results may be output, and the viewer may select which of the multiple output results is the highest quality output, without correcting it. In this case, the learning unit 224 can improve the quality of the output of the trained model by re-learning which output result was selected by the viewer (for example, the selected output result and the number of people who selected it).

[0094] The response unit 221 can also output multiple responses to a question from the trained model and automatically select the output with the highest quality as the final response. The quality can be determined based on the acceptability to the user or the validity of the answer based on a rule base or preset conditions.

[0095] <References> The response unit 221 may provide information that may be useful for correction together with the response from the trained model. Information that may be useful for correction can be specified by, for example, collecting reference information from information sources such as websites, blog information, and papers, determining the similarity between the collected data and the response from the trained model, and selecting information whose similarity is equal to or greater than a predetermined value. The response unit 221 may display a list of the response from the trained model and the selected reference information.

[0096] In addition, the information that the corrector used as a reference or basis for the correction may be provided at the time of correction and used as re-learning data. [Explanation of symbols]

[0097] 1 User terminal 2 Answering Machine 21 Web Server 22 AP Server 23 Database Server

Claims

1. An input data receiving unit that receives input data from the first user, A processing unit that provides the aforementioned input data to a machine learning model that has been trained by machine learning and outputs output data, A correction result receiving unit that receives correction results from a second user who has viewed the output data, A determination unit provides the aforementioned input data or keywords contained in the aforementioned input data to a search engine to obtain search results, and determines whether the aforementioned editing result is public information based on whether the obtained search results contain content similar to the aforementioned editing result. An information processing system characterized by comprising the following features.

2. An input data receiving unit that receives input data from the first user, A processing unit that provides the aforementioned input data to a machine learning model that has been trained by machine learning and outputs output data, A correction result receiving unit that receives correction results from a second user who has viewed the output data, A determination unit determines whether the editing result is public information based on whether content similar to the editing result received from the second user is registered in a public information storage unit that stores public information, An information processing system characterized by comprising the following features.

3. An information processing system according to claim 1 or 2, The system includes an update unit that updates the learning model using the correction results if the correction results are not publicly available information. An information processing system characterized by the following.

4. The first step is to receive input data from the first user, The steps include: providing the aforementioned input data to a machine learning model that has been trained to produce output data; The steps include receiving the revised output data from a second user who has viewed the output data, and The steps include: providing the input data or keywords contained in the input data to a search engine to obtain search results, and determining whether the edited result is public information based on whether the obtained search results contain content similar to the edited result; A program that causes a computer to execute something.

5. The first step is to receive input data from the first user, The steps include: providing the aforementioned input data to a machine learning model that has been trained to produce output data; The steps include receiving the revised output data from a second user who has viewed the output data, and The steps include determining whether the editing result is public information based on whether content similar to the editing result received from the second user is registered in the public information storage unit that stores public information, and A program that causes a computer to execute something.

6. A program according to claim 4 or 5, wherein the computer further performs the step of updating the learning model using the correction result if the correction result is not public information.

7. The first step is to receive input data from the first user, The steps include: providing the aforementioned input data to a machine learning model that has been trained to produce output data; The steps include receiving the revised output data from a second user who has viewed the output data, and The steps include: providing the input data or keywords contained in the input data to a search engine to obtain search results, and determining whether the edited result is public information based on whether the obtained search results contain content similar to the edited result; A method characterized by a computer executing the following.

8. The first step is to receive input data from the first user, The steps include: providing the aforementioned input data to a machine learning model that has been trained to produce output data; The steps include receiving the revised output data from a second user who has viewed the output data, and The steps include determining whether the editing result is public information based on whether content similar to the editing result received from the second user is registered in the public information storage unit that stores public information, and A method characterized by a computer executing the following.

9. A method according to claim 7 or 8, wherein the computer further performs the step of updating the learning model using the correction result if the correction result is not public information.

Images