Information processing systems, information processing methods, and programs

Abstract

To enable us to provide experts who meet your specific needs. [Solution] An information processing system comprising: an answer creation unit that creates answers to questions using a machine learning model; an edit acquisition unit that receives edit results for answers from an editer; an edit storage unit that stores information identifying the editer and the edit results in association; and an editer presentation unit that, when a new question is received from the questioner, searches the evaluation storage unit for an editer who has provided an edit result related to the new question and presents information identifying the retrieved editer to the questioner.

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing method, and a program.

Background Art

[0002] Patent Document 1 discloses a system that matches an expert suitable for the needs of a prospective customer.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] It is necessary to preset the business scope and field of expertise of an expert on the system side.

[0005] The present invention has been made in view of such a background, and an object thereof is to provide a technology capable of presenting an expert that meets needs.

Means for Solving the Problems

[0006] The main invention of the present invention for solving the above problems is an information processing system, comprising: an answer creation unit that creates an answer to a question using a machine learning model; a proofreading acquisition unit that receives a proofreading result for the answer from a proofreader; a proofreading storage unit that stores the information identifying the proofreader and the proofreading result in association with each other; and a proofreader presentation unit that, when receiving the question from a questioner, searches the evaluation storage unit for the proofreader who provided the proofreading result related to the question and presents information identifying the searched proofreader to the questioner.

[0007] Further issues and solutions disclosed in this application will be made clear in the section on embodiments of the invention and in the drawings. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide experts who meet the needs. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the overall configuration of an information processing system. [Figure 2] This figure shows an example of the hardware configuration of management server 2. [Figure 3] This figure shows an example of the software configuration for management server 2. [Figure 4] This diagram illustrates the operation of management server 2. [Modes for carrying out the invention]

[0010] <System Overview> The following describes an information processing system according to one embodiment of the present invention. The information processing system of this embodiment aims to match a user (questioner) with an expert. In the information processing system of this embodiment, an answer to the question is created using AI, and the answer generated by the AI ​​is reviewed by an expert reviewer, and the review results are evaluated. For a user with a new question, an expert can be matched based on the evaluation of past reviews (for example, an evaluation that is above a predetermined value).

[0011] Figure 1 shows an example of the overall configuration of an information processing system. The information processing system in this embodiment includes a management server 2. The management server 2 is connected to the user terminal 1 via a communication network. The communication network is, for example, the internet and is constructed using public telephone networks, mobile phone networks, wireless communication channels, Ethernet (registered trademark), etc.

[0012] User terminal 1 is a computer operated by the user. User terminal 1 can be, for example, a smartphone, a tablet computer, or a personal computer. Users may include a user with a question (the questioner) and an expert who can provide an answer to that question. The expert also acts as a proofreader, correcting the AI's answer to the question.

[0013] The management server 2 may be a general-purpose computer such as a workstation or personal computer, or it may be logically implemented through cloud computing.

[0014] <Management Server> Figure 2 shows an example of the hardware configuration of the management server 2. Note that the illustrated configuration is just one example, and other configurations are also possible. The management server 2 includes a CPU 201, memory 202, storage device 203, communication interface 204, input device 205, and output device 206. The storage device 203 stores various data and programs, such as a hard disk drive, solid-state drive, or flash memory. The communication interface 204 is an interface for connecting to a communication network, such as an adapter for connecting to Ethernet®, a modem for connecting to a public telephone network, a wireless communication device for wireless communication, or a USB (Universal Serial Bus) connector or RS232C connector for serial communication. The input device 205 is for inputting data, such as a keyboard, mouse, touch panel, button, or microphone. The output device 206 is for outputting data, such as a display, printer, or speaker. Furthermore, each functional unit of the management server 2, as described later, is realized by the CPU 201 reading programs stored in the storage device 203 into memory 202 and executing them, and each storage unit of the management server 2 is realized as part of the storage area provided by memory 202 and storage device 203.

[0015] Figure 3 shows an example of the software configuration of the management server 2. The management server 2 includes a known information storage unit 231, an answer storage unit 232, an correction storage unit 233, an answer creation unit 211, a correction acquisition unit 212, an evaluation acquisition unit 213, an information registration unit 214, and a corrector presentation unit 215.

[0016] <Storage section> The known information storage unit 231 stores various types of knowledge as known information. Known information may be, for example, PDF files, word processor files, or text files. Image data and audio data can also be used as known information. The known information storage unit 231 can also be implemented by, for example, a file system. In this embodiment, the known information storage unit 231 is assumed to consist of a file system that stores known information as files and a vector database that stores vector data in which the contents of the files (part or all) have been vectorized by embedding. By searching the vector database, it becomes possible to efficiently search for known information that has similar content to the search target.

[0017] The answer storage unit 232 stores information related to the answer created by the AI ​​(hereinafter referred to as answer information). The answer information may include the question and the answer. The answer information may also include information that identifies the questioner and the date and time the question or answer was asked.

[0018] The correction memory unit 233 stores information regarding the results of corrections made by experts (hereinafter referred to as correction information). The correction information includes information identifying the corrector, the correction result, the evaluation value, and the aggregated value of the evaluation value. The correction result may be the path or URL to the file storing the correction result. The correction information may also include vector data in which the correction result has been embedded, either in place of or in addition to the correction result. The evaluation value may be a list of evaluation values ​​from multiple users (evaluators). The aggregated value of the evaluation value may be, for example, the average value. The correction information may also include information identifying the question (information identifying the answer information).

[0019] <Functional Unit> The answer creation unit 211 creates an answer to a question. The answer creation unit 211 can create an answer to a question using a machine learning model. In the present embodiment, the machine learning model is assumed to be a pre-trained large language model (LLM) that is a generator, and it is assumed that the LLM is provided in an external server, and by accessing an API provided by the external server and giving a prompt to the LLM, the LLM is caused to generate an answer. Note that the management server 2 may be provided with a learning model storage unit that stores the machine learning model.

[0020] In the present embodiment, the answer creation unit 211 can create an answer by so-called RAG (Retrieval-Augmented Generation). The answer creation unit 211 searches the known information storage unit 231 for known information related to the question, and generates an answer by giving a prompt including the retrieved known information, the question, and an instruction to create an answer to the question based on the known information to the LLM.

[0021] Whether it is related to the question can be determined, for example, by whether the distance between the vector data obtained by embedding the question and the vector data stored in the known information storage unit 231 is close. For example, the question can be tokenized using a morphological analyzer, and each token can be converted into a feature vector. For the conversion into a feature vector, a distributed representation learning algorithm such as word2vec or GloVe can be used. Using the feature vector obtained from the question, a question vector is generated. The question vector can be obtained by averaging the feature vectors of each token included in the question. Alternatively, a weighted average may be calculated according to the importance of the tokens. The answer creation unit 211 also vectorizes the known information in the same manner. That is, after tokenizing by a morphological analyzer and converting each token into a feature vector, a known information vector is generated.

[0022] The relationship between a question vector and a known information vector can be determined by calculating the cosine similarity between these vectors. Cosine similarity takes a value between -1 and 1, with a value closer to 1 indicating a higher degree of similarity. In this embodiment, if the cosine similarity between the question vector and the known information vector is greater than or equal to a predetermined threshold (e.g., 0.7), they are considered to be related. If multiple pieces of known information exceed the threshold, a predetermined number (e.g., 5) can be selected as related in order of decreasing cosine similarity.

[0023] Furthermore, the answer creation unit 211 can register the question and the created answer in the answer storage unit 232.

[0024] The correction acquisition unit 212 receives correction results for answers from experts (correctors). The answer information registered in the answer storage unit 232 is made available for viewing by experts, and the correction acquisition unit 212 can receive correction results sent from the user terminal 1 of the expert (corrector) who has viewed the answer. It may also be configured to receive information based on the question from the expert's user terminal 1.

[0025] The correction acquisition unit 212 may also have experts make corrections to the answers and questions created by the answer creation unit 211 by sending a message to the expert instructing them to make corrections. The correction acquisition unit 212 can send the message to the expert's user terminal 1 and receive the correction results from the expert's user terminal 1.

[0026] The correction acquisition unit 212 registers the acquired correction results in the correction storage unit 233. The correction acquisition unit 212 can register correction information, including information identifying the corrector, the correction result, an empty evaluation value, and a NULL evaluation value, in the correction storage unit 233.

[0027] The evaluation acquisition unit 213 acquires evaluation values ​​for the correction results. The correction information (and corresponding response information) stored in the correction storage unit 233 is made available for viewing by the user (evaluator). The evaluation acquisition unit 213 can receive evaluation values ​​for the correction results from the user terminal 1 of the evaluator who has viewed the correction results. The user (evaluator) can, for example, give a "like" to the correction results, and the act of giving a "like" can be counted as 1 point in the evaluation value. The evaluation acquisition unit 213 may acquire the evaluation value as a score, such as out of 5 points, out of 10 points, or out of 100 points.

[0028] The evaluation acquisition unit 213 may determine an evaluation value according to the popularity of the edited result. The evaluation acquisition unit 213 can acquire the number of views of the edited result and determine the popularity of the edited result according to the number of views. The number of views may be determined as popularity, or a weighted number of views may be calculated by weighting according to the viewer, and the weighted number of views may be used as popularity.

[0029] The evaluation unit 213 may determine an evaluation value according to the quality of the editing result. For example, the evaluation unit 213 can calculate the quality by dividing the number of "likes" given to the editing result (hereinafter referred to as the evaluation count) by the number of views (i.e., the percentage of "likes" given among those who viewed the result). The evaluation unit 213 may use the number of unique users who viewed the result as the number of views, rather than the total number of views. Each user may be allowed to give a "like" to a single editing result only once. The evaluation unit 213 may calculate the quality by dividing the sum of the evaluation scores by the number of views.

[0030] The evaluation unit 213 may determine an evaluation value based on the popularity and quality of the correction results in the manner described below.

[0031] The popularity of the edited results can be measured using indicators such as the number of views, downloads, and "likes." The evaluation unit 213 can weight these indicators and calculate the weighted sum as the popularity score. For example, by assigning a weight of 0.5 to the number of views, a weight of 0.3 to the number of downloads, and a weight of 0.2 to the number of "likes," the popularity score can be calculated using the following formula. Popularity Score = 0.5 × Views + 0.3 × Downloads + 0.2 × Likes Note that these weights are just examples and can be changed.

[0032] The quality of the editing results can be measured using, for example, expert evaluations, user evaluations, or automated evaluation metrics. Expert evaluations allow experts familiar with the field (may include or exclude the editors) to evaluate the accuracy and usefulness of the editing results on a 5-point scale. User evaluations allow users who view the editing results to rate them as "helpful" or similar on a 5-point scale. Automated evaluation metrics can quantitatively measure the quality of the editing results using, for example, natural language processing evaluation metrics such as BERT scores or ROUGE scores. The evaluation acquisition unit 213 can calculate a quality score by weighting these evaluations. For example, by assigning a weight of 0.4 to expert evaluations, a weight of 0.4 to user evaluations, and a weight of 0.2 to automated evaluations, the quality score can be calculated using the following formula. Quality Score = 0.4 × Expert Rating + 0.4 × User Rating + 0.2 × Automated Rating Note that these weights are just examples and can be changed.

[0033] The evaluation unit 213 may integrate the popularity score and the quality score to determine an overall evaluation value. For example, the evaluation unit 213 can add the popularity score and the quality score with weights assigned to them. By assigning a weight of 0.3 to the popularity score and a weight of 0.7 to the quality score, the evaluation value can be calculated using the following formula. Rating = 0.3 × Popularity Score + 0.7 × Quality Score Note that these weights are just examples and can be changed.

[0034] The evaluation acquisition unit 213 can update the aggregated evaluation values ​​by adding the acquired evaluation values ​​(or determined evaluation values) to the evaluation values ​​of the correction information corresponding to the correction results of the subject being evaluated.

[0035] The information registration unit 214 registers the correction results in the known information storage unit 231. The information registration unit 214 can decide whether or not to register the correction results depending on the evaluation value. The information registration unit 214 can register the correction results in the known information storage unit 231 if the evaluation value is equal to or greater than a predetermined value.

[0036] The proofreader suggestion unit 215 can suggest experts to a user (questioner) who has a question. The experts are those who have previously provided proofreading. When the proofreader suggestion unit 215 receives a question from the questioner, it can search the evaluation storage unit 213 for proofreaders who have previously provided proofreading results related to that question, and can present the questioner with information that identifies some or all of the found proofreaders. The answer creation unit 211 may create an answer only if the proofreader suggestion unit 215 is unable to suggest an expert (i.e., unable to find proofreaders who have provided relevant proofreading results), or it may create an answer even if an expert is able to be suggested.

[0037] Regarding whether or not the information is relevant to the question, the editor information unit 215 can, for example, determine whether the distance between the vector data with the question embedded and the vector data with the editing results embedded is close. The editor information unit 215 can use the cosine similarity between the feature vector of the editing results and the question vector, similar to the processing described above for known information. The editor information unit 215 can search for a predetermined number of editing information in order of similarity and provide the questioner with information identifying the editors included in the searched editing information.

[0038] The proofreader presentation unit 215 can extract a predetermined number of proofreaders in order of their evaluation values ​​from the searched proofreaders and provide the questioner's user terminal 1 with information identifying the extracted proofreaders. The proofreader presentation unit 215 may present experts and ask the user whether or not to have the AI ​​generate an answer, and the answer creation unit 211 may create an answer in response to the user's instruction to have an answer generated, or the answer creation unit 211 may create an answer without asking whether or not an answer needs to be generated.

[0039] <Operation> Figure 4 is a diagram illustrating the operation of the management server 2.

[0040] When the management server 2 receives a question from the questioner (S301), it searches for known information related to the question (S302), and gives the LLM a prompt containing the searched known information and instructions to create an answer, causing it to generate an answer (S303). The management server 2 searches for past editing results related to the question (S304), and presents the questioner with the experts who provided the searched editing results (S305). When the management server 2 obtains the editing results for the answer from the editor (S306), it obtains the evaluation value for the editing results from the evaluator and updates the evaluation value of the editing results (S307), and if the evaluation value is above a predetermined value (S308:YES), it registers the editing results as known information (S309).

[0041] As described above, the information processing system of this embodiment makes it possible to match the questioner with an expert who has a high reputation for past editing work.

[0042] Although these embodiments have been described above, they are intended to facilitate understanding of the present invention and are not intended to limit its interpretation. The present invention can be modified and improved without departing from its spirit, and equivalents thereof are also included.

[0043] For example, the processing performed by each functional unit of the management server 2 described above may be executed by any of the functional units. Furthermore, different functional units may be added to perform some of the processing performed by each of the functional units described above. Also, the functional units of the management server 2 may be distributed across multiple computers.

[0044] Furthermore, the information stored in each memory unit of the management server 2 may be stored in any of the memory units. That is, the information stored in the multiple memory units mentioned above may be stored in a single memory unit, or a portion of the information stored in one memory unit may be stored in another memory unit.

[0045] <Example 1> In the above embodiment, the reviewer was evaluated based solely on the quality of the editing results. However, evaluation can also be performed using the reviewer's profile information in addition to this. Profile information may include, for example, the reviewer's work history, field of expertise, degree, qualifications, affiliated academic societies, number of publications, and number of patents.

[0046] Evaluation using profile information can be carried out using the following procedure. For example, profile information of the proofreader is collected and registered in a database. Next, weights are assigned to each item of the profile information. For example, a high weight may be assigned if the area of ​​expertise matches the area of ​​the question, a medium weight if the person has a long career, a medium weight if they have relevant degrees or qualifications, and a low weight if they have a large number of publications or patents.

[0047] For each proofreader, a profile score can be calculated by multiplying the value of each item in their profile information by its weight and then summing the results. For example, the profile score can be calculated using the following formula. Profile score = Σ (item value × weight)

[0048] The quality score and profile score can be combined to determine the overall evaluation score of the proofreader. For example, by assigning a weight of 0.6 to the quality score and a weight of 0.4 to the profile score (these weights are just examples and can be changed), the evaluation score can be calculated using the following formula. Rating = 0.6 × Quality Score + 0.4 × Profile Score

[0049] Furthermore, the weighting of profile information can be dynamically adjusted according to the field and difficulty level of the question. For example, for highly specialized questions, the weighting of the field of expertise can be increased, while for basic questions, the weighting of the work history can be increased.

[0050] <Modification 2> In the above embodiment, vectorization and similarity calculation were used to determine the relationship between the question and the correction result, but it is also possible to combine this with semantic analysis using natural language processing. By performing semantic analysis, it becomes possible to determine the relationship not only at the superficial level of word matching, but also at a deeper level of meaning based on context.

[0051] For example, syntactic analysis and semantic role labeling can be performed on the text of a question and its correction results. Syntactic analysis analyzes the syntactic structure of a sentence (subject, predicate, object, etc.), while semantic role labeling determines the semantic role (agent, object, place, etc.) of words and phrases in a sentence. Natural language processing libraries and toolkits (such as Stanford CoreNLP and SpaCy) can be used for these processes.

[0052] Next, based on the results of syntactic analysis and semantic role labeling, we can extract the semantic expressions of the question and the corrected result. Semantic expressions represent the semantic roles of subjects, predicates, objects, etc., and their relationships. For example, the semantic expression of the sentence "Yesterday, I walked my dog ​​in the park" would be something like "I (agent) walked my dog ​​(object) in the park (place) yesterday (time) (action)."

[0053] Next, we can compare the semantic expressions of the question and the corrected results to calculate their semantic relationship. Indicators such as the degree of agreement in semantic roles and the similarity of predicates (calculated using a thesaurus such as WordNet) can be used to compare the semantic expressions. By combining these indicators, an overall relationship score can be calculated, and if the score is above a certain threshold, a relationship can be determined.

[0054] Finally, the results of the correlation determination using vectorization and similarity calculation are integrated with the correlation determination using semantic analysis. For example, a combined score may be calculated by weighting the correlation scores of both methods to determine whether or not there is a final correlation.

[0055] <Variation 3> In the above embodiment, the mean was used as the aggregated value of the evaluation, but the median and mode may also be calculated and made available for the questioner to select.

[0056] For example, regarding the evaluation values ​​of the correction results, not only the mean but also the median and mode can be calculated, and the questioner can be presented with options for these aggregated values. For instance, a message such as "Please choose how to aggregate the evaluation values: 1. Mean, 2. Median, 3. Mode" can be displayed, allowing the questioner to make a selection. The questioner can then choose the appropriate aggregation method according to the aspects they consider important.

[0057] Next, based on the aggregation method selected by the questioner, the system ranks the graders' evaluation scores and recommends the top graders. For example, if the questioner selects the mean, the graders can be ranked in descending order of average evaluation score; if the median is selected, in descending order of median score; and if the mode is selected, in descending order of mode score, with the top few graders presented to the questioner.

[0058] Furthermore, multiple aggregate values ​​can be combined when recommending proofreaders. For example, the final ranking could be determined by combining the ranks of the mean and median, or proofreaders with high scores in both the mean and mode could be given priority in the recommendation process.

[0059] <Modification 4> In the above embodiment, the correction results were presented directly to the questioner, but it is also possible to add a function that explains the corrections in an easy-to-understand way for the questioner.

[0060] For example, by comparing the original answer before revision with the revised answer, it is possible to identify the changes. A string difference detection algorithm (such as the Diff algorithm) can be used to identify the changes.

[0061] Next, for the identified changes, it is possible to estimate the intent and effect of those changes. Natural language processing techniques can be used for this estimation. For example, by analyzing the context before and after the changes, it is possible to determine the purpose of the changes (correction of typos, simplification of expression, addition of information, etc.) and evaluate the quality of the revised expression (readability, accuracy, appropriateness, etc.).

[0062] Next, we generate a sentence that clearly explains the intent and effect of the estimated change to the questioner. Template-based methods or machine learning-based sentence generation models (e.g., T5 or BART) can be used to generate this explanatory sentence. For example, the following explanatory sentence can be generated: "This section contained a lot of technical jargon, so we've changed it to simpler language that's easy for the general public to understand." "The previous answer did not directly address the core of the question, so I have revised it to reflect the main points of the question." "This section has been corrected due to a typo. The correct wording is XX."

[0063] Finally, the generated explanation can be presented to the questioner along with the corrected answer.

[0064] Another possible method for presenting explanations is to directly append them to the changes in the corrected answer. For example, the changes could be highlighted, and when the mouse hovers over them, an explanation could pop up.

[0065] Furthermore, the process of generating explanatory text can also be configured to generate explanatory text by providing the LLM with a prompt that includes the original answer, the revised answer, and instructions to explain the differences.

[0066] <Disclosure Items> Furthermore, this disclosure also includes the following configurations. [Item 1] An answer generation unit that uses a machine learning model to create answers to questions, A correction acquisition unit that receives the correction results for the aforementioned answer from the corrector, A correction storage unit that stores information identifying the person who made the correction and the correction results in association, When a new question is received from the questioner, the editor presenting unit searches the evaluation storage unit for the editor who provided the editing results related to the new question, and presents the questioner with information identifying the editor that was found. An information processing system characterized by comprising the following features. [Item 2] The information processing system described in item 1, The system includes an evaluation unit that acquires an evaluation value for the aforementioned editing results, The correction storage unit stores information identifying the corrector, the correction result or information based on the correction result, and a value based on the evaluation value. The aforementioned proofreader presentation unit selects some of the proofreaders found based on the evaluation value and provides the questioner with information identifying the selected proofreaders. An information processing system characterized by the following. [Item 3] The information processing system described in item 2, The evaluation unit determines the evaluation value according to the popularity of the correction result. An information processing system characterized by the following. [Item 4] The information processing system described in item 2, The evaluation unit determines the evaluation value according to the quality of the correction result. An information processing system characterized by the following. [Item 5] The steps involve creating answers to questions using machine learning models, The steps include receiving the corrected version of the answer from the proofreader, The steps include storing information identifying the proofreader and the proofreading results in the proofreading memory unit in association with each other, When a new question is received from the questioner, the system retrieves the editor who provided the editing results related to the new question from the editing storage unit, and presents the questioner with information identifying the editor that was retrieved. An information processing method characterized by a computer executing the following. [Item 6] The steps involve creating answers to questions using machine learning models, The steps include receiving the corrected version of the answer from the proofreader, The steps include storing information identifying the proofreader and the proofreading results in the proofreading memory unit in association with each other, When a new question is received from the questioner, the system retrieves the editor who provided the editing results related to the new question from the editing storage unit, and presents the questioner with information identifying the editor that was retrieved. A program that causes a computer to execute something. [Explanation of symbols]

[0067] 1 User terminal 2 Management Server

Claims

1. An answer generation unit that uses a machine learning model to create answers to questions, A correction acquisition unit that receives the correction results for the aforementioned answer from the corrector, A correction storage unit that stores information identifying the person who made the correction and the correction results in association, When a new question is received from the questioner, the editor presentation unit searches the evaluation storage unit for the editor who provided the editing results related to the new question, and presents the questioner with information identifying the editor that was found. An information processing system characterized by comprising the following features.

2. The information processing system according to claim 1, The system includes an evaluation unit that acquires an evaluation value for the aforementioned editing results, The correction storage unit stores information identifying the corrector, the correction result or information based on the correction result, and a value based on the evaluation value. The aforementioned proofreader presentation unit selects some of the proofreaders found based on the evaluation value and provides the questioner with information identifying the selected proofreaders. An information processing system characterized by the following.

3. The information processing system according to claim 2, The evaluation unit determines the evaluation value according to the popularity of the correction result. An information processing system characterized by the following.

4. The information processing system according to claim 2, The evaluation unit determines the evaluation value according to the quality of the correction result. An information processing system characterized by the following.

5. The steps involve creating answers to questions using machine learning models, The steps include receiving the corrected version of the answer from the proofreader, The steps include storing information identifying the proofreader and the proofreading results in the proofreading memory unit in association with each other, When a new question is received from the questioner, the system retrieves the editor who provided the editing results related to the new question from the editing storage unit, and presents the questioner with information identifying the editor that was retrieved. An information processing method characterized by a computer executing the following.

6. The steps involve creating answers to questions using machine learning models, The steps include receiving the corrected version of the answer from the proofreader, The steps include storing information identifying the proofreader and the proofreading results in the proofreading memory unit in association with each other, When a new question is received from the questioner, the system retrieves the editor who provided the editing results related to the new question from the editing storage unit, and presents the questioner with information identifying the editor that was retrieved. A program that causes a computer to execute something.

Images