Information processing device, information processing method, and information processing program

Abstract

Provided are an information processing device, an information processing method, and an information processing program in which is generated a question not having a response described in text. Using text to be analyzed as input, a question generation unit (12) uses a pre-trained machine learning model to generate generated text, which is text pertaining to the content of the text to be analyzed, and a classification type, which is information indicating whether the generated text is a question about information not containing an answer in the text to be analyzed. A question file creation unit (13) outputs the generated text and classification type generated by the question generation unit (12).

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Question generation technology is a technology that takes text as input and generates questions in natural language about that text. Many of the generated questions are used as scenarios or training data for question-answering systems. Therefore, there are few mechanisms that provide value from the questions generated by question generation technology. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-135456 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when considering a device intended to present the generated questions themselves, conventional question generation technology has the problem that when generating questions from the source text, it generates questions even if the answers are already contained in the text. [Means for solving the problem]

[0005] In order to solve the above-mentioned problems and achieve the object, the information processing device has the following units: A question generation unit receives a sentence to be analyzed as input, and generates a question about the content of the sentence to be analyzed using a machine learning model that has been trained in advance. question a question output unit that outputs the generated sentence and the type of the generated sentence, the question being a question about information for which the answer is not included in the sentence to be analyzed; [Effects of the Invention]

[0006] According to the present invention, a question sentence is not generated if the answer is already included in the text, and a question sentence can be generated if the answer is not included in the text. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram of an information processing apparatus according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing the details of the question generation unit. [Figure 3] FIG. 3 is a block diagram of a machine learning device that trains a question generation model. [Figure 4] FIG. 4 is a diagram illustrating an example of question-answering learning data. [Figure 5] FIG. 5 is a conceptual diagram of learning data for training a question generation model. [Figure 6] FIG. 6 is a diagram illustrating an example of a question created by the information processing device according to the embodiment. [Figure 7] FIG. 7 is a flowchart of a question generation process performed by the information processing device according to the embodiment. [Figure 8] FIG. 8 is a flowchart of the machine learning process performed by the machine learning device according to the embodiment. [Figure 9] FIG. 9 is a diagram showing the results of an experiment using the information processing device according to the embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a computer that executes an information processing program. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of an information processing device, an information processing method, and an information processing program disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the information processing device, the information processing method, and the information processing program disclosed in the present application are not limited to the following embodiments.

[0009] [Configuration of the embodiment] 1 is a block diagram of an information processing device according to an embodiment, and the configuration of the information processing device 1 according to the embodiment will be described with reference to FIG.

[0010] The information processing device 1 is connected to a text data DB 2. The text data DB 2 is a database in which various document files are stored. As shown in FIG. 1 , the information processing device 1 includes a text data DB (Data Base) management unit 11, a question generation unit 12, and a question file creation unit 13.

[0011] The text data DB management unit 11 monitors the text data DB 2. When a new file entry is added to the text data DB 2, the text data DB management unit 11 acquires the text portion of the newly added entry. The text data DB management unit 11 then transfers the acquired text to the question generation unit 12.

[0012] Here, the text data DB management unit 11 can operate by setting the monitoring frequency of the text data DB2 to a predetermined cycle. For example, the text data DB management unit 11 may monitor the text DB2 every five minutes and transfer the text to the question generation unit 12. The text data DB management unit 11 may also process all entries added that day as a batch process overnight. Alternatively, the text data DB management unit 11 may acquire text upon receiving a notification that an entry has been added by a signal output from the text data DB2. Furthermore, the text data DB management unit 11 may transfer text according to predetermined conditions. For example, the text data DB management unit 11 may be configured to extract text from files created by people who meet predetermined conditions.

[0013] The question generation unit 12 receives text input from the text data DB management unit 11. Then, the question generation unit 12 generates a question based on the acquired text. Details of question generation by the question generation unit 12 are explained below. FIG. 2 is a block diagram showing details of the question generation unit. As shown in FIG. 2, the question generation unit 12 has a text processing unit 121 and a question generation model 122. Hereinafter, the target text from which a question is generated will be referred to as the "analysis target text."

[0014] The text processing unit 121 receives an input of the analysis target text T acquired from the text data DB management unit 11. The text processing unit 121 also receives an input of the viewpoint label P from an external device (not shown). The viewpoint label P may be input in advance using the external device. Both the analysis target text T and the viewpoint label P are character strings.

[0015] The viewpoint label P is a label that indicates the viewpoint of the question, such as "money" or "legal affairs." The viewpoint label P can be freely set as long as it is appropriate for the content of the text that will be used as training data. For example, the viewpoint label P may be an abstract viewpoint such as "money," or it may be a person's name if a collection of questions from the same person is used as training data.

[0016] In this embodiment, the case where the viewpoint label P is input together with the analysis target text T has been described as an example, but the question generation unit 12 does not have to use the viewpoint label P when generating a question. In that case, the analysis target text T is input to the text processing unit 121, but the viewpoint label P is not input.

[0017] The text processing unit 121 performs a process of converting character strings representing text into a vector format that can be input to a deep learning model. For example, the text processing unit 121 divides the input analysis target text T into tokens, which are predetermined processing units. Any word, such as a morpheme, word, or subword, can be used as the predetermined processing unit. In this embodiment, as an example of processing, the text processing unit 121 performs morphological analysis when dividing the text into tokens. Then, the text processing unit 121 creates a one-hot vector by assigning an ID (Identifier) ​​corresponding to each token. In this way, the text processing unit 121 converts the analysis target text T into a format that can be input to the question generation model 122. Here, any method can be used for the morphological analysis and ID assignment. The text processing unit 121 also converts the viewpoint label P into a vector format.

[0018] Thereafter, the text processing unit 121 outputs the analysis target text T and viewpoint label P converted into vector format to the question generation model 122.

[0019] The question generation model 122 is a deep learning generative model that includes a text generation layer 123. There are no particular restrictions on the model that can be adopted as the question generation model 122, as long as it is a neural network model that can take text as input and output text.

[0020] The question generation model 122 receives as input a viewpoint label P and a text T to be analyzed from the text processing unit 121. The text T to be analyzed is text extracted from a document stored in the text data DB2 by the text data DB management unit 11 and converted into a vector format by the text processing unit 121.

[0021] The question generation model 122, which is a pre-trained neural network model including the text generation layer 123, receives as input the viewpoint label P and the analysis target text T converted into vector format, and performs calculations to generate the category type Ty and the result text O. Here, if the viewpoint label P is not used, the question generation model 122 receives as input the analysis target text T converted into vector format, and generates the category type Ty and the result text O.

[0022] The type Ty is information for determining whether the result text O is a question or extracted text, which is a character string extracted from the analysis target text. Furthermore, when the viewpoint label P is not used, the type Ty is information for determining whether the result text O is a question. The question generation model 122 generates the character string "question" as the type Ty if the result text O is a question sentence, and generates the character string "extraction" if the result text O is extracted text. Note that the character string used here as information representing the type Ty is just an example, and other character strings may be set.

[0023] As described above, the question generation unit 12 receives an analysis target sentence as input and generates a generated sentence and a type of category, which is information indicating whether the generated sentence is a question about information not included in the analysis target sentence, using a pre-trained machine learning model. The question generation unit 12 also acquires a viewpoint, which is information indicating a tendency of the content of the generated sentence, and receives the analysis target sentence and the viewpoint as input to generate a generated sentence and a type of category. Here, "representing a tendency" means indicating to what category the content belongs, such as "money" or "legal affairs," and may be abstract information such as "money," or specific information such as a person's name.

[0024] Furthermore, before the question generator 12 performs the above-described question generation process, the question generation model 122 is trained in advance. The training process of the question generation model 122 will be described below. FIG. 3 is a block diagram of a machine learning device that trains the question generation model. Here, in this embodiment, for ease of understanding, the question generation model 122 is trained using the machine learning device 20 shown in FIG. 3, but the machine learning function of the machine learning device 20 may be installed in the information processing device 1. In that case, the information processing device 1 has two operating phases: a learning phase and an inference phase. The information processing device 1 executes the function of the machine learning device 20 in the learning phase and performs question generation processing in the inference phase.

[0025] The machine learning device 20 can use text data for question-answering model training, such as machine reading comprehension, to train the question generation model 122. FIG. 4 is a diagram showing an example of question-answering training data. As shown in FIG. 4, the question-answering training data, which is text data for training, is formed as a set of three pieces of information: training text, question sentence, and answer sentence. Here, the training text is a text that is a partial excerpt from the text to be analyzed. The machine learning device 20 has a training data generation unit 21, a question generation unit 22, and a parameter update unit 23.

[0026] The learning data generation unit 21 acquires question-answering learning data such as that shown in Fig. 4. Fig. 5 is an image diagram of learning data for training a question generation model. Here, the learning data generation unit 21 simultaneously creates two types of learning data such as that shown in Fig. 5 from one piece of question-answering learning data such as that shown in Fig. 4.

[0027] The first training data #1 at the top of FIG. 5 is training data for question generation. When creating training data for question generation, the training data generation unit 21 uses the training text and question sentence of the question-answering training data. First, the training data generation unit 21 deletes the answer sentence portion from the training text in the question-answering training data to generate training text in the training data for question generation. Next, the training data generation unit 21 sets the classification type in the training data for question generation to "question." Next, the training data generation unit 21 sets the question sentence of the question-answering training data as the result text of the training data for question generation. As a result, the training data generation unit 21 sets the training text as text missing the answer portion, and sets the question sentence that questions the missing portion of the training text as the correct answer generation sentence.

[0028] The second training data #2 at the bottom of the page in Figure 4 is training data for generating an extracted text. When creating training data for generating an extracted text, the training data generation unit 21 uses the training text and answer sentence of the question-answer training data. In this case, the training data generation unit 21 sets the training text of the question-answer training data as is as the training text in the training data for generating an extracted text. Next, the training data generation unit 21 sets the type of the training data for generating an extracted text to "extraction." Next, the training data generation unit 21 sets the answer sentence as the result text of the training data for generating an extracted text. As a result, the training data generation unit 21 sets the answer sentence as the correct generated sentence.

[0029] Thereafter, the training data generation unit 21 generates training data by assigning viewpoint labels to the training data including the training text, the type of correct answer, and the result text of the correct answer.The training data generation unit 21 then outputs the type of correct answer and the result text of the correct answer to the parameter update unit 23.The training data generation unit 21 also outputs the training text with the viewpoint labels added to the question generation unit 22.

[0030] As described above, the training data generation unit 21 acquires the analysis target sentence, the question sentence related to the content of the analysis target sentence, and the answer sentence that is an answer to the question sentence and includes information contained in the analysis target sentence. When generating a generated sentence having a type indicating that the generated sentence is a question about information not contained in the training text, the training data generation unit 21 removes the information that is an answer to the question sentence from the analysis target sentence and treats the question sentence as the correct answer generated sentence. When generating a generated sentence having a type indicating that the generated sentence is not a question about information not contained in the analysis target sentence, the training data generation unit 21 treats the answer sentence as the correct answer generated sentence. Here, the training text is an example of the analysis target sentence, and the result text is an example of a generated sentence. Note that an answer sentence that is an answer to a question sentence and includes information contained in the analysis target sentence is an answer sentence that is an answer sentence to the question sentence and includes information contained in the analysis target sentence. Furthermore, when a generated sentence is to be generated, the type indicates that the generated sentence, which is the resultant text generated by the information processing device 1, has a type indicating that it is a question about information not included in the sentence to be analyzed, i.e., the type is "question." Similarly, when a generated sentence is to be generated, the type indicates that the generated sentence, which is the resultant text generated by the information processing device 1, has a type indicating that it is not a question about information not included in the sentence to be analyzed, i.e., the type is "extraction."

[0031] The question generation unit 22 has a text processing unit 221 and a question generation model 222. The question generation model 222 is a neural network model that has not yet completed learning and includes a text generation layer 223. The question generation unit 22 performs the same processing as the question generation unit 12 shown in Fig. 2. However, the question generation unit 22 performs the processing based on parameters that are currently being learned.

[0032] The text processing unit 221 performs the same processing as the text processing unit 121 shown in Fig. 2. The viewpoint labels and training text output from the text processing unit 221 are input to the question generation model 222.

[0033] The question generation model 222 performs the same processing as the question generation model 122 shown in Fig. 2 on the input viewpoint labels and training text based on the parameters being trained. Then, the question generation model 222 outputs the generated category type and result text to the parameter update unit 23.

[0034] The parameter update unit 23 receives an input of the correct type and result text of the correct answer from the learning data generation unit 21. The parameter update unit 23 also receives an input of the type and result text generated by the question generation model 222. The parameter update unit 23 then compares the type and result text generated by the question generation model 222 with the correct type and result text of the correct answer. Thereafter, the parameter update unit 23 updates the parameters of the question generation model 222 in accordance with the comparison result. Here, the parameter update unit 23 can use a general learning technique in machine learning.

[0035] Through the above processing, the machine learning device 20 performs learning such that, when analysis target text that does not include an answer sentence is input, the type is set to "question" and result text that is a question sentence is generated. Furthermore, when analysis target text that includes an answer sentence is input, the machine learning device 20 performs learning such that the type is set to "extraction" and result text corresponding to the answer sentence is generated. After that, the machine learning device 20 transmits the trained question generation model 222 to the information processing device 1 to use it as the question generation model 122.

[0036] In this way, the question generation unit 22 and the parameter update unit 23 are a machine learning execution unit that performs learning of a machine learning model based on the learning data generated by the learning data generation unit 21.

[0037] Here, the question generation units 12 and 22 may use a binary classification model for the category type instead of a character string. In this case, the binary classification model represents two values: whether to extract or not, and a method can be used in which, for example, 1 is assigned to extraction and 0 is assigned to classification. Furthermore, the question generation units 12 and 22 may employ a learning method in which the category type and the result text are concatenated to generate a single character string.

[0038] Furthermore, any suitable existing technology can be used to assign viewpoint labels to the training data. For example, viewpoint labels can be set manually, or the question generators 12 and 22 can automatically set viewpoint labels using keyword matching or a classification model.

[0039] Returning to FIG. 1 , the explanation continues. The question file creation unit 13 creates a file that describes the question generated by the question generation unit 12. Then, the question file creation unit 13 outputs the created file. For example, the question file creation unit 13 performs a process such as uploading the created file to a shared folder for uploading documents. Here, the format and contents of the file created by the question file creation unit 13 can be freely set. For example, the question file creation unit 13 may describe both a question statement and a viewpoint in the file that describes the question, or may describe only the question statement. Furthermore, the output of the file created by the question file creation unit 13 is not limited to uploading it to a folder. Alternatively, the question file creation unit 13 may send the created file by email to the person who uploaded the document, or may output the file contents to a chatbot or the like to notify the person in a chat format.

[0040] The question file creation unit 13 is an example of a “question output unit.” That is, the question file creation unit 13 outputs the generated sentences and categories generated by the question generation unit 12.

[0041] 6 is a diagram showing an example of question creation by an information processing device according to an embodiment. Here, a case will be described in which the file 200 shown in FIG. 6 is prepared. Here, sentences included in the prepared file 200 are stored sentence by sentence in the text data DB2 as entries 201 and 202. However, the storage method is not limited to this, and for example, all of the sentences in the file 200 may be stored in one entry.

[0042] The text data DB management unit 11 of the information processing device 1 monitors the text data DB2 and acquires the analysis target text from the entry 201 of a new file. For example, the text data DB management unit 11 acquires the following character string as the analysis target text: "Digital transformation is the concept that the penetration of IT will change people's lives in every aspect for the better. Digital shift has the same meaning." Then, the text data DB management unit 11 transfers the acquired analysis target text to the question generation unit 12.

[0043] When "money" is input as the viewpoint label, the question generation unit 12 outputs "question" as the type, and generates and outputs a question sentence 203 as the result text from the analysis target text. When "term" is input as the viewpoint label, the question generation unit 12 outputs "extraction" as the type, and generates and outputs extracted text 204 as the result text from the analysis target text.

[0044] The question file creation unit 13 generates a file by assigning a label of [money-question] to the question sentence 203 generated by the question generation unit 12. This indicates that the perspective label is money in the question sentence 203, and that a question has been generated. The question file creation unit 13 also generates a file by assigning a label of [term-extraction] to the extracted text 204 generated by the question generation unit 12. This indicates that the extracted text 204 uses a perspective related to the definition of a term, and that the relevant part exists in the text to be analyzed, and therefore the description in the text to be analyzed is used as is.

[0045] As described above, the question file creation unit 13 outputs the result text and type, which are the generated sentences generated by the question generation unit 12. The question file creation unit 13 also outputs the result text and type with added viewpoint information.

[0046] [Processing of the embodiment] 7 is a flowchart of the question generation process by the information processing device according to the embodiment. Next, the flow of the question generation process by the information processing device 1 according to the embodiment will be described with reference to FIG.

[0047] The text data DB management unit 11 monitors the text data DB2 (step S1).

[0048] Then, text data DB management unit 11 determines whether or not an entry for a new file has been added to text data DB2 (step S2). If an entry for a new file has not been added (step S2: No), text data DB management unit 11 returns to step S1 and waits until an entry for a new file is added.

[0049] On the other hand, if a new file entry has been added (step S2: Yes), text data DB management unit 11 acquires the text portion of the newly added entry as the analysis target text, and then transfers the acquired analysis target text to question generation unit 12 (step S3).

[0050] The question generation unit 12 acquires the text to be analyzed from the text data DB management unit 11. The question generation unit 12 also acquires viewpoint labels from an external device or the like (step S4).

[0051] The text processing unit 121 of the question generation unit 12 converts the analysis target text and viewpoint labels into vector format. Then, the text processing unit 121 inputs the vectorized analysis target text and viewpoint labels to the question generation model 122. The question generation model 122 receives the vectorized analysis target text and viewpoint labels as input and generates a category type and result text using the text generation layer 123 (step S5).

[0052] The question file creation unit 13 receives input of the category type and the result text from the question generation unit 12. Then, the question file creation unit 13 assigns a viewpoint label and a category type to the result text and creates a file in which the result text is registered. Thereafter, the question file creation unit 13 outputs the created file (step S6).

[0053] [Processing of the embodiment] 8 is a flowchart of the machine learning process performed by the machine learning device according to the embodiment. Next, the flow of the machine learning process performed by the machine learning device 20 according to the embodiment will be described with reference to FIG.

[0054] The learning data generation unit 21 acquires question-answer learning data including a learning text, a question sentence, and an answer sentence (step S11).

[0055] Next, the learning data generating unit 21 selects one unselected type from the type "question" and type "extract" (step S12).

[0056] Next, the learning data generating unit 21 determines whether the selected category type is a "question" (step S13).

[0057] If the selected category type is "question" (step S13: Yes), the learning data generation unit 21 deletes the answer portion from the learning text (step S14).

[0058] Next, the learning data generation unit 21 sets the question sentence as the correct answer generation sentence (step S15). After that, the learning data generation unit 21 proceeds to step S17.

[0059] On the other hand, if the selected type is "extraction" and not "question" (step S13: No), the learning data generation unit 21 sets the answer sentence as the correct answer generation sentence (step S16). After that, the learning data generation unit 21 proceeds to step S17.

[0060] Next, the learning data generation unit 21 generates learning data by assigning viewpoint labels to the learning data including the learning text, the type of correct answer, and the result text of the correct answer (step S17).

[0061] Next, the learning data generation unit 21 determines whether or not both category types have been selected for the current question-answer learning data (step S18). If an unselected category type remains (step S18: No), the learning data generation unit 21 returns to step S12.

[0062] On the other hand, if both types have been selected (step S18: Yes), the learning data generation unit 21 determines whether or not the generation of learning data has been completed for all question-answering learning data (step S19).If there is still question-answering learning data for which learning data generation has not been performed (step S19: No), the learning data generation unit 21 returns to step S11.

[0063] On the other hand, if the generation of learning data has been completed for all the question-answering learning data (step S19: Yes), the learning data generation unit 21 outputs the generated learning data to the question generation unit 22. The question generation unit 22 and the parameter update unit 23 use the learning data to perform learning of the question generation model 222 (step S20).

[0064] [Experimental Results] Next, an experiment on question generation by the information processing device 1 according to this embodiment will be described. Here, an evaluation corpus was created manually and using a viewpoint classifier, and model learning and evaluation were performed.

[0065] More specifically, the evaluation corpus was generated using the following procedure. First, the target text was obtained from a specific website, from which questions were generated. Next, questions were manually created for the obtained target text. Next, if the answer to the created question was present in the target text, it was labeled "extract," which indicates a type of answer. If the answer to the created question was not present in the target text, it was manually labeled "question," which indicates a type of answer. Next, if an answer was present, the answer was manually extracted from the target text as the "result text." Next, perspectives were assigned to the created questions. Here, four categories mechanically created using keyword-based classification for the experiment and five perspectives for one other category were used as perspectives for the experimental questions. The four perspectives were money, law, performance, and reason. Money was a category that included keywords such as how much, fee, cost, and reward. Law was a category that included keywords such as illegal, rule, contract, and audit. Performance was a category that included keywords such as evaluation, experiment, ability, comparison, and performance. The reason is a perspective that includes keywords such as why, cause, how, and causal relationship. However, this perspective assignment can also be done manually. As a result, the evaluation corpus used all or part of the text and question pairs created manually using the above procedure.

[0066] Additionally, the following two types of language generation models were used as the question generation model 122. One is Hobbyiest, a dialogue model developed by NTT Communication Science Laboratories (https: / / github.com / nttcslab / japanese-dialog-transformers / blob / main / README-jp.md). The other is Japanese T5, a model from the Huggingface model hub (https: / / huggingface.co / sonoisa / t5-base-japanese). Furthermore, the following model training corpora were used to train the question generation model 122. A machine reading comprehension corpus with 195,053 entries was used as question-answering training data. In addition, 9,755 questions created by the same people as the evaluation questions were used. Of these, the total number of model training corpora actually used was 204,394.

[0067] The following evaluation experiment was then conducted on the trained question generation model 122 using an evaluation corpus. First, data with "no answer," i.e., data with a category type of "question," was selected from the evaluation corpus. This data included manually created questions, analysis target text, and viewpoint labels. The viewpoint labels and analysis target text were then input into the question generation model 122. After that, it was verified whether questions were generated as the output of the question generation model 122 and whether the generated questions were close to questions generated manually.

[0068] Next, data with "answers included," i.e., data with a classification type of "extraction," is selected from the evaluation corpus. This data includes manually created questions, analysis target text, viewpoint labels, and answer sentences. The viewpoint labels and analysis target text are then input into the question generation model 122. After that, it was verified whether the same answer sentences as those in the evaluation corpus could be extracted as the output of the question generation model 122.

[0069] The evaluation criteria used were the question accuracy rate, average BERT score for questions, average question ranking, and perspective accuracy rate. Here, the evaluation was performed using the five highest-rated output results. The question accuracy rate, average BERT score for questions, and average question ranking are evaluation criteria when using the "no answer" data in the evaluation corpus. The perspective accuracy rate is also an evaluation criterion when using the "answered" data in the evaluation corpus.

[0070] The question correctness rate is the rate at which the result text contains correct questions. In other words, the question correctness rate is the rate at which the result text output by the question generation model 122 contains manually created questions in the evaluation corpus.

[0071] The average question BERT score is the semantic similarity between the manually created question and the result text. In other words, the question BERT score is the text similarity. For example, consider the BERT score for a question such as, "How many resources do you estimate are required to implement DTC?" In this case, the BERT score for a question such as, "How much investment is required to implement DTC?" is 0.775. The BERT score for a question such as, "Are there any legal issues with DTC?" is 0.721. The BERT score for a question such as, "Is NTT promoting IWON?" is 0.693.

[0072] The question mean reverse rank is the reciprocal of the rank of the first correct question among the five highest-rated output results. For example, if the correct question appears first, the question mean reverse rank is 1. If the correct question appears second, the question mean reverse rank is 1. If the correct question appears third, the question mean reverse rank is 0.33 (= 1 / 3). In other words, the question mean reverse rank is an evaluation criterion in which the higher the correct answer is, the closer the score is to 1, and the higher the evaluation.

[0073] The viewpoint accuracy rate is the rate at which text with the same viewpoint as the correct answer is output. As described above, the viewpoint accuracy rate is used in evaluations when using the “answered” data in the evaluation corpus.

[0074] FIG. 9 is a diagram showing experimental results using an information processing device according to an embodiment. FIG. 9 shows the question accuracy rate, question BERT score, question reverse ranking, and viewpoint accuracy rate for each of five viewpoint labels when Japanese T5 and Hobbyiest are used as the question generation model 122. In this case, Japanese T5 is superior whether the result text is a question sentence or an extracted text. Hobbyiest often outputs only a question or a viewpoint during N-best. This is likely due to the influence of the pre-training model. When using Japanese T5, questions with content relatively similar to questions created manually were generated.

[0075] [Effects of the embodiment] As described above, the information processing device 1 according to this embodiment acquires the analysis target text registered in the text data DB2 and generates a question together with a type. This makes it possible to determine whether the answer to the created question is included in the analysis target text. Using the result of this determination, it is possible to extract text that does not include an answer, and not generate a question if the answer is already included in the text, and to generate a question if the answer is not included in the text.

[0076] To more reliably resolve the problem of generating questions from any text when generating questions from text, the information processing device 1 according to this embodiment introduces a "perspective label" into the input. The information processing device 1 acquires the analysis target text registered in the text data DB2 and generates a question according to the perspective of the specified perspective label. The information processing device 1 also extracts and outputs statements corresponding to the perspective of the specified perspective label from the analysis target text. As a result, the information processing device 1 according to this embodiment generates a question related to the input perspective if the text does not contain a statement related to the input perspective. If the text already contains a statement related to the input perspective, the information processing device 1 extracts the relevant portion of the text to generate the main text. In other words, it is possible to more reliably generate questions whose answers are not contained in the text. Furthermore, by generating sentences based on perspectives, when a sentence with the "extract" type is generated, it is possible to know that the necessary information is present in the text and how it is written for that perspective. This can help users understand why a question is not generated. Furthermore, by using a specific person as a perspective, it is possible to create a collection of anticipated questions for that person.

[0077] The information processing device 1 and machine learning device 20 according to this embodiment provide specific improvements over conventional question generation techniques that, when generating questions from text, generate questions even if the answer is already included in the text, and represent an improvement in the technical field related to question generation techniques when it is unclear whether the answer is included in the text.

[0078] [System configuration, etc.] Furthermore, the components of each device shown in the figure are conceptual functional units and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of the devices can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic. Furthermore, each processing function performed by each device can be realized using a GPU (Graphics Processing Unit).

[0079] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0080] [program] In one embodiment, the information processing device 1 and the machine learning device 20 can be implemented by installing an information processing program that executes the above-described question generation process as package software or online software on a desired computer. For example, by executing the above-described information processing program on a computer, the computer can function as the information processing device 1 or the machine learning device 20. The term "computer" as used herein includes desktop and notebook personal computers. Other examples of computers also include mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handy-phone Systems), as well as slate terminals such as PDAs (Personal Digital Assistants). The information processing device 1 may be implemented as a web server or as a cloud that provides services related to the above-described management process through outsourcing.

[0081] 10 is a diagram showing an example of a computer that executes an information processing program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0082] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to an input unit 1200 such as a mouse 1110 or a keyboard 1120. The video adapter 1060 is connected to an output unit 1300 such as a display 1130.

[0083] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the information processing device 1 or the machine learning device 20, which has functions equivalent to those of the information processing device 1 or the machine learning device 20, is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing processes similar to those of the functional configuration of the information processing device 1 or the machine learning device 20 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0084] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes the processing of the above-described embodiment.

[0085] The program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090, but may also be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0086] (Additional note 1) Memory and at least one processor coupled to said memory; Including, The processor: using a pre-trained machine learning model with a target sentence as input, to generate a generated sentence that is a sentence related to the content of the target sentence and a type that is information indicating whether the generated sentence is a question about information whose answer is not included in the target sentence; Output the generated statement and the type. Information processing device.

[0087] (Additional note 2) A non-transitory storage medium storing a program executable by a computer to perform information processing, The information processing using a pre-trained machine learning model with a target sentence as input, to generate a generated sentence that is a sentence related to the content of the target sentence and a type that is information indicating whether the generated sentence is a question about information whose answer is not included in the target sentence; Output the generated statement and the type. Non-transitory storage medium.

[0088] (Additional note 3) Memory and at least one processor coupled to said memory; Including, The processor: a sentence to be analyzed, a question sentence related to the content of the sentence to be analyzed, and an answer sentence that is an answer to the question sentence and includes information included in the sentence to be analyzed; when a generated sentence having a type indicating that the generated sentence is a question sentence about information for which the answer is not included in the sentence to be analyzed is to be generated, the information that is the answer to the question sentence is removed from the sentence to be analyzed, and the question sentence is used as a correct answer generated sentence; when a generated sentence having the type indicating that the generated sentence is not a question sentence about information for which the answer is not included in the sentence to be analyzed is to be generated, the answer sentence is used as a correct answer generated sentence, and training data including the sentence to be analyzed, the type type, and the correct answer generated sentence is generated; The machine learning model is trained based on the generated training data. Information processing device. (Here, an answer sentence that is an answer to a question sentence and includes information that is included in the sentence to be analyzed is an answer sentence that is an answer sentence to a question sentence and includes information that is included in the sentence to be analyzed. Furthermore, when a generated sentence having a type indicating that it is a question about information whose answer is not included in the sentence to be analyzed is to be generated, it means that the generated sentence generated by the information processing device has a type indicating that it is a question about information that is not included in the sentence to be analyzed. Similarly, when a generated sentence having a type indicating that it is not a question about information whose answer is not included in the sentence to be analyzed is to be generated, it means that the generated sentence generated by the information processing device has a type indicating that it is not a question about information that is not included in the sentence to be analyzed.)

[0089] (Additional note 4) A non-transitory storage medium storing a program executable by a computer to perform information processing, The information processing a sentence to be analyzed, a question sentence related to the content of the sentence to be analyzed, and an answer sentence that is an answer to the question sentence and includes information included in the sentence to be analyzed; when a generated sentence having a type indicating that the generated sentence is a question sentence about information for which the answer is not included in the sentence to be analyzed is to be generated, the information that is the answer to the question sentence is removed from the sentence to be analyzed, and the question sentence is used as a correct answer generated sentence; when a generated sentence having the type indicating that the generated sentence is not a question sentence about information for which the answer is not included in the sentence to be analyzed is to be generated, the answer sentence is used as a correct answer generated sentence, and training data including the sentence to be analyzed, the type type, and the correct answer generated sentence is generated; The machine learning model is trained based on the generated training data. Non-transitory storage medium.

[0090] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0091] 1. Information processing equipment 2. Text Data DB 11 Text Data DB Management Department 12 Question generator 13 Question File Creation Department 20 Machine Learning Device 21 Learning data generation unit 22 Question generation part 23 Parameter update section 121 Text Processing Unit 122 Question Generation Model 123 Text Generation Layer 221 Text Processing Unit 222 Question Generation Model 223 Text Generation Layer

Claims

1. a question generation unit that receives an analysis target sentence as input and uses a pre-trained machine learning model to generate a generated sentence that is a question regarding the content of the analysis target sentence and a type of the generated sentence that is information indicating whether the generated sentence is a question regarding information whose answer is not included in the analysis target sentence; a question output unit that outputs the generated sentence and the type generated by the question generation unit; An information processing device comprising:

2. The information processing device according to claim 1, characterized in that the question generation unit acquires a perspective, which is information that represents a tendency of the question content of the generated sentence, and generates the generated sentence and the classification type using the sentence to be analyzed and the perspective as input.

3. The information processing apparatus according to claim 2 , wherein the question output unit outputs the generated sentence and the category type to which the viewpoint information is added.

4. An information processing method executed by an information processing device, comprising: using a pre-trained machine learning model with an analysis target sentence as input, to generate a generated sentence that is a question regarding the content of the analysis target sentence and a type that is information indicating whether the generated sentence is a question regarding information whose answer is not included in the analysis target sentence; Output the generated statement and the type.

1. An information processing method comprising:

5. using a pre-trained machine learning model with an analysis target sentence as input, to generate a generated sentence that is a question regarding the content of the analysis target sentence and a type that is information indicating whether the generated sentence is a question regarding information whose answer is not included in the analysis target sentence; Output the generated statement and the type. An information processing program that causes a computer to execute a process.

6. a learning data generation unit that acquires an analysis target sentence, a question sentence related to the content of the analysis target sentence, and an answer sentence that is an answer to the question sentence and includes information included in the analysis target sentence, determines whether or not to generate a generated sentence having a type that indicates that the generated sentence is a question sentence about information whose answer is not included in the analysis target sentence, removes information corresponding to the answer sentence from the analysis target sentence when the generated sentence has a type that indicates that the generated sentence is a question sentence about information whose answer is not included in the analysis target sentence, and generates learning data including the analysis target sentence, the type type, and the correct answer generated sentence, with the question sentence as the correct answer generated sentence when the generated sentence has the type that indicates that the generated sentence is not a question sentence about information whose answer is not included in the analysis target sentence; a machine learning execution unit that learns a machine learning model based on the learning data generated by the learning data generation unit; An information processing device comprising:

7. An information processing method executed by an information processing device, comprising: obtain an analysis target sentence, a question sentence related to the content of the analysis target sentence, and an answer sentence that is an answer to the question sentence and includes information included in the analysis target sentence, determine whether or not to generate a generated sentence having a type indicating that the generated sentence is a question sentence about information whose answer is not included in the analysis target sentence, remove information corresponding to the answer sentence from the analysis target sentence when the generated sentence has a type indicating that the generated sentence is a question sentence about information whose answer is not included in the analysis target sentence, and generate learning data including the analysis target sentence, the type type, and the correct answer generated sentence, with the question sentence as the correct answer generated sentence, The machine learning model is trained based on the generated training data.

1. An information processing method comprising:

8. obtain an analysis target sentence, a question sentence related to the content of the analysis target sentence, and an answer sentence that is an answer to the question sentence and includes information included in the analysis target sentence, determine whether or not to generate a generated sentence having a type indicating that the generated sentence is a question sentence about information whose answer is not included in the analysis target sentence, remove information corresponding to the answer sentence from the analysis target sentence when the generated sentence has a type indicating that the generated sentence is a question sentence about information whose answer is not included in the analysis target sentence, and generate learning data including the analysis target sentence, the type type, and the correct answer generated sentence, with the question sentence as the correct answer generated sentence, The machine learning model is trained based on the generated training data. An information processing program that causes a computer to execute a process.

Images