Information processing device, information processing method, and program

Abstract

In order to provide a question / correct answer pair appropriate for learning of a question answering system, an information processing device (1) is provided with: an acquisition unit (11) that acquires a target pair, which is a question / correct answer pair to be determined; and a determination unit (12) that determines the appropriateness of the target pair acquired by the acquisition unit (11) by referring to information indicating the appropriateness of questions and / or correct answers included in one or more similar pairs similar to the target pair.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] A technique has been disclosed for automatically generating pairs of questions serving as training data and correct answers to the questions in order to train a question-answering system that outputs correct answers to input questions.

[0003] Patent Document 1 discloses a technology related to active learning that selects pairs with high learning effectiveness from a plurality of pairs of unlabeled questions and correct answers. The technology in Patent Document 1 discloses a configuration in which a plurality of unlabeled pairs are classified in a feature space, and characteristic pairs are selected from the classified pairs using the value of a loss function for each question. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] US Patent Application Publication No. 2016 / 0253596 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology of Patent Document 1 uses the loss function value for each question to select characteristic pairs. Generally, calculation of the loss function value involves complex processes such as converting the pair into feature quantities and processing using a model trained by machine learning. Therefore, depending on the input pair, the calculated loss function value may be a result unintended by the designer. In other words, the technology of Patent Document 1 has a problem in that if the calculation result of the loss function value is unintended, the result is used to select characteristic pairs, which may result in the selection of pairs with low learning effectiveness.

[0006] One aspect of the present invention has been made in view of the above-mentioned problems, and one example of its purpose is to provide pairs of questions and correct answers that are appropriate for training a question-answering system. [Means for solving the problem]

[0007] An information processing device according to one aspect of the present invention includes an acquisition means for acquiring a target pair, which is a pair of a question to be judged and a correct answer, and a judgment means for judging the appropriateness of the target pair by referring to information indicating the appropriateness of at least one of the question and the correct answer contained in one or more similar pairs that are similar to the target pair acquired by the acquisition means.

[0008] An information processing method according to one aspect of the present invention includes an information processing device acquiring a target pair, which is a pair of a question to be judged and a correct answer, and judging the appropriateness of the target pair by referring to information indicating the appropriateness of at least one of the question and the correct answer contained in one or more similar pairs that are similar to the target pair acquired in the acquisition.

[0009] A program according to one aspect of the present invention is a program that causes a computer to function as an information processing device, and the program causes the computer to function as an acquisition means that acquires a target pair, which is a pair of a question and a correct answer to be judged, and a judgment means that judges the appropriateness of the target pair by referring to information indicating the appropriateness of at least one of the question and the correct answer contained in one or more similar pairs that are similar to the target pair acquired by the acquisition means. [Effects of the Invention]

[0010] According to one aspect of the present invention, it is possible to provide pairs of questions and correct answers that are suitable for training a question-answering system. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a configuration of an information processing device according to a first exemplary embodiment of the present invention. [Figure 2] FIG. 2 is a conceptual diagram showing an example of a target pair and a similar pair in the first exemplary embodiment of the present invention. [Figure 3] 1 is a flowchart showing the flow of an information processing method according to a first exemplary embodiment of the present invention. [Figure 4] FIG. 10 is a block diagram showing the configuration of an information processing device according to a second exemplary embodiment of the present invention. [Figure 5] FIG. 10 is a diagram illustrating an example of processing executed by an information processing device according to a second exemplary embodiment of the present invention. [Figure 6] FIG. 10 is a block diagram showing the configuration of an information processing device according to a third exemplary embodiment of the present invention. [Figure 7] FIG. 10 is a diagram showing an example of processing executed by an information processing device according to the present exemplary embodiment 3 of the present invention. [Figure 8] FIG. 10 is a block diagram showing the configuration of an information processing device according to a fourth exemplary embodiment of the present invention. [Figure 9] FIG. 10 is a diagram showing an example of display content displayed by an information processing device according to a fourth exemplary embodiment of the present invention. [Figure 10] FIG. 2 is a block diagram showing an example of a hardware configuration of an information processing device according to each exemplary embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0012] Exemplary Embodiment 1 A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described below.

[0013] (Overview of information processing device 1) The information processing device 1 according to this exemplary embodiment is a device that determines the appropriateness of pairs of questions and correct answers used in training a question answering system.

[0014] A question-answering system is a system that outputs a correct answer to an input question. One example of a question-answering system is a learning model that has been trained using training data that pairs questions with correct answers.

[0015] Although the specific configuration of the question answering system is not limited, for example, a convolution neural network (CNN), a recurrent neural network (RNN), a transformer, or a combination thereof can be used. Also, a non-neural network model such as a random forest or a support vector machine can be used.

[0016] An example of the appropriateness of a question and correct answer pair used in training a question-answering system is the usefulness of the question and correct answer pair in training the question-answering system. Here, whether the question and correct answer pair is useful or not may differ depending on the question-answering system, even for the same question and correct answer pair. As an example, if a question and correct answer pair is included in the trained teacher data of a question-answering system, the question and correct answer pair is not useful for the question-answering system. On the other hand, if the question and correct answer pair is not included in the trained teacher data of a question-answering system different from the given question-answering system, the question and correct answer pair is useful for the other question-answering system.

[0017] Another example of the appropriateness of question and correct answer pairs used to train a question-answering system is the subjective appropriateness of the correct answer to a question (whether it is appropriate or not as viewed by an expert in the field), such as the question and correct answer pairs used in FAQs (Frequently Asked Questions).

[0018] On the other hand, examples of inappropriate question and answer pairs used in training a question-answering system include questions where the content of the question cannot be interpreted, the question is not grammatically formed, and the correct answer to the question is incorrect.

[0019] (Configuration of information processing device 1) The configuration of an information processing device 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of an information processing device 1 according to this exemplary embodiment.

[0020] 1, the information processing device 1 includes an acquisition unit 11 and a determination unit 12. In this exemplary embodiment, the acquisition unit 11 and the determination unit 12 are components that respectively realize an acquisition means and a determination means.

[0021] The acquisition unit 11 acquires a target pair, which is a pair of a question to be judged and a correct answer, and supplies the acquired target pair to the judgment unit 12.

[0022] The judgment unit 12 judges the appropriateness of the target pair by referring to information indicating the appropriateness of at least one of the question and the correct answer contained in one or more similar pairs similar to the target pair acquired by the acquisition unit 11.

[0023] Examples of one or more similar pairs similar to the target pair include a pair that includes a question similar to the question of the target pair, a pair that includes a correct answer similar to the correct answer of the target pair, or a pair that includes a question and a correct answer similar to the question and correct answer of the target pair.

[0024] As one example, the determination unit 12 determines the appropriateness of the target pair by referring to information indicating the appropriateness of a question of a similar pair that includes a question similar to the question of the target pair. As another example, the determination unit 12 determines the appropriateness of the target pair by referring to information indicating the appropriateness of a correct answer of a similar pair that includes a correct answer similar to the correct answer of the target pair. As yet another example, the determination unit 12 determines the appropriateness of the target pair by referring to information indicating the appropriateness of a question and a correct answer of a similar pair that includes a question and a correct answer similar to the question and correct answer of the target pair.

[0025] An example of one or more similar pairs similar to the target pair will be described with reference to Fig. 2. Fig. 2 is a conceptual diagram showing an example of the target pair and similar pairs in this exemplary embodiment.

[0026] 2, the determination unit 12 locates a target pair TP of a question Q and a correct answer A in a feature space in which pairs SP1 to SP6 of questions q1 to q6 and correct answers a1 to a6 are located. As an example, the determination unit 12 determines the distance between each pair SP1 to SP6 in the feature space and the target pair TP as the similarity, and identifies pairs whose distance from the target pair TP is closer than a predetermined distance as similar pairs. In the diagram shown in FIG. 2, the determination unit 12 identifies similar pairs SP1, SP2, and SP3 that are included in a similar region SR that is a predetermined distance from the target pair TP.

[0027] 2, the information indicating the appropriateness of at least one of the questions q1 to q3 and the correct answers a1 to a3 included in each of similar pairs SP1, SP2, and SP3 is a score s1, a score s2, and a score s3. As an example, the determination unit 12 determines the appropriateness of the target pair TP by referring to the score s1, the score s2, and the score s3.

[0028] As described above, the information processing device 1 according to this exemplary embodiment is configured to include an acquisition unit 11 that acquires a target pair, which is a pair of a question to be judged and a correct answer, and a judgment unit 12 that judges the appropriateness of the target pair by referring to information indicating the appropriateness of at least one of the question and the correct answer contained in one or more similar pairs that are similar to the target pair acquired by the acquisition unit 11.

[0029] Therefore, according to the information processing device 1 of this exemplary embodiment, the appropriateness of the target pair is determined by referring to information indicating the appropriateness of at least one of the question and correct answer contained in a similar pair that is similar to the target pair, so the possibility of determining an inappropriate target pair as appropriate is lower than when the appropriateness is determined by referring only to the target pair.

[0030] As a more specific example, when determining the appropriateness of a target pair using the loss function value for each question, the loss function value is calculated through complex processing, which may result in an unintended value being calculated. In this configuration, there is a high possibility that an inappropriate target pair will be determined to be appropriate.

[0031] On the other hand, according to the information processing device 1 of this exemplary embodiment, the appropriateness of the target pair is determined by referring to information indicating the appropriateness of not only the target pair but also at least one of the question and correct answer included in a similar pair similar to the target pair, so that it is possible to provide pairs of questions and correct answers that are appropriate for learning the question answering system.

[0032] (Flow of information processing method S1) The flow of the information processing method S1 according to this exemplary embodiment will be described with reference to Fig. 3. Fig. 3 is a flow diagram showing the flow of the information processing method S1 according to this exemplary embodiment.

[0033] (Step S11) In step S11, the acquisition unit 11 acquires a target pair, which is a pair of a question to be judged and a correct answer. The acquisition unit 11 supplies the acquired target pair to the judgment unit 12.

[0034] (Step S12) In step S12, the judgment unit 12 judges the appropriateness of the target pair by referring to information indicating the appropriateness of at least one of the question and the correct answer contained in one or more similar pairs that are similar to the target pair acquired by the acquisition unit 11 in step S11.

[0035] As described above, the information processing method S1 according to this exemplary embodiment employs a configuration in which, in step S11, the acquisition unit 11 acquires a target pair, which is a pair of a question and a correct answer to be judged, and in step S12, the judgment unit 12 judges the appropriateness of the target pair by referring to information indicating the appropriateness of at least one of the question and the correct answer included in one or more similar pairs similar to the target pair acquired in step S11. Therefore, the information processing method S1 according to this exemplary embodiment can achieve the same effects as the information processing device 1 described above.

[0036] Exemplary Embodiment 2 A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are given the same reference numerals, and their description will be omitted as appropriate.

[0037] (Configuration of information processing device 1A) The configuration of the information processing device 1A according to this exemplary embodiment will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the configuration of the information processing device 1A according to this exemplary embodiment.

[0038] As shown in FIG. 4, the information processing device 1A includes a control unit 10A, a database 20A, an input / output unit 30A, and a communication unit 40A.

[0039] The database 20A stores data referenced by the control unit 10A. As an example, as shown in Fig. 4, the database 20A stores pairs P of a question and a correct answer and scores S, which are an example of information indicating the appropriateness of the pairs P, in association with each other. As another example, as shown in Fig. 4, the database 20A stores determined pairs DP, the appropriateness of which has been determined by the control unit 10A.

[0040] Here, the score S, which is an example of information indicating the appropriateness of the question and correct answer pair P, may be expressed as a real number or a categorical value taking a discrete value. The process by which the information processing device 1A acquires the score S will be described later.

[0041] The input / output unit 30A is an interface for acquiring or outputting data. The input / output unit 30A supplies acquired data to the control unit 10A and outputs data supplied from the control unit 10A. Examples of the input / output unit 30A include a touchpad and a keyboard for acquiring data, and a display device for outputting data.

[0042] The communication unit 40A is a communication module that communicates with other devices via a network (not shown). For example, the communication unit 40A outputs data supplied from the control unit 10A to other devices via the network, and acquires data output from other devices via the network and supplies the data to the control unit 10A.

[0043] The specific configuration of the network through which the communication unit 40A communicates with other devices does not limit this embodiment, but as an example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these networks can be used.

[0044] (Function of control unit 10A) The control unit 10A controls each component of the information processing device 1A. As shown in Fig. 4, the control unit 10A also functions as an acquisition unit 11 and a determination unit 12. In this exemplary embodiment, the acquisition unit 11 and the determination unit 12 are configured to respectively realize an acquisition means and a determination means.

[0045] The acquisition unit 11 acquires a target pair, which is a pair of a question to be judged and a correct answer, and supplies the acquired target pair to the judgment unit 12.

[0046] The determination unit 12 determines the appropriateness of the target pair by referring to information indicating the appropriateness of at least one of the question and the correct answer included in one or more similar pairs similar to the target pair acquired by the acquisition unit 11. The determination unit 12 stores the determined pair DP in the database 20A.

[0047] Here, an example of information indicating the appropriateness of at least one of the question and correct answer included in a similar pair is information indicating the usefulness of the similar pair when used for training a question-answering system. In this case, the determination unit 12 determines the usefulness of the target pair when used for training a question-answering system as the appropriateness of the target pair. With this configuration, the determination unit 12 can provide pairs of questions and correct answers that are useful for training a question-answering system.

[0048] Another example of the usefulness of using similar pairs in training a question-answering system is whether using the similar pairs in addition to training the question-answering system makes it easier to obtain a question-answering system with higher performance.

[0049] Here, the information indicating the appropriateness of the similar pair may be obtained from a user or calculated by a predetermined algorithm. Also, the process described below as an example of the process by which the information processing device 1A obtains the score S may be used.

[0050] Another example of information indicating the appropriateness of at least one of the question and correct answer included in a similar pair is information indicating the subjective appropriateness of the correct answer to the question (whether it is appropriate in the opinion of an expert in the field).

[0051] 4, the determination unit 12 also functions as a search unit 121 and an evaluation unit 122. In this exemplary embodiment, the search unit 121 and the evaluation unit 122 are configured to respectively realize a search means and an evaluation means.

[0052] The search unit 121 refers to a database 20A that stores a plurality of pairs P of a question and a correct answer and a score S as information indicating the appropriateness of each of the plurality of pairs P, and searches for one or more similar pairs whose similarity to the target pair is equal to or greater than a predetermined level. An example of the process by which the search unit 121 searches for similar pairs will be described later.

[0053] The evaluation unit 122 evaluates the appropriateness of the target pair from the score S, which is information indicating the appropriateness of each similar pair searched by the search unit 121. An example of the process by which the evaluation unit 122 evaluates the appropriateness of the target pair will be described later.

[0054] (An example of a process in which the information processing device 1A acquires the score S of the pair P) An example of a process in which the information processing device 1A acquires the score S of the pair P will be described below.

[0055] As an example of a process in which the information processing device 1A acquires the score S of the pair P, the information processing device 1A may be configured to acquire the score S from the user via the input / output unit 30A. For example, the information processing device 1A may output an image via the input / output unit 30A that prompts the user to input the score S of the pair P, and acquire the user's input on the image.

[0056] As another example of the process in which the information processing device 1A acquires the score S of the pair P, the information processing device 1A may be configured to use a rule-based program. One example of the rule-based program is a rule-based program that calculates the score S depending on whether a predetermined word is included in at least one of the question and the correct answer. For example, the information processing device 1A may be configured to input at least one of the question and the correct answer of the pair P into the rule-based program and acquire the calculated score S.

[0057] As yet another example of the process in which information processing device 1A acquires the score S of pair P, information processing device 1A may be configured to use an existing question-answering system. For example, information processing device 1A may be configured to input a question of pair P to an existing question-answering system and acquire, as score S, the degree to which a correct answer output from the question-answering system matches the correct answer of pair P. In this configuration, information processing device 1A may further acquire score S according to the degree of certainty of the correct answer output from the question-answering system. For example, information processing device 1A may be configured to acquire a higher score S the higher the degree of certainty of the correct answer output from the question-answering system.

[0058] As yet another example of the process in which the information processing device 1A acquires the score S of the pair P, the information processing device 1A may be configured to use a prediction model that receives the pair P as input and outputs the score S of the pair P. In this configuration, the information processing device 1A may be configured to input the pair P to the prediction model and acquire the score S output from the prediction model.

[0059] (An example of a process in which the search unit 121 searches for similar pairs) An example of the process performed by the search unit 121 to search for similar pairs will be described below.

[0060] For example, the search unit 121 may refer to the similarity between the plurality of pairs and the target pair in order to search for one or more similar pairs from the plurality of pairs. For example, the search unit 121 may be configured to function as a similarity calculation means for calculating the similarity, or may be configured to refer to the similarity calculated by another device.

[0061] In the case where the search unit 121 also functions as a similarity calculation means, an example of a process in which the search unit 121 calculates the similarity between the pair P and the target pair will be described below.

[0062] An example of the process in which the search unit 121 calculates the similarity between pair P and the target pair is a configuration in which the number of matching character strings between character strings included in the question or correct answer of pair P and character strings included in the question or correct answer of the target pair is calculated as the similarity between the target pair and pair P. In this configuration, the search unit 121 identifies pair P having a similarity equal to or greater than a predetermined level (for example, five or more matching character strings) as a similar pair. In other words, in this configuration, the search unit 121 searches for similar pairs including questions or correct answers that have many matching character strings with character strings included in the question or correct answer of the target pair.

[0063] Another example of the process in which the search unit 121 calculates the similarity between the pair P and the target pair is a configuration using a similarity prediction model that receives a plurality of question and correct answer pairs as input and outputs the similarity between the plurality of question and correct answer pairs. Specifically, the search unit 121 may be configured to input the target pair and the pair P into a similarity prediction model and obtain the output similarity. A known learning model can be used as the similarity prediction model used by the search unit 121, and one example is a known natural language processing model.

[0064] When the search unit 121 uses a similarity prediction model, the search unit 121 may also be configured to function as a similarity prediction model learning means that learns the similarity prediction model using training data that includes multiple sets of two pairs and the similarities between the two pairs.

[0065] As yet another example of the process in which the search unit 121 calculates the similarity between the pair P and the target pair, as described with reference to Fig. 2, the search unit 121 may calculate the distance between the target pair and the pair P in the feature space as the similarity. In this configuration, the search unit 121 identifies the pair P whose similarity is equal to or greater than a predetermined degree (for example, the distance between the target pair and the pair P in the feature space is equal to or less than a predetermined distance) as a similar pair.

[0066] (An example of an evaluation process in which the evaluation unit 122 evaluates the appropriateness of the target pair) An example of an evaluation process in which the evaluation unit 122 evaluates the appropriateness of the target pair will be described below.

[0067] As an example of the evaluation process performed by the evaluation unit 122, if the score S, which is information indicating the appropriateness of each of the target pair and the similar pair, is equal to or greater than a predetermined value, the evaluation unit 122 may evaluate the target pair as appropriate.

[0068] Another example of the evaluation process performed by the evaluation unit 122 may include a process of calculating the average or weighted average of the scores S as information indicating the appropriateness of each of the multiple similar pairs searched by the search unit 121.

[0069] For example, the evaluation unit 122 calculates a simple average of the scores S of the multiple similar pairs. If the calculated simple average is equal to or greater than a predetermined value, the evaluation unit 122 may evaluate the target pair as appropriate.

[0070] Furthermore, the evaluation unit 122 may refer to the similarity referred to by the search unit 121 and calculate a weighted average of the score S using a weighting coefficient corresponding to the similarity of multiple similar pairs. As an example, the evaluation unit 122 may set a larger weighting coefficient for a similar pair that is relatively more similar than other similar pairs, and a smaller weighting coefficient for a similar pair that is relatively less similar than other similar pairs, and calculate a weighted average of multiple similar pairs. If the calculated weighted average is equal to or greater than a predetermined value, the evaluation unit 122 may evaluate the target pair as appropriate.

[0071] In this way, the evaluation unit 122 can appropriately evaluate the appropriateness of the target pair by taking the average or weighted average of the scores S as information indicating the appropriateness of each of the multiple similar pairs searched by the search unit 121.

[0072] (An example of processing executed by information processing device 1A) An example of processing executed by the information processing device 1A will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of processing executed by the information processing device 1A according to this exemplary embodiment.

[0073] First, the acquisition unit 11 acquires a target pair, which is a pair of a question to be judged and a correct answer. As an example, the acquisition unit 11 acquires a target pair of a question "What is the capital of the United States?" and a correct answer "Washington, DC," as shown in FIG. 5. The acquisition unit 11 supplies the acquired target pair to the judgment unit 12.

[0074] The search unit 121 of the determination unit 12 refers to the database 20A to search for one or more similar pairs whose similarity to the target pair supplied from the acquisition unit 11 is equal to or greater than a predetermined level. Then, the search unit 121 supplies the searched one or more similar pairs to the evaluation unit 122. As an example, as shown in FIG. 5 , the search unit 121 refers to the database 20A to search for pair 1 and pair 2, which are similar pairs. The process by which the search unit 121 searches for similar pairs is as described above. Then, the search unit 121 supplies the searched pair 1 and pair 2 to the evaluation unit 122.

[0075] The evaluation unit 122 evaluates the appropriateness of the target pair from the information indicating the appropriateness of each of the similar pairs supplied from the search unit 121. As an example, as shown in FIG. 5 , the evaluation unit 122 obtains appropriateness scores of "0" and "0.1", which are information indicating the appropriateness of each of the similar pairs, Pair 1 and Pair 2, supplied from the search unit 121. Then, the evaluation unit 122 calculates an appropriateness score of "0.05", which is information indicating the appropriateness of the target pair, and evaluates the appropriateness of the target pair. The process of obtaining the score and the evaluation process of evaluating the appropriateness by the evaluation unit 122 are as described above.

[0076] (Effects of information processing device 1A) As described above, the information processing device 1A according to this exemplary embodiment is configured to include a search unit 121 that refers to a database 20A storing multiple pairs P of questions and correct answers and information indicating the appropriateness of each of the multiple pairs P, and searches for one or more similar pairs whose similarity to the target pair is equal to or greater than a predetermined level, and an evaluation unit 122 that evaluates the appropriateness of the target pair based on the information indicating the appropriateness of each of the similar pairs searched by the search unit 121.

[0077] Therefore, according to the information processing device 1A of this exemplary embodiment, similar pairs similar to the target pair are searched for in the database 20A, and the appropriateness of the target pair is evaluated based on information indicating the appropriateness of the similar pairs. This makes it possible to provide pairs of questions and correct answers that are more appropriate for learning the question-answering system than when only referring to information indicating the appropriateness of the target pair.

[0078] Exemplary Embodiment 3 A third exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the above exemplary embodiment are denoted by the same reference numerals, and their description will not be repeated.

[0079] (Configuration of information processing device 1B) The configuration of an information processing device 1B according to this exemplary embodiment will be described with reference to Fig. 6. Fig. 6 is a block diagram showing the configuration of an information processing device 1B according to this exemplary embodiment.

[0080] 6, the information processing device 1B includes a control unit 10B, a database 20A, an input / output unit 30A, and a communication unit 40A. The database 20A, the input / output unit 30A, and the communication unit 40A are as described above.

[0081] (Function of control unit 10B) The control unit 10B controls each component of the information processing device 1B. As shown in Fig. 6, the control unit 10B also functions as an acquisition unit 11 and a determination unit 12. In this exemplary embodiment, the acquisition unit 11 and the determination unit 12 are configured to respectively realize an acquisition means and a determination means.

[0082] The acquisition unit 11 acquires a target pair, which is a pair of a question to be judged and a correct answer, and supplies the acquired target pair to the judgment unit 12.

[0083] The determination unit 12 determines the appropriateness of the target pair by referring to information indicating the appropriateness of at least one of the question and the correct answer included in one or more similar pairs similar to the target pair acquired by the acquisition unit 11. The determination unit 12 stores the determined pair DP in the database 20A.

[0084] 6, the determination unit 12 also functions as a prediction unit 125 and an update unit 126. In this exemplary embodiment, the prediction unit 125 and the update unit 126 are configured to respectively realize a prediction means and an update means.

[0085] The prediction unit 125 receives a question-answer pair as input, and uses a prediction model that outputs a score as information indicating the appropriateness of the pair to predict the appropriateness of the target pair supplied from the acquisition unit 11. A known learning model can be used as the prediction model used by the prediction unit 125.

[0086] The specific configuration of the prediction model is not particularly limited, but examples include CNN, RNN, Transformer, or a combination thereof. Non-neural network models such as random forests and support vector machines may also be used.

[0087] The update unit 126 updates the parameters of the prediction model by referring to a database 20A that stores a plurality of pairs P of a question and a correct answer, and a score S as information indicating the appropriateness of each of the plurality of pairs P. In other words, the update unit 126 updates the parameters of the prediction model so that when the pair P is input to the prediction model, the score S is output from the prediction model.

[0088] (An example of processing executed by information processing device 1B) An example of processing executed by the information processing device 1B will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of processing executed by the information processing device 1B according to this exemplary embodiment.

[0089] First, the acquisition unit 11 acquires a target pair, which is a pair of a question to be judged and a correct answer. As an example, the acquisition unit 11 acquires a target pair of the question "What is the capital of the United States?" and the correct answer "Washington, DC," as shown in FIG. 7. The acquisition unit 11 supplies the acquired target pair to the judgment unit 12.

[0090] The prediction unit 125 of the determination unit 12 inputs the target pair supplied from the acquisition unit 11 into a prediction model and obtains a score as information indicating the appropriateness of the target pair. As an example, the prediction unit 125 inputs the target pair of the question "What is the capital of the United States?" and the correct answer "Washington DC" into the prediction model and obtains a score of "0.05."

[0091] The process in which the determining unit 12 determines the appropriateness of the target pair by referring to the database 20A is as described above.

[0092] Furthermore, the update unit 126 of the determination unit 12 updates the parameters of the prediction model by referring to a database 20A that stores a plurality of pairs of questions and correct answers and scores as information indicating the appropriateness of each of the plurality of pairs. As an example, the update unit 126 updates the parameters of the prediction model by referring to a database 20A that stores pairs 1 to 10 and appropriateness scores indicating the appropriateness of each of pairs 1 to 10.

[0093] (Effects of information processing device 1B) As described above, the information processing device 1B according to this exemplary embodiment is configured to include a prediction unit 125 that predicts the appropriateness of a target pair using a prediction model that takes a question and correct answer pair as input and outputs a score as information indicating the appropriateness of the pair, and an update unit 126 that updates the parameters of the prediction model by referring to a database 20A that stores multiple pairs P of questions and correct answers and a score S as information indicating the appropriateness of each of the multiple pairs P.

[0094] Therefore, in addition to the effects achieved by the information processing device 1 and information processing device 1A according to the exemplary embodiments described above, the information processing device 1B according to this exemplary embodiment has the effect of being able to appropriately derive information indicating the appropriateness of a pair of a question and a correct answer.

[0095] Exemplary Embodiment 4 A fourth exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the above embodiments are denoted by the same reference numerals, and their description will not be repeated.

[0096] (Configuration of information processing system 100) The configuration of the information processing system 100 according to this exemplary embodiment will be described with reference to Fig. 8. Fig. 8 is a block diagram showing the configuration of the information processing system 100 according to this exemplary embodiment.

[0097] 8, an information processing system 100 according to this exemplary embodiment includes an information processing device 1C and a question answering system 50. The information processing device 1C and the question answering system 50 are connected to each other so that they can communicate with each other.

[0098] In the information processing system 100, an information processing device 1C is configured to train a question answering system 50 that outputs a correct answer to an input question. The information processing device 1C also displays a target pair and information indicating the determined appropriateness of the target pair. An example of the display content displayed by the information processing device 1C will be described later.

[0099] (Question Answering System 50) The question answering system 50 is a system that outputs a correct answer to an input question. The specific configuration of the question answering system 50 is not particularly limited, but as an example, a CNN, an RNN, a Transformer, or a combination thereof can be used. Also, a non-neural network model such as a random forest or a support vector machine may be used.

[0100] (Configuration of information processing device 1C) 8, the information processing device 1C includes a control unit 10C, a database 20A, an input / output unit 30A, and a communication unit 40A. The database 20A, the input / output unit 30A, and the communication unit 40A are as described above. In this exemplary embodiment, the input / output unit 30A is configured to realize a display means.

[0101] (Function of control unit 10C) The control unit 10C controls the components of the information processing device 1C. As shown in Fig. 8, the control unit 10C also functions as an acquisition unit 11, a judgment unit 12, a teacher data generation unit 13, and a learning unit 14. In this embodiment, the acquisition unit 11, the judgment unit 12, the teacher data generation unit 13, and the learning unit 14 are configured to realize an acquisition means, a judgment means, a teacher data generation means, and a learning means, respectively.

[0102] The acquisition unit 11 acquires a target pair, which is a pair of a question to be judged and a correct answer, and supplies the acquired target pair to the judgment unit 12.

[0103] The determination unit 12 determines the appropriateness of the target pair by referring to information indicating the appropriateness of at least one of the question and the correct answer included in one or more similar pairs similar to the target pair acquired by the acquisition unit 11. The determination unit 12 stores the determined pair DP in the database 20A. The determination unit 12 also supplies the target pair and information indicating the determined appropriateness of the target pair to the input / output unit 30A.

[0104] Here, the determination unit 12 may be configured to function as the search unit 121 and evaluation unit 122 in the above-described embodiment, or may be configured to function as the prediction unit 125 and update unit 126.

[0105] The teacher data generation unit 13 generates teacher data including target pairs determined to be appropriate by the determination unit 12. As an example, the teacher data generation unit 13 may generate teacher data including, in addition to the target pairs determined to be appropriate by the determination unit 12, pairs P of question and correct answer stored in the database 20A, whose associated score S is equal to or greater than a predetermined value. As another example, the teacher data generation unit 13 may generate teacher data including, in addition to the target pairs determined to be appropriate by the determination unit 12, determined pairs DP determined to be appropriate, among determined pairs DP stored in the database 20A.

[0106] The learning unit 14 trains the question answering system 50 using the teacher data generated by the teacher data generation unit 13. As an example, the learning unit 14 trains the question answering system 50 by outputting the teacher data to the question answering system 50 via the communication unit 40A.

[0107] (Example of display content displayed by information processing device 1C) An example of the display content displayed by the information processing device 1C will be described with reference to Fig. 9. Fig. 9 is a diagram showing an example of the display content displayed by the information processing device 1C according to this exemplary embodiment.

[0108] The determination unit 12 of the information processing device 1C supplies the target pair and information indicating the determined appropriateness of the target pair to the input / output unit 30A. The input / output unit 30A outputs the target pair acquired from the determination unit 12 and the information indicating the determined appropriateness of the target pair.

[0109] As an example, when the input / output unit 30A is a display device that displays images, the input / output unit 30A displays the display content shown in Fig. 9. Specifically, the input / output unit 30A displays the target pair of the question "What is the capital of the United States?" and the correct answer "Washington, DC", as well as the display content including the appropriateness of "0.05", which is information indicating the appropriateness of the target pair.

[0110] Furthermore, when the determination unit 12 acquires the similar pair referred to when determining the appropriateness of the target pair and information indicating the appropriateness of the similar pair, the input / output unit 30A may be configured to include the similar pair and the information indicating the appropriateness of the similar pair in the display content. As an example, as shown in Fig. 9, the input / output unit 30A may display display content including similar pair 1 of the question "What is the capital of Japan?" and the correct answer "Tokyo", an appropriateness score of "0" that is information indicating the appropriateness of similar pair 1, and similar pair 2 of the question "What is the capital of the United Kingdom?" and the correct answer "London", and an appropriateness score of "0.1" that is information indicating the appropriateness of similar pair 2.

[0111] (Effects of information processing device 1C) As described above, the information processing device 1C according to this exemplary embodiment is configured to include a teacher data generation unit 13 that generates teacher data including target pairs that have been determined to be appropriate by the determination unit 12, and a learning unit 14 that uses the teacher data to train the question answering system 50.

[0112] Therefore, according to the information processing device 1C of this exemplary embodiment, in addition to the effects achieved by the information processing device 1, information processing device 1A, and information processing device 1B of the exemplary embodiments described above, the information processing device 1C has the effect of being able to train the question answering system 50 using appropriate training data.

[0113] Furthermore, the information processing device 1C according to this exemplary embodiment is configured to include an input / output unit 30A that displays the target pair and information indicating the appropriateness of the target pair determined by the determination unit 12. Therefore, the information processing device 1C according to this exemplary embodiment can present the target pair and information indicating the appropriateness of the determined target pair to the user.

[0114] [Software implementation example] Some or all of the functions of the information processing devices 1, 1A, 1B, and 1C may be realized by hardware such as an integrated circuit (IC chip), or may be realized by software.

[0115] In the latter case, the information processing devices 1, 1A, 1B, and 1C are realized, for example, by a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in FIG. 10. The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for operating the computer C as the information processing devices 1, 1A, 1B, and 1C. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing each function of the information processing devices 1, 1A, 1B, and 1C.

[0116] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0117] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, mouse, display, and printer.

[0118] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0119] [Appendix 1] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiments are also included in the technical scope of the present invention.

[0120] [Appendix 2] Some or all of the above-described embodiments can also be described as follows: However, the present invention is not limited to the following described aspects.

[0121] (Appendix 1) An information processing device comprising: an acquisition means for acquiring a target pair, which is a pair of a question to be judged and a correct answer; and a judgment means for judging the appropriateness of the target pair by referring to information indicating the appropriateness of at least one of the question and the correct answer contained in one or more similar pairs similar to the target pair acquired by the acquisition means.

[0122] (Appendix 2) The information indicating the appropriateness is information indicating the usefulness of the similar pair when used for training a question-answering system, and the determination means determines the usefulness of the target pair when used for training a question-answering system as the appropriateness of the target pair.

[0123] (Appendix 3) The information processing device according to claim 1 or 2, wherein the determination means includes a search means that refers to a database storing a plurality of pairs of a question and a correct answer and information indicating the appropriateness of each of the plurality of pairs, and searches for one or more similar pairs whose similarity to the target pair is equal to or greater than a predetermined degree, and an evaluation means that evaluates the appropriateness of the target pair based on the information indicating the appropriateness of each of the similar pairs searched by the search means. (Appendix 4) An information processing device as described in Appendix 3, wherein the evaluation process by the evaluation means includes a process of taking an average or weighted average of scores as information indicating the appropriateness of each of the multiple similar pairs searched by the search means.

[0124] (Appendix 5) The information processing device according to claim 1 or 2, wherein the determination means is provided with a prediction means for predicting the appropriateness of the target pair using a prediction model that receives a pair of a question and a correct answer as input and outputs a score as information indicating the appropriateness of the pair.

[0125] (Appendix 6) The information processing device described in Appendix 5, wherein the determination means further includes an update means for updating parameters of the prediction model by referring to a database storing multiple pairs of questions and correct answers and information indicating the appropriateness of each of the multiple pairs.

[0126] (Appendix 7) 7. The information processing device according to any one of claims 1 to 6, further comprising: a training data generating unit configured to generate training data including a target pair determined to be appropriate by the determining unit.

[0127] (Appendix 8) 8. The information processing device according to claim 7, further comprising a learning means for learning a question answering system using the training data.

[0128] (Appendix 9) 9. The information processing device according to any one of appendices 1 to 8, further comprising a display means for displaying the target pair and information indicating the appropriateness of the target pair determined by the determination means.

[0129] (Appendix 10) An information processing method including: an information processing device acquiring a target pair, which is a pair of a question to be judged and a correct answer; and judging the appropriateness of the target pair by referring to information indicating the appropriateness of at least one of the question and the correct answer contained in one or more similar pairs that are similar to the target pair acquired in the acquisition.

[0130] (Appendix 11) A program that causes a computer to function as an information processing device, the program causing the computer to function as: an acquisition means that acquires a target pair, which is a pair of a question and a correct answer to be judged; and a judgment means that judges the appropriateness of the target pair by referring to information indicating the appropriateness of at least one of the question and the correct answer contained in one or more similar pairs that are similar to the target pair acquired by the acquisition means.

[0131] [Appendix 3] Some or all of the above-described embodiments can also be expressed as follows.

[0132] An information processing device comprising at least one processor, which executes an acquisition process to acquire a target pair, which is a pair of a question to be judged and a correct answer, and a judgment process to judge the appropriateness of the target pair by referring to information indicating the appropriateness of at least one of the question and the correct answer contained in one or more similar pairs that are similar to the target pair acquired in the acquisition process.

[0133] The information processing device may further include a memory that stores a program for causing the processor to execute the acquisition process and the determination process. The program may be recorded on a computer-readable, non-transitory, tangible recording medium. [Explanation of symbols]

[0134] 1, 1A, 1B, 1C Information processing equipment 10A, 10B, 10C control section 11 Acquisition Department 12 Judgment section 13 Teacher data generation unit 14 Learning Department 121 Search Department 122 Evaluation Department 125 Prediction Department 126 Update Department

Claims

1. an acquisition means for acquiring a target pair, which is a pair of a question to be judged and a correct answer; a determination means for determining the appropriateness of the target pair by referring to information indicating the appropriateness of at least one of a question and a correct answer included in one or more similar pairs similar to the target pair acquired by the acquisition means; An information processing device comprising:

2. the information indicating the appropriateness is information indicating usefulness when the similar pair is used for training a question-answering system; The determining means determines the suitability of the target pair by determining the usefulness of the target pair when used for training a question-answering system. The information processing device according to claim 1 .

3. The determination means a search means for searching for one or more similar pairs having a similarity to the target pair that is equal to or greater than a predetermined level, by referring to a database that stores a plurality of pairs of questions and correct answers and information indicating the appropriateness of each of the plurality of pairs; evaluation means for evaluating the appropriateness of the target pair based on information indicating the appropriateness of each similar pair retrieved by the retrieval means; 3. The information processing device according to claim 1, further comprising:

4. The evaluation process by the evaluation means includes: The process includes a process of calculating an average or a weighted average of the scores as information indicating the appropriateness of each of the plurality of similar pairs searched by the search means. The information processing device according to claim 3 .

5. The determination means A prediction means for predicting the appropriateness of the target pair using a prediction model that receives a question and a correct answer pair as input and outputs a score as information indicating the appropriateness of the pair.

3. The information processing device according to claim 1, further comprising:

6. The determination means an update means for updating the parameters of the prediction model by referring to a database storing a plurality of pairs of questions and correct answers and information indicating the appropriateness of each of the plurality of pairs; The information processing apparatus according to claim 5 , further comprising:

7. The information processing apparatus according to claim 1 , further comprising: a training data generating unit configured to generate training data including the target pair determined to be appropriate by the determining unit.

8. A learning means is provided for training the question answering system using the training data. The information processing device according to claim 7 .

9. The information processing device Obtaining a target pair, which is a pair of a question to be judged and a correct answer; determining the appropriateness of the target pair by referring to information indicating the appropriateness of at least one of a question and a correct answer included in one or more similar pairs similar to the target pair obtained in the obtaining; An information processing method including:

10. A program that causes a computer to function as an information processing device, The program causes the computer to: an acquisition means for acquiring a target pair, which is a pair of a question to be judged and a correct answer; a determination means for determining the appropriateness of the target pair by referring to information indicating the appropriateness of at least one of a question and a correct answer included in one or more similar pairs similar to the target pair acquired by the acquisition means; A program that functions as a

Images