Question text creation device, question text creation method, and program

Abstract

To provide a question sentence preparation technique for a fill-in-blank question using a natural language processing model.SOLUTION: This question sentence preparation device comprises: a sentence acquisition unit that acquires a first sentence including a word to be learned from a data set; a certainty factor calculation unit that masks the word in the first sentence, and calculates a certainty factor of the word in a masked place of the first sentence and a certainty factor of a candidate word which is another candidate allowed to enter the masked place; and a question sentence output unit that outputs the masked sentence as a fill-in-blank question when the certainty factor of the word is larger by a prescribed value than the certainty factor of the candidate word.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to a problem statement creation device, a problem statement creation method, and a program.

Background Art

[0002] With the development of information technology in recent years, information technology has been applied in various fields. For example, various learning support tools using information technology have been developed. As an example of such a learning support tool, a learner can learn English or the like by operating a smartphone, a tablet, a personal computer, etc. and using a learning support app or the like to solve problems displayed on the learning support app.

[0003] As an example of the problem format provided by such a learning support app, there is a fill-in-the-blank problem, and a problem statement creation method for automatically generating such a fill-in-the-blank problem has been proposed. A fill-in-the-blank problem is a type of problem that allows a learner to infer or select from options the words or terms to fill in the blank based on the context before and after.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In recent years, artificial intelligence has been used in fields such as natural language processing, and various natural language processing models such as BERT (Bidirectional Encoder Representations from Transformers) by Google have been developed, and the high performance of these natural language processing models in natural language processing has attracted attention.

[0006] An object of the present disclosure is to provide a problem statement creation technique for fill-in-the-blank problems using a natural language processing model.

Means for Solving the Problems

[0007] To solve the above problems, one aspect of the present invention relates to a problem statement creation device having: a sentence acquisition unit that acquires a first sentence including a word to be learned from a dataset; a confidence calculation unit that masks the word in the first sentence and calculates, using a prediction model, the confidence of the word at the masked position in the first sentence and the confidence of candidate words that are other candidates that can be inserted into the masked position; and a problem statement output unit that outputs the masked sentence as a fill-in-the-blank problem when the confidence of the word is greater than or equal to a predetermined value more than the confidence of the candidate words.

Effects of the Invention

[0008] According to the present disclosure, it is possible to provide a problem statement creation technique for fill-in-the-blank problems using a natural language processing model.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Mode for Carrying Out the Invention

[0010] <Explanation of Terms> · In this specification, the confidence level is the legitimate (prediction) probability of the Mask prediction task (word filling-in-the-blank task), and the confidence level of a word is the probability (confidence level) that the word fits in the corresponding position in the sentence. The "confidence level of a word" is calculated based on one or more other words in the sentence. Also, the "confidence level of a word" may be calculated based on one or more words in the sentences before and after the corresponding sentence. The higher the value (that is, the value of the confidence level), the more likely it is that the corresponding word is used by analogy from other words. Note that as long as the confidence level can be calculated, it is not necessary to use a specific calculation means. In this embodiment, the confidence level calculated by BERT is used for explanation.

[0011] In the following embodiments, a problem sentence creation device for a filling-in-the-blank problem is disclosed.

[0012] As shown in FIG. 1, when the problem sentence creation device 100 according to an embodiment of the present disclosure receives a word to be learned and a sentence including the word as inputs, it outputs a filling-in-the-blank problem for the input word using an appropriate natural language processing model such as BERT. For example, as shown in FIG. 2, when learner A learns English using a learning application operated by a learning support system, the learning support system acquires information about the English words that learner A has already acquired from learner A's learning history and the like, and creates a filling-in-the-blank problem for the unacquired English words from the English language content obtained from a database or the web.

[0013] Specifically, as shown in FIG. 3, the problem sentence creation device 100 masks the English words to be learned in the sentence (S12 in FIG. 3, S32 in FIG. 7) obtained from a dataset (S11 in FIG. 3, S31 in FIG. 7 described later), etc. (S13 in FIG. 3, S33 in FIG. 7), inputs the sentence including the masked portion into a natural language processing model such as BERT (S14 in FIG. 3, S34 in FIG. 7), and estimates candidates for the English words to be entered in the masked portion. According to the present disclosure, the problem sentence creation device 100 estimates the confidence level of each candidate for the English word estimated from the natural language processing model, and when the difference between the highest first confidence level, that is, the confidence level of the masked English word to be learned, and the next highest second confidence level is equal to or greater than a predetermined value, outputs the sentence including the masked portion as a fill-in-the-blank problem for the English word having the first confidence level (S15 in FIG. 3, S35 in FIG. 7). Thereby, it becomes possible to output, as a problem sentence, a sentence including an English word with a high confidence level having a significant difference from other English word candidates as the English word to be entered in the masked portion, and to create a fill-in-the-blank problem having only the English word to be learned as the correct answer.

[0014] As will be described later, the problem sentence creation device 100 may be a server or the like that provides problem sentences to a learner's smartphone, tablet, personal computer, etc. via a learning application or the like. For example, the problem sentence creation device 100 may be a server of a company, vendor, etc. that operates a learning application, and acquires a sentence to be processed from a database that stores sentences held by the company, vendor, etc. as a dataset, and creates a problem sentence as will be described later. Preferably, the dataset is composed of copyright-managed sentences.

[0015] The problem sentence creation device 100 may have a hardware configuration including a processor 101 such as a CPU (Central Processing Unit), a memory 102 such as a RAM (Random Access Memory) and a flash memory, a storage 103 such as a hard disk, and an input / output (I / O) interface 104, as shown in FIG. 4, for example.

[0016] The processor 101 executes various processes of the problem statement creation device 100 described later.

[0017] The memory 102 stores various data and programs in the problem statement creation device 100, and functions as a working memory, particularly for working data, programs being executed, etc. Specifically, the memory 102 stores programs for executing and controlling various processes described later that are loaded from the storage 103, and functions as a working memory during the execution of the programs by the processor 101.

[0018] The storage 103 stores various data and programs in the problem statement creation device 100.

[0019] The I / O interface 104 is an interface for receiving commands, input data, etc. from the user, displaying and playing back output results, and inputting and outputting data to and from an external device. For example, the I / O interface 104 may be a device for inputting and outputting various data such as USB (Universal Serial Bus), communication lines, keyboards, mice, displays, microphones, speakers, etc.

[0020] However, the problem statement creation device 100 according to the present disclosure is not limited to the above-described hardware configuration, and may have any other appropriate hardware configuration. For example, one or more of the various processes by the problem statement creation device 100 may be realized by a processing circuit or an electronic circuit wired to realize this.

[0021] Next, with reference to FIGS. 5 to 9, the problem statement creation device 100 according to an embodiment of the present disclosure will be described in more detail. The problem statement creation device 100 receives, as inputs, words to be learned (e.g., English words, terms, personal names, places, etc.) and sentences, and outputs a fill-in-the-blank problem with the word as a blank. In the following embodiments, English words are applied as the words to be learned, and a fill-in-the-blank problem with the English word masked is output, but the present disclosure is not limited to this, and can be applied to the learning of terms in any other subject other than English learning.

[0022] FIG. 5 is a block diagram showing a functional configuration of the problem statement creation device 100 according to an embodiment of the present disclosure. As shown in FIG. 5, the problem statement creation device 100 includes a sentence acquisition unit 110, a confidence calculation unit 120, and a problem statement output unit 130.

[0023] The sentence acquisition unit 110 acquires a sentence including a word to be learned from a dataset. In this specification, a sentence is composed of one or more words ending with a period "。" or a full stop ".", and a sentence is a plurality of consecutive sentences having a coherent meaning. The sentence or sentence to be acquired may be acquired from, for example, a database or the web. Preferably, the sentence or sentence to be acquired may be extracted from a dataset of a database whose copyright is managed.

[0024] The confidence calculation unit 120 masks the word to be learned in the acquired sentence and calculates the confidence of the word to be learned at the masked position in the sentence and the confidence of candidate words that are other candidates that can enter the masked position using an inference model. Specifically, the confidence calculation unit 120 converts a sentence or sentence including the word to be learned into a sentence or sentence with a blank at the masked position by masking the word. Then, the confidence calculation unit 120 inputs the sentence or sentence including the masked position into an appropriate natural language processing model such as BERT. As shown in FIG. 3, BERT outputs candidate words to be entered at the masked position and the confidence of each candidate word.

[0025] In the illustrated specific example, for the masked word "continue", BERT outputs candidate words and confidence levels for the masked position, "continue 0.9373···", "cease 0.0506···", "begin 0.0025···", from the input sentence containing the masked position, "This city will cease to be and will ( ) to grow!". Here, the confidence level indicates the probability that the word will fit into the masked position. For example, the confidence level of the masked word is the highest. Note that it is also possible that the confidence level of the masked word may not be the highest.

[0026] Also, for the masked word "hurt", BERT outputs candidate words and confidence levels for the masked position, "hurt 0.9465···", "wounded 0.0167···", "offended 0.0062···", from the input sentence containing the masked position, "I apologize if my actions ( ) your pride."

[0027] Note that if the masked word (for example, "continue" or "hurt" in the above-described embodiments) is not inferred as a candidate that can fit into the masked position, the problem sentence creation device 100 obtains another sentence as an error process.

[0028] In the above-described embodiments, one sentence is input to BERT, but the present disclosure is not limited to this. In one embodiment, a passage composed of a plurality of consecutive sentences may be input to BERT. For example, as shown in FIG. 6, the confidence level calculation unit 120 may mask the word to be learned (S23 in FIG. 6) for a passage (S21 and S22 in FIG. 6) composed of the sentence containing the word to be learned and the sentences before and after the sentence, and input the passage containing the masked position to BERT (S24 in FIG. 6). Similarly, BERT outputs candidate words and confidence levels for the masked position from the input sentence containing the masked position (S25 in FIG. 6).

[0029] When the difference between the confidence level of a word and the confidence level of a candidate word is equal to or greater than a predetermined value, the problem statement output unit 130 outputs the masked sentence as a fill-in-the-blank problem. Specifically, the problem statement output unit 130 compares the confidence levels of each candidate word at the masked position, and calculates the difference between the highest confidence level (for example, the confidence level of the masked word) and the next highest confidence level. Then, when the difference between the highest confidence level and the next highest confidence level is equal to or greater than a predetermined threshold, the problem statement output unit 130 determines that the input sentence is suitable as a fill-in-the-blank problem for the word, and outputs the masked sentence as a fill-in-the-blank problem for the word. On the other hand, when the difference between the highest confidence level and the next highest confidence level is less than the predetermined threshold, the problem statement output unit 130 determines that the input sentence is not suitable as a fill-in-the-blank problem for the word, and does not output the masked sentence as a fill-in-the-blank problem for the word. That is, when there is no significant difference in these confidence levels, it is difficult to uniquely identify the correct answer to the fill-in-the-blank problem from the input sentence, and it is considered not suitable as a fill-in-the-blank problem. Similarly, when the highest confidence level is not the confidence level of the masked word, the input sentence is also considered not suitable as a fill-in-the-blank problem. That is, the problem statement output unit 130 outputs the masked sentence as a fill-in-the-blank problem when the highest confidence level is the confidence level of the masked word and the difference between the confidence level of the masked word and the confidence level of the candidate word is equal to or greater than a predetermined value.

[0030] In one embodiment, when the difference between the confidence of a word and the confidence of a candidate word is less than a predetermined value, the sentence acquisition unit 110 acquires a sentence adjacent to the acquired sentence (i.e., the sentence including the word to be learned from the dataset), and the confidence calculation unit 120 uses the inference model to calculate the confidence of the word to be learned and the confidence of the candidate word at the masked position of the sentence composed of the acquired sentence and the sentence adjacent to the acquired sentence. When the difference between the confidence of the word to be learned and the confidence of each candidate word is greater than or equal to the predetermined value, the problem sentence output unit 130 may output the masked sentence as a fill-in-the-blank problem for the word. For example, when the input sentence to BERT is not suitable as a fill-in-the-blank problem, the sentences before and after the input sentence may be searched, and the sentence composed of the input sentence and the sentences before and after may be input to BERT. Generally, it is considered that as the context range expands, the candidate words to be entered at the masked position are narrowed down, and it is assumed that the difference between the confidence of the word to be learned and the next highest confidence becomes larger. Therefore, when the input sentence is not suitable as a fill-in-the-blank problem, the sentence acquisition unit 110 extracts the sentences before and after the input sentence from the dataset, the confidence calculation unit 120 masks the word to be learned in the extracted sentence, inputs the sentence including the masked position to BERT, and may calculate the candidate words and the confidence of each candidate word. When the difference between the highest confidence of the word to be learned and the next highest confidence becomes greater than or equal to the predetermined value, the problem sentence creation device 100 may output the sentence as a fill-in-the-blank problem. That is, when the difference between the confidence of the word and the confidence of the candidate word in the nth (n is an integer of 1 or more) sentence is less than the predetermined value, the sentence acquisition unit 110 acquires the (n + 1)th sentence adjacent to the nth sentence, and the confidence calculation unit 120 uses the inference model to calculate the confidence of the word to be learned and the confidence of the candidate word at the masked position of the sentence composed of the acquired sentence (the nth sentence) and the sentence adjacent to the acquired sentence (the (n + 1)th sentence). When the difference between the confidence of the word to be learned and the confidence of each candidate word is greater than or equal to the predetermined value, the problem sentence output unit 130 can output the masked sentence as a fill-in-the-blank problem for the word (for example, the acquisition of adjacent sentences may be repeated up to a specified number of times, or until the difference between the confidence of the word to be learned and the confidence of each candidate word becomes greater than or equal to the predetermined value).

[0031] Also, in one embodiment, the problem sentence output unit 130 may output a word and a candidate word associated with the word as options in a fill-in-the-blank problem. That is, the problem sentence output unit 130 may automatically generate options for the fill-in-the-blank problem. For example, the problem sentence output unit 130 may select, as options for the blank, the word to be learned and derivative words or words of different parts of speech of the word as options for the blank in the fill-in-the-blank problem. Specifically, when the word to be learned is a verb, the problem sentence output unit 130 may refer to a dictionary database or the like and determine corresponding nouns, adjectives, adverbs, etc. of the verb as options.

[0032] Also, in one embodiment, the problem sentence output unit 130 may display one or more characters of a word in a fill-in-the-blank problem. For example, the problem sentence output unit 130 may display, as a hint in the blank of the fill-in-the-blank problem, the first character of the correct word (for example, when the correct word is "continue", the first character "c").

[0033] Also, in one embodiment, the problem sentence output unit 130 may display a sentence and a sentence associated with the sentence in a fill-in-the-blank problem. For example, as shown in FIG. 7, in an English fill-in-the-blank problem, the problem sentence output unit 130 may display an English sentence and its translation.

[0034] Also, in one embodiment, the problem sentence output unit 130 may display a sentence and an image associated with the sentence in a fill-in-the-blank problem. For example, in a fill-in-the-blank problem "Carbon dioxide is a substance in which one carbon atom and two ( ) are bonded.", a diagram "O=C=O" in which a carbon atom and two oxygen atoms are bonded may be displayed together.

[0035] Also, in one embodiment, the problem sentence creation device 100 may further include, as shown in FIG. 8, a word providing unit 140 that provides a word to be learned. For example, the word providing unit 140 may determine a word to be learned based on the learning history and acquisition level of the learner, and provide the word to be learned to the sentence acquisition unit 110, the confidence calculation unit 120, and / or the problem sentence output unit 130.

[0036] In addition, the above-described embodiments may be combined as appropriate.

[0037] FIG. 9 is a flowchart showing a problem sentence creation process according to an embodiment of the present disclosure. The problem sentence creation process is executed by the above-described problem sentence creation device 100, and can be particularly realized by a processor of the problem sentence creation device 100 executing a program.

[0038] As shown in FIG. 9, in step S101, the problem sentence creation device 100 acquires a sentence including a word to be learned from a dataset. For example, the problem sentence creation device 100 extracts a sentence or passage including a word to be learned from a copyright-managed database. Specifically, when the word to be learned is a high school entrance examination English word, the problem sentence creation device 100 extracts an English sentence including these English words from the database.

[0039] In step S102, the problem sentence creation device 100 masks the word to be learned in the acquired sentence. For example, when acquiring a sentence including a word to be learned, the problem sentence creation device 100 makes the location of the word in the acquired sentence into a blank.

[0040] In step S103, the problem sentence creation device 100 calculates the confidence level of the word at the masked location of the masked sentence and the confidence level of the candidate words using a speculation model. For example, the problem sentence creation device 100 inputs the sentence including the masked location into BERT and acquires candidate words that can be entered at the masked location and the confidence level of each candidate word. For example, the confidence level of the masked word becomes the highest.

[0041] In step S104, when the difference between the confidence level of the masked word and the confidence level of other candidate words is equal to or greater than a predetermined value, the problem sentence creation device 100 outputs the masked sentence as a fill-in-the-blank problem. Note that the predetermined value may be set to any appropriate value that can significantly distinguish the masked word from the candidate words.

[0042] As described above, the embodiments of the present invention have been described in detail. However, the present invention is not limited to the specific embodiments described above, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

Explanation of Reference Numerals

[0043] 100 Problem statement creation device 110 Sentence acquisition unit 120 Confidence calculation unit 130 Problem statement output unit 140 Word providing unit

Claims

1. A sentence acquisition unit that acquires a first sentence including a word to be learned from a dataset; A confidence calculation unit that masks the word in the first sentence and calculates, using a prediction model, the confidence of the word at the masked position in the first sentence and the confidence of candidate words that can be entered at the masked position; A problem sentence output unit that sets a predetermined value for distinguishing the word from the candidate words, and outputs the masked sentence as a fill-in-the-blank problem when the confidence of the word is greater than or equal to the predetermined value than the confidence of the candidate words; A problem sentence creation device having the above.

2. When the difference between the confidence of the word and the confidence of the candidate word is less than the predetermined value, The sentence acquisition unit acquires a second sentence adjacent to the first sentence in the sentences in the dataset, The confidence calculation unit calculates, using the prediction model, the confidence of the word and the confidence of the candidate word at the masked position of the sentence composed of the first sentence and the second sentence, The problem sentence output unit outputs the masked sentence as a fill-in-the-blank problem for the word when the confidence of the word is greater than or equal to the predetermined value than the confidence of the candidate word, according to the problem sentence creation device described in claim 1.

3. When the difference between the confidence of the word and the confidence of the candidate word is less than the predetermined value, The sentence acquisition unit acquires an (n + 1)-th sentence adjacent to the n-th sentence in the sentences in the dataset, where n is an integer greater than or equal to 1, The confidence calculation unit calculates, using the prediction model, the confidence of the word and the confidence of the candidate word at the masked position of the sentence composed of the n-th sentence and the (n + 1)-th sentence, The problem sentence output unit outputs the masked sentence as a fill-in-the-blank problem for the word when the confidence of the word is greater than or equal to the predetermined value than the confidence of the candidate word, according to the problem sentence creation device described in claim 1.

4. The problem sentence output unit outputs the word and the selected candidate words associated with the word as options in the fill-in-the-blank problem, according to the problem sentence creation device described in any one of claims 1 to 3.

5. The problem sentence output unit displays one or more characters of the word in the fill-in-the-blank problem, according to the problem sentence creation device described in any one of claims 1 to 4.

6. The problem sentence output unit displays the masked sentence and the sentence associated with the masked sentence in the fill-in-the-blank problem, and the problem sentence creation device according to any one of claims 1 to 5.

7. The problem sentence output unit displays the masked sentence and the image associated with the masked sentence in the fill-in-the-blank problem, and the problem sentence creation device according to any one of claims 1 to 5.

8. The problem sentence creation device according to any one of claims 1 to 7, further comprising a word providing unit that provides the word to be learned.

9. A step of obtaining a first sentence including a word to be learned from a dataset; Masking the word in the first sentence and calculating, using a prediction model, the confidence of the word at the masked position in the first sentence and the confidence of a candidate word that is another candidate that can be inserted into the masked position; Setting a predetermined value for distinguishing the word and the candidate word, and when the confidence of the word is greater than or equal to the predetermined value than the confidence of the candidate word, outputting the masked sentence as a fill-in-the-blank problem; A problem sentence creation method executed by a computer.

10. A process of obtaining a first sentence including a word to be learned from a dataset; A process of masking the word in the first sentence and calculating, using a prediction model, the confidence of the word at the masked position in the first sentence and the confidence of a candidate word that is another candidate that can be inserted into the masked position; A process of setting a predetermined value for distinguishing the word and the candidate word, and when the confidence of the word is greater than or equal to the predetermined value than the confidence of the candidate word, outputting the masked sentence as a fill-in-the-blank problem; A program for causing a computer to execute.

Images