Data expansion information acquisition device and data expansion information acquisition program

Abstract

To provide a data expansion information acquisition device capable of calculating a degree of similarity even when the number of tokens differs before and after token replacement, and performing further versatile data expansion.SOLUTION: The present invention includes a pre-learning model that is trained using a mask language model, a tokenizer unit that decomposes an input text into tokens, an embedding unit that calculates a word dispersion representation of the entire input text, a mask prediction unit that predicts a masked part in a token string obtained by decomposing an input text using the pre-learning model, and a processing unit that, upon replacing one or more tokens in the token string with one or more mask tokens, after the embedding unit calculates the degree of similarity between a generated text that is generated by repeatedly predicting tokens corresponding to the masked part using the mask prediction unit and the input text, outputs a predetermined number of generated texts in descending order of the degree of similarity.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a data extension information acquisition device and a data extension information acquisition program. [Background technology]

[0002] Conventionally, in machine learning such as deep learning, a large amount of training data is required to achieve high accuracy, so data augmentation techniques are generally used. For example, in image processing, data augmentation is achieved by increasing the amount of training data by creating images that are processed by rotating, zooming, translating, etc. from the original image. Also, in the field of natural language processing, data augmentation has been attempted using various methods, and data augmentation techniques are disclosed in, for example, Patent Document 1, Non-Patent Document 1, etc.

[0003] In recent years, as a natural language processing technology, the technology described in Non-Patent Document 2 has been attracting attention. Non-Patent Document 2 shows that highly accurate learning is possible by the following procedure.

[0004] First, a large amount of document text is prepared, and this is used as unsupervised learning data for pre-learning using MLM (Masked Language Model) and NSP (Next Sentence Prediction) techniques to create a pre-learned model.

[0005] Next, supervised training data for the required task is prepared and fine-tuned (trained) together with the pre-trained model.

[0006] The above-mentioned MLM is a language model that masks the input data on a token-by-token basis and predicts the masked parts. Therefore, the pre-trained model created by the above procedure is capable of predicting the masked parts of the input data. On the other hand, the above-mentioned NSP is a method that literally learns to predict the following sentence, although details are omitted.

[0007] Furthermore, an example of a tokenizer for deep learning is the technology described in Non-Patent Document 3. Non-Patent Document 3 describes a method of further decomposing words into subwords, which are then used as tokens. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2020-126360 A [Non-patent literature]

[0009] [Non-Patent Document 1] Shinnosuke Nishimoto, Hiroshi Noji, Yuji Matsumoto, "Aspect Estimation for Sentiment Analysis by Data Augmentation", Proceedings of the 23rd Annual Conference of the Association for Natural Language Processing, pp. 581-584, March 2017, http: / / www.anlp.jp / proceedings / annual_meeting / 2017 / pdf_dir / P10-4.pdf [Non-Patent Document 2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” arXiv:1810.04805 (2018). [Non-Patent Document 3] Taku Kudo, John Richardson, “SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing” arXiv:1808.06226 (2018) Summary of the Invention [Problem to be solved by the invention]

[0010] However, regarding data augmentation in natural language processing, the technology (method) described in Patent Document 1 is a method specialized for translation and cannot be applied to other tasks.

[0011] On the other hand, Non-Patent Document 1 lists a technology based on the similarity of word vectors. In the case of natural language processing using deep learning, tokens are treated as vectors, so this method is very convenient, but there is a problem when using a tokenizer such as that shown in Non-Patent Document 3. The following describes this problem using a concrete example.

[0012] Assume that there are two texts: "A restaurant with a reputation for being delicious" (hereafter referred to as "Tx1") and "A restaurant with a reputation for being delicious" (hereafter referred to as "Tx2"). Examples of decomposing these texts into words are "A restaurant with a reputation for being delicious" (hereafter referred to as "Ts1") and "A restaurant with a reputation for being delicious" (hereafter referred to as "Ts2"). If it is found that the words "delicious" and "delicious" are semantically similar, it is possible to create Tx2 from Tx1 by replacing one word. This shows that if words are tokens and there is a word distributed representation (vector) for each token, data augmentation can be performed using similarity.

[0013] On the other hand, for example, if a tokenizer such as that shown in Non-Patent Document 3 is used, subwords obtained by further breaking down words become tokens. An example of the result of breaking down Tx2 into tokens using this tokenizer is "a restaurant with a good reputation for being delicious" (hereinafter referred to as "Ts2'"). The "##" at the beginning of "##good" indicates that the word has been further broken down. Note that when Tx1 is broken down using this tokenizer, it becomes Ts1, just like when it is broken down into words.

[0014] In this case, it is not possible to create Ts2 from Ts1 by replacing one token. It is necessary to replace the token "delicious" with two tokens, "oi" and "##shii". When calculating the one-to-one similarity of vectors, it can be easily calculated using cosine similarity, but in the case of one-to-two, it is necessary to define a method for calculating the similarity, and finding this replacement is very difficult and problematic.

[0015] In natural language processing using deep learning, it is becoming mainstream to use subwords as tokens, so this problem is very serious and needed to be solved.

[0016] Therefore, there is a demand for an information acquisition device for data augmentation and an information acquisition program for data augmentation that can calculate similarity even if the number of tokens is different before and after token replacement and can perform data augmentation in a more general manner.

[0017] The present invention has been made in consideration of this problem, and aims to enable calculation of similarity even when the number of tokens differs before and after replacement, thereby implementing more versatile data expansion. [Means for solving the problem]

[0018] A first information acquisition device for data augmentation of the present invention is characterized in that it comprises: (1) a pre-trained model trained using a mask language model; (2) a tokenizer unit that decomposes an input text into tokens; (3) an embedding unit that calculates word embeddings for the entire input text; (4) a mask prediction unit that predicts a masked portion in a token sequence obtained by decomposing the input text using the pre-trained model; and (5) a processing unit that calculates a similarity between a created text and the input text in the embedding unit and outputs a predetermined number of the created texts in order of highest similarity.

[0019] The second information acquisition program for data augmentation of the present invention is characterized in that it causes a computer to function as: (1) a pre-trained model trained using a mask language model; (2) a tokenizer unit that decomposes input text into tokens; (3) an embedding unit that calculates word embeddings for the entire input text; (4) a mask prediction unit that predicts a masked portion in a token sequence obtained by decomposing the input text using the pre-trained model; and (5) a processing unit that calculates a similarity between the input text and a created text created by repeatedly replacing one or more tokens in the token sequence with one or more mask tokens and predicting tokens corresponding to the masked portions using the mask prediction unit, and outputs a predetermined number of the created texts in order of highest similarity. Effect of the Invention

[0020] According to the present invention, it is possible to calculate the similarity even if the number of tokens before and after token replacement is different, and data expansion can be implemented in a more general manner. [Brief description of the drawings]

[0021] [Figure 1] 1 is a block diagram showing a configuration of a data extension information acquisition device according to an embodiment. FIG. [Diagram 2] 10 is a flowchart illustrating a characteristic operation of the data extension information acquisition device according to the embodiment. [Diagram 3] FIG. 11 is an explanatory diagram illustrating an example of a token ID sequence obtained using a tokenizer according to an embodiment. [Figure 4] FIG. 11 is an explanatory diagram showing an example in which a part of a token ID string according to the embodiment is masked. [Diagram 5] 11 is an explanatory diagram showing an example of an array storing prediction results of a masked portion of a token ID string according to the embodiment; FIG. [Figure 6] FIG. 11 is an explanatory diagram showing an example of a token ID sequence in which a masked portion is replaced with a prediction result according to the embodiment. [Figure 7] 7 is an explanatory diagram showing an example of a similarity table showing the similarity of the masked portion of each token ID string (FIG. 6) according to the embodiment. FIG. [Figure 8] FIG. 4 is an explanatory diagram illustrating an example of an input file according to the embodiment. [Figure 9] FIG. 11 is an explanatory diagram illustrating an example of an output file according to the embodiment. [Figure 10] FIG. 11 is an explanatory diagram showing an example in which data augmentation is applied to training data for a sentence classification task according to an embodiment. [Figure 11] FIG. 11 is an explanatory diagram showing an example in which data augmentation is applied to training data for a named entity extraction task according to an embodiment. [Figure 12] FIG. 11 is an explanatory diagram showing an example in which an arbitrary number of tokens in a token ID string according to an embodiment are replaced with an arbitrary number of mask portions. [Figure 13] FIG. 11 is an explanatory diagram (part 1) showing an example of an array storing prediction results of a masked portion of a token ID string according to a modified embodiment. [Figure 14] FIG. 11 is an explanatory diagram (part 1) showing an example in which a token ID string according to a modified embodiment is partially masked. [Figure 15] FIG. 11 is an explanatory diagram (part 2) showing an example of an array storing prediction results of a masked portion of a token ID string according to a modified embodiment. [Figure 16] FIG. 11 is an explanatory diagram (part 2) showing an example in which a token ID string according to a modified embodiment is partially masked. [Figure 17] FIG. 13 is a block diagram showing a configuration of a data extension information acquisition device according to a modified embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] (A) Main embodiment Hereinafter, an embodiment of a data extension information acquisition device and a data extension information acquisition program according to the present invention will be described in detail with reference to the drawings.

[0023] (A-1) Configuration of the embodiment FIG. 1 is a block diagram showing a configuration of a data extension information acquisition device according to an embodiment.

[0024] In FIG. 1, a data augmentation information acquisition device 1 according to the embodiment includes a pre-learning model 11, a tokenizer 12, an embedding unit 13, a mask prediction unit 14, a processing unit 15, an input file 16, and an output file 17.

[0025] The language processing device according to the embodiment may be configured as hardware, such as a dedicated IC chip equipped with each component shown in FIG. 1, or may be configured as software, centered around a CPU and a program executed by the CPU, but functionally it can be represented as shown in FIG. 1.

[0026] The data augmentation information acquisition device 1 of this embodiment is characterized in that it can obtain similarity even if the number of tokens before and after token replacement of the input text is different. Specifically, the data augmentation information acquisition device 1 uses the similarity of the entire text instead of calculating the similarity of the tokens of the replaced part. Since the text is composed of the same tokens except for the replaced part, it can be said that the similarity of the word embedded representation (vector) representing the entire text is derived from the difference in the replaced part.

[0027] Note that word embeddings representing the entire text can be obtained, for example, by using BERT (Bidirectional Encoder Representations from Transformers) described in Non-Patent Document 2. In this case, since word embeddings for the entire text cannot be calculated until the tokens are actually replaced, a very large amount of calculation is required to consider all possibilities, so that efficient processing is achieved by performing masked prediction using a pre-training model using MLM (masked language model) and limiting calculations to tokens that are likely to fit the replaced parts. Each configuration will be described below.

[0028] The pre-training model 11 is a functional unit that stores the results of pre-training. The pre-training model 11 can be created using BERT (i.e., using MLM). Hereinafter, the pre-training model 11 created using BERT will also be referred to as a "BERT-trained model." In addition, in this embodiment, the pre-training model 11 is assumed to hold the results of learning a Japanese language model from a large amount of Japanese text.

[0029] The tokenizer 12 is a functional unit that breaks down text into tokens (including conversion to token IDs). Similarly to the technology described in Non-Patent Document 3, the tokenizer 12 breaks down text into words and further breaks down into subwords. As described above, the subwords here become tokens. Note that in order to utilize the BERT trained model (pre-training model 11), the broken down tokens are converted into token IDs to form the token ID string.

[0030] The embedding unit 13 is a functional unit that calculates word embeddings (vectors) from a token ID string. The beginning of the token ID string is the [CLS] token, and the vector corresponding to this token is the embedding of the entire text.

[0031] The mask prediction unit 14 is a functional unit that predicts tokens that are likely to correspond to a mask portion by using the pre-learning model 11. A mask portion is a portion in a token ID string whose value is the [MASK] token.

[0032] The above-mentioned tokenizer 12, embedding unit 13, and mask prediction unit 14 can be realized using BERT.

[0033] The processing unit 15 is a functional unit that creates information for data extension by controlling the tokenizer 12, the embedding unit 13, and the mask prediction unit 14. Details of the processing unit 15 will be described in the section on operation.

[0034] The input file 16 is a file (e.g., a txt file) that is the source of data expansion. In this embodiment, the input file 16 is assumed to be text data (txt file). The input file 16 is, for example, data as shown in FIG. 8.

[0035] The output file 17 is a file for recording the processing results. The output file 17 is, for example, data as shown in FIG.

[0036] (A-2) Operation of the embodiment Next, the operation of the data extension information acquisition device 1 according to the embodiment having the above configuration will be described.

[0037] (A-2-1) Characteristic operation of the data expansion information acquisition device 1 2 is a flowchart showing the characteristic operation of the data extension information acquisition device according to the embodiment. In the following, for the sake of simplicity, an example will be shown in which the processing unit 15 processes (processes and writes to the output file 17) one line of text in the input file 16. Note that when all lines of the input file 16 are to be processed, the following series of processes are simply repeated.

[0038] <Step S101> First, the processing unit 15 breaks down the text of the input file 16 into tokens using the tokenizer 12, and acquires a token ID string. Note that it is assumed that at this point, a [CLS] token is added to the beginning of the token string.

[0039] Fig. 3 is an explanatory diagram showing an example of a token ID string obtained using a tokenizer according to an embodiment. In Fig. 3, text ( / delicious / and / reputation / of / store) broken down into tokens is shown together with the leading [CLS] token. Note that the slash " / " indicates a separator between tokens. That is, in the text "A store with a reputation for deliciousness," "delicious," "and," "reputation," "of," and "store" are broken down into five tokens in addition to the [CLS] token.

[0040] <Step S102> Next, the processing unit 15 acquires word embeddings (vectors) of the token ID string acquired in the above-mentioned step S101, using the embedding unit 13. At this time, the processing unit 15 stores the word embeddings of the entire text, which are acquired as vectors of [CLS] tokens.

[0041] <Step S103> Next, the processing unit 15 replaces any one token in the token ID string acquired in the above-mentioned step S101 with two [MASK] tokens, and performs prediction in the mask prediction unit 14. Fig. 4 is an explanatory diagram showing an example in which a part of the token string according to the embodiment is masked. Fig. 4 shows an example in which the token "delicious" is replaced with two [MASK] tokens.

[0042] <Step S104> Next, the processing unit 15 stores the top three prediction results in order of likelihood, with the first prediction result of the mask designated as mask A and the second prediction result designated as mask B, in an array.

[0043] Fig. 5 is an explanatory diagram showing an example of an array storing prediction results of masked portions of a token ID string according to an embodiment. Fig. 5 shows an example in which prediction results of "oi", "mai", and "uma" are stored in mask A, and "##shii", "#i", and "da" are stored in mask B. Note that Fig. 5 is only an example, and the format (data format) for storing the prediction results is not particularly limited.

[0044] <Steps S105 to S110> The processing unit 15 creates a token ID string by replacing the masked portion while changing the subscript of mask A from 1 to 3 (step S105) and similarly changing the subscript of mask B from 1 to 3 (step S106), and calculates a vector of the entire text in the same manner as in step S102 described above (step S107). The processing unit 15 calculates the similarity (here, cosine similarity is used, but the comparison means is not particularly limited) between the calculated vector and the vector of the original text stored in step S102 described above, and stores it as a similarity table (step S108).

[0045] The processing unit 15 repeats the above process by returning to step S106 until B (mask B) is finished (step S109), and similarly returns to step S105 and repeats the above process until A (mask A) is finished (step S110).

[0046] Fig. 6 is an explanatory diagram showing an example of a token ID string in which the masked portion according to the embodiment is replaced with a predicted result. Note that, although what is actually created here is a token ID string, in Fig. 6, the leading [CLS] token is omitted for ease of explanation. Fig. 7 is an explanatory diagram showing an example of a similarity table showing the similarity of the masked portion of each token ID string in Fig. 6. Fig. 7 shows the cosine similarity for combinations (6 out of a total of 9) of the predicted results of three masks A and the predicted results of three masks B.

[0047] <Step S111> Finally, the processing unit 15 creates text from the token ID string with the highest similarity by referring to the similarity table created in step S108 described above. This text and information on which part was changed and how it was changed are written to the output file 17. For example, in the case of Fig. 7, A=1, B=1 have the highest similarity, so the text "A restaurant with a reputation for being delicious" is created as can be seen from Fig. 6. Furthermore, since "delicious" in the original text "A restaurant with a reputation for being delicious" has been replaced with "delicious", these are written to the output file 17 together with the created text.

[0048] This completes the operation of the data extension information acquisition device 1.

[0049] (A-2-2) Supplementary explanation of data expansion The following provides a supplementary explanation of data expansion (e.g., how to utilize the output file 17).

[0050] Fig. 9 is an explanatory diagram showing an example of the output file 17 created by the processes in steps S101 to S111 described above from the input file (two lines of data) in Fig. 8. There are various tasks in natural language processing, and the format of the training data differs depending on the task, but data expansion can be easily performed with data such as that shown in Fig. 9.

[0051] For example, the case where this is applied to training data for a text classification task is shown in Fig. 10. Fig. 10 shows that created data 22 has been created from original data 21. By combining the original data 21 and the created data 22 in Fig. 10, the amount of data doubles, achieving data expansion.

[0052] By extracting only the text from the original data 21 in Fig. 10, text data (input file 16) as shown in Fig. 8 can be created, and by processing this text data in sequence from step S101 described above, data (output file 17) as shown in Fig. 9 can be obtained. By using the obtained output file 17 and replacing the text portion in the original data 21 with the created text in Fig. 9 and using the labels as they are, created data 22 (new training data) can be obtained.

[0053] Furthermore, an example of application to training data for a named entity extraction task is shown in Fig. 11. Fig. 11 shows that created data 24 has been created from original data 23. In this example, tokens are in character units, and one token is one line, but by concatenating the tokens, text data as shown in Fig. 8 can be created, and by processing this text data in sequence from step S101 described above, data as shown in Fig. 9 can be obtained. Using this, tokens can be replaced with tokens obtained when the created text in Fig. 9 is decomposed, and label correspondence processing can be performed to obtain created data 24.

[0054] The above-mentioned process can be realized, for example, by a data augmentation information acquisition device 1A shown in Fig. 17. The data augmentation information acquisition device 1A has teacher data 30, a text data extraction unit 31, a data reconstruction unit 32, and new teacher data 33 in addition to the configuration of the data augmentation information acquisition device 1.

[0055] The teacher data 30 corresponds to teacher data such as the original data 21 in Fig. 10 or the original data 23 in Fig. 11. The new teacher data 33 corresponds to teacher data such as the created data 22 in Fig. 10 or the created data 24 in Fig. 11.

[0056] The text data extraction unit 31 is a functional unit that extracts text portions (text data) from the teacher data 30. For example, if the original data 21 includes text and labels, the text data extraction unit 31 extracts only text data as described above. Also, if the original data 23 includes character-based tokens and labels for each token, the text data extraction unit 31 extracts text data by concatenating the tokens. In either case, the means for extracting text data from the teacher data 30 is not particularly limited. The extracted text data becomes the input file 16.

[0057] The data reconstruction unit 32 is a functional unit that creates new teacher data 33 (reconstructs teacher data) based on the output file 17 and teacher data 30. For example, if the original data 21 includes text and labels, the data reconstruction unit 32 creates new teacher data 33 by replacing the existing text portion in the teacher data 30 with the created text in the output file 17 (the created text in FIG. 9 ), as described above.

[0058] (A-3) Effects of the embodiment According to this embodiment, the following effects are obtained.

[0059] The data augmentation information acquisition device 1 predicts the masked portion based on the language model acquired by pre-learning, and therefore can create text appropriate for the language in which pre-learning was performed. In other words, it is possible to prevent non-sentences from being created.

[0060] Furthermore, since the data augmentation information acquisition device 1 uses the similarity of the entire text before and after token replacement, it is possible to prevent the creation of text with a significantly different meaning.

[0061] Furthermore, the data augmentation information acquisition device 1 has a wide range of application, being applicable to any tokenizer, not limited to the use of the tokenizer described in Non-Patent Document 3. In addition, the data augmentation information acquisition device 1 has no restrictions on the number of tokens before and after token replacement, and includes a method of replacing conventional tokens one-to-one, making it highly versatile and enabling more flexible data expansion.

[0062] Furthermore, by performing data augmentation using the information acquisition device for data augmentation 1, a large amount of teacher data can be created, thereby enabling highly accurate learning to be achieved in machine learning such as deep learning.

[0063] (B) Other embodiments Although various modified embodiments have been mentioned in the above-described embodiment, the present invention can also be applied to the following modified embodiments.

[0064] (B-1) In the above embodiment, the data expansion information acquisition device 1 treats a subword as one token, but is not limited to this. That is, a token may be, for example, a word unit or a character unit.

[0065] (B-2) In the above embodiment, the case where one token is replaced with two masks has been described as an example, but the present invention is not limited to this. The same processing can be performed when replacing any number of tokens with any number of masks. However, in the above embodiment, since there are two masks, the processing is a double loop (steps S105, S106, S109, and S110 described above), so it is clear that when replacing with N masks, it will be an N-fold loop. Figure 12 shows an example of the number of tokens to be replaced and the token string after replacement when the number of tokens to be replaced is changed in various ways. Figure 12 shows an example in which a certain token in the token ID string shown in Figure 3 above is changed. For example, when the number of tokens to be replaced is "2" and the number of tokens to be replaced is "3", an example is shown in which two tokens, "delicious" and "to", are replaced with three [MASK] tokens.

[0066] (B-3) In the above embodiment, the data augmentation information acquisition device 1 is configured to output the one with the highest similarity from the top three most likely combinations, but this is not limited to this. It is also possible to set in advance how many of the top items to handle, and further, it is also possible to set in advance how many items to output will be adopted from the top with the highest similarity.

[0067] (B-4) In the above embodiment, the data augmentation information acquisition device 1 is configured to set two masks and perform mask prediction only once, but this is not limited to this. That is, the data augmentation information acquisition device 1 may be configured to predict the remaining mask again while one of the masks is fixed in sequence by the prediction result.

[0068] An example of processing in this case is shown in Fig. 13 to Fig. 16. In this example, since the predicted value of "##し" is the highest in Fig. 13, a token string (Fig. 14) is created in which mask B is fixed at "##し", and mask A is predicted again. The prediction result is shown in Fig. 15. In this way, token strings are created in order from the highest predicted value in Fig. 13 with one mask fixed (Fig. 16), and the remaining masks are predicted again. Note that at this point, it is possible to limit the number of top results if necessary. Processing according to this procedure increases the number of mask predictions, which increases the processing time, but it is effective when a large number of cases are used, as it further suppresses the creation of non-sentences.

[0069] (B-5) Also, in the case of the modified example of (B-4), the number of masks and the top number of masks to be handled may be freely set (pre-set). Therefore, the data augmentation information acquisition device 1 may be configured to predict the masks once, and then predict the remaining masks again while fixing some masks in sequence as the prediction results. [Explanation of symbols]

[0070] 1, 1A...information acquisition device for data augmentation, 11...pre-learning model, 12...tokenizer, 13...embedding unit, 14...mask prediction unit, 15...processing unit, 16...input file, 17...output file, 21, 23...original data, 22, 24...created data, 30...teaching data, 31...text data extraction unit, 32...data reconstruction unit, 33...new teaching data

Claims

1. A pre-trained model that has been trained using a masked language model; a tokenizer that breaks down input text into tokens; an embedding unit that calculates embedded word representations of the entire input text; a mask prediction unit that predicts a mask portion in a token sequence obtained by decomposing the input text using the pre-learning model; a processing unit that calculates a similarity between a generated text created by repeatedly replacing one or more tokens in the token string with one or more mask tokens and predicting tokens corresponding to the masked portions using the mask prediction unit in the embedding unit and the input text, and outputs a predetermined number of the generated texts in descending order of the similarity; A data extension information acquisition device comprising:

2. 2. The data augmentation information acquisition device according to claim 1, wherein, when predicting a token corresponding to a masked portion using the mask prediction unit, the processing unit creates the created text by fixing a part of the masked portion as a prediction result and repeating the prediction of the remaining masked portion.

3. A text data extraction unit that extracts text data from the training data; The input text is the text data extracted from the teacher data by the text data extraction unit, a data reconstruction unit that reconstructs new teacher data from the created text created by the processing unit and the teacher data; 3. The data extension information acquisition device according to claim 1, further comprising:

4. The information acquisition device for data augmentation according to claim 3 , wherein the teaching data includes text and a label assigned to each text.

5. The information acquisition device for data augmentation according to claim 3 , wherein the teaching data includes tokens and labels assigned to each token.

6. Computer, A pre-trained model that has been trained using a masked language model; a tokenizer that breaks down input text into tokens; an embedding unit that calculates embedded word representations of the entire input text; a mask prediction unit that predicts a mask portion in a token sequence obtained by decomposing the input text using the pre-learning model; a processing unit that calculates a similarity between a generated text created by repeatedly replacing one or more tokens in the token string with one or more mask tokens and predicting tokens corresponding to the masked portions using the mask prediction unit in the embedding unit and the input text, and outputs a predetermined number of the generated texts in descending order of the similarity; The information acquisition program for data expansion is characterized by functioning as follows.

Images