Learning device, learning method, and program

Abstract

This learning device is provided with: a conversion unit that converts inputted text data to a feature quantity sequence on the basis of a language model; and a parameter update unit that updates parameters of the language model on the basis of the text data, the feature quantity sequence, and a pre-trained word vector.

Description

[Technical Field]

[0001] The present invention relates to a learning device, a learning method, and a program. [Background technology]

[0002] Pre-trained language models, which are models that have been pre-trained (pre-training 1) using a large corpus, are known to perform well in various tasks in natural language processing. Generally, pre-training 1 requires a huge amount of computational resources. Pre-trained language models trained using corpora related to general domains are publicly available and can be downloaded and used. However, if the target task belongs to a specialized domain such as medicine or law, pre-trained language models specialized for that domain are not publicly available, and so attempting to train pre-training 1 using a corpus specialized for that domain requires a huge amount of training time.

[0003] Therefore, attempts have been made to obtain a learning model specialized for a specialized domain with higher performance in a short learning time. For example, Non-Patent Document 1 discloses a method of performing pre-learning 2 using training data for a target task between pre-learning 1 and fine-tuning. Note that fine-tuning is a procedure of performing learning using training data for the target task and retraining a language model into a model for the target task. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Suchin Gururangan, Ana Marasovic, SwabhaSwayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. In ACL, pp.8342-8360, July 2020. Summary of the Invention [Problem to be solved by the invention]

[0005] However, the performance of the language model in Non-Patent Document 1 is still insufficient compared to the performance of pre-trained language models specialized in specialized domains.

[0006] The disclosed technology aims to obtain a language model with higher performance while suppressing the lengthening of the learning time. [Means for solving the problem]

[0007] The disclosed technology includes a conversion unit that converts input text data into a feature sequence based on a language model, and a text data processing unit that converts input text data into a feature sequence based on a language model. 、 and the word vectors of the target task domain is used as an input, and a feature sequence that is the output of the language model for the input is Based on Calculating a loss and, based on the slope of the loss, an intermediate representation of the language model and, The word vector Similar to and a parameter update unit that updates the parameters of the language model in the manner described above. [Effects of the Invention]

[0008] It is possible to obtain a language model with higher performance while suppressing the lengthening of the learning time. [Brief explanation of the drawings]

[0009] [Figure 1]FIG. 1 is a first diagram illustrating the functional configuration of a learning device. [Figure 2] FIG. 2 is a second diagram showing the functional configuration of the learning device. [Figure 3] FIG. 10 is a third diagram showing the functional configuration of the learning device. [Figure 4] 10 is a flowchart showing an example of the flow of a parameter update process in pre-learning 1 according to the prior art. [Figure 5] 10 is a flowchart showing an example of the flow of parameter update processing in pre-learning 2 in the prior art. [Figure 6] 10 is a flowchart showing an example of the flow of a parameter update process in pre-learning 2 according to the present embodiment. [Figure 7] 10 is a flowchart showing an example of the flow of a similarity calculation process according to Modification 1. [Figure 8] 10 is a flowchart showing an example of the flow of a similarity calculation process according to Modification 2. [Figure 9] 10 is a flowchart showing an example of the flow of a parameter update process in Fine-Tuning. [Figure 10] FIG. 2 illustrates an example of a hardware configuration of a learning device. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0011] In the following description, references are indicated as [1], [2], etc. The names of the references are listed at the end of the specification. In this specification, a character with an X followed by a "-" is referred to as " - It is written as "X".

[0012] (Functional configuration of the learning device) The learning device 10 according to this embodiment updates the parameters of a language model through machine learning using word vectors.

[0013] 1 to 3 are diagrams illustrating the functional configuration of the learning device.

[0014] The learning device 10 is a device for learning a language model 112. The language model 112 to be learned is used to convert an input text sequence into a feature sequence (sequence length is 1 or more). The learning device 10 learns the language model 112 as shown in FIG. 1 or FIG. 2, and performs fine-tuning on the learned language model 112 (trained language model) using training data for a target task as shown in FIG. 3. This makes it possible to obtain a model corresponding to any target task. The target task may be, for example, text classification, text extraction such as reading comprehension, or text generation such as translation or summarization.

[0015] Specifically, the learning device 10 updates the parameters of the language model 112 by performing machine learning based on text extracted from a corpus 20 as shown in Figure 1 (hereinafter referred to as pre-learning 1), machine learning based on text data extracted from teacher data 30 as shown in Figure 2 (hereinafter referred to as pre-learning 2), or fine-tuning based on text data extracted from teacher data 30 and correct answer data as shown in Figure 3.

[0016] A corpus 20 is a collection of text data related to a general domain (topic or field) or a specialized domain.

[0017] The training data 30 is training data for Fine-Tuning, and is data in which input text and correct answer data are associated with each other.

[0018] The learning device 10 includes a storage unit 11, an acquisition unit 12, a mask unit 13, a conversion unit 14, a word vector acquisition unit 15, and a parameter update unit 16.

[0019] The storage unit 11 stores various data, specifically, a language model 112 and a word vector 111.

[0020] In pre-learning 1, the acquisition unit 12 acquires text data from the corpus 20, and in pre-learning 2, acquires text data included in the teacher data of the target task from the teacher data 30. In addition, in Fine-Tuning, the acquisition unit 12 acquires the teacher data of the target task from the teacher data 30. Note that in pre-learning 2, the correct answer data included in the teacher data 30 is not necessary, and therefore the acquisition unit 12 may acquire text data in the same domain as the teacher data 30 from data other than the teacher data 30, instead of the text data included in the teacher data 30 of the target task.

[0021] The masking unit 13 masks a part of the input text obtained from the training data 30 in pre-learning 1 or pre-learning 2. The masking unit 13 performs a masking process using a technique called a masked language model [2].

[0022] In Fine-Tuning, the conversion unit 14 converts the text data acquired by the acquisition unit 12. In pre-learning 1 or pre-learning 2, the conversion unit 14 converts the text data masked by the mask unit 13.

[0023] The word vector acquisition unit 15 acquires word vectors to be used in pre-learning 1 or pre-learning 2. A word vector is a vector that represents the meaning of a word, and known examples include word2vec[4]. It is known that word vectors can represent the meaning of a word in a vector space; for example, if you subtract the vector representing "man" from the vector representing "king" and add the vector representing "woman", a vector similar to the vector representing "queen" will appear.

[0024] Specifically, the word vector acquisition unit 15 may acquire a word vector that has already been generated as a word vector specialized for the domain belonging to the target task, or may acquire the vector by machine learning based on a corpus or the like.

[0025] When word vectors are acquired by machine learning, the word vector acquisition unit 15 acquires word vectors that represent the meanings of words in the domain of the target task by learning word vectors using a collection of input texts of training data for the target task as a corpus.

[0026] The word vector acquisition unit 15 can use any method for learning word vectors. In particular, fastText [5] is known to be capable of learning from a small corpus. By using publicly available fastText word vectors as initial values ​​and training fastText on a set of input texts in the training data for the target task, it is possible to efficiently acquire word vectors for the target task.

[0027] The learning device 10 does not have to include the word vector acquisition unit 15, and for example, another device (for example, a word vector acquisition device) may have the same function as the word vector acquisition unit 15.

[0028] The parameter update unit 16 updates the parameters of the language model 112 in pre-learning 1, pre-learning 2, or fine-tuning.

[0029] (Traditional pre-learning 1) Next, a description will be given of conventional pre-learning 1, which serves as a basis for comparison. Fig. 4 is a flowchart showing an example of the flow of parameter update processing in pre-learning 1 in the prior art.

[0030] In the conventional pre-learning 1, the acquisition unit 12 acquires a set of data of input text X from the corpus 20 (step S11). Next, the learning device 10 selects one input text X from the set of data as a processing target (step S12). The masking unit 13 masks the input text X and generates the masked text X. - X is obtained (step S13).

[0031] The conversion unit 14 converts the masked text based on the parameters of the language model 112 stored in the storage unit 11, and converts the feature sequence H( - X) is obtained (step S14).

[0032] The parameter update unit 16 calculates the loss from the feature sequence H and the input text X, and updates the parameters of the language model 112 stored in the storage unit 11 based on the gradient of the loss (step S15).

[0033] The learning device 10 determines whether all input texts X included in the set data have been processed (step S16). If the learning device 10 determines that any of the input texts X included in the set data have not been processed (step S16: No), the learning device 10 returns to the processing of step S12 and selects the next input text X.

[0034] When the learning device 10 determines that all input texts X included in the set data have been processed (step S16: Yes), it further determines whether they have been processed a specified number of times (step S17). When the learning device 10 determines that they have not been processed the specified number of times (step S17: No), it regards all input texts X as unprocessed (step S18). It then returns to the processing of step S12.

[0035] If the learning device 10 determines that the process has been performed the specified number of times (step S18: Yes), the process ends.

[0036] (Traditional pre-learning 2) 5 is a flowchart showing an example of the flow of parameter update processing in prior art pre-learning 2. Pre-learning 2 is executed on the language model 112 trained in pre-learning 1.

[0037] The acquisition unit 12 acquires a set of data of input text X from the training data 30 (step S21). Next, the learning device 10 selects one input text X from the set of data as a processing target (step S22). The masking unit 13 masks the input text X and generates the masked text X. - X is obtained (step S23).

[0038] The conversion unit 14 converts the masked text based on the parameters of the language model 112 stored in the storage unit 11, and converts the feature sequence H( - X) is obtained (step S24).

[0039] The parameter update unit 16 calculates the loss from the feature sequence H and the input text X, and updates the parameters of the language model 112 stored in the storage unit 11 based on the gradient of the loss (step S25).

[0040] The learning device 10 determines whether all input texts X included in the set data have been processed (step S26). If the learning device 10 determines that any of the input texts X included in the set data have not been processed (step S26: No), the learning device 10 returns to the processing of step S22 and selects the next input text X.

[0041] When the learning device 10 determines that all input texts X included in the set data have been processed (step S26: Yes), it further determines whether they have been processed a specified number of times (step S27). When the learning device 10 determines that they have not been processed the specified number of times (step S27: No), it regards all input texts X as unprocessed (step S28). It then returns to the processing of step S22.

[0042] If the learning device 10 determines that the process has been performed the specified number of times (step S27: Yes), the process ends.

[0043] (Pre-learning 2 according to this embodiment) 6 is a flowchart showing an example of the flow of parameter update processing in pre-learning 2 according to this embodiment. Pre-learning 2 according to this embodiment is executed on the language model 112 trained in conventional pre-learning 1. Hereinafter, when it is necessary to distinguish between the prior art and this embodiment, pre-learning 2 will be referred to as "conventional pre-learning 2" and "pre-learning 2 according to this embodiment." When a description is common to both, it will be referred to as "pre-learning 2."

[0044] The acquisition unit 12 acquires a set of data of input text X and output text Y from the training data 30 (step S31). Here, the input text X is in the same domain as the text to be processed in the target task. Next, the learning device 10 selects one input text X from the set of data as the processing target (step S32). The masking unit 13 masks the input text X and generates the masked text Y. - X is obtained (step S33).

[0045] The conversion unit 14 converts the masked text based on the parameters of the language model 112 stored in the storage unit 11, and converts the feature sequence H( - X) is obtained (step S34).

[0046] The parameter update unit 16 calculates a loss from the feature sequence H, the input text X, and the word vector E, and updates the parameters of the language model 112 stored in the storage unit 11 based on the gradient of the loss (step S35). Here, the parameter update unit 16 uses the word vectors acquired by the word vector acquisition unit 15. A specific parameter update method will be described later.

[0047] The learning device 10 determines whether all input texts X included in the set data have been processed (step S36). If the learning device 10 determines that any of the input texts X included in the set data have not been processed (step S36: No), the learning device 10 returns to the processing of step S32 and selects the next input text X.

[0048] When learning device 10 determines that all input texts X included in the set data have been processed (step S36: Yes), it further determines whether they have been processed a specified number of times (step S37). When learning device 10 determines that they have not been processed the specified number of times (step S37: No), it regards all input texts X as unprocessed (step S38). It then returns to the processing of step S32.

[0049] If the learning device 10 determines that the process has been performed the specified number of times (step S37: Yes), the process ends.

[0050] Note that the conventional pre-learning 1 and pre-learning 2, and the pre-learning 2 according to this embodiment, employ a method called a masked language model. In the masked language model, a vocabulary is V, and text is a sequence of words X∈V l where l is the word sequence length. The input of the masked language model is a sequence of X with perturbations (such as masking some tokens). - X. The masked language model is - It is a language model that predicts the unperturbed token sequence X from X.

[0051] The output of the masked language model is the matrix

[0052]

number

[0053]

number

[0054] Next, the parameter update process in step S35 of pre-learning 2 according to this embodiment will be described in detail.

[0055]

number

[0056] Here, d1 is the dimension of the intermediate representation. Any intermediate representation of a language model can be used as Z. Here, the output of the word embedding layer is used, but the output of any layer can be used. Note that the intermediate representation may include representations of each layer, including the final layer. Note that the prediction score of the final layer can also be considered an intermediate representation, since it is subsequently linearly converted to the dimension of the vocabulary size.

[0057] The matrix of word vectors acquired by the word vector acquisition unit 15 is

[0058]

number

[0059]

number

[0060] Also, another function,

[0061]

number

[0062] Here, two types of functions are exemplified. t Any function can be used as long as it allows learning to proceed so that the word vectors of the two words are similar.

[0063] According to the learning device 10 of this embodiment, parameter updating using word vectors is applied to pre-learning 2. While learning a language model itself in the domain to which the target task belongs requires computational costs, the cost of learning word vectors is low, making it relatively easy to acquire domain-specific word vectors. Furthermore, by using domain-specific word vectors, the depth of learning in pre-learning 2 is improved, and a language model with higher performance can be obtained.

[0064] For example, typical word vector learning methods use a language model consisting of two layers: an input layer and an output layer. In contrast, pre-trained language models are composed of ultra-multilayered neural networks; for example, BERT is a 14-layer neural network that combines a 12-layer neural network with a word embedding layer (input layer) and an output layer. Therefore, compared to language models, word vectors can be learned with a smaller amount of data, and the computational cost per data point is also lower.

[0065] The technology of this embodiment uses separately acquired word vectors that represent the meanings of words in the target task domain to perform pre-training on a pre-trained language model that is trained using a masked language model, taking into account word vectors, thereby making it possible to train a language model that can solve the target task with high accuracy.

[0066] This embodiment is characterized by the use of word vectors, which are easier to learn than when the language model 112 is trained by conventional pre-training 2. The ability to train from a small amount of data is an important feature in setting up pre-training 2, which trains using text from the domain of the target task, which has a small amount of data. This embodiment has the effect of efficiently learning the meanings of words in the domain even from a small amount of data by explicitly learning the similarity with word vectors when performing pre-training 2.

[0067] This embodiment can also be applied to conventional pre-learning 1. When this embodiment is applied to conventional pre-learning 1, the use of word vectors has the effect of reducing the enormous amount of calculation time and corpus size required for pre-learning 1. Furthermore, the objective function of this embodiment can be considered as regularization using word vectors, which has the effect of promoting generalization of learning and improving performance in the target task. Hereinafter, when it is necessary to distinguish between the prior art and this embodiment, pre-learning 1 will be referred to as "conventional pre-learning 1" and "pre-learning 1 according to this embodiment." When a description is common to both, it will be referred to as "pre-learning 1."

[0068] Furthermore, in training word vectors, for example, the corpus (text data) used for training fastText [5] can be larger than the total corpus used for training a language model. By using a larger corpus for training, word vectors can acquire knowledge that a language model cannot learn.

[0069] (Variation 1) The granularity of word segmentation in a language model and the granularity of word segmentation in a word vector may differ. This is because many language models use subwords as the granularity of word segmentation. For example, the word "suddenly" may not be segmented in the word segmentation in a word vector, and a vector equivalent to "suddenly" exists, but the word segmentation in a language model may further segment it into "sudden" and "ly." This section describes a modified example that corresponds to the case where a word sequence X is segmented at a finer granularity than words as a result of word segmentation in a language model.

[0070] 7 is a flowchart showing an example of the flow of a similarity calculation process according to Modification 1. Modification 1 is an example of a case where a learning method capable of acquiring word vectors in subword units, such as fastText, is employed.

[0071] In the similarity calculation according to the first modification, the parameter update unit 16 determines whether the input text X being processed is divided into words at a finer granularity than the words registered in the language model 112 (step S41). If the parameter update unit 16 determines that the input text X is divided into words at a finer granularity (step S41: Yes), the parameter update unit 16 acquires vectors at a subword level granularity for both the word vectors and the language model intermediate representation (step S42). Then, the parameter update unit 16 calculates the similarity of the vectors at the subword level according to the loss function for each pattern described above (step S43).

[0072] If the parameter update unit 16 determines that the word vectors and the language model intermediate representation are not divided into finer granularities (step S41: No), the parameter update unit 16 acquires vectors at the word level granularity for both the word vectors and the language model intermediate representation (step S44).Then, the parameter update unit 16 calculates the similarity of the vectors at the word level according to the loss function for each pattern described above (step S45).

[0073] 8 is a flowchart showing an example of the flow of a similarity calculation process according to Modification 2. Modification 2 is an example of a case where a learning method that cannot acquire word vectors in subword units is employed.

[0074] In the similarity calculation according to the second modification, the parameter update unit 16 determines whether the input text X being processed is divided into words at a finer granularity than the words registered in the language model 112 (step S51). If the parameter update unit 16 determines that the input text X is divided into words at a finer granularity (step S51: Yes), the parameter update unit 16 acquires word vectors at a word-level granularity and vectors at a subword-level granularity for the language model intermediate representation (step S52).

[0075] Then, the parameter update unit 16 calculates the average value of the language model intermediate representations for the subwords belonging to the same word, and sets the average value as the word-level language model intermediate representation (step S53).The parameter update unit 16 calculates the similarity of the vectors at the word level according to the loss function for each pattern described above (step S54).

[0076] If the parameter update unit 16 determines that the word vectors and the language model intermediate representation are not divided into finer granularities (step S51: No), the parameter update unit 16 acquires vectors at the word level granularity for both the word vectors and the language model intermediate representation (step S55).Then, the parameter update unit 16 calculates the similarity of the vectors at the word level according to the loss function for each pattern described above (step S56).

[0077] In addition, L WELM and L WENorm calculates the average value for all words, but it is also possible to narrow down the target words to be calculated. For example, let S be the set of target words, and m be the number of words in S in the word sequence X. In this case, each function is

[0078]

number

[0079]

number

[0080] The set of target words S can be "low-frequency words whose occurrence count is below a threshold" or "words that are divided into smaller words by the word division of the language model." This can avoid negatively affecting function words that are unrelated to the domain, such as "is," or words that have been sufficiently trained in pre-training 1.

[0081] During training, the loss function of the original masked language model is used by adding a function that takes into account word vectors. For example, L WELM +λL WENorm λ is a hyperparameter that indicates the strength of considering word vectors, and is, for example, 1.

[0082] 9 is a flowchart showing an example of the flow of parameter update processing in Fine-Tuning. Fine-Tuning is a process of adjusting the language model 112 trained by pre-training 2 so that it can perform processing specialized for the target task (specifically, after removing or adding layers from the trained language model in accordance with the target task, re-training is performed using training data for the target task).

[0083] The acquisition unit 12 acquires set data of input text X and output text Y from the training data 30 (step S61). Next, the learning device 10 selects one input text X from the set data as a processing target (step S62).

[0084] The conversion unit 14 performs language processing on the input text X to obtain a feature sequence H'(X) (step S63). Note that the format of the feature sequence score H'(X) during Fine-Tuning differs depending on the target task.

[0085] The parameter update unit 16 calculates the loss from the feature sequence H'(X) and the output text Y, and updates the parameters based on the gradient of the loss (step S64).

[0086] The learning device 10 determines whether all input texts X included in the set data have been processed (step S65). If the learning device 10 determines that any of the input texts X included in the set data have not been processed (step S65: No), the learning device 10 returns to the processing of step S62 and selects the next input text X.

[0087] When the learning device 10 determines that all input texts X included in the set data have been processed (step S65: Yes), it further determines whether they have been processed a specified number of times (step S66). When the learning device 10 determines that they have not been processed the specified number of times (step S66: No), it regards all input texts X as unprocessed (step S67). It then returns to the processing of step S62.

[0088] If the learning device 10 determines that the process has been performed the specified number of times (step S66: Yes), the process ends.

[0089] (Example of hardware configuration) The learning device 10 can be realized, for example, by causing a computer to execute a program that describes the processing described in this embodiment. Note that this "computer" may be a physical machine or a virtual machine on the cloud. When a virtual machine is used, the "hardware" described here is virtual hardware.

[0090] The above program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The above program can also be provided via a network such as the Internet or email.

[0091] Fig. 10 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 10 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B.

[0092] A program for realizing processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0093] When a program startup instruction is received, the memory device 1003 reads and stores the program from the auxiliary storage device 1002. The CPU 1004 implements the functions of the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network. The display device 1006 displays a program-based GUI (Graphical User Interface), etc. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs calculation results. Note that the computer may be equipped with a GPU (Graphics Processing Unit) instead of the CPU 1004, or may be equipped with a GPU in addition to the CPU 1004. In this case, processing may be shared, for example, with the GPU executing processing requiring special calculations such as neural networks, and the CPU 1004 executing other processing.

[0094] (Experimental results) We conducted experiments on the BioASQ5b dataset [6], a question-answering task in the medical domain. The results are shown in Table 1. The evaluation metrics are SACC (the accuracy rate of the top-1 prediction), LACC (the rate at which the correct answer is included in the top-5 predictions), and MRR (the average of the reciprocal of the rank of the correct answer). For all metrics, higher numbers indicate better performance.

[0095] [Table 1]

[0096] The methods compared are (1) the simplest method (conventional pre-training 1 + Fine-Tuning), (2) a domain-specific model that uses a large amount of computational resources (pre-training 1 on a domain corpus + Fine-Tuning), and (3) conventional technology (conventional pre-training 1 + conventional pre-training 2 + Fine-Tuning).

[0097] The techniques according to this embodiment are (4) and (5), both of which are (conventional pre-learning 1 + pre-learning 2 according to this embodiment + Fine-Tuning).

[0098] Domain-specific models undergo pre-training specific to the domain, and therefore significantly outperform other models. Conventional techniques outperform the simplest methods by performing pre-training2, but fall short of the domain-specific models.

[0099] The learning device 10 according to this embodiment achieves performance superior to that of conventional techniques by performing pre-learning that takes into account word vectors instead of pre-learning 2, and has succeeded in narrowing the gap with domain-specific models.

[0100] In terms of computation time, this embodiment, like the prior art, completed the computation within a day using one GTX1080 Titan GPU (11GB). However, the domain-specific model [3] required more than 10 days of computation using eight V100 GPUs (16GB).

[0101] Furthermore, when using the published fastText as word vectors and performing training using the learning device 10 according to this embodiment as a replacement for the conventional pre-training 1:BERT, it was confirmed that the performance exceeded that of BERT. Specifically, in the question answering task SQuAD [7] in the Wikipedia domain, the performance of BERT and the learning device 10 according to this embodiment in SQuAD was 81.97% and 82.29% in exact match rate, and 89.34% and 89.50% in partial match rate (F1).

[0102] Since fastText's training cost is much lower than that of BERT, it is possible to train using a corpus that is too large for BERT, and as a result, it is assumed that it can learn knowledge that BERT cannot acquire. Note that BERT's training corpus is 3B tokens, while fastText's training corpus is 600B tokens.

[0103] In the prior art pre-training 2, a learning method that is originally based on learning from a huge corpus is applied to a small corpus (a collection of input texts of training data for the target task), so it is thought that the performance improvement is limited.

[0104] The learning device 10 according to this embodiment uses word vectors specific to the domain of the target task to efficiently insert domain knowledge of the target task into a language model from a small corpus. By obtaining word vectors from training data for the target task, word vectors that express the meaning of words in the target domain / task can be acquired. Furthermore, in pre-learning 2, the intermediate representation of the language model is trained to include word vector information, thereby achieving efficient learning with a small corpus.

[0105] In this embodiment, the explanation has been given mainly using an encoder-type language model (which can be used for target tasks such as text classification and text extraction) as an example, but it is also possible to apply the present invention to an encoder-decoder-type language model (which can be used for target tasks such as text generation).

[0106] [References] [1] Suchin Gururangan, Ana Marasovic, SwabhaSwayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. In ACL, pp. 8342-8360, July 2020. (Non-patent document 1) [2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL-HLT, pp. 4171-4186, 2019. [3] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, Vol. 36, No. 4, pp. 1234-1240, 2020. [4] Thomas Mikolov, Greg Corrado, Kai Chen, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space, pp. 107-111. 1-12, [5] Piotr Bojanowski, Edward Grave, Armand Joulin, and Thomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, Vol. 5, pp. 135–146, [6] George Tsatsaronis, George Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, Yannis Almirantis, John Pavlopoulos, Nicolas Basciotis, Patrick Gallinari, Thierry Artieres, Axel Ngonga, Norman Heino, Eric Gaussier, Liliana Barrio-Alvers, Michael Schroeder, Ion Androutsopoulos, and Georgios Paliouras. An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition. BMCBioinformatics, Vol. 16, p. 138 , [7] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, pp. 2383-2392, 2016.

[0107] (Addendum) This specification discloses at least the learning device, learning method, program, and storage medium described in the following appended claims. (Additional note 1) Memory and at least one processor coupled to said memory; Including, The processor: The input text data is converted into a feature sequence based on a language model, updating parameters of the language model based on the text data, the feature sequence, and pre-trained word vectors; Learning device. (Additional note 2) The word vectors are pre-trained based on a general corpus, The corpus is larger than the total amount of the input text data. Item 1. A learning device according to item 1. (Additional note 3) The input text data is from the same domain as the text data to be processed in the target task. Item 1. A learning device according to item 1. (Additional note 4) The processor: When the word vectors include word vectors of subwords obtained by dividing words, and the language model divides the text data into granularities finer than words, parameters of the language model are updated based on the word vectors of the subwords. 4. The learning device according to any one of claims 1 to 3. (Additional note 5) The processor: When the language model divides the text data into subwords that are finer than words, the language model updates parameters based on an average value of intermediate representations of the language model and the word vectors for subwords that belong to the same word. 4. The learning device according to any one of claims 1 to 3. (Additional note 6) 1. A computer-implemented method comprising: converting input text data into a feature sequence based on a language model; updating parameters of the language model based on the text data, the feature sequence, and pre-trained word vectors; How to learn. (Additional note 7) A program for causing a computer to function as each part of the learning device described in any one of appendixes 1 to 5. (Additional note 8) A non-transitory storage medium storing a program executable by a computer to perform a learning process, In the learning process, the computer The input text data is converted into a feature sequence based on a language model, updating parameters of the language model based on the text data, the feature sequence, and pre-trained word vectors; Non-transitory storage medium.

[0108] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0109] 10 Learning Device 11 Storage section 12 Acquisition Department 13 Mask section 14 Conversion unit 15 Word vector acquisition section 16 Parameter update section 20 Corpus 30 Training data

Claims

1. a conversion unit that converts input text data into a feature sequence based on a language model; a parameter update unit that receives text data in a domain of a target task and word vectors in the domain of the target task as input, calculates a loss based on a feature sequence that is an output of the language model for the input, and updates parameters of the language model based on a gradient of the loss so that an intermediate representation of the language model and the word vectors are similar to each other. Learning device.

2. The input text data is from the same domain as the text data to be processed in the target task. The learning device according to claim 1 .

3. the parameter update unit updates parameters of the language model based on the word vectors of the subwords when the word vectors include word vectors of subwords obtained by dividing a word, and the language model divides the text data into granularities finer than words. The learning device according to claim 1 or 2.

4. when the language model divides the text data into granularities finer than words, the parameter update unit updates parameters of the language model based on an average value of intermediate representations of the language model and the word vectors for subwords belonging to the same word. The learning device according to claim 1 or 2.

5. 1. A computer-implemented method comprising: converting input text data into a feature sequence based on a language model; receiving text data in a domain of a target task and word vectors in the domain of the target task as inputs, calculating a loss based on a feature sequence that is an output of the language model for the inputs, and updating parameters of the language model based on a gradient of the loss so that an intermediate representation of the language model and the word vectors are similar to each other; How to learn.

6. A program for causing a computer to function as each unit in the learning device according to any one of claims 1 to 4.

Images