Learning device, learning method and program

Abstract

A training device according to an embodiment comprises: a changing unit which changes a position number of a leading token of a series of tokens included in given training data, and calculates a series of position numbers in which position numbers are sequentially set from the changed position number; and a training unit which trains language model parameters including a position vector, which are a feature amount corresponding to the position number, or language model parameters for obtaining the position vector by using the series of position numbers and the series of tokens.

Description

[Technical Field]

[0001] The present invention relates to a learning device, a learning method, and a program. [Background technology]

[0002] In recent years, there has been a lot of research into language models, including BERT (Bidirectional Encoder Representations from Transformers). A language model is a neural network model that obtains distributed representations of tokens. Since a language model does not input a single token but the entire text divided into tokens as a token sequence, it is possible to obtain distributed representations that reflect the semantic relationships with other tokens in the token sequence. Such distributed representations are also called context features, and the step of learning them is called pre-training. A token is a unit of distributed representation, such as a word or subword.

[0003] Moreover, pre-trained distributed representations can be used to solve various tasks such as text classification and question answering. The process of solving such tasks is called fine-tuning.

[0004] Incidentally, BERT is based on a model called the Transformer, which is described in Non-Patent Document 1. The Transformer is a model that stacks layers composed of an attention mechanism, a fully connected layer, and the like. The Transformer model uses a mechanism called scaled dot-product attention for attention, but this mechanism does not take into account the order relationship of tokens. For this reason, the Transformer model defines a feature that represents the position of a token, called Position Encoding, and adds it to the token feature to take into account the order relationship of tokens. Hereinafter, such a feature that represents the token position will be referred to as a "position vector." A position vector is a vector that is uniquely determined for the position number i of a token in a token sequence. In addition to Position Encoding, distributed representation of the position number of a token in a token sequence and Position Embedding used in BERT are also specific examples of position vectors. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Proceedings of the NIPS 2017, pp. 5998-6008, 2017. Summary of the Invention [Problem to be solved by the invention]

[0006] Here, to learn the position vector of position number i, a valid token must exist at position number i in the token sequence. A valid token is a token used for model training, such as a word or subword obtained by segmenting natural language text. On the other hand, special tokens such as padding tokens are not valid tokens.

[0007] However, since training data generally includes both long and short texts, when position number i is large, there may be no valid token at that position number i. Therefore, when position number i is large, the position vector at that position number i may not be sufficiently trained.

[0008] For example, when adjusting the length of text included in training data to a certain fixed length, padding tokens are added to text whose length is less than L. Therefore, when L is a relatively large value, the closer position number i is to L, the more likely it is that no valid tokens exist, and the more likely it is that the position vector of position number i cannot be sufficiently trained.

[0009] An embodiment of the present invention has been made in view of the above points, and aims to realize sufficient learning of position vectors that represent token positions in a token sequence. [Means for solving the problem]

[0010] In order to achieve the above object, a learning device according to one embodiment includes: a modification unit that modifies the position number of the first token in a token sequence included in given learning data, and calculates a position number sequence in which position numbers are set in order from the modified position number; and a learning unit that uses the position number sequence and the token sequence to learn language model parameters that include a position vector, which is a feature corresponding to the position number, or language model parameters for obtaining the position vector. [Effects of the Invention]

[0011] A sufficient learning of the position vector representing the token position in the token sequence can be achieved. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 10 is a diagram illustrating an example of changing a position number. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a learning device according to the present embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of the functional configuration of a learning device according to the present embodiment. [Figure 4] 10 is a flowchart illustrating an example of a learning process according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] An embodiment of the present invention will be described below. In this embodiment, a learning device 10 will be described that can achieve sufficient learning of position vectors, which are features that represent token positions in a token sequence, when learning a model to solve a certain task. Note that a token refers to a predetermined processing unit, such as a word or subword, used when dividing text. When processing text information using a neural network, each token is converted into a distributed representation for use. Furthermore, a position vector is a feature vector that is uniquely determined for the position number i of a token in a token sequence. Specific examples include Position Encoding described in Non-Patent Document 1, a distributed representation of the position number of a token in a token sequence, and Position Embedding used in BERT described in Reference 1. The position vector may be part of the model parameters of a language model, such as Position Embedding described in Reference 1, or may be a feature vector such as Position Encoding PE described in Non-Patent Document 1. (pos,2i) and P.E. (pos,2i+1) It can also be calculated from model parameters as follows:

[0014] Here, during model training, the learning device 10 according to this embodiment sets the position number of the first token in a token sequence to a random value and starts allocating position vectors from that position number. For example, as shown in FIG. 1, a token sequence of "XYZ," "cooking," "of," "time," "desu," ".", "now," ",", "today," "of," "cooking," and "is" is input to a language model. In this case, in the prior art, position numbers are set in order starting from "1," such that the position number of the first token "XYZ" is "1," the position number of the next token "cooking" is "2," the position number of the token after that is "of" is "3," and so on. In contrast, the learning device 10 according to this embodiment sets the position numbers in order starting from the random value "j," such that the position number of the first token is set to a random value j, the position number of the next token is "j+1," the position number of the token after that is "j+2," and so on. In the example shown in FIG. 1, the random value j is set to 91.

[0015] This increases the proportion of valid tokens that appear at larger position numbers i, making it possible to fully learn the position vector of that position number i. For example, in the example shown in Figure 1, a valid token "XYZ" exists at position number "91," making it possible to fully learn the position vector of position number "91" compared to conventional technology. Note that token position numbers are set sequentially from the first token, so hereinafter the position of the first token will be referred to as the "start position" and that number as the "start number."

[0016] Note that a valid token is a token used in model training, such as a word or subword obtained by segmenting natural language text. On the other hand, special tokens such as padding tokens are not valid tokens. A padding token is a token added to the end of a token sequence in padding processing to align the token sequence of training data to a certain length (e.g., a length specified as a training condition, the maximum length within a mini-batch when performing mini-batch training, etc.). Since this padding token is added for purposes such as improving the efficiency of calculations in model training, the calculation result is set to 0 so that it is not used in model training. In other words, tokens such as padding tokens that result in a calculation result of 0 in model training are not valid tokens, and all other tokens are valid tokens.

[0017] <Hardware configuration of the learning device 10> The hardware configuration of learning device 10 according to this embodiment is shown in Figure 2. As shown in Figure 2, learning device 10 according to this embodiment is realized by the hardware configuration of a general computer or computer system, and includes input device 101, display device 102, external I / F 103, communication I / F 104, processor 105, and memory device 106. Each of these pieces of hardware is connected to each other via bus 107 so that they can communicate with each other.

[0018] The input device 101 is, for example, a keyboard, a mouse, a touch panel, a physical button, etc. The display device 102 is, for example, a display, a display panel, etc. Note that the study device 10 does not necessarily have to have at least one of the input device 101 and the display device 102, for example.

[0019] The external I / F 103 is an interface with an external device such as a recording medium 103a. The learning device 10 can read from and write to the recording medium 103a via the external I / F 103. Examples of the recording medium 103a include a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

[0020] The communication I / F 104 is an interface for connecting the learning device 10 to a communication network. The processor 105 is, for example, a variety of computing devices such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). The memory device 106 is, for example, a variety of storage devices such as a HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, a RAM (Random Access Memory), or a ROM (Read Only Memory).

[0021] The learning device 10 according to this embodiment can perform the learning process described below by having the hardware configuration shown in Fig. 2. Note that the hardware configuration shown in Fig. 2 is an example, and the learning device 10 may have, for example, multiple processors 105, multiple memory devices 106, or various other hardware components in addition to the hardware shown in the figure.

[0022] <Functional configuration of learning device 10> The functional configuration of the learning device 10 according to this embodiment is shown in Fig. 3. As shown in Fig. 3, the learning device 10 according to this embodiment includes an input unit 201, a start number changing unit 202, a language model unit 203, a classification unit 204, an update unit 205, and an output unit 206. Each of these units is realized, for example, by a process in which one or more programs installed in the learning device 10 are executed by the processor 105. Note that these one or more programs may be stored in the memory device 106 via a recording medium 103a or the like, or may be downloaded from a server device or the like connected via a communication network and then stored in the memory device 106.

[0023] The learning device 10 according to this embodiment also includes a storage unit 207. The storage unit 207 is realized, for example, by the memory device 106. Note that the storage unit 207 may also be realized, for example, by a storage device such as a database connected to the learning device 10 via a communication network.

[0024] Here, it is assumed that a set of training data (also called a "training dataset") for a target task (hereinafter also called a target task) is provided to the learning device 10. The training data includes a token sequence created from text data and its correct answer data.

[0025] The input unit 201 inputs the learning data given to the learning device 10 .

[0026] The start number changing unit 202 sets the position number (start number) of the start position of a token sequence included in the training data input by the input unit 201 to a random value, and calculates a position number sequence starting from that start number. The position number sequence is a sequence expressed as {j, j+1, . . . , J} where j is the start number. Here, if the sequence length of the token sequence is L and the maximum sequence length that can be handled by the language model is Z, then J = min(j+L-1, Z). In other words, if the start number is set to a random value and the final position number of the position number sequence exceeds the maximum sequence length Z, the position number sequence is truncated at position number Z.

[0027] The language model unit 203 is realized by a language model (for example, BERT) including a neural network. The language model unit 203 calculates context features from a token sequence whose start numbers have been changed by the start number changing unit 202. Hereinafter, the model parameters of the language model that realizes the language model unit 203 will be referred to as "first model parameters."

[0028] The classification unit 204 includes a neural network and is realized by a classification model for solving a target task (for example, a natural language processing task such as a word fill-in-the-blank task or a text classification task). The classification unit 204 calculates predicted features of the target task from the context features calculated by the language model unit 203. Hereinafter, the model parameters of the classification model that realizes the classification unit 204 will be referred to as "second model parameters."

[0029] The update unit 205 updates the first model parameters and the second model parameters through supervised learning using the predicted features calculated by the classification unit 204 and the correct answer data included in the learning data input by the input unit 201.

[0030] The output unit 206 outputs the trained first model parameters trained by the update unit 205. The output unit 206 may output the trained first model parameters to any predetermined output destination. Examples of such output destinations include an auxiliary storage device such as an HDD or SSD, a display, another program, another device or terminal connected via a communication network, and the like.

[0031] The storage unit 207 stores the first model parameters and the second model parameters. In addition to these, the storage unit 207 may also store, for example, intermediate calculation results when calculating context features, predicted features, etc., intermediate calculation results when updating the first model parameters and the second model parameters, etc.

[0032] <Learning data> Below, we will explain how to create training data for a word fill-in-the-blank task, assuming that the target task is a word fill-in-the-blank task. For details about the word fill-in-the-blank task, please refer to, for example, "Task #1: Masked LM" in Reference 1.

[0033] First, prepare text data. The text data can be any text data. For example, each paragraph of text published on the Web, such as an online encyclopedia, can be extracted as a single piece of text data.

[0034] Next, each piece of text data is divided into tokens using an appropriate tokenizer, and padding tokens are added to the end so that the length of the token sequence is a predetermined length L. As a result, for example, if each piece of text data is n=1, , N, a token sequence {x n1 ,···,x nL} is obtained.

[0035] Next, for each token sequence {x n1 ,···,x nL}, and each of the selected tokens is either replaced with a mask token [MASK] or another randomly selected token, or is left as is. Here, the conditions for replacement or retention can be the same as those in "Task #1: Masked LM" described in Reference 1. This allows us to n1 ,···,x nL} is a token sequence {x' n1 ,···,x' nL} is obtained.

[0036] Then, the token sequence {x' n1 ,···,x' nL}, the original token sequence {x n1 ,···,x nL}, the tokens selected for replacement or maintenance are used as the correct answer data, and the token sequence {x'n1 , ···, x' nL} and the correct data thereof are used as learning data n. Hereinafter, the token sequence {x' n1 , ···, x' nL} is referred to as a "replaced token sequence", and its correct data is referred to as a "correct token". Note that when there are multiple tokens selected as the object of replacement or maintenance, it should be noted that there are also multiple correct tokens.

[0037] <Learning Process> Next, the learning process according to the present embodiment will be described with reference to FIG. 4. Hereinafter, a word filling task is assumed as the target task, and the case of simultaneously learning the first model parameter and the second model parameter will be described. Also, it is assumed that the first model parameter and the second model parameter are initialized with appropriate values (for example, random values).

[0038] The input unit 201 inputs the given learning data (step S101).

[0039] Next, the start number changing unit 202 sets the position number (start number) of the start position of the replaced token sequence included in the learning data input in step S101 above to a random value, and calculates a position number sequence starting from that start number (step S102). At this time, the start number changing unit 202 randomly selects a start number j such that m ≤ j ≤ M using preset constants m and M (where m < M). As a result, a position number sequence {j, j + 1, ···, J} is obtained. However, for the sequence length L of the token sequence and the maximum sequence length Z that the language model can handle, J = min(j + L - 1, Z).

[0040] Note that m and M can be set arbitrarily, but for example, when a position vector is one of the first model parameters as in Reference 1, M can be set to a value about half that of Z in order to prevent the first model parameter from increasing. On the other hand, m is usually set to 1, but it is also possible to set m to a value greater than 1 when, for example, it is desired to focus on learning position vectors with large position numbers.

[0041] Next, the language model unit 203 uses the first model parameters and the position number sequence {j, j+1, . . . , J} calculated in step S102 to calculate context features from the replaced token sequence included in the training data input in step S101 (step S103). At this time, the language model unit 203 may calculate the context features by the same processing as that of a known language model, except that it uses a position vector sequence corresponding to the position number sequence {j, j+1, . . . , J}.

[0042] For example, when the position vector is the Position Embedding of Reference 1, since it has a dictionary-like format in which one position number corresponds to one position vector, the language model unit 203 obtains a position vector sequence by searching for the position vector corresponding to each position number included in the position number sequence {j, j+1, , J}. Thereafter, as in Reference 1, the language model unit 203 inputs the position vector sequence (Position Embedding) and the replaced token sequence into BERT and calculates context features.

[0043] On the other hand, for example, when the position vector is Positional Encoding of Non-Patent Document 1, the language model unit 203 calculates a PE from each position number included in the position number sequence {j, j+1, . . . , J}. (pos,2i) and P.E. (pos,2i+1) Then, as in Non-Patent Document 1, the language model unit 203 inputs the position vector sequence and the replaced token sequence into the Transformer model and calculates the context feature.(pos,2i) and P.E. (pos,2i+1) Note that pos represents the position number and i represents the dimension.

[0044] Next, the classification unit 204 calculates predicted features for the word fill-in-the-blank task from the context features calculated in step S103 using the second model parameters (step S104). The classification model calculates predicted features for the word fill-in-the-blank task from the context features calculated in step S103 using the context features h k The classification model predicts the token at position k from a predetermined vocabulary set based on the context feature h. k The predicted feature y k ∈R d' This converts the predicted feature y k Among the elements of, the vocabulary corresponding to the index of the element with the maximum value can be predicted as the token at position number k.

[0045] Next, the update unit 205 updates the first model parameters and the second model parameters through supervised learning using the predicted feature calculated in step S104 and the correct tokens included in the learning data input in step S101 (step S105). Note that the learning method may be a commonly known method for learning neural networks. The updating of the first model parameters and the second model parameters is repeatedly performed until a learning convergence condition is satisfied. Examples of the learning convergence condition include that the learning has been repeated a predetermined number of times, or that the amount of parameter update before and after the repetition is equal to or less than a predetermined threshold, etc.

[0046] Then, the output unit 206 outputs the trained first model parameters trained in the above step S105 (step S106). This results in the trained language model being output. Note that the output unit 206 may output not only the trained first model parameters but also the trained second model parameters.

[0047] <Summary> As described above, the learning device 10 according to this embodiment sets a random value to the position number of the first token in a token sequence input to a language model, and then learns model parameters from the token sequence. This increases the proportion of valid tokens at large position numbers, even when the token sequence is padded. This makes it possible to sufficiently learn position vectors at large position numbers, thereby enabling the acquisition of a highly accurate language model.

[0048] While the present embodiment has been described with respect to learning the first and second model parameters, the learning device 10 according to the present embodiment may further use the trained first and second model parameters to perform inference of a target task. In this case, the learning device 10 may be referred to as, for example, an "inference device." Note that when the learning device 10 functions as an inference device, the start number changing unit 202 and the updating unit 205 are unnecessary, and it is sufficient for the learning device 10 to include the input unit 201, the language model unit 203, the classification unit 204, the output unit 206, and the storage unit 207.

[0049] The learning device 10 of this embodiment provides specific improvements over conventional learning methods for position vectors described in Non-Patent Document 1, Reference Document 1, etc., and represents an advancement in the technical field related to performance evaluation of machine learning models (especially machine learning models for solving natural language processing tasks such as word fill-in-the-blank tasks and text classification tasks).

[0050] The following additional notes are provided regarding the above-described embodiments. (Additional note 1) Memory and at least one processor coupled to said memory; Including, The processor: The position number of the first token in the token sequence included in the given learning data is changed, and a position number sequence is calculated in which the position numbers are set in order from the changed position number, using the position number sequence and the token sequence, learning a language model parameter including a position vector that is a feature corresponding to the position number or a language model parameter for obtaining the position vector; Learning device. (Additional note 2) The processor: The position number j of the first token is changed to a random value, The learning device described in appendix 1, wherein when L is the sequence length of the token sequence, Z is the maximum sequence length that can be input to the language model, and J = min(j+L-1,Z), {j, j+1,...,J} is calculated as the position number sequence. (Additional note 3) The processor: 3. The learning device according to claim 1 or 2, wherein the position number of the first token is changed to a random value between m and M, both inclusive, using preset constants m and M. (Additional note 4) The constant m is set to a value of 1 or more, 4. The learning device according to claim 3, wherein, when the language model parameters include the position vector, the constant M is set to half the maximum sequence length Z that can be input to the language model or a value around said half. (Additional note 5) A non-transitory storage medium storing a program executable by a computer to perform a learning process, The learning process includes: The position number of the first token in the token sequence included in the given learning data is changed, and a position number sequence is calculated in which the position numbers are set in order from the changed position number, using the position number sequence and the token sequence, learning a language model parameter including a position vector that is a feature corresponding to the position number or a language model parameter for obtaining the position vector; Non-transitory storage medium.

[0051] The present invention is not limited to the above-described specifically disclosed embodiments, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims.

[0052] [References] Reference 1: Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language, arXiv:1810.04805 [cs.CL] [Explanation of symbols]

[0053] 10 Learning Device 101 Input Device 102 Display device 103 External I / F 103a Recording media 104 Communication I / F 105 processors 106 Memory Device 107 Bus 201 Input section 202 Starting Number Change Part 203 Language Model Section 204 Classification Department 205 Update Department 206 Output section 207 Memory section

Claims

1. a modification unit that modifies the position number of a first token in a token sequence included in the given learning data, and calculates a position number sequence in which position numbers are set in order from the modified position number; a learning unit that uses the position number sequence and the token sequence to learn a language model parameter including a position vector that is a feature corresponding to the position number or a language model parameter for obtaining the position vector; and The change unit The position number j of the first token is changed to a random value, A learning device that calculates {j, j+1, ..., J} as the position number sequence, where L is the sequence length of the token sequence, Z is the maximum sequence length that can be input to the language model, and J = min(j+L-1, Z).

2. The change unit 2. The learning device according to claim 1, wherein the position number of the first token is changed to a random value between m and M, both inclusive, using preset constants m and M.

3. The constant m is set to a value of 1 or more, The learning device according to claim 2 , wherein when the position vector is included in the language model parameters, the constant M is set to a value equal to or around half of a maximum sequence length Z that can be input to the language model.

4. A modification procedure for modifying the position number of the first token in the token sequence included in the given learning data, and calculating a position number sequence in which the position numbers are set in order from the modified position number; a learning procedure for learning language model parameters including a position vector, which is a feature corresponding to the position number, or language model parameters for obtaining the position vector, using the position number sequence and the token sequence; The computer executes The change procedure is as follows: The position number j of the first token is changed to a random value, A learning method in which, when L is the sequence length of the token sequence, Z is the maximum sequence length that can be input to the language model, and J = min(j+L-1, Z), {j, j+1, ..., J} is calculated as the position number sequence.

5. A program that causes a computer to function as the learning device according to any one of claims 1 to 3.

Images