Learning device, learning method, and learning program

Abstract

To execute pre-learning of a language model with appropriate granularity.SOLUTION: A learning device having a processor for executing a program and a storage device for storing the program, stores a word dictionary associating each word of a word group and a token as a character string unique to the word in the storage unit. The processor performs, in accordance with a variable rule, extraction processing for extracting a character string of a part from each word of the word group, overlapping removal processing for removing overlapping when a plurality of character strings of the same part extracted by the extraction processing exist, application processing for applying a unique first correct label to the character string of the part, and first generation processing for generating a label dictionary associating the first correct label applied by the application processing and the token associated with a word of an extraction source of the character string of the part to which the first correct label has been applied in the word dictionary.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a learning device, a learning method, and a learning program for learning a language model.

Background Art

[0002] A method of learning based on text data obtained from a large amount of WEB, using labels generated mechanically rather than labels given manually to the inherent properties therein, is called self-supervised learning. Currently, many language models using Transformer are pre-trained by self-supervised learning. The pre-trained language model is considered to have semantic and syntactic language knowledge. When applying the pre-trained model to an application task (fine-tuning), it is known that learning can be executed with a smaller amount of data and in a shorter time than a model without pre-training, and high task performance can be shown.

[0003] Patent Document 1 discloses "a learning method for a natural language processing system including a front-end device that executes natural language processing using a language model for a natural sentence, and a back-end device that executes a back-end task using the processing result of the natural language processing, the front-end device including a front-end processing step of executing natural language processing using a language model for a learning natural sentence, the back-end device including a back-end processing step of executing a back-end task using the result of the natural language processing, an evaluation step of evaluating the output of the back-end task, and a learning step of simultaneously learning the language model and the back-end model using the evaluation result."

[0004] Patent Document 2 discloses that "this two-stage process enables technical use cases that were previously impossible. Firstly, it can generate high-quality summaries for certain categories of documents where relevant ground truth summaries are expensive or difficult to collect, for use in supervised learning training of networks. For example, it can generate concise and accurate summaries for bills or corporate emails. Secondly, the adaptive process is far less computationally intensive than the pre-training process because the amount of labeled data required is significantly reduced."

[0005] Non-patent document 1 discloses that, using the BERT (Bidirectional Encoder Representations from Transformers) architecture, token-level pre-training tasks with relatively low computational cost other than Masked Language Modeling are effective as alternative tasks. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2022-77202 [Patent Document 2] U.S. Patent Publication No. 10885436 [Non-patent literature]

[0007] [Non-Patent Document 1] Atsuki Yamaguchi, George Chrysostomou, Katerina Margatina and Nikolaos Aletras, "Frustratingly Simple Pretraining Alternatives to Masked Language Modeling", Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP2021), November 7, 2021 pages 3116-3125 [Overview of the project] [Problems that the invention aims to solve]

[0008] Pre-training language models requires not only large amounts of text data but also hardware with sufficient performance. Here, hardware includes the Central Processing Unit (CPU) and Graphics Processing Unit (GPU). In order to make effective use of finite hardware resources, techniques for training language models at low cost have attracted attention in recent years. There are attempts to reduce computational load by improving the architecture proposed in BERT (Bidirectional Encoder Representations from Transformers), and there are movements to explore alternative tasks other than Masked Language Modeling.

[0009] However, with the conventional techniques described above, it is difficult to flexibly adjust the training cost of the language model. Therefore, in cases where available computing resources are limited, it may not be possible to train the language model.

[0010] The present invention aims to make it possible to adjust the granularity of the dictionary used for training a language model. [Means for solving the problem]

[0011] A learning device, which is one aspect of the invention disclosed in this application, is a generating device having a processor that executes a program and a storage device that stores the program, wherein the storage device stores a word dictionary that associates each word of a word group with a token which is a string of characters unique to that word, and the processor processes each word of the word group according to a variable rule. ra The method is characterized by performing an extraction process to extract a part of a string; a duplicate removal process to remove duplicates if there are multiple identical parts of the string extracted by the extraction process; an assignment process to assign a unique first correct label to the part of the string; and a first generation process to generate a label dictionary that associates the first correct label assigned by the assignment process with the token associated in the word dictionary with the source word of the part of the string to which the first correct label was assigned. [Effects of the Invention]

[0012] According to one embodiment of the present invention, the granularity of the dictionary used for training a language model can be adjusted. Other problems, configurations, and effects will be revealed by the following description of embodiments. [Brief explanation of the drawing]

[0013] [Figure 1] Figure 1 is a block diagram showing an example of the functional configuration of the learning device according to Example 1. [Figure 2] Figure 2 is an explanatory diagram showing an example of prediction task information. [Figure 3] Figure 3 is an explanatory diagram showing an example of text information. [Figure 4] Figure 4 is a block diagram showing an example of the functional configuration of the label dictionary generation unit. [Figure 5] Figure 5 is an explanatory diagram showing an example of obtaining a label dictionary by the label dictionary generation unit. [Figure 6] Figure 6 is a block diagram showing an example of the functional configuration of the pre-learning unit. [Figure 7] Figure 7 is an explanatory diagram showing an example of pre-training performed by the pre-training unit. [Figure 8] FIG. 8 is an explanatory diagram showing an example of a setting screen for replacement ratio. [Figure 9] FIG. 9 is a block diagram showing a functional configuration example of the learning device according to Example 2. [Figure 10] FIG. 10 is an explanatory diagram showing an example of required accuracy setting information. [Figure 11] FIG. 11 is an explanatory diagram showing an example of computer resource usage status setting information. [Figure 12] FIG. 12 is an explanatory diagram showing an example of personal information protection requirement degree setting information. [Figure 13] FIG. 13 is an explanatory diagram showing an example of predicted character direction setting information. [Figure 14] FIG. 14 is an explanatory diagram showing an example of a setting screen for the number of predicted characters and the predicted character direction. [Figure 15] FIG. 15 is a block diagram showing a functional configuration example of the learning device according to Example 3. [Figure 16] FIG. 16 is an explanatory diagram showing an example of computer resource usage status setting information. [Figure 17] FIG. 17 is a block diagram showing a functional configuration example of the pre-learning unit of the learning device according to Example 3. [Figure 18] FIG. 18 is a flowchart showing an example of a processing procedure by the tokenizer unit. [Figure 19] FIG. 19 is a block diagram showing a functional configuration example of the learning device according to Example 4. [Figure 20] FIG. 20 is an explanatory diagram showing an example of adaptation task information. [Figure 21] FIG. 21 is an explanatory diagram showing an example of a setting screen when performing additional pre-learning and fine-tuning. [Figure 22] FIG. 22 is a block diagram showing a hardware configuration example of a computer that realizes the learning device according to Examples 1 to 4. <X

DETAILED DESCRIPTION OF THE INVENTION

[0014] The following examples will be described with reference to the attached drawings. In the following, each example and each modification may be combined in whole or in part without departing from the spirit of the present invention. [Examples]

[0015] <Example of functional configuration of a learning device> Figure 1 is a block diagram showing an example of the functional configuration of the learning device 100 according to Embodiment 1. The learning device 100 can take in a tokenizer 101, prediction task information 102, and text information 103 as input data. The learning device 100 includes a label dictionary generation unit 110 and a pre-training unit 600. The learning device 100 outputs a pre-trained language model 141.

[0016] The learning device 100 can handle any language, but Example 1 describes an example that handles English. Note that the pre-training method in Example 1 only changes the dimension of the model's output layer compared to mask language modeling, so it is possible to simultaneously apply acceleration and lightweight techniques to any architecture that would normally be applied to mask language modeling.

[0017] The tokenizer 101 is a converter that transforms text information 103 into a sequence of numerical values ​​called tokens, based on a defined algorithm and dictionary. Possible tokenization methods include, for example, Byte-level Byte Pair Encoding (BPE), SentencePiece, and WordPiece, but the specific method is not limited. In Example 1 below, for convenience, an example using a BPE-based tokenizer as tokenizer 101 will be described.

[0018] The prediction task information 102 is input data used in the label dictionary generation unit 110.

[0019] [Predicted Task Information 102] Figure 2 is an explanatory diagram showing an example of prediction task information 102. Prediction task information 102 includes the number of predicted characters 201 and the prediction character direction 202 as the prediction task. The number of predicted characters 201 stores an arbitrary natural number n. The prediction character direction 202 stores the "first" or "last" word. For example, if the prediction character direction 202 is "first", the n characters from the beginning to the nth character of the string indicated by the target word are the target of the prediction task. Also, if the prediction character direction 202 is "last", the n characters from the end to the nth character of the string indicated by the target word are the target of the prediction task. In Figure 2, the number of predicted characters 201 is set to n=1 and the prediction character direction 202 is set to "first".

[0020] In Example 1, the input data used by the learning device 100 can be represented in any data structure, regardless of the data structure used. For example, information can be stored using data structures such as lists, databases, queues, stacks, or tables.

[0021] Figure 2 shows an example where the prediction task information 102 is stored as table data. Furthermore, the prediction task information 102 is not limited to the example that handles the predicted number of characters 201 and the predicted character direction 202. For example, the label dictionary generation unit 110 may extract characters according to an extraction rule that extracts only consonant characters or only vowel characters, and construct the label dictionary 121.

[0022] Returning to Figure 1, the text information 103 is the input data used in the pre-training unit 600.

[0023] [Text Information 103] Figure 3 is an explanatory diagram showing an example of text information 103. As shown in Figure 3, text information 103 includes sentences 301, 302, ... each containing one or more text sentences. Note that the sentences stored in each line do not need to be single sentences; they may consist of multiple text sentences as long as they are excerpts from any sequence of text sentences.

[0024] Returning to Figure 1, the label dictionary generation unit 110 generates a label dictionary 121 using the tokenizer 101 and the prediction task information 102.

[0025] [Label dictionary generation unit 110] Figure 4 is a block diagram showing an example of the functional configuration of the label dictionary generation unit 110. Figure 5 is an explanatory diagram showing an example of obtaining a label dictionary 121 by the label dictionary generation unit 110. The label dictionary generation unit 110 has, for example, an extraction unit 401 up to a specified character, a duplicate removal unit 402, an identifier assignment unit 403, and a label dictionary generation unit 404, and outputs a label dictionary 121.

[0026] The extraction unit 401 extracts a portion of a string from each word in the word group of the dictionary held by the tokenizer 101, according to a variable rule. The variable rule is a changeable extraction rule regarding the characters within the words in the dictionary held by the tokenizer 101. The extraction unit 401 generates a list that takes into account the predicted number of characters 201 and the predicted character direction 202 of the prediction task information 102. The predicted number of characters 201 and the predicted character direction 202 define the variable rule. Specifically, for example, in Figure 5, the word dictionary 500 held by the tokenizer 101 has a key 501 and a value 502. The key 501 represents a word, and the value 502 represents a token, i.e., a numerical value unique to the word.

[0027] The words shown in key 501 include at least the words contained in text information 103. The words shown in key 501 may also be generated from words that appear in text information 103.

[0028] In Figure 5, the extraction unit 401 converts the word dictionary 500 into a list 510 that takes into account the predicted number of characters 201 and the predicted character direction 202. In the example in Figure 2, the predicted number of characters 201 is n=1 and the predicted character direction 202 is "beginning", so the extraction unit 401 extracts element 511, which is the string from the beginning to the first character of each word shown in key 501, i.e., the first character, from the word dictionary 500 and generates a list 510 consisting of the first characters of each word shown in key 501.

[0029] In Figure 4, the duplicate removal unit 402 removes duplicates from element 511 if duplicates exist in the list generated by the extraction unit 401. Specifically, for example, if identical strings exist in the list, the duplicate removal unit 402 removes all but one of the multiple identical strings.

[0030] In the case of Figure 5, list 510 contains two instances of "p," namely the first letter "p" of "people" and the first letter "p" of "pool." Therefore, the duplicate removal unit 402 deletes the other "p," leaving one of them. The list 510 after duplicate removal by the duplicate removal unit 402 is designated as list 520. Note that the list 510 without any duplicates is also designated as list 520 for convenience.

[0031] In Figure 4, the identifier assignment unit 403 assigns a unique identifier to each element 511 in the list 520 output from the duplicate removal unit 402. In Figure 5, the identifier assignment unit 403 assigns a unique identifier 531 to each element 511 of the list 520 and outputs the list 530.

[0032] In Figure 4, the label dictionary generation unit 404 generates a label dictionary 121 using the list 530 to which the identifier 531 has been assigned by the identifier assignment unit 403, and the word dictionary 500 of the tokenizer 101. Specifically, for example in Figure 5, the label dictionary generation unit 404 generates a label dictionary 121 using the value 502 (token) of the word dictionary 500 of the tokenizer 101 as the key 541 and the identifier 531 as the value 542 (correct label).

[0033] In other words, the value 542 for token "82", which has a value of 502 for the first letter "p" of "people", and the value 542 for token "28749", which has a value of 502 for the first letter "p" of "pool", both result in the value "12" for the unique identifier 531 of element 511, which is the first letter "p".

[0034] In this way, the label dictionary generation unit 110 can generate a label dictionary 121 from the tokenizer 101 and the prediction task information 102. Furthermore, by incorporating the prediction character direction 202 in the label dictionary generation unit 110, it is effective even in languages ​​where word endings play an important role in a sentence, such as German.

[0035] The label dictionary generation unit 110 may receive the generated label dictionary 121, as described above, from an external computer, or it may read the label dictionary 121 from an external memory if an external memory for storing the label dictionary 121 is installed in the learning device 100.

[0036] Returning to Figure 1, the pre-training unit 130 trains a language model using the label dictionary 121 and text information 103. The trained language model is the pre-trained language model 141.

[0037] [Pre-learning section 130] Figure 6 is a block diagram showing an example of the functional configuration of the pre-training unit 130. Figure 7 is an explanatory diagram showing an example of pre-training by the pre-training unit 130. In Figure 6, the pre-training unit 130 includes a tokenizing unit 601, a label generation unit 602, a masking unit 603, and a model learning unit 604.

[0038] The tokenizing unit 601 receives the tokenizer 101 and the text information 103 as input, and converts the text in the text information 103 into a sequence of numbers according to the tokenizer 101's predetermined algorithm.

[0039] Specifically, for example, in Figure 7, taking sentence 301 as an example, the pre-training unit 130 refers to the word dictionary 500 of the tokenizer 101 and converts each word in sentence 301 into a numerical value (token) of value 502, thereby generating a token sequence 701. The leading "0" in the token sequence 701 is a token indicating the start of sentence 301, and the trailing "2" in the token sequence 701 is a token indicating the end of sentence 301.

[0040] In Figure 6, the label generation unit 602 refers to the label dictionary 121 and converts each token in the token sequence 701 into the corresponding value 542 of the label dictionary 121, which is the correct label, thereby generating the correct label sequence. Specifically, for example in Figure 7, the label generation unit 602 converts each token in the token sequence 701 into the value 542 (correct label) corresponding to the key 541 that matches the token, thereby generating the correct label sequence 702, which is a sequence of correct labels.

[0041] In Figure 6, the masking unit 603 randomly replaces a certain percentage of the tokens in the token sequence 701 with mask tokens to generate a masked token sequence. Specifically, for example in Figure 7, the masking unit 603 randomly replaces a certain percentage of the tokens in the token sequence 701 with the mask token 704, which is the numerical value "50264" (this value is just an example), to generate a masked token sequence 703. The mask token is a numerical value that is not assigned to any word in the word dictionary 500.

[0042] In Figure 6, the model learning unit 604 takes the ground truth label sequence 702 and the masked token sequence 703 as input, learns a machine learning model based on a neural network, and outputs a pre-trained language model 141. Specifically, for example, in Figure 7, the model learning unit 604 refers to the numerical values ​​"50264" of the masked tokens 731 to 733 contained in the masked token sequence 703, and learns a multi-class classification problem to predict the numerical values ​​721 to 723 (in Figure 7, "72", "9", and "235") that will be the labels for the masked tokens 731 to 733 in the ground truth label sequence 702, using a cross-entropy loss function and an arbitrary optimizer such as Adam, and outputs a pre-trained language model 141.

[0043] More specifically, the model learning unit 604 uses the masked token sequence 703 as training data and the ground truth label sequence 702 as ground truth data, inputting these into an arbitrary machine learning model to predict the key 541 that matches the numerical value "50264" of the masked tokens 731-733.

[0044] The model learning unit 604 can use any architecture as a machine learning model based on a neural network, but for example, it can use a Transformer-type encoder language model such as BERT, or an Encoder-Decoder model such as BART (Bidirectional Auto-Regressive Transformer).

[0045] The pre-training unit 600 is not necessarily limited to untrained machine learning models; it can be applied to any machine learning model, such as Task Adaptive Pretraining (TAPT). This means that, even in situations where computing resources are limited, a large-scale pre-trained language model can be further trained based on Example 1.

[0046] Furthermore, the substitution ratio for mask tokens 731-733 can be any ratio, and is not limited to the substitution ratio shown in Figure 7. In addition, if computing resources are abundant, it is possible to train the model on additional tasks such as mask language modeling, in addition to pre-training using the label dictionary 121.

[0047] [Example of settings screen] Figure 8 is an explanatory diagram showing an example of a screen for setting the replacement ratio. The setting screen 800 includes a replacement ratio slider 801, a predicted character count input field 802, a predicted character direction input field 803, and a setting button 804.

[0048] The replacement ratio slider 801 is a user interface that allows the user to set the replacement ratio to a value greater than 0 and less than or equal to 1. The predicted character count input field 802 is an interface that allows the user to input the predicted character count of 201. The predicted character direction input field 803 is an interface that allows the user to input the predicted character direction of 202. The setting button 804 is a user interface that allows the user to confirm the settings made in the replacement ratio slider 801, the predicted character count input field 802, and the predicted character direction input field 803.

[0049] Thus, according to Example 1, the learning device 100 utilizes a label dictionary 121 constructed according to the user-specified number of predicted characters 201 and predicted character direction 202, randomly replaces a certain percentage of each text sentence in the text information 103 with mask tokens, and has the machine learning model solve the task of inferring the label, which is the value 542 of the label dictionary 121 corresponding to the replaced portion. This makes it possible to perform pre-training of the machine learning model with less computational effort than conventional mask language modeling.

[0050] Furthermore, the pre-training method according to Example 1 can reduce memory usage to an appropriate level by using a relatively small specified number of characters. In other words, the smaller the number of predicted characters 201, the more duplication is performed by the duplication elimination unit 402. Therefore, it has the effect of making the operation of language models lighter on edge devices with limited memory space, such as robots.

[0051] As a concrete example, let's assume 32-bit floating-point arithmetic and use the tokenizer 101 provided in the existing BPE-based RoBERTa model. Let's also assume that the output layer dimension of the pre-trained language model 141 is 768, the number of words in the word dictionary 500 of the tokenizer 101 is 50265, and the number of words in the label dictionary 121, obtained by extracting only the first character (n=1) from the word dictionary 500 of the tokenizer 101, is 256.

[0052] In this case, the memory usage of the output layer by the pre-training method of the pre-training unit 130 is (768 × 256 + 256) × 2 × 4 = 1.57 MB. In the case of mask language modeling, the memory usage of the output layer is (768 × 50265 + 50265) × 2 × 4 = 309 MB, which reduces the memory usage by about 307 MB. [Examples]

[0053] Next, Example 2 will be described. The learning device in Example 2 performs pre-training of a language model according to the user's purpose of use. In Example 2, the explanation will focus on the differences from Example 1, so the same reference numerals will be used for components identical to those in Example 1, and their descriptions will be omitted.

[0054] <Example of functional configuration of a learning device> Figure 9 is a block diagram showing an example of the functional configuration of the learning device according to Embodiment 2. The learning device 900 according to Embodiment 2 has the same configuration as the learning device 100 according to Embodiment 1, with the addition of required accuracy setting information 901, computing resource usage setting information 902, personal information protection requirement setting information 903, predicted character direction setting information 904, and a prediction task information generation unit 910.

[0055] The learning device 900 takes the tokenizer 101, text information 103, required accuracy setting information 901, computing resource usage setting information 902, and personal information protection requirement setting information 903 as input data to generate a pre-trained language model 141.

[0056] In Example 2, the prediction task information generation unit 910 generates prediction task information 102 based on the word dictionary 500 of the tokenizer 101, the required accuracy setting information 901, the computing resource usage setting information 902, the personal information protection requirement setting information 903, and the prediction character direction setting information 904. This makes it possible to perform pre-training of a language model that is tailored to the purpose of use.

[0057] [Required accuracy setting information 901] Figure 10 is an explanatory diagram showing an example of the requested accuracy setting information 901. The requested accuracy setting information 901 includes the requested accuracy 1001 and the number of predicted characters 1002. The requested accuracy 1001 is the accuracy that the user requests from the pre-trained language model 141. The requested accuracy 1001 is represented by uppercase letters. "S" is the highest and "E" is the lowest.

[0058] The higher the required precision (1001), the greater the value of the predicted character count (1002). The range below the uppercase letters represents the range of values ​​for the required precision (1001) that will be used as weights when calculating the weighted average of the predicted character count (1002). The predicted character count (1002) is the candidate value for the predicted character count (201) corresponding to the required precision (1001).

[0059] The required accuracy setting information 901 may be created manually. Alternatively, the learning device 900 may automatically generate the required accuracy setting information 901. Specifically, for example, the learning device 900, using the configuration of Embodiment 1, increases the number of predicted characters 201 in ascending order from n=1, generates a label dictionary 121 for each number of predicted characters 201, and identifies the degree of increase in the size of the label dictionary 121. The learning device 900 associates the number of predicted characters 201 with the degree of increase in the size of the label dictionary 121 as the range of the required accuracy 1001 value, and generates the required accuracy setting information 901.

[0060] [Computer resource usage settings information 902] Figure 11 is an explanatory diagram showing an example of computer resource usage setting information 902. The computer resource usage setting information 902 includes computer resource usage status 1101 and predicted character count 1102. Computer resource usage status 1101 is the usage status, i.e., utilization rate, of computer resources (e.g., processor or memory) used by the learning device 900, which is a computer.

[0061] The lower the computing resource usage status 1101 (the more resources available), the higher the value of the predicted number of characters 1102. The lower range of computing resource usage status 1101 is the range of computing resource usage rates that will be used as weights when calculating the weighted average of the predicted number of characters 1102. The predicted number of characters 1102 is the candidate value of the predicted number of characters 201 corresponding to the computing resource usage status 1101.

[0062] The computer resource usage setting information 902 may be created manually. Alternatively, the learning device 900 may automatically generate the computer resource usage setting information 902. Specifically, for example, the learning device 900 increases the number of predicted characters 201 in ascending order from n=1 according to the configuration of Embodiment 1, generates a label dictionary 121 for each number of predicted characters 201, and identifies the degree of increase in the size of the label dictionary 121. The learning device 900 associates the computer resource usage status from the software that manages computer resources when the label dictionary 121 was created with the degree of increase in the size of the label dictionary 121, associates the number of predicted characters 201 as a range of values ​​for the computer resource usage status 1101, and generates the computer resource usage setting information 902.

[0063] [Personal information protection requirement setting information 903] Figure 12 is an explanatory diagram showing an example of personal information protection requirement setting information 903. Personal information protection requirement setting information 903 includes a personal information protection requirement 1201 and a predicted number of characters 1202. The personal information protection requirement 1201 is the degree to which the user requests protection of their personal information.

[0064] The higher the personal information protection requirement score (1201), the lower the predicted character count (1202) value becomes from the perspective of preventing data leaks. The range below the personal information protection requirement score (1201) is the range of values ​​for the personal information protection requirement score (1201) that will be used as weights when calculating the weighted average of the predicted character count (1202). The predicted character count (1202) is a candidate value for the predicted character count (201) corresponding to the personal information protection requirement score (1201).

[0065] The personal information protection requirement setting information 903 may be created manually. Alternatively, the learning device 900 may automatically generate the personal information protection requirement setting information 903 in the same manner as the requirement precision setting information 901.

[0066] Furthermore, in the aforementioned required accuracy setting information 901, computing resource usage setting information 902, and personal information protection requirement setting information 903, if the number of predicted characters is increased to 1002, 1102, or 1202, the label dictionary 121 approaches an injective relationship with the word dictionary 500 of the tokenizer 101, and the output dimension of the pre-trained language model 141 approaches the number of words in the tokenizer 101.

[0067] As a result, the task of predicting the keys 541 of the label dictionary 121 corresponding to mask tokens 731-733 (the numerical values ​​721-723 in Figure 7, representing "72", "9", and "235") becomes similar to the task of predicting the values ​​502 obtained from the word dictionary 500 of the tokenizer 101 corresponding to mask tokens 731-733 (Masked Language Modeling). This increases the likelihood that sensitive information may be reconstructed, especially when training language models that handle sensitive information, depending on the content of the text information 103 provided as input.

[0068] [Predictive text direction setting information 904] Figure 13 is an explanatory diagram showing an example of predictive character direction setting information 904. The predictive character direction setting information 904 includes a language type 1301 and a predictive character direction 202. The language type 1301 is the type of language used in the word dictionary 500. A predictive character direction 202 is associated with each language type 1301.

[0069] [Settings screen] Figure 14 is an explanatory diagram showing an example of a setting screen for the number of predicted characters and the direction of predicted characters. The setting screen 1400 includes a required accuracy setting unit 1401, a computer resource usage setting unit 1402, a personal information protection requirement setting unit 1403, a predicted character count display unit 1404, a predicted character direction display unit 1405, a setting button 1406, and a learning execution button 1407.

[0070] The requested accuracy setting unit 1401 is a user interface for setting the requested accuracy 1001 via user operation or the prediction task information generation unit 910. The requested accuracy setting unit 1401 includes a slider 1411 and a slider bar 1412. The slider 1411 can be moved along the slider bar 1412 via user operation or the prediction task information generation unit 910. The slider bar 1412 indicates the range of weights that will be the value of the requested accuracy 1001.

[0071] The computer resource usage setting unit 1402 is a user interface for setting the computer resource usage status 1101 by the prediction task information generation unit 910. The computer resource usage setting unit 1402 has a slider 1411 and a slider bar 1412. The slider 1411 is movable on the slider bar 1412 by the prediction task information generation unit 910. The slider bar 1412 indicates the range of weights that will be the value of the computer resource usage status 1101.

[0072] The personal information protection requirement setting unit 1403 is a user interface for setting the personal information protection requirement 1201 via user operation or the predictive task information generation unit 910. The personal information protection requirement setting unit 1403 includes a slider 1431 and a slider bar 1432. The slider 1431 can be moved on the slider bar 1432 via user operation or the predictive task information generation unit 910. The slider bar 1432 indicates the range of weights that represent the value of the personal information protection requirement 1201.

[0073] The predicted character count display unit 1404 displays the predicted character count 201 generated by the prediction task information generation unit 910. The predicted character direction display unit 1405 displays the predicted character direction 202 generated by the prediction task information generation unit 910.

[0074] The setting button 1406 is a user interface that allows the user to initiate the generation of prediction task information 102 by the prediction task information generation unit 910.

[0075] The learning execution button 1407 is a user interface that allows the user to initiate pre-training on the pre-training unit 130.

[0076] Returning to Figure 9, the prediction task information generation unit 910 obtains the computing resource usage status 1101 from known software that manages the computing resources used by the learning device 900, and moves the slider 1421 to the position that indicates the obtained computing resource usage status 1101.

[0077] Furthermore, the prediction task information generation unit 910 obtains the required accuracy 1001 and the personal information protection requirement 1201 from the positions of the sliders 1411 and 1413 positioned by user operation.

[0078] The prediction task information generation unit 910 refers to the requested precision setting information 901 and reads out the predicted number of characters 1002 corresponding to the acquired requested precision 1001. The prediction task information generation unit 910 refers to the computer resource usage setting information 902 and reads out the predicted number of characters 1102 corresponding to the acquired computer resource usage 1101. The prediction task information generation unit 910 refers to the personal information protection requirement setting information 903 and reads out the predicted number of characters 1202 corresponding to the acquired personal information protection requirement 1201.

[0079] The prediction task information generation unit 910 adjusts the request precision 1001 and the personal information protection requirement 1201 so that the sum of the request precision 1001, the computing resource usage status 1101, and the personal information protection requirement 1201 is 1. In this case, the prediction task information generation unit 910 may set either the request precision 1001 or the personal information protection requirement 1201 to a value such that the other value becomes 0.0, or it may allocate the values ​​evenly.

[0080] In this case, the prediction task information generation unit 910 may accept either the requested precision 1001 or the personal information protection requirement 1201 through user operation. In this case, the prediction task information generation unit 910 may set the range of movement of the sliders 1411 and 1413 to a range where the other value is 0.0. When either the requested precision 1001 or the personal information protection requirement 1201 is set through user operation, the prediction task information generation unit 910 sets the other value such that the sum of the requested precision 1001, the computing resource usage status 1101, and the personal information protection requirement 1201 is 1.

[0081] The prediction task information generation unit 910 calculates a weighted average value using the acquired request precision 1001, computing resource usage status 1101, and personal information protection requirement degree 1201 as weights for the predicted number of characters 1002, 1102, and 1202. If the weighted average value has decimal parts, the prediction task information generation unit 910 rounds or truncates the decimal part of the weighted average value to make it a natural number. The prediction task information generation unit 910 outputs the weighted average value, which has become a natural number, as the predicted number of characters 201 and displays it on the predicted number of characters display unit 1404.

[0082] Furthermore, the prediction task information generation unit 910 refers to the word dictionary 500 of the tokenizer 101 and uses a known algorithm to identify the language type 1301 of the word indicated by key 501. The prediction task information generation unit 910 refers to the prediction character direction setting information 904 to identify the prediction character direction 202 corresponding to the identified language type 1301 and displays it on the prediction character direction display unit 1405.

[0083] Known algorithms include, for example, machine learning models trained on a training dataset where the words shown in key 501 of the word dictionary 500 of tokenizer 101 are used as training data, and the language species 1301 are used as ground truth data.

[0084] Thus, according to Example 2, the label dictionary 121 can be generated by adjusting the variable rule (for example, the number of predicted characters 201) according to the user's purpose of use. Specifically, for example, the smaller the number of predicted characters 201, the more duplicate removal is performed by the duplicate removal unit 402, which increases the number of values ​​542 (correct labels) to which multiple keys 541 (tokens) are assigned, and also increases the number of keys 541 (tokens) assigned to the values ​​542 (correct labels).

[0085] On the other hand, the larger the predicted number of characters 201, the less duplication is performed by the duplication removal unit 402. As a result, the number of values ​​542 (ground truth labels) to which multiple keys 541 (tokens) are assigned decreases, and the number of keys 541 (tokens) assigned to values ​​542 (ground truth labels) also decreases. Therefore, it is possible to perform pre-training of the language model according to the user's purpose.

[0086] Furthermore, the smaller the number of predicted characters (201), the fewer entries there are for key (541) and value (542) (ground truth label). This reduces the size of the label dictionary (121), saving memory and lowering the computational load during language model training. Conversely, the larger the number of predicted characters (201), the fewer entries there are for key (541) and value (542) (ground truth label). This results in finer granularity in the label dictionary (121), improving training accuracy. [Examples]

[0087] Next, Example 3 will be described. The learning device of Example 3 implements a real-time learning adjustment function according to the hardware status. In Example 3, the explanation will focus on the differences from Example 1, so the same reference numerals will be used for components identical to those in Example 1, and their descriptions will be omitted.

[0088] <Example of functional configuration of a learning device> Figure 15 is a block diagram showing an example of the functional configuration of the learning device according to Example 3. The learning device 1500 according to Example 3 has the same configuration as the learning device 100 according to Example 1, with the addition of computer resource usage status setting information 1502.

[0089] The learning device 1500 according to Example 3 receives a tokenizer 101, prediction task information 102, text information 103, and computing resource usage setting information 1502 as input data, and generates a pre-trained language model 141.

[0090] The learning device 1500 additionally inputs computing resource usage setting information 902 to the pre-training unit 1530. This allows the learning device 1500 to periodically monitor computing resource usage during pre-training and dynamically adjust the input length of text information 103 to the machine learning model, enabling efficient pre-training of the language model.

[0091] [Computer resource usage settings information 1502] Figure 16 is an explanatory diagram showing an example of computer resource usage setting information 1502. Computer resource usage setting information 1502 includes computer resource usage status 1101, predicted number of characters 1102, and maximum input length 1600. The maximum input length 1600 is the upper limit of the length required to tokenize the text sentence of the text information 103. The unit of the maximum input length 1600 is elements (tokens or words). In Examples 1 and 2, this length is fixed, but in Example 3, it becomes variable according to the computer resource usage status 1101. Specifically, for example, the more congested the computer resource usage status 1101 becomes, the shorter the maximum input length becomes.

[0092] [Example of functional configuration of pre-learning unit 1530] Figure 17 is a block diagram showing an example of the functional configuration of the pre-learning unit 1530 of the learning device 1500 according to Embodiment 3. The tokenizing unit 1701 within the pre-learning unit 1530 receives not only the tokenizer 101 and text information 103, but also computer resource usage status setting information 1502.

[0093] Figure 18 is a flowchart showing an example of the processing procedure by the tokenization unit 1701. The tokenization unit 1701 performs sentence splitting (step S1801). Sentence splitting (step S1801) is a process that divides the text of each line of the text information 103 into one or more text sentences. Possible splitting methods include methods using regular expressions and context-aware methods using SpaCy, but the specific method is not limited.

[0094] Next, the tokenizing unit 1701 determines the maximum input length 1600 (step S1802). Specifically, for example, the tokenizing unit 1701 refers to the computer resource usage status setting information 1502 to identify the maximum input length 1600 corresponding to the current computer resource usage status 1101.

[0095] Next, the tokenization unit 1701 performs tokenization and sentence sampling (step S1803). Specifically, for example, the tokenization unit 1701 tokenizes one or more text sentences obtained in sentence segmentation (step S181), that is, converts the words in the text sentences into tokens (numerical values ​​of value 502), as shown in Figure 5.

[0096] The tokenizing unit 1701 then checks whether the converted token sequence is less than or equal to the maximum input length of 1600. If it is less than or equal to the maximum input length of 1600, it extracts it as token sequence 701 (step S1803). If the converted token sequence is greater than the maximum input length of 1600, the tokenizing unit 1701 performs sentence splitting again on the one or more text sentences obtained in sentence splitting (step S1801) (step S1801), and then retries tokenizing and sentence sampling.

[0097] Thus, according to Example 3, the pre-training of the pre-trained language model 141 can be efficiently performed while taking into account the utilization of computing resources. Therefore, situations in which pre-training cannot be performed due to insufficient memory capacity can be avoided. [Examples]

[0098] Next, Example 4 will be described. The learning device in Example 4 will be described as an example that performs additional pre-training and fine-tuning. Note that in Example 4, the explanation will focus on the differences from Example 1, so the same reference numerals will be used for components identical to those in Example 1, and their descriptions will be omitted.

[0099] <Example of functional configuration of a learning device> Figure 19 is a block diagram showing an example of the functional configuration of the learning device according to Embodiment 4. The learning device 1900 according to Embodiment 4 is configured by adding a label prediction learning unit 1930 and adaptive task information 1931 to the learning device 100 according to Embodiment 1, and replacing the text information 103 with labeled text information 1903.

[0100] For example, before operation begins, the learning device 1900 uses the label dictionary generation unit 110 and the pre-training unit 130 to train a pre-trained language model 141, enabling it to predict correct labels. Furthermore, even after operation begins, additional training is possible using the label dictionary generation unit 110 and the pre-training unit 130.

[0101] The learning device 1900 receives a tokenizer 101, prediction task information 102, a pre-trained language model 141, labeled text information 1903, and adaptive task information 1931 as input data, and generates a fine-tuned language model 1941.

[0102] In Example 4, the pre-trained language model 141 may be the pre-trained language model 141 obtained using the learning devices 100, 900, and 1500 described in Examples 1 to 3, or it may be any pre-trained language model.

[0103] Labeled text information 1903 is data in which each line of text information 103 is assigned a correct label (simply referred to as "label" to distinguish it from value 542 (correct label) in label dictionary 121) related to that text. Labeled text information 1903 can be represented as data in any format, but for example, each line of text and label can be represented as tabular data.

[0104] Furthermore, the pre-trained language model 141 can be any pre-trained language model, such as a model trained using mask language modeling or a model trained using a Shuffled Word Detection task like ELECTRA.

[0105] [Adaptive Task Information 1931] Figure 20 is an explanatory diagram showing an example of adaptive task information 1931. Adaptive task information 1931 is information about the label prediction method and includes task type 2001 and number of classes 2002. Task type 2001 indicates the type of task to be performed by the pre-trained language model 141. Number of classes 2002 is the number of classes classified by task type 2001 and matches the number of classes in labeled text information 1903.

[0106] The pre-training unit 130 acquires an arbitrary pre-trained language model 141 and performs pre-training on the pre-trained language model 141 based on the label dictionary 121. Therefore, in Example 4, the pre-training unit 130 does not initialize and train the parameters of the machine learning model, but further trains the pre-trained model 141 specified as input.

[0107] The label prediction learning unit 1930 adds an output layer to the pre-trained language model 141 that outputs a class number of 2002. The label prediction learning unit 1930 inputs the sentences from each line of the labeled text information 1903 into the pre-trained language model 141 with the added output layer, and fine-tunes the model using the output from the added output layer and a loss function based on the labels to obtain a fine-tuned language model 1941. For example, if the task type 2001 is "classification", the cross-entropy function can be used, and if the task type 2001 is "regression", the Mean Squared Error function can be used, and in this case the class number 2002 is conveniently set to "1".

[0108] [Settings screen] Figure 21 is an explanatory diagram showing an example of a settings screen when performing additional pre-training and fine-tuning. The settings screen 2100 includes a load button 2101, a pre-training task display unit 2102, an additional training task display unit 2103, an applied task display unit 2104, and a training execution button 2106.

[0109] The load button 2101 is a user interface for loading a pre-trained language model 141, a tokenizer 101, prediction task information 102, labeled text information 1903, and adaptive task information 1931 via user operation.

[0110] The pre-training task display unit 2102 is an area that displays pre-training tasks. In Figure 21, the pre-training task display unit 2102 displays "MLM," which means that the pre-training unit 130 is executing MLM (Masked Language Modeling).

[0111] The additional learning task display unit 2103 is an area that displays additional learning tasks. In Figure 21, the additional learning task display unit 2103 displays "4 characters|beginning", which means that the number of predicted characters 201 in the prediction task information 102 is "4" and the predicted character direction 202 is "beginning".

[0112] The applied task display unit 2104 is an area that displays the applied task. In Figure 21, the applied task display unit 2104 displays "Classification|6", which means that the task type 2001 of the applied task information 1931 is "Classification" and the number of classes 2002 is "6".

[0113] The learning execution button 2106 is a user interface for performing additional pre-training and fine-tuning via user operation.

[0114] When loading is complete by pressing the load button 2101, the pre-training tasks are displayed in the pre-training task display unit 2102, the additional training tasks are displayed in the additional training task display unit 2103, and the applied task display unit 2104 is displayed.

[0115] When the learning execution button 2105 is pressed, the learning device 1900 performs additional pre-training and fine-tuning on the adaptive task based on the conditions displayed on the settings screen 2100. The information can be updated by pressing the load button 2101 again and selecting a different prediction task information 102 or adaptive task information 1931.

[0116] Thus, according to Example 4, the learning device 1900 can further pre-train the pre-trained language model 141 and fine-tune it at the target of the adaptive task, allowing the existing language model to be trained on a scale appropriate to the user's environment.

[0117] <Example hardware configuration for a computer that implements learning devices 100, 900, 1500, and 1900> Figure 22 is a block diagram showing an example hardware configuration of a computer 2200 that implements the learning devices 100, 900, 1500, and 1900 according to Examples 1 to 4. The computer 2200 is composed of, for example, a processor (CPU) 2201, an auxiliary storage device 2202, a memory 2203, an input device 2204, an output device 2205, and a communication interface 2206. The components of the example hardware configuration of the computer 2200, from the processor 2201 to the communication interface 2206, are interconnected and can communicate as needed. The auxiliary storage device 2202 and the memory 2203 are collectively referred to as storage devices.

[0118] The processor 2201 executes the program stored in the memory 2203. The processor 2201 may consist of, for example, a single arithmetic unit and processing unit, or any number of arithmetic units and processing units. The memory 2203 includes a non-volatile memory element, ROM (Read Only Memory), and a volatile memory element, RAM (Random Access Memory). The ROM stores immutable programs, etc. The RAM temporarily stores the program executed by the processor 2201 and data used when the program is executed.

[0119] The auxiliary storage device 2202 is a high-capacity, non-volatile storage device such as a magnetic storage device (Hard Disk Drive) or flash memory (Solid State Drive). The auxiliary storage device 2202 also stores the program executed by the processor 2201, and the data used during program execution. In other words, the program is loaded from the auxiliary storage device 2202, read into memory 2203, and executed by the processor 2201.

[0120] Specifically, for example, the processing units of the learning devices 100, 900, 1500, and 1900 shown in Figures 1, 9, 15, and 19 are realized by loading a non-temporary program stored in the auxiliary storage device 2202 into the memory 2203, and the processor 2201 executing the loaded program.

[0121] Furthermore, the tokenizer 101, prediction task information 102, text information 103, label dictionary 121, pre-trained language model 141, required accuracy setting information 901, computing resource usage setting information 902, 1502, personal information protection requirement setting information 903, pre-trained language model 141, labeled text information 1903, adaptive task information 1931, and fine-tuned language model 1941 data used in each processing unit of Examples 1 to 4 shown in Figures 1, 9, 15, and 19 are stored, for example, in auxiliary storage device 2202.

[0122] Computer 2200 may have an input device 2204. The input device 2204 is a device on which the user inputs prediction task information 102, text information 103, prediction task information 102, required accuracy setting information 901, computer resource usage setting information 902, 1502, personal information protection requirement setting information 903, labeled text information 1903, and adaptive task information 1931 to learning devices 100, 900, 1500, and 1900. The input device 2204 may be, for example, a keyboard or a mouse. The input from the input device 2204 may be stored in auxiliary storage device 2202 or memory 2203.

[0123] Computer 2200 may have an output device 2205. The output device 2205 may be connected to, for example, a display or a printer, and will present the execution results output by the learning devices 100, 900, 1500, and 1900 to the user. If the output device 2205 is a display, for example, it can display the setting screens 800, 1400, and 2100 shown in Figures 8, 14, and 21.

[0124] The communication interface 2206 is a network interface device that controls communication with other devices according to a predetermined protocol. The communication interface 2206 includes, for example, a serial interface such as USB. The learning devices 100, 200, 300, and 400 can send and receive data from any terminal via the network.

[0125] In learning devices 100, 900, 1500, and 1900, for example, the program executed by processor 2201 may include an OS (Operating System) or arbitrary software. In this case, the OS or arbitrary software is stored, for example, in auxiliary storage device 2202 and loaded into memory 2203 as needed.

[0126] Various forms are possible for the computer 2200. For example, the learning devices 100, 900, 1500, and 1900 can be implemented on a single physical computer, or on a computer system consisting of multiple logically or physically configured computers. They may also operate on a virtual computer realized on multiple physical computing resources.

[0127] Furthermore, the numerical portion expressed as "numerical value" or "numerical value sequence" in Examples 1 to 4 described above is not limited to numerical values; it may also consist of strings of characters such as alphabets and symbols, or strings that include both characters and numerical values.

[0128] Furthermore, in the above-described embodiments 1 to 4, the label dictionary generation unit 110 and the pre-training unit 130 were implemented on the same computer 2200, but the label dictionary generation unit 110 and the pre-training unit 130 may be implemented on separate computers 2200.

[0129] As explained above, the granularity of the label dictionary 121 can be adjusted according to Examples 1 to 4 described above. Specifically, if the granularity of the label dictionary 121 is finer, the learning accuracy of the language model can be improved, and if the granularity of the label dictionary 121 is coarser, the memory usage of the label dictionary 121 can be reduced and the computational load during language model training can be reduced. Furthermore, by using such a label dictionary 121, it becomes possible to perform pre-training of the language model with an appropriate granularity.

[0130] It should be noted that the present invention is not limited to the embodiments described above, but includes various modifications and equivalent configurations within the spirit of the attached claims. For example, the embodiments described above are described in detail to make the present invention easier to understand, and the present invention is not necessarily limited to having all of the described configurations. Furthermore, some of the configurations of one embodiment may be replaced with those of another embodiment. Furthermore, some of the configurations of one embodiment may be added to those of another embodiment. Furthermore, some of the configurations of each embodiment may be added, deleted, or replaced with other configurations.

[0131] Furthermore, each of the aforementioned configurations, functions, processing units, and processing means may be implemented in hardware, for example, by designing them as integrated circuits, or they may be implemented in software by having a processor interpret and execute programs that realize each function.

[0132] Information such as programs, tables, and files that implement each function can be stored in memory, hard disks, SSDs (Solid State Drives), or on recording media such as IC (Integrated Circuit) cards, SD cards, and DVDs (Digital Versatile Discs).

[0133] Furthermore, the control lines and information lines shown are those deemed necessary for explanation purposes and do not necessarily represent all control lines and information lines required for implementation. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0134] 100 Learning Devices 101 Tokenizer 102 Predictive Task Information 103 Text Information 110 Label dictionary generation unit 121 Label Dictionary 130 Pre-learning Section 141 Pre-trained language models 201 Predicted character count 202 Predicted text direction 401 Extraction part 402 Deduplication section 403 Identifier Assignment Section 404 Label Dictionary Generation Unit 500 dictionaries 541 Key (Token) 542 Value (Correct Label) 600 Pre-learning Section 601 Tokenization Department 602 Label generation unit 603 Masking section 604 Model Learning Department 701 Token Sequence 702 Correct Label Sequence 703 Masked Token Column 704 Mask Tokens 900 Learning Device 901 Request accuracy setting information 902 Computer Resource Usage Settings Information 903 Personal information protection requirement setting information 904 Predictive text direction setting information 910 Prediction Task Information Generation Unit 1500 Learning Devices 1502 Computer Resource Usage Settings Information 1530 Pre-learning session 1701 Tokenization Department 1900 Learning device 1903 Labeled text information 1930 Label Prediction Learning Unit 1941 Fine-tuned language model

Claims

1. A learning device having a processor for executing a program and a storage device for storing the program, The memory device stores a word dictionary that associates each word in a word group with a token, which is a unique string of characters associated with that word. The aforementioned processor, An extraction process that extracts a portion of a string from each word in the aforementioned group of words according to a variable rule, A duplicate removal process is performed to remove duplicates when there are multiple identical strings extracted by the extraction process, A labeling process that assigns a unique correct label to some of the aforementioned strings, A first generation process generates a label dictionary that associates the first correct label assigned by the assignment process with the token associated with the source word of the partial string to which the first correct label was assigned in the word dictionary, A learning device characterized by performing the following actions.

2. A learning device according to claim 1, The aforementioned processor, Based on the required precision specified externally, a configuration process is executed to set the variable rule. In the extraction process, the processor extracts a portion of the string from each word in the word group according to the variable rule set by the setting process. A learning device characterized by the following features.

3. A learning device according to claim 1, The aforementioned processor, Based on the usage status of computing resources within the learning device, a setting process is executed to set the variable rules. In the extraction process, the processor extracts a portion of the string from each word in the word group according to the variable rule set by the setting process. A learning device characterized by the following features.

4. A learning device according to claim 1, The aforementioned processor, Based on the level of personal information protection requirements specified externally, the system executes a configuration process to set the aforementioned variable rules. In the extraction process, the processor extracts a portion of the string from each word in the word group according to the variable rule set by the setting process. A learning device characterized by the following features.

5. A learning device according to claim 1, The aforementioned variable rule is a rule concerning the characters within the aforementioned word. A learning device characterized by the following features.

6. A learning device according to claim 5, The aforementioned variable rule is a rule that extracts a predetermined number of characters from a character at a specific position in the word. A learning device characterized by the following features.

7. A learning device according to claim 6, The aforementioned variable rule is a rule that extracts a predetermined number of characters from the first character of the word. A learning device characterized by the following features.

8. A learning device according to claim 6, The aforementioned variable rule is to extract a predetermined number of characters from a string, working backward from the last character of the word. A learning device characterized by the following features.

9. A learning device according to claim 6, The aforementioned processor, A setting process that determines the character at the specific position to be either the first or last character of the word based on the language type of the word group, and sets the variable rule. A learning device characterized by performing the following actions.

10. A learning device according to claim 5, The aforementioned variable rule is a rule that extracts specific characters contained in the aforementioned word. A learning device characterized by the following features.

11. A learning device according to claim 1, A second generation process that generates a token sequence by converting words in a text sentence into tokens by referring to the aforementioned word dictionary, A third generation process that generates a sequence of correct labels by referring to the label dictionary and converting each token in the token sequence into the first correct label, A fourth generation process that generates a masked token sequence by replacing some of the tokens in the aforementioned token sequence with mask tokens, which are strings that are not assigned to any of the words in the aforementioned word group. A second learning process involves inputting the masked token sequence into a first language model that predicts the first ground truth label, causing the first language model to output a first prediction result for a specific first ground truth label corresponding to the tokens replaced by the masked tokens in the ground truth label sequence, and learning the first language model based on the first prediction result and the specific first ground truth label. A learning device characterized by performing the following actions.

12. A learning device according to claim 11, In the second generation process, the processor adjusts the length of the token sequence based on the usage of computing resources within the learning device. A learning device characterized by the following features.

13. A learning device according to claim 11, The aforementioned processor, A second learning process involves obtaining text information relating the aforementioned text sentence to a second correct label, obtaining task information regarding a method for predicting the second correct label, generating a second language model that predicts the updated second correct label from the first language model based on the task information, inputting the text sentence into the second language model, outputting a second prediction result for the second correct label from the second language model, and learning the second language model based on the second prediction result and the second correct label. A learning device characterized by performing the following actions.

14. A learning method performed by a learning device having a processor for executing a program and a storage device for storing the program, The memory device stores a word dictionary that associates each word in a word group with a token, which is a unique string of characters associated with that word. The aforementioned processor, An extraction process that extracts a portion of a string from each word in the aforementioned group of words according to a variable rule, A duplicate removal process is performed to remove duplicates when there are multiple identical strings extracted by the extraction process, A labeling process that assigns a unique first correct label to the aforementioned string, A first generation process generates a label dictionary that associates the first correct label assigned by the assignment process with the token associated with the source word of the partial string to which the first correct label was assigned in the word dictionary, A learning method characterized by performing the following.

15. In the processor, An extraction process that extracts a portion of a string from each word in a word dictionary, which associates each word in a word group with a token that is a unique string associated with that word, according to a variable rule. A duplicate removal process is performed to remove duplicates when there are multiple identical strings extracted by the extraction process, A labeling process that assigns a unique first correct label to the aforementioned string, A first generation process generates a label dictionary that associates the first correct label assigned by the assignment process with the token associated with the source word of the partial string to which the first correct label was assigned in the word dictionary, A learning program characterized by causing the execution of [a specific action].

Images