Learning method determination device, learning method determination method, and learning method determination program

Abstract

To efficiently train language models. [Solution] The learning method determination device includes: an acquisition unit that acquires a first computational resource amount, which is a constraint on the amount of computational resources used in the learning process of a language model, and a target language resource amount, which is the amount of available resources for the target language; a threshold determination unit that refers to the first computational resource amount and determines a first threshold that is referenced to determine the schedule of the learning process; a comparison unit that compares the target language resource amount and the first threshold; and a schedule determination unit that refers to the comparison result and determines the schedule.

Description

Technical Field

[0001] The present disclosure relates to a learning method determination device, a learning method determination method, and a learning method determination program.

Background Art

[0002] Techniques related to the learning of language models are known. For example, Non-Patent Document 1 discloses a technique for training (two-stage training) a language model trained with an English corpus using a Japanese corpus.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technique described in Non-Patent Document 1, it has not been examined how to change the ratio of the English corpus and the Japanese corpus in learning for better efficiency. In other words, in the technique described in Non-Patent Document 1, regarding the corpus (language resource amount) used in learning, the ratio between Japanese, which is the target language of the language model, and English, which is used as an auxiliary language, has not been examined. Therefore, there is a need for a technique to train a language model more efficiently than the technique described in Non-Patent Document 1.

[0005] This disclosure has been made in view of the above-mentioned issues, and one exemplary purpose is to provide a technique for efficiently training language models. [Means for solving the problem]

[0006] A learning method determination device relating to an exemplary aspect of this disclosure includes: acquisition means for acquiring a first amount of computational resources, which is a constraint on the amount of computational resources used in the learning process of a language model for a target language, and a target language resource amount, which is the amount of resources of the target language available in the learning process; threshold determination means for determining a first threshold, which is referenced to determine a schedule for the learning process of the language model, in which the proportion of the amount of language resources used in the learning of the language model that the target language resource amount is used is, by referring to the first amount of computational resources; comparison means for comparing the target language resource amount with the first threshold; and schedule determination means for determining the schedule by referring to the comparison result by the comparison means.

[0007] A learning method determination method relating to an exemplary aspect of the present disclosure includes: an acquisition process in which at least one processor acquires a first amount of computational resources, which is a constraint on the amount of computational resources used for training a language model for a target language, and a target language resource amount, which is the amount of resources of the target language available in the training process; a threshold determination process in which the at least one processor refers to the first amount of computational resources and determines a first threshold, which is referenced to determine a schedule for training the language model, in which the proportion of the amount of language resources used in training the language model that the target language resource amount is used is determined; a comparison process in which the at least one processor compares the target language resource amount with the first threshold; and a schedule determination process in which the at least one processor refers to the comparison result in the comparison process and determines the schedule.

[0008] An exemplary aspect of the present disclosure relates to a learning method determination program, which is a program that causes a computer to function as a learning method determination device, and the computer is configured to function as: an acquisition means for acquiring a first amount of computational resources, which is a constraint on the amount of computational resources used in the learning process of a language model for a target language, and a target language resource amount, which is the amount of resources of the target language available in the learning process; a threshold determination means for determining a first threshold, which is referenced to determine a schedule for the learning process of the language model, in which the proportion of the amount of language resources used in the learning process of the language model that the target language resource amount is used is, by referring to the first amount of computational resources; a comparison means for comparing the target language resource amount with the first threshold; and a schedule determination means for determining the schedule by referring to the comparison result of the comparison means. [Effects of the Invention]

[0009] One exemplary effect of this disclosure is that it can provide a technique for efficiently training language models. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram showing the configuration of the learning method determination device related to this disclosure. [Figure 2] This flowchart shows the flow of the learning method determination process related to this disclosure. [Figure 3] This graph shows the relationship between the number of unique characters in the text corpus of the target language and the loss, as disclosed in this document. [Figure 4] This is a block diagram showing the configuration of the learning method determination device related to this disclosure. [Figure 5] This flowchart shows the flow of the learning method determination process related to this disclosure. [Figure 6] This diagram shows the configuration of a computer that functions as a learning method determination device related to this disclosure. [Modes for carrying out the invention]

[0011] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining some or all of the technologies (things or methods) employed in each of the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technologies employed in each of the exemplary embodiments shown below may also be included in the scope of the present invention. In addition, the effects mentioned in each of the exemplary embodiments shown below are examples of effects that can be expected in that exemplary embodiment and do not define the scope of the present invention. That is, embodiments that do not produce the effects mentioned in each of the exemplary embodiments shown below may also be included in the scope of the present invention.

[0012] [First Exemplary Embodiment] A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is the basic form for each of the exemplary embodiments described later. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems occur. Furthermore, each technology shown in the drawings referenced to explain this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems occur.

[0013] (Configuration of Learning Method Determination Device 1) The configuration of the learning method determination device 1 will be described with reference to Figure 1. Figure 1 is a block diagram showing the configuration of the learning method determination device 1. As shown in Figure 1, the learning method determination device 1 includes an acquisition unit 11, a threshold determination unit 12, a comparison unit 13, and a schedule determination unit 14.

[0014] (Acquisition part 11) The acquisition unit 11 acquires a first computational resource amount, which is a constraint on the amount of computational resources used for the learning process of the language model for the target language, and a target language resource amount, which is the amount of resources of the target language available in the learning process. The acquisition unit 11 supplies the acquired first computational resource amount to the threshold determination unit 12. Further, the acquisition unit 11 supplies the acquired target language resource amount to the comparison unit 13.

[0015] (Threshold determination unit 12) The threshold determination unit 12 refers to the first computational resource amount and determines a first threshold that is referred to for determining a schedule of the learning process of the language model, which is a schedule of the ratio of the target language resource amount used among the language resource amounts used in the learning process of the language model. The threshold determination unit 12 supplies the determined first threshold to the comparison unit 13.

[0016] (Comparison unit 13) The comparison unit 13 compares the target language resource amount with the first threshold. The comparison unit 13 supplies the comparison result to the schedule determination unit 14.

[0017] (Schedule determination unit 14) The schedule determination unit 14 refers to the comparison result by the comparison unit 13 and determines a schedule of the ratio of the target language resource amount used among the language resource amounts used in the learning process of the language model.

[0018] (Effect of the learning method determination device 1) As described above, the learning method determination device 1 employs a configuration comprising: an acquisition unit 11 that acquires a first computational resource amount, which is a constraint on the amount of computational resources used in the learning process of a language model for a target language, and a target language resource amount, which is the amount of target language resources available in the learning process; a threshold determination unit 12 that refers to the first computational resource amount and determines a first threshold that is referenced to determine the schedule for the language model learning process, which is the proportion of the amount of language resources used in the language model learning process in which the target language resources are used; a comparison unit 13 that compares the target language resources with the first threshold; and a schedule determination unit 14 that refers to the comparison result by the comparison unit 13 and determines the schedule for the proportion of the amount of language resources used in the language model learning process in which the target language resources are used.

[0019] Therefore, the learning method determination device 1 has the effect of enabling efficient training of the language model.

[0020] (Flowchart for determining learning methods) The flow of the learning method determination method S1 will be explained with reference to Figure 2. Figure 2 is a flowchart showing the flow of the learning method determination method S1. As shown in Figure 2, the learning method determination method S1 includes an acquisition process S11, a threshold determination process S12, a comparison process S13, and a schedule determination process S14.

[0021] (Acquisition process S11) In acquisition process S11, the acquisition unit 11 acquires a first computational resource amount, which is a constraint on the amount of computational resources used for the training process of a language model for the target language, and a target language resource amount, which is the amount of resources of the target language available in the training process. The acquisition unit 11 supplies the acquired first computational resource amount to the threshold determination unit 12. The acquisition unit 11 also supplies the acquired target language resource amount to the comparison unit 13.

[0022] (Threshold determination process S12) In threshold determination process S12, the threshold determination unit 12 refers to the first amount of computational resources and determines a first threshold that is referenced to determine the schedule for the language model learning process, specifically the schedule for determining the proportion of the total amount of language resources used in the language model learning process that is the target amount of language resources. The threshold determination unit 12 supplies the determined first threshold to the comparison unit 13.

[0023] (Comparison process S13) In comparison processing S13, the comparison unit 13 compares the amount of target language resources with a first threshold. The comparison unit 13 supplies the comparison result to the schedule determination unit 14.

[0024] (Schedule determination process S14) In the schedule determination process S14, the schedule determination unit 14 refers to the comparison result from the comparison unit 13 and determines the schedule for the proportion of the target language resource used in the language model learning process.

[0025] (Effect of Learning Method Determination Method S1) As described above, the learning method determination method S1 employs a configuration that includes: an acquisition process S11 in which the acquisition unit 11 acquires a first computational resource amount, which is a constraint on the amount of computational resources used in the learning process of a language model for the target language, and a target language resource amount, which is the amount of target language resources available in the learning process; a threshold determination process S12 in which the threshold determination unit 12 refers to the first computational resource amount and determines a first threshold that is referenced to determine the schedule for the language model learning process, which is the proportion of the amount of language resources used in the language model learning process in which the target language resources are used; a comparison process S13 in which the comparison unit 13 compares the target language resources with the first threshold; and a schedule determination process S14 in which the schedule determines the proportion of the amount of language resources used in the language model learning process in which the target language resources are used, referring to the comparison result by the comparison unit 13.

[0026] [Second exemplary embodiment] A second exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same function as those described in the above-described exemplary embodiment are denoted by the same reference numerals, and their descriptions are omitted as appropriate. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise. Furthermore, each technology shown in the drawings referenced to describe this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise.

[0027] Training language models (hereinafter also referred to as "LLMs (Large Language Models)") requires a large amount of text corpus. However, languages ​​other than English have relatively few text corpora. Therefore, the following methods are known for training language models for languages ​​with limited text corpus resources. • A method of learning by repeatedly using the same text corpus multiple times (multi-epoch learning) • A method of learning that uses text corpora of other languages ​​in addition to the target language (multilingual learning). • A two-stage learning method in multilingual learning, where the proportion of the target language and other languages ​​is gradually changed. However, when training a language model using a combination of the methods described above, the number of training settings (hyperparameters) increases, making exhaustive exploration costly.

[0028] Therefore, engineers training language models heuristically narrowed down the search space based on previously obtained analysis results regarding how the performance of language models changes depending on the training settings. However, the analysis of language model training settings that has been conducted in the past has been limited, and a problem has arisen in that the optimal search space cannot be narrowed down when training an LLM by combining the above methods.

[0029] Therefore, the inventors are investigating how to narrow down the search space for learning settings that are expected to yield high performance when training a language model targeting a language with limited resources using some or all of the multi-epoch learning, multilingual learning, and two-stage learning described above.

[0030] As an example, the inventors have found that when the amount of computational resources used for training an LLM is fixed to a certain value, the optimal training method changes depending on whether the number of unique elements in the text corpus of the target language used for training (the number of unique elements excluding repetitions if there are multiple epochs) is above or below a certain threshold.

[0031] Figure 3 shows a graph that forms the basis of the findings obtained by the inventors. Figure 3 is a graph showing the relationship between the number of unique elements in the text corpus of the target language and the loss. The graph in Figure 3 is for the case where the target language is Japanese.

[0032] In the graph shown in Figure 3, the horizontal axis represents the base-2 logarithm of the uniqueness quantity of the target language text corpus relative to the baseline quantity. The vertical axis represents the minimum LLM loss achievable for the corresponding uniqueness quantity of the text corpus; a smaller value indicates better LLM performance. The dashed line represents multi-epoch learning using only the target language, the dotted line represents multilingual learning, and the solid line represents two-stage learning.

[0033] Figure 3 shows that when the horizontal axis value is between -5 and -3 (when the number of unique characters in the text corpus is small), LLM performance is best when using two-stage learning, and when the horizontal axis value is -3 or greater (when the number of unique characters in the text corpus is large), the performance of all learning methods is the same. Here, multi-epoch learning, multilingual learning, and two-stage learning are inclusion relationships, with two-stage learning including multilingual learning, and multilingual learning including multi-epoch learning. Therefore, when the horizontal axis value is -3 or greater and the performance of all learning methods is the same, multi-epoch learning is the optimal learning method.

[0034] In other words, the inventors found that multilingual learning is the optimal learning method when the number of unique elements in the text corpus is below a certain threshold, and multi-epoch learning is the optimal learning method when the number of unique elements in the text corpus is above a certain threshold.

[0035] The learning method determination device 1A and each process performed by the learning method determination device 1A, as described below, are based on the knowledge mentioned above and are based on the inventor's unique perspective.

[0036] (Overview of Learning Method Determination Device 1A) The learning method determination device 1A is a device that determines the appropriate learning method for learning an LLM. An appropriate learning method is one that minimizes learning loss. In this disclosure, as an example, the learning method determination device 1A determines whether multi-epoch learning or two-stage learning is the appropriate learning method. In other words, the learning method determination device 1A is a device that determines, in learning an LLM, what schedule is appropriate (minimizes loss) for changing the proportion of the target language resources, which are the resources of the target language of the LLM, that are used in the total amount of language resources used for learning.

[0037] Specifically, the learning method determination device 1A refers to the first computational resource amount CR1, which is a constraint on the amount of computational resources used in the learning process of the LLM for the target language, and determines the first threshold TH1. Then, the learning method determination device 1A determines an appropriate learning method based on the comparison result between the first threshold TH1 and the target language resource amount T_unique, which is the amount of target language resources available in the learning process.

[0038] The first computational resource constraint, CR1, which is the amount of computational resources used in the training process of an LLM for a target language, is the amount of resources that the device performing the LLM training can use for the training process. One example is the total amount of computation that can be used for the training process, measured in units of FLOP (Floating-point Operation).

[0039] The target language resource quantity T_unique, which is the amount of target language resources available in the learning process, is the unique quantity of the target language text corpus that has been collected and is available for use in the LLM learning process (excluding repetitions if there are multiple epochs). One example is the unique quantity of text corpora of all target languages ​​that exist on Earth.

[0040] Furthermore, the term "language" in this disclosure includes not only natural languages ​​such as Japanese and English, but also dialects and words and sentences used in specific fields (domains) such as medicine.

[0041] Furthermore, the term "language resource volume" in this disclosure is not particularly limited and may refer to the resource volume of any language, or the resource volume of any language used in any processing (processing of the entire learning process, processing in one or more epochs, processing in one or more stages, processing in one or more steps, etc.). Also, if the number of epochs is two or more, it may refer to the resource volume excluding duplicates, or the resource volume including duplicates.

[0042] Another example of a learning method determined by the learning method determination device 1A is a method of learning in multilingual learning in which the language ratio of the target language and several other languages ​​is changed in multiple stages. For example, this could be a three-stage learning method in which the language ratio of the target language and two other languages ​​is changed in three stages, or a four-stage learning method in which the language ratio of the target language and three other languages ​​is changed in four stages. Yet another example is a method in which the language ratio is changed in step units.

[0043] Furthermore, the learning method determination device 1A may be configured to determine multiple first thresholds TH1. For example, if the learning method determination device 1A selects one of the multiple learning methods, it may be configured to determine multiple first thresholds TH1 corresponding to the number of learning methods.

[0044] (Configuration of Learning Method Determination Device 1A) The configuration of the learning method determination device 1A will be described with reference to Figure 4. Figure 4 is a block diagram showing the configuration of the learning method determination device 1A. As shown in Figure 4, the learning method determination device 1A includes a control unit 10, a storage unit 20, an input / output unit 21, and a communication unit 22.

[0045] (Storage unit 20) The memory unit 20 stores data that the control unit 10 references. As an example, as shown in Figure 4, the memory unit 20 stores the machine learning model TM, training data TD, first computational resource amount CR1, target language resource amount T_unique, and first threshold TH1.

[0046] The machine learning model TM is a machine learning model (regression model) trained using the training data TD to take the amount of computational resources as input and output a threshold corresponding to that amount of computational resources. When the machine learning model TM is said to be stored in the memory unit 20, it means that the parameters defining the machine learning model TM are stored in the memory unit 20.

[0047] The training data TD is the data used to train the machine learning model TM. As shown in Figure 4, the training data TD contains multiple sets of second computational resources CR2 and second thresholds TH2. The value of the second computational resource CR2 is not particularly limited, but as an example, the second computational resource CR2 is less than the first computational resource CR1. The process of training the machine learning model TM using the training data TD will be described later.

[0048] The first computational resource quantity CR1 and the target language resource quantity T_unique are as described above.

[0049] The first threshold TH1 is a schedule for the LLM training process, and is a threshold referenced to determine the schedule for the proportion of the total amount of language resources used in the LLM training process that uses the target language resource T_unique. The method for determining the first threshold TH1 will be described later.

[0050] (Input / output section 21) The input / output unit 21 is an interface to input devices that accept data input and output devices that output data. Examples of input devices include, but are not limited to, microphones, cameras, eye-tracking devices, keyboards, and touchpads. Examples of output devices include, but are not limited to, speakers and liquid crystal displays.

[0051] (Communications Section 22) The communication unit 22 is an interface for sending and receiving data over a network. Examples of the communication unit 22 include, but are not limited to, communication chips in various communication standards such as Ethernet®, Wi-Fi®, and wireless communication standards for mobile data communication networks, as well as USB-compliant connectors.

[0052] (Control Unit 10) The control unit 10 controls each component of the learning method determination device 1A. As shown in Figure 4, the control unit 10 also includes an acquisition unit 11, a threshold determination unit 12, a comparison unit 13, a schedule determination unit 14, a learning unit 15, and an output unit 16. In this exemplary embodiment, the acquisition unit 11, threshold determination unit 12, comparison unit 13, schedule determination unit 14, and output unit 16 each realize an acquisition means, a threshold determination means, a comparison means, a schedule determination means, and an output means, respectively.

[0053] (Acquisition part 11) The acquisition unit 11 acquires data supplied from the input / output unit 21 or the communication unit 22. The acquisition unit 11 stores the acquired data in the storage unit 20. As an example, the acquisition unit 11 acquires the first computational resource amount CR1 and the target language resource amount T_unique. As another example, the acquisition unit 11 acquires the training data TD, which consists of the second computational resource amount CR2 and the second threshold TH2.

[0054] (Threshold determination unit 12) The threshold determination unit 12 determines the first threshold TH1. The threshold determination unit 12 stores the determined first threshold TH1 in the storage unit 20. As an example, the threshold determination unit 12 determines the first threshold TH1 by referring to the first computational resource amount CR1. As an example of this configuration, the threshold determination unit 12 determines the first threshold TH1 using the training data TD. With this configuration, the threshold determination unit 12 can determine the first threshold TH1 by referring to the value of the training data TD calculated by preliminary experiments, etc., and thus can determine an appropriate first threshold TH1.

[0055] As an example of how the threshold determination unit 12 determines the first threshold TH1 using the training data TD, the threshold determination unit 12 determines the first threshold TH1 using a machine learning model TM trained on the training data TD. More specifically, the threshold determination unit 12 inputs the first computational resource amount CR1 to the machine learning model TM1 and sets the threshold output from the machine learning model TM as the first threshold TH. With this configuration, the threshold determination unit 12 can determine an appropriate first threshold TH1.

[0056] Another example of how the threshold determination unit 12 determines the first threshold TH1 using the training data TD is that if the second computational resource amount CR2 and the second threshold TH2 included in the training data TD follow a power law, the threshold determination unit 12 determines the first threshold TH using a power law model. More specifically, the value T of the second computational resource amount CR2 * The value of the second computing resource CR2 is C * Let's assume the following case. In this case, logT * logC * If the expression is a linear function, the threshold determination unit 12 determines the first threshold TH1 by inputting the value of the first computational resource amount CR1 into the linear function.

[0057] (Comparison Section 13) The comparison unit 13 compares the target language resource amount T_unique with the first threshold TH1 and supplies the comparison result to the schedule determination unit 14. In other words, the comparison unit 13 supplies the schedule determination unit 14 with a comparison result indicating whether or not the target language resource amount T_unique is greater than or equal to the first threshold TH1.

[0058] (Schedule determination section 14) The schedule determination unit 14 refers to the first threshold TH1 and determines the schedule for the proportion of the total amount of language resources used in the LLM learning process that will be the target language resource T_unique.

[0059] More specifically, if the comparison result from the comparison unit 13 indicates that the target language resource amount T_unique is greater than or equal to the first threshold TH1, the schedule determination unit 14 determines the schedule for training the LLM to be a multi-epoch learning schedule, which is a one-stage learning method that uses only the target language in the LLM training process. On the other hand, if the comparison result from the comparison unit 13 indicates that the target language resource amount T_unique is less than the first threshold TH1, the schedule determination unit 14 determines the schedule for training the LLM to be a two-stage learning schedule, which is a two-stage learning method that uses multiple languages, including the target language and languages ​​other than the target language, in the LLM training process.

[0060] With this configuration, the schedule determination unit 14 can determine whether multi-epoch learning or two-stage learning is appropriate, based on the first computational resource amount CR1 and the target language resource amount T_unique.

[0061] (Learning Section 15) The learning unit 15 trains a machine learning model. For example, the learning unit 15 trains the machine learning model TM using the training data TD stored in the memory unit 20. More specifically, when the learning unit 15 inputs the second computational resource amount CR2 contained in the training data TD to the machine learning model TM, it trains the machine learning model TM so that the threshold output from the machine learning model TM becomes the second threshold TH2 associated with the input second computational resource amount CR2.

[0062] Furthermore, as mentioned above, the second computational resource amount CR2 included in the training data TD may be less than the first computational resource amount CR1. With this configuration, the learning unit 15 can efficiently train the machine learning model TM because the first computational resource amount CR1 is less than the second computational resource amount CR2.

[0063] Furthermore, the learning unit 15 may train the machine learning model TM using training data TD, which includes the number of epochs, the model size of the LLM, the number of training steps, the ratio of the length of the first training stage, and the ratio of the amount of target language between the first and second stages, in addition to the second computational resource amount CR2 and the second threshold TH2.

[0064] As another example, the learning unit 15 trains the LLM using the first computational resource CR1 and the target language resource T_unique according to the schedule determined by the schedule determination unit 14. Known methods may be used as the method by which the learning unit 15 trains the LLM according to the schedule determined by the schedule determination unit 14. Also, when the learning unit 15 trains the LLM, the learning unit 15 performs a process to determine the learning settings such as the model size, and then trains the LLM according to the schedule determined by the schedule determination unit 14.

[0065] As another example, the learning unit 15 may instruct an external device, different from the learning method determination device 1A, to train the LLM according to the schedule determined by the schedule determination unit 14. In this case, the learning unit 15 may instruct the external device to narrow the range of learning settings to select using the schedule determined by the schedule determination unit 14. This configuration allows the learning unit 15 to reduce the search space for the external device.

[0066] (Output section 16) The output unit 16 outputs data via the input / output unit 21 or the communication unit 22. For example, the output unit 16 outputs information including at least one of the first threshold TH1 and the schedule determined by the schedule determination unit 14. With this configuration, the output unit 16 can notify the user of at least one of the threshold at which the proportion of target language resource T_unique used in training changes, and the schedule for the proportion of target language resource T_unique used in training.

[0067] (Processing performed by the learning method determination device 1A) The flow of the process (learning method determination method S1A) executed by the learning method determination device 1A will be explained with reference to Figure 5. Figure 5 is a flowchart showing the flow of the learning method determination method S1A.

[0068] (Acquisition process S11) In the acquisition process S11, the acquisition unit 11 acquires the first computational resource amount CR1 and the target language resource amount T_unique. The acquisition unit 11 stores the acquired first computational resource amount CR1 and target language resource amount T_unique in the storage unit 20.

[0069] (Threshold determination process S12) In the threshold determination process S12, the threshold determination unit 12 refers to the first computational resource amount CR1 and determines the first threshold TH1. The threshold determination unit 12 stores the determined first threshold TH1 in the storage unit 20. An example of the process by which the threshold determination unit 12 determines the first threshold TH1 is as described above.

[0070] (Comparison process S13) In comparison processing S13, the comparison unit 13 compares the target language resource amount T_unique with the first threshold TH1 determined by the threshold determination unit 12 in threshold determination processing S12, and supplies the comparison result to the schedule determination unit 14.

[0071] (Schedule determination process S14) In the schedule determination process S14, the schedule determination unit 14 determines the schedule for the proportion of the target language resource T_unique used in the learning of the LLM. As an example, in the schedule determination process S14, the schedule determination unit 14 executes the following steps S141 to S143.

[0072] (Step S141) In step S141, the schedule determination unit 14 refers to the comparison result and determines whether the target language resource amount T_unique is greater than or equal to the first threshold TH1.

[0073] (Step S142) In step S141, if it is determined that the target language resource amount T_unique is greater than or equal to the first threshold TH1 (step S141: YES), the schedule determination unit 14 determines the schedule for training the LLM to be a multi-epoch learning schedule, which is a one-stage learning method that uses only the target language in the LLM training.

[0074] (Step S143) In step S141, if it is determined that the target language resource amount T_unique is less than the first threshold TH1 (step S141: NO), the schedule determination unit 14 determines the schedule for training the LLM to be a two-stage learning schedule, which is a two-stage learning method that uses multiple languages, including the target language and a different language, in the training of the LLM.

[0075] (Output processing S15) In output processing S15, the output unit 16 outputs information that includes at least one of the first threshold TH1 and the schedule determined by the schedule determination unit 14.

[0076] (Specific example) A specific example of the processing performed by the learning method determination device 1A is described below.

[0077] For example, in the acquisition process S11 described above, the acquisition unit 11 acquires 10^18 FLOPs as the first computational resource amount CR1 and 2 × 10^10 tokens as the target language resource amount T_unique.

[0078] Next, in threshold determination process S12, the threshold determination unit 12 refers to the first computational resource amount CR1 and determines the first threshold TH1.

[0079] For example, if the first threshold TH1 is 10^9 tokens, in the comparison process S13, the comparison unit 13 supplies the schedule determination unit 14 with a comparison result indicating that the target language resource amount T_unique is greater than or equal to the first threshold TH1.

[0080] In this case, during the schedule determination process S14, the schedule determination unit 14 determines the schedule for training the LLM to be a multi-epoch learning schedule, which is a one-stage learning method that uses only the target language in the LLM training (in other words, the proportion of the target language is 100%).

[0081] As another example, in acquisition process S11, if the target language resource amount T_unique acquired by the acquisition unit 11 is 5 × 10^8 tokens, then in comparison process S13, the comparison unit 13 supplies the schedule determination unit 14 with a comparison result indicating that the target language resource amount T_unique is less than the first threshold TH1.

[0082] In this case, during the schedule determination process S14, the schedule determination unit 14 determines the schedule for training the LLM to be a two-stage learning schedule, which is a two-stage learning method that uses multiple languages, including the target language and a different language, in the LLM training. As an example, the schedule determination unit 14 determines that the proportion of the target language in the first stage is 100%, and the proportion of the target language in the second stage is 0%.

[0083] Furthermore, if the schedule determination unit 14 has decided on a two-stage learning schedule for training the LLM, it may perform multiple pre-training sessions with different values ​​set for the proportion in which the target language resource amount T_unique is used, and then determine the schedule for training the LLM based on the proportion setting that yielded the best performance (lowest loss) among these multiple pre-training sessions.

[0084] Then, in output processing S15, the output unit 16 outputs information that includes at least one of the first threshold TH1 and the schedule determined by the schedule determination unit 14.

[0085] (Effects of the learning method determination device 1A) As described above, the learning method determination device 1A determines the learning schedule for the LLM based on the comparison result between the first threshold TH1, which is determined by referring to the first computational resource amount CR1, and the target language resource amount T_unique. As mentioned above, the appropriate learning method changes depending on whether the target language resource amount T_unique used for learning is greater than or equal to the first threshold TH1 or less than it. Since the learning method determination device 1A can determine an appropriate learning schedule by referring to the comparison result between the first threshold TH1 and the target language resource amount T_unique, the LLM can be trained efficiently.

[0086] Furthermore, by determining the learning schedule for the LLM, the learning method determination device 1A can reduce the search space required by the device training the LLM to determine other learning settings (e.g., model size). Therefore, the learning method determination device 1A can reduce the processing load on the device training the LLM.

[0087] [Examples of implementation using software] Some or all of the functions of the learning method determination devices 1 and 1A (hereinafter also referred to as "the above devices") may be implemented by hardware such as integrated circuits (IC chips) or by software.

[0088] In the latter case, each of the above devices is implemented, for example, by a computer that executes instructions for a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as Computer C) is shown in Figure 6. Figure 6 is a block diagram showing the hardware configuration of Computer C, which functions as each of the above devices.

[0089] Computer C comprises at least one processor C1 and at least one memory C2. Memory C2 stores a program P that causes computer C to operate as each of the above-mentioned devices. In computer C, processor C1 reads program P from memory C2 and executes it, thereby realizing each of the above-mentioned devices.

[0090] For processor C1, for example, a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof can be used. For memory C2, for example, flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof can be used.

[0091] Computer C may also be equipped with RAM (Random Access Memory) for loading program P at runtime and for temporarily storing various data. Furthermore, computer C may be equipped with communication interfaces for sending and receiving data with other devices. Additionally, computer C may be equipped with input / output interfaces for connecting input / output devices such as keyboards, mice, displays, and printers.

[0092] Furthermore, program P can be recorded on a non-temporary, tangible recording medium M that is readable by computer C. Such a recording medium M could be, for example, tape, disk, card, semiconductor memory, or programmable logic circuitry. Computer C can acquire program P via such a recording medium M. Program P can also be transmitted via a transmission medium. Such a transmission medium could be, for example, a communication network or broadcast waves. Computer C can also acquire program P via such a transmission medium.

[0093] Furthermore, each of the above functions of each of the above devices may be implemented by a single processor in a single computer, by multiple processors in a single computer working together, or by multiple processors in each of multiple computers working together. In addition, the programs for implementing each of the above functions in each of the above devices may be stored in a single memory in a single computer, distributed and stored in multiple memories in a single computer, or distributed and stored in multiple memories in each of multiple computers.

[0094] [Additional Note A] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0095] (Note A1) Acquisition means for acquiring a first computational resource amount, which is a constraint on the amount of computational resources used in the training process of a language model for the target language, and a target language resource amount, which is the amount of resources of the target language available in the training process, A threshold determination means that, referring to the first amount of computational resources, determines a first threshold that is referenced to determine a schedule for the training process of the language model, which is the schedule for determining the proportion of the amount of target language resources used in the training process of the language model that is used, A comparison means for comparing the amount of target language resources with the first threshold, A schedule determination means that determines the schedule by referring to the comparison results of the comparison means, A learning method determination device equipped with the following features.

[0096] (Appendix A2) The acquisition means further acquires training data consisting of a second computational resource amount and a second threshold, The threshold determination means determines the first threshold using the training data. A learning method determination device as described in Appendix A1.

[0097] (Note A3) The system further includes a learning unit that takes the amount of computing resources as input and trains a machine learning model using the training data to output a threshold corresponding to the amount of computing resources, The threshold determination means determines the first threshold using the machine learning model. The learning method determination device described in Appendix A2.

[0098] (Note A4) The second amount of computing resources is less than the first amount of computing resources. A learning method determination device as described in Appendix A2 or A3.

[0099] (Note A5) If the comparison result indicates that the amount of target language resources is equal to or greater than the first threshold, the schedule determination means determines the schedule to be a one-stage learning method schedule that uses only the target language in the language model learning process. If the comparison result indicates that the amount of target language resources is less than the first threshold, the schedule determination means determines the schedule to be a two-stage learning method using multiple languages, including the target language and languages ​​other than the target language, in the language model learning process. A learning method determination device as described in one of the appendices A1 to A4.

[0100] (Note A6) The system further includes an output means that outputs information indicating at least one of the first threshold and the schedule determined by the schedule determination means. A learning method determination device as described in any one of the appendices A1 to A5.

[0101] [Additional Note B] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0102] (Note B1) At least one processor performs an acquisition process to acquire a first amount of computational resources, which is a constraint on the amount of computational resources used for training a language model for the target language, and a target language resource amount, which is the amount of resources of the target language available in the training process. The at least one processor performs a threshold determination process that refers to the first amount of computational resources and determines a first threshold which is referenced to determine a schedule for the language model training process, in which the proportion of the amount of language resources used in the language model training process that the target language resources are used is determined. The at least one processor performs a comparison process that compares the amount of target language resources with the first threshold, The at least one processor performs a schedule determination process that refers to the comparison results in the comparison process and determines the schedule, A method for determining learning methods, including the method itself.

[0103] (Note B2) In the acquisition process described above, the at least one processor further acquires training data consisting of a second computational resource amount and a second threshold, In the threshold determination process, the at least one processor determines the first threshold using the training data. The method for determining the learning method is described in Appendix B1.

[0104] (Note B3) The at least one processor further includes a learning process that takes a computational resource amount as input and trains a machine learning model using the training data to output a threshold corresponding to the computational resource amount, In the threshold determination process, the at least one processor determines the first threshold using the machine learning model. The method for determining the learning method is described in Appendix B2.

[0105] (Note B4) The second amount of computing resources is less than the first amount of computing resources. The method for determining the learning method is as described in Appendix B2 or B3.

[0106] (Note B5) If the comparison result indicates that the amount of target language resources is equal to or greater than the first threshold, then in the schedule determination process, at least one processor determines the schedule to be a schedule for a one-stage learning method that uses only the target language in the language model learning process. If the comparison result indicates that the amount of target language resources is less than the first threshold, then in the schedule determination process, at least one processor determines the schedule to be a two-stage learning method using multiple languages, including the target language and languages ​​other than the target language, in the language model learning process. The method for determining the learning method is described in one of the appendices B1 through B4.

[0107] (Note B6) The at least one processor further includes an output process that outputs information indicating at least one of the first threshold and the schedule determined by the schedule determination process, The method for determining the learning method is described in one of the appendices B1 through B5.

[0108] [Additional Note C] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0109] (Note C1) A program that makes a computer function as a learning method determination device, The aforementioned computer, Acquisition means for acquiring a first computational resource amount, which is a constraint on the amount of computational resources used in the training process of a language model for the target language, and a target language resource amount, which is the amount of resources of the target language available in the training process, A threshold determination means that, referring to the first amount of computational resources, determines a first threshold that is referenced to determine a schedule for the training process of the language model, which is the schedule for determining the proportion of the amount of target language resources used in the training process of the language model that is used, A comparison means for comparing the amount of target language resources with the first threshold, A schedule determination means that determines the schedule by referring to the comparison results of the comparison means, A learning method determination program that functions as such.

[0110] (Note C2) The acquisition means further acquires training data consisting of a second computational resource amount and a second threshold, The threshold determination means determines the first threshold using the training data. The learning method determination program described in Appendix C1.

[0111] (Note C3) The system further includes a learning unit that takes the amount of computing resources as input and trains a machine learning model using the training data to output a threshold corresponding to the amount of computing resources, The threshold determination means determines the first threshold using the machine learning model. The learning method determination program described in Appendix C2.

[0112] (Note C4) The second amount of computing resources is less than the first amount of computing resources. The learning method determination program described in Appendix C2 or C3.

[0113] (Note C5) If the comparison result indicates that the amount of target language resources is equal to or greater than the first threshold, the schedule determination means determines the schedule to be a one-stage learning method schedule that uses only the target language in the language model learning process. If the comparison result indicates that the amount of target language resources is less than the first threshold, the schedule determination means determines the schedule to be a two-stage learning method using multiple languages, including the target language and languages ​​other than the target language, in the language model learning process. The learning method determination program described in one of the appendices C1 to C4.

[0114] (Appendix C6) The aforementioned computer, Further configured as an output means that outputs information indicating at least one of the first threshold and the schedule determined by the schedule determination means, The learning method determination program described in one of the appendices C1 to C5.

[0115] [Additional Note D] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0116] (Note D1) It comprises at least one processor, and the at least one processor is An acquisition process to obtain a first computational resource amount, which is a constraint on the amount of computational resources used in the training process of a language model for the target language, and a target language resource amount, which is the amount of resources of the target language available in the training process, A threshold determination process that refers to a first computational resource amount and determines a first threshold which is referenced to determine a schedule for the language model learning process, in which the proportion of the language resources used in the language model learning process is the target language resource amount, A comparison process that compares the amount of target language resources with the first threshold, A schedule determination process that determines the schedule by referring to the comparison results in the comparison process, A learning method determination device that performs the following actions.

[0117] The learning method determination device may also include a memory. Furthermore, the memory may store a program for causing at least one processor to execute each of the aforementioned processes.

[0118] (Note D2) In the acquisition process described above, the at least one processor further acquires training data consisting of a second computational resource amount and a second threshold, In the threshold determination process, the at least one processor determines the first threshold using the training data. The learning method determination device described in Appendix D1.

[0119] (Note D3) The at least one processor further performs a learning process to train a machine learning model using the training data, taking the amount of computing resources as input and outputting a threshold corresponding to the amount of computing resources. In the threshold determination process, the at least one processor determines the first threshold using the machine learning model. The learning method determination device described in Appendix D2.

[0120] (Note D4) The second amount of computing resources is less than the first amount of computing resources. A learning method determination device as described in Appendix D2 or D3.

[0121] (Note D5) If the comparison result indicates that the amount of target language resources is equal to or greater than the first threshold, then in the schedule determination process, at least one processor determines the schedule to be a schedule for a one-stage learning method that uses only the target language in the language model learning process. If the comparison result indicates that the amount of target language resources is less than the first threshold, then in the schedule determination process, at least one processor determines the schedule to be a two-stage learning method using multiple languages, including the target language and languages ​​other than the target language, in the language model learning process. A learning method determination device described in any one of the appendices D1 to D4.

[0122] (Note D6) The aforementioned at least one processor, Further output processing is performed to output information indicating at least one of the first threshold and the schedule determined by the schedule determination process. A learning method determination device described in any one of the appendices D1 to D5.

[0123] [Additional Note E] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0124] (Note E1) A program that makes a computer function as a learning method determination device, To the aforementioned computer, An acquisition process to obtain a first computational resource amount, which is a constraint on the amount of computational resources used in the training process of a language model for the target language, and a target language resource amount, which is the amount of resources of the target language available in the training process, A threshold determination process that refers to a first computational resource amount and determines a first threshold which is referenced to determine a schedule for the language model learning process, in which the proportion of the language resources used in the language model learning process is the target language resource amount, A comparison process that compares the amount of target language resources with the first threshold, A schedule determination process that determines the schedule by referring to the comparison results in the comparison process, A non-temporary recording medium that records a learning method determination program that executes the program. [Explanation of symbols]

[0125] 1.1A Learning Method Determination Device 11 Acquisition Department 12. Threshold determination unit 13. Comparison Section 14. Schedule Determination Department 15. Learning Department 16 Output section CR1 1st computational resource amount CR2 2nd computational resources TH1 First threshold TH2 Second threshold T_unique Target language resource quantity TD training data TM Machine Learning Model

Claims

1. Acquisition means for acquiring a first computational resource amount, which is a constraint on the amount of computational resources used in the training process of a language model for the target language, and a target language resource amount, which is the amount of resources of the target language available in the training process, A threshold determination means that, referring to the first amount of computational resources, determines a first threshold that is referenced to determine a schedule for the learning process of the language model, which is the schedule for determining the proportion of the amount of target language resources used in the learning process of the language model that is used, A comparison means for comparing the amount of target language resources with the first threshold, A schedule determination means that determines the schedule by referring to the comparison results of the comparison means, A learning method determination device equipped with the following features.

2. The acquisition means further acquires training data consisting of a second computational resource amount and a second threshold, The threshold determination means determines the first threshold using the training data. A learning method determination device according to claim 1.

3. The system further includes a learning unit that takes the amount of computing resources as input and trains a machine learning model using the training data to output a threshold corresponding to the amount of computing resources, The threshold determination means determines the first threshold using the machine learning model. A learning method determination device according to claim 2.

4. The second amount of computing resources is less than the first amount of computing resources. A learning method determination device according to claim 2 or 3.

5. If the comparison result indicates that the amount of target language resources is equal to or greater than the first threshold, the schedule determination means determines the schedule to be a one-stage learning method schedule that uses only the target language in the language model learning process. If the comparison result indicates that the amount of target language resources is less than the first threshold, the schedule determination means determines the schedule to be a two-stage learning method schedule that uses multiple languages, including the target language and languages ​​other than the target language, in the language model learning process. A learning method determination device according to any one of claims 1 to 3.

6. The system further includes an output means that outputs information indicating at least one of the first threshold and the schedule determined by the schedule determination means. A learning method determination device according to any one of claims 1 to 3.

7. At least one processor performs an acquisition process to acquire a first amount of computational resources, which is a constraint on the amount of computational resources used for training a language model for a target language, and a target language resource amount, which is the amount of resources of the target language available in the training process. The at least one processor performs a threshold determination process that refers to the first amount of computing resources and determines a first threshold which is referenced to determine a schedule for the language model learning process, in which the proportion of the amount of language resources used in the language model learning process that the target language resources are used is determined. The at least one processor performs a comparison process that compares the amount of target language resources with the first threshold, The at least one processor performs a schedule determination process that refers to the comparison results in the comparison process and determines the schedule, A method for determining learning methods, including the method itself.

8. A program that makes a computer function as a learning method determination device, The aforementioned computer, Acquisition means for acquiring a first computational resource amount, which is a constraint on the amount of computational resources used in the training process of a language model for the target language, and a target language resource amount, which is the amount of resources of the target language available in the training process, A threshold determination means that, referring to the first amount of computational resources, determines a first threshold that is referenced to determine a schedule for the learning process of the language model, which is the schedule for determining the proportion of the amount of target language resources used in the learning process of the language model that is used, A comparison means for comparing the amount of target language resources with the first threshold, A schedule determination means that determines the schedule by referring to the comparison results of the comparison means, A learning method determination program that functions as such.

Images