Prediction device, prediction method, and prediction program

Abstract

This provides a technique for efficiently setting the appropriate number of epochs in language model training. [Solution] The prediction device includes an acquisition unit that acquires a first pair including the amount of computational resources, which is a constraint on the amount of computational resources used in the learning process of a language model, and the amount of target language resources, which is the amount of available resources for the target language; and a prediction unit that refers to a combination of a second pair and a second epoch number and predicts a range of first epoch numbers in which the loss in learning using the amount of computational resources and target language resources included in the first pair becomes smaller.

Description

Technical Field

[0001] The present disclosure relates to a prediction device, a prediction method, and a prediction program.

Background Art

[0002] Techniques related to the setting of the number of epochs in machine learning are known. For example, Patent Document 1 discloses an information processing apparatus that sets the number of epochs based on the variance value of the accuracy of a plurality of neural networks immediately before the start of a loop and the results of neural network learning when implementing a loop of a genetic algorithm on the plurality of neural networks.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the information processing apparatus described in Patent Document 1 does not assume learning of a language model. In language model learning, for example, the appropriate number of epochs changes according to the amount of text corpus used for learning. Therefore, there is a need for a technique to efficiently set an appropriate number of epochs in language model learning.

[0005] The present disclosure has been made in view of the above problems, and an exemplary object thereof is to provide a technique for efficiently setting an appropriate number of epochs in language model learning.

Means for Solving the Problems

[0006] An illustrative aspect of the present disclosure includes: an acquisition means for acquiring a first pair including a computational resource amount, which is a constraint on the amount of computational resources used in the learning process of a language model for a target language, and a target language resource amount, which is the amount of resources of the target language available in the learning process; and a prediction means for predicting a range of first epoch numbers in which the loss in learning using the computational resource amount and target language resource amount included in the first pair becomes smaller, by referring to a combination of a second pair in which at least one of the computational resource amount and the target language resource amount included in the first pair is different, and a second epoch number in which the loss in learning using the computational resource amount and target language resource amount included in the second pair becomes smaller.

[0007] An illustrative aspect of the present disclosure includes an acquisition process in which at least one processor acquires a first pair comprising a computational resource amount, which is a constraint on the amount of computational resources used in the training process of a language model for a target language, and a target language resource amount, which is the amount of resources of the target language available in the training process; and a prediction process in which the at least one processor refers to a combination of a second pair in which at least one of the computational resource amounts and the target language resource amounts included in the first pair is different, and a second epoch number in which the loss in training using the computational resource amounts and target language resource amounts included in the second pair becomes smaller, and predicts a range of first epoch numbers in which the loss in training using the computational resource amounts and target language resource amounts included in the first pair becomes smaller.

[0008] An illustrative aspect of this disclosure relates to a prediction program, which is a program that causes a computer to function as a prediction device, and the computer functions as: an acquisition means that acquires a first pair including a computational resource amount, which is a constraint on the amount of computational resources used in the learning process of a language model for a target language, and a target language resource amount, which is the amount of resources of the target language available in the learning process; a prediction means that refers to a combination of a second pair in which at least one of the computational resource amount and the target language resource amount included in the first pair is different, and a second epoch number in which the loss in learning using the computational resource amount and target language resource amount included in the second pair becomes smaller, and predicts a range of first epoch numbers in which the loss in learning using the computational resource amount and target language resource amount included in the first pair becomes smaller. [Effects of the Invention]

[0009] One exemplary aspect of this disclosure is that it can provide a technique for efficiently setting an appropriate number of epochs in the training of a language model. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram showing the configuration of the prediction device related to this disclosure. [Figure 2] This flowchart shows the flow of the prediction method related to this disclosure. [Figure 3] This graph shows how the LLM loss changes when the number of epochs is varied for each unique character in the text corpus of the target language related to this disclosure. [Figure 4] This graph shows the relationship between the number of unique characters in the target language's text corpus and the appropriate number of epochs when the amount of computational resources used in the process of training the LLM related to this disclosure is changed. [Figure 5] This is a block diagram showing the configuration of the prediction device related to this disclosure. [Figure 6] This flowchart shows the flow of the prediction method related to this disclosure. [Figure 7]This is a block diagram showing the configuration of a computer that functions as a prediction device related to this disclosure. [Modes for carrying out the invention]

[0011] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining some or all of the technologies (things or methods) employed in each of the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technologies employed in each of the exemplary embodiments shown below may also be included in the scope of the present invention. In addition, the effects mentioned in each of the exemplary embodiments shown below are examples of effects that can be expected in that exemplary embodiment and do not define the scope of the present invention. That is, embodiments that do not produce the effects mentioned in each of the exemplary embodiments shown below may also be included in the scope of the present invention.

[0012] [First Exemplary Embodiment] A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is the basic form for each of the exemplary embodiments described later. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems occur. Furthermore, each technology shown in the drawings referenced to explain this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems occur.

[0013] (Configuration of prediction device 1) The configuration of the prediction device 1 will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configuration of the prediction device 1. As shown in FIG. 1, the prediction device 1 includes an acquisition unit 11 and a prediction unit 12. In this exemplary embodiment, the acquisition unit 11 and the prediction unit 12 respectively implement acquisition means and prediction means.

[0014] (Acquisition Unit 11) The acquisition unit 11 acquires a first pair including a computational resource amount, which is a constraint on the amount of computational resources used for the learning process of the language model for the target language, and a target language resource amount, which is the amount of resources of the target language available in the learning process. The acquisition unit 11 supplies the acquired first pair to the prediction unit 12.

[0015] (Prediction Unit 12) The prediction unit 12 refers to the combination of a second pair in which at least one of the computational resource amount and the target language resource amount included in the first pair acquired by the acquisition unit 11 is different, and a second epoch number in which the loss in the learning using the computational resource amount and the target language resource amount included in the second pair is smaller, and predicts the range of the first epoch number in which the loss in the learning using the computational resource amount and the target language resource amount included in the first pair is smaller.

[0016] (Effect of Prediction Device 1) As described above, in the prediction device 1, an acquisition unit 11 that acquires a first pair including a computational resource amount, which is a constraint on the amount of computational resources used for the learning process of the language model for the target language, and a target language resource amount, which is the amount of resources of the target language available in the learning process; a second pair in which at least one of the computational resource amount and the target language resource amount included in the first pair acquired by the acquisition unit 11 is different; and a prediction unit 12 that refers to the combination of the second pair and the second epoch number in which the loss in the learning using the computational resource amount and the target language resource amount included in the second pair is smaller, and predicts the range of the first epoch number in which the loss in the learning using the computational resource amount and the target language resource amount included in the first pair is smaller, are adopted.

[0017] Therefore, according to the prediction device 1, an effect can be obtained that an appropriate number of epochs can be efficiently set in the learning of the language model.

[0018] (Flow of prediction method S1) The flow of the prediction method S1 will be described with reference to FIG. 2. FIG. 2 is a flowchart showing the flow of the prediction method S1. As shown in FIG. 2, the prediction method S1 includes an acquisition process S11 and a prediction process S12.

[0019] (Acquisition process S11) In the acquisition process S11, the acquisition unit 11 acquires a first pair including the amount of computing resources that is a constraint on the amount of computing resources used in the learning process of the language model for the target language and the amount of target language resources that is the amount of resources of the target language available in the learning process. The acquisition unit 11 supplies the acquired first pair to the prediction unit 12.

[0020] (Prediction process S12) The prediction unit 12 refers to a combination of a second pair in which at least one of the amount of computing resources and the amount of target language resources included in the first pair acquired by the acquisition unit 11 is different, and a second number of epochs in which the loss in learning using the amount of computing resources and the amount of target language resources included in the second pair is smaller, and predicts a range of the first number of epochs in which the loss in learning using the amount of computing resources and the amount of target language resources included in the first pair is smaller.

[0021] (Effect of prediction method S1) As described above, the prediction method S1 employs a configuration that includes an acquisition process S11 in which the acquisition unit 11 acquires a first pair including the amount of computational resources, which is a constraint on the amount of computational resources used in the learning process of a language model for the target language, and the amount of target language resources, which is the amount of resources of the target language available in the learning process; and a prediction process S12 in which the prediction unit 12 refers to a combination of a second pair in which at least one of the amount of computational resources and the amount of target language resources included in the first pair acquired by the acquisition unit 11 is different, and a second epoch number in which the loss in learning using the amount of computational resources and the amount of target language resources included in the second pair becomes smaller, and predicts a range of first epoch numbers in which the loss in learning using the amount of computational resources and the amount of target language resources included in the first pair becomes smaller. Therefore, the same effect as the prediction device 1 described above can be obtained with the prediction method S1.

[0022] [Second exemplary embodiment] A second exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same function as those described in the above-described exemplary embodiment are denoted by the same reference numerals, and their descriptions are omitted as appropriate. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise. Furthermore, each technology shown in the drawings referenced to describe this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise.

[0023] Training language models (hereinafter also referred to as "LLMs (Large Language Models)") requires a large amount of text corpus. However, languages ​​other than English have relatively few text corpora. Therefore, the following methods are known for training language models for languages ​​with limited text corpus resources. • A method of learning by repeatedly using the same text corpus multiple times (multi-epoch learning) • A method of learning that uses text corpora of other languages ​​in addition to the target language (multilingual learning). • A two-stage learning method in multilingual learning, where the proportion of the target language and other languages ​​is gradually changed. However, when training a language model using a combination of the methods described above, the number of training settings (hyperparameters) increases, making exhaustive exploration costly.

[0024] Therefore, engineers training language models heuristically narrowed down the search space based on previously obtained analysis results regarding how the performance of language models changes depending on the training settings. However, the analysis of language model training settings that has been conducted in the past has been limited, and a problem has arisen in that the optimal search space cannot be narrowed down when training an LLM by combining the above methods.

[0025] Therefore, the inventors are investigating how to narrow down the search space for learning settings that are expected to yield high performance when training a language model targeting a language with limited resources using some or all of the multi-epoch learning, multilingual learning, and two-stage learning described above.

[0026] As an example, the inventors have found that as the number of unique elements in the target language's text corpus (the number of elements excluding repetitions when there are multiple epochs) decreases, the appropriate number of epochs that minimizes learning loss increases monotonically. Furthermore, the inventors have found that as the amount of computational resources required for training the LLM decreases, the appropriate number of epochs also decreases monotonically.

[0027] Figures 3 and 4 show graphs that form the basis of the findings obtained by the inventors. Figure 3 is a graph showing how the LLM loss changes when the number of epochs is changed for each unique quantity in the text corpus of the target language. Figure 4 is a graph showing the relationship between the unique quantity in the text corpus of the target language and the appropriate number of epochs when the amount of computational resources used in training the LLM is changed.

[0028] In the graph shown in Figure 3, the horizontal axis represents the logarithmic value with base 2 (epoch number). The vertical axis represents the LLM loss, with smaller values ​​indicating better LLM performance. The upper graph in Figure 3 shows the case of one-stage multilingual (including monolingual) learning, and the lower graph in Figure 3 shows the case of two-stage multilingual learning.

[0029] Furthermore, the curves L1_1 to L1_7 and L2_1 to L2_7 in the graph shown in Figure 3 are curves that fit the change in the LLM loss when the LLM is trained with different unique quantities (unique quantities in the text corpus of the target language) to a quadratic function with respect to the base 2 logarithmic value of the number of epochs.

[0030] Furthermore, the curves L1 and L2 in the graph shown in Figure 3 are curves formed by connecting the minimum points of curves fitted with a quadratic function for each unique quantity in the text corpus of the target language (however, since the number of epochs cannot be less than 1, the minimum points in the range where the horizontal axis value is 0 or greater are connected).

[0031] Furthermore, in the graph shown in Figure 4, the horizontal axis represents the base-2 logarithm of the uniqueness quantity of the text corpus of the target language (a quantity where the number of uniqueness in the text corpus is sufficient is set to 0) relative to the base quantity. The vertical axis represents the base-2 logarithm of the appropriate number of epochs.

[0032] Furthermore, line L3_1 shows the case of multilingual single-stage learning with a computational resource of 10^18 FLOPS, and line 3_2 shows the case of multilingual two-stage learning with a computational resource of 10^18 FLOPS. Also, line L4_1 shows the case of multilingual single-stage learning with a computational resource of 1 / 4 × 10^18 FLOPS, and line 4_2 shows the case of multilingual two-stage learning with a computational resource of 1 / 4 × 10^18 FLOPS. Furthermore, line L5_1 shows the case of multilingual single-stage learning with a computational resource of 1 / 16 × 10^18 FLOPS, and line 5_2 shows the case of multilingual two-stage learning with a computational resource of 1 / 16 × 10^18 FLOPS.

[0033] As shown in Figure 4, the appropriate number of epochs increases as the number of unique elements in the target language's text corpus decreases, and decreases as the computational resources required to train the LLM decrease. Furthermore, as shown in Figure 4, this feature is independent of the training method.

[0034] Furthermore, from Figure 4, the appropriate number of epochs can be calculated using the following formula (1). The appropriate number of epochs = f(logT-g(logCR)) ... (1) Here, T is the number of unique elements in the text corpus of the target language, and CR is the amount of computational resources required to train the LLM. Furthermore, g(logCR) is calculated by the following equation (2). g(logCR) = alogCR ... (2) Here, a is a coefficient.

[0035] The prediction device 1A and the processes performed by the prediction device 1A, as described below, are based on the knowledge mentioned above and on the inventor's own unique perspective.

[0036] (Overview of Prediction Device 1A) Prediction device 1A is a device that predicts the appropriate range of epoch numbers for LLM training. An appropriate number of epochs is the number of epochs at which the training loss is minimized. The appropriate range of epoch numbers is the range that includes the appropriate number of epochs, and may be greater than or equal to a certain value, less than or equal to a certain value, a combination of these, or the appropriate number of epochs itself.

[0037] Specifically, the prediction device 1A predicts the range RA of the first epoch number in which the loss in learning is minimized, using the computational resource amount CR, which is a constraint on the amount of computational resources used in the learning process of the LLM for the target language, and the target language resource amount T, which is the amount of target language resources available in the learning process. The first epoch number may also be the epoch number in which the loss in learning is minimized using the computational resource amount CR and the target language resource amount T. With this configuration, the prediction device 1A can predict the range of the epoch number in which the loss in learning is minimized.

[0038] Computational resource constraints (CR) used in the training process of an LLM for a target language refer to the amount of resources that the device performing the LLM training can use for the training process. One example is the total amount of computation that can be used for the training process, measured in units of FLOPs (Floating-point Operations).

[0039] The target language resource quantity T, which is the amount of target language resources available in the learning process, is the unique quantity of target language text corpora that have been collected and are available for use in the LLM learning process (excluding repetitions if there are multiple epochs). One example of this is the unique quantity of text corpora of all target languages ​​that exist on Earth.

[0040] Furthermore, the prediction device 1A refers to a combination of a second pair PA2 in which at least one of the computational resource amount CR and target language resource amount T included in the first pair PA1 is different, and a second epoch number in which the loss in learning using the computational resource amount CR and target language resource amount T included in the second pair PA2 becomes smaller, and predicts the range RA of the first epoch number.

[0041] The computational resources CR and target language resources T included in the second pair PA2 are not particularly limited, but as an example, the computational resources CR included in the second pair PA2 are less than the computational resources CR included in the first pair PA1. As another example, the target language resources T included in the second pair PA2 are less than the target language resources T included in the first pair PA1.

[0042] Furthermore, as described above, the range RA of the first epoch number predicted by the prediction device 1A is applicable to the number of epochs in any learning method. For example, the range RA of the first epoch number predicted by the prediction device 1A may be configured to predict the range of the first epoch number when learning an LLM using multiple languages.

[0043] (Configuration of prediction device 1A) The configuration of the prediction device 1A will be described with reference to Figure 5. Figure 5 is a block diagram showing the configuration of the prediction device 1A. As shown in Figure 5, the prediction device 1A includes a control unit 10, a storage unit 20, an input / output unit 21, and a communication unit 22.

[0044] (Storage unit 20) The memory unit 20 stores data that the control unit 10 references. For example, the memory unit 20 stores multiple epoch numbers (second epoch number) EN.

[0045] Multiple second epoch numbers EN represent the epoch number at which the loss in learning using the computational resources CR and target language resources T contained in each of the multiple second pairs PA2 becomes smaller. Hereafter, the second epoch number EN at which the loss in learning using the computational resources CR and target language resources T contained in the second pair PA2 becomes smaller will also be referred to as the second epoch number EN corresponding to the computational resources CR and target language resources T.

[0046] For example, epoch number EN1_1 is the second epoch number corresponding to computational resource amount CR1 and target language resource amount T1. Another example is epoch number EN2_1, which is the second epoch number corresponding to computational resource amount CR2 and target language resource amount T1. Yet another example is epoch number EN1_2, which is the second epoch number corresponding to computational resource amount CR1 and target language resource amount T2.

[0047] Each of the multiple second epoch numbers EN may be calculated by the control unit 10 from each of the multiple second pairs PA2, or it may be stored in the memory unit 20 in advance.

[0048] In a configuration where the control unit 10 calculates from each of the multiple second pairs PA2, the control unit 10 trains the LLM by changing the hyperparameters for each of the multiple second pairs PA2 using the computational resource amount CR and target language resource amount T included in the second pair PA2. Examples of hyperparameters include the number of epochs, the model size of the LLM, and the number of training steps (in the case of two-stage learning, the ratio of the length of the first stage of learning and the ratio of the target language resource amount T between the first and second stages). The control unit 10 then stores the number of epochs that minimizes the loss as the second epoch number EN in the memory unit 20. That is, the second epoch number EN is the number of epochs that minimizes the loss in learning using the computational resource amount CR and target language resource amount T included in the second pair. With this configuration, the prediction device 1A can also predict the range of epochs that minimize the loss in learning for the first epoch number that references the second epoch number EN.

[0049] As another example, the memory unit 20 stores a first pair PA1, a second pair PA2, and a range RA of the first epoch number, which are not shown in Figure 5.

[0050] (Input / output section 21) The input / output unit 21 is an interface to input devices that accept data input and output devices that output data. Examples of input devices include, but are not limited to, microphones, cameras, eye-tracking devices, keyboards, and touchpads. Examples of output devices include, but are not limited to, speakers and liquid crystal displays.

[0051] (Communications Section 22) The communication unit 22 is an interface for sending and receiving data over a network. Examples of the communication unit 22 include, but are not limited to, communication chips in various communication standards such as Ethernet®, Wi-Fi®, and wireless communication standards for mobile data communication networks, as well as USB-compliant connectors.

[0052] (Control Unit 10) The control unit 10 controls each component of the prediction device 1A. The control unit 10 also includes an acquisition unit 11, a prediction unit 12, and an output unit 13. In this exemplary embodiment, the acquisition unit 11, the prediction unit 12, and the output unit 13 implement the acquisition means, prediction means, and output means, respectively.

[0053] (Acquisition part 11) The acquisition unit 11 acquires data output from the input / output unit 21 or the communication unit 22. The acquisition unit 11 stores the acquired data in the storage unit 20. As an example, the acquisition unit 11 acquires a first pair PA1 which includes the computational resource amount CR, which is a constraint on the amount of computational resources used in the learning process of the LLM for the target language, and the target language resource amount T, which is the amount of resources of the target language available in the learning process.

[0054] (Prediction Unit 12) The prediction unit 12 predicts the range RA of the first epoch number. The prediction unit 12 stores the predicted range RA of the first epoch number in the storage unit 20. As an example, the prediction unit 12 refers to the combination of the second pair PA2 and the second epoch number EN, and predicts the range RA of the first epoch number.

[0055] As an example, the prediction unit 12 predicts the range RA of the first epoch number by using the fact that the first epoch number monotonically changes with respect to each of the calculation resource amount CR and the target language resource amount T.

[0056] For example, taking the first epoch number as k, if the target language resource amount T included in the first pair PA1 is the same as the target language resource amount T included in the second pair PA2, the calculation resource amount included in the second pair PA2 is the calculation resource amount CR' which is less than the calculation resource amount CR included in the first pair PA1, and the value of the second epoch number EN corresponding to the second pair PA2 is k', then the prediction unit 12 predicts k>k' as the range RA of the first epoch number k. In this configuration, since the prediction unit 12 predicts the range RA of the first epoch number k by referring to the second epoch number EN with the calculation resource amount CR' less than the calculation resource amount CR included in the first pair PA1, the range RA of the first epoch number k can be predicted efficiently.

[0057] On the other hand, taking the first epoch number as k, if the target language resource amount T included in the first pair PA1 is the same as the target language resource amount T included in the second pair PA2, the calculation resource amount included in the second pair PA2 is the calculation resource amount CR' which is more than the calculation resource amount CR included in the first pair PA1, and the value of the second epoch number EN corresponding to the second pair PA2 is k', then the prediction unit 12 predicts k<k' as the range RA of the first epoch number k.

[0058] Also, with the number of first epochs being k, when the amount of computing resources CR included in the first pair PA1 is the same as the amount of computing resources CR included in the second pair PA2, and the value of the number of second epochs EN corresponding to the amount of target language resources T' that is less than the amount of target language resources T included in the first pair PA1 is k', the prediction unit 12 predicts k > k' as the range RA of the number of first epochs k.

[0059] On the other hand, with the number of first epochs being k, when the amount of computing resources CR included in the first pair PA1 is the same as the amount of computing resources CR included in the second pair PA2, and the value of the number of second epochs EN corresponding to the amount of target language resources T' that is more than the amount of target language resources T included in the first pair PA1 is k', the prediction unit 12 predicts k < k' as the range RA of the number of first epochs k.

[0060] As another example, for instance, the prediction unit 12 predicts the range RA of the number of first epochs using a calculation model that calculates the following formula (3) with the number of first epochs being k. k = f(CR, T) = exp(p_0 × max(p_1 × logCR + p_2 - logT, 0)) ···(3) Here, p_0, p_1, and p_2 are parameters. As an example of a method for estimating the parameters, when the amount of computing resources is CR j and the amount of target language resources is T j,i and the value of the corresponding number of second epochs ENj_i is k j,i a method for estimating the parameters is to update the values of the parameters by a gradient method or the like so that the following formula (4) becomes small using the least squares method.

[0061]

Equation

[0062] In this way, the prediction unit 12 can efficiently predict the range RA of the number of first epochs by predicting the range RA of the number of first epochs using the fact that the number of first epochs changes monotonically with respect to each of the amount of computing resources CR and the amount of target language resources T.

[0063] (Output section 13) The output unit 13 outputs data via the input / output unit 21 or the communication unit 22. For example, the output unit 13 outputs at least one of the referenced second pair PA2 and the second epoch number EN corresponding to the second pair PA2, and the range RA of the first epoch number. With this configuration, the output unit 13 can notify the user of the predicted range RA of the first epoch number and at least one of the referenced second pair PA2 and the second epoch number EN corresponding to the second pair PA2 used to predict the range RA of the first epoch number.

[0064] (Processing performed by prediction device 1A) The flow of the process (prediction method S1A) executed by the prediction device 1A will be explained with reference to Figure 6. Figure 6 is a flowchart showing the flow of prediction method S1A.

[0065] (Acquisition process S11) In acquisition process S11, the acquisition unit 11 acquires a first pair PA1 which includes the computational resource amount CR, which is a constraint on the amount of computational resources used for the learning process of the LLM for the target language, and the target language resource amount T, which is the amount of resources of the target language available in the learning process. The acquisition unit 11 stores the acquired first pair PA1 in the storage unit 20.

[0066] (Prediction processing S12) In prediction processing S12, the prediction unit 12 refers to the combination of the second pair PA2 and the second epoch number EN and predicts the range RA of the first epoch number. The prediction unit 12 stores the predicted range RA of the first epoch number in the storage unit 20. As described above, the prediction unit 12 may predict the range RA of the first epoch number by using the fact that the first epoch number changes monotonically with respect to the computational resource amount CR and the target language resource amount T included in the second pair PA2, or it may predict the range RA of the first epoch number using a computation model.

[0067] (Output processing S13) In output processing S13, the output unit 13 outputs at least one of the referenced second pair PA2, the second epoch number EN corresponding to the second pair PA2, and the range RA of the first epoch number.

[0068] (Example of application of prediction device 1A) For example, the prediction device 1A may train an LLM using the computational resources CR and target language resources T included in the first pair PA1, as well as the range RA of the first epoch number. Known methods may be used as the method for training the LLM. Furthermore, when the prediction device 1A trains an LLM, it may perform a process to determine various training settings such as the model size of the LLM before training the LLM. This allows the prediction device 1A to reduce the search space in which it changes the number of epochs in order to generate a higher-performance LLM.

[0069] Furthermore, the prediction device 1A may instruct an external device different from the prediction device 1A to train the LLM. In this case, the prediction device 1A may instruct the external device to narrow the range of various learning settings using the predicted range RA of the first epoch number. With this configuration, the prediction device 1A can reduce the search space for the external device to change the number of epochs.

[0070] (Effect of prediction device 1A) As described above, the prediction device 1A refers to the combination of the computational resource amount CR and target language resource amount T included in the second pair PA2 and the corresponding second epoch number, and predicts the range RA of the first epoch number in which the loss in learning using the computational resource amount CR and target language resource amount T included in the first pair is minimized. As mentioned above, as the computational resource amount CR decreases, the appropriate epoch number also decreases monotonically, and as the target language resource amount T decreases, the appropriate epoch number increases monotonically. Therefore, the prediction device 1A can compare the computational resource amount CR included in the second pair PA2 and the computational resource amount CR included in the first pair PA1 with the target language resource amount T included in the second pair PA2 and the target language resource amount T included in the first pair PA1, and predict the range of the first epoch number from the comparison result and the second epoch number, thereby narrowing the search range and efficiently setting an appropriate epoch number in LLM learning.

[0071] [Examples of implementation using software] Some or all of the functions of prediction devices 1 and 1A (hereinafter also referred to as "each of the above devices") may be implemented by hardware such as integrated circuits (IC chips) or by software.

[0072] In the latter case, each of the above devices is implemented, for example, by a computer that executes instructions for a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as Computer C) is shown in Figure 7. Figure 7 is a block diagram showing the hardware configuration of Computer C, which functions as each of the above devices.

[0073] Computer C comprises at least one processor C1 and at least one memory C2. Memory C2 stores a program P that causes computer C to operate as each of the above-mentioned devices. In computer C, processor C1 reads program P from memory C2 and executes it, thereby realizing each of the above-mentioned devices.

[0074] For processor C1, for example, a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof can be used. For memory C2, for example, flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof can be used.

[0075] Computer C may also be equipped with RAM (Random Access Memory) for loading program P at runtime and for temporarily storing various data. Furthermore, computer C may be equipped with communication interfaces for sending and receiving data with other devices. Additionally, computer C may be equipped with input / output interfaces for connecting input / output devices such as keyboards, mice, displays, and printers.

[0076] Furthermore, program P can be recorded on a non-temporary, tangible recording medium M that is readable by computer C. Such a recording medium M could be, for example, tape, disk, card, semiconductor memory, or programmable logic circuitry. Computer C can acquire program P via such a recording medium M. Program P can also be transmitted via a transmission medium. Such a transmission medium could be, for example, a communication network or broadcast waves. Computer C can also acquire program P via such a transmission medium.

[0077] Furthermore, each of the above functions of each of the above devices may be implemented by a single processor in a single computer, by multiple processors in a single computer working together, or by multiple processors in each of multiple computers working together. In addition, the programs for implementing each of the above functions in each of the above devices may be stored in a single memory in a single computer, distributed and stored in multiple memories in a single computer, or distributed and stored in multiple memories in each of multiple computers.

[0078] [Additional Note 1] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0079] [Additional Note A] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0080] (Note A1) An acquisition means for acquiring a first pair including the amount of computational resources, which is a constraint on the amount of computational resources used in the training process of a language model for the target language, and the amount of target language resources, which is the amount of resources of the target language available in the training process; Prediction means predicts a range of first epoch numbers in which the loss in learning using the computational resources and target language resources is smaller, by referring to a combination of a second pair in which at least one of the computational resources and target language resources included in the first pair is different, and a second epoch number in which the loss in learning using the computational resources and target language resources included in the second pair is smaller. A prediction device equipped with the following features.

[0081] (Appendix A2) The prediction means predicts the range of the first epoch number by using the fact that the first epoch number changes monotonically with respect to the amount of computational resources and the amount of target language resources, respectively. The prediction device described in Appendix A1.

[0082] (Note A3) The amount of computing resources included in the second pair is less than the amount of computing resources included in the first pair. The prediction device described in Appendix A1.

[0083] (Note A4) The first epoch number is the number of epochs in which the loss in learning using the amount of computational resources and the amount of target language resources included in the first pair is minimized. The second epoch number is the number of epochs in which the loss in learning using the amount of computational resources and the amount of target language resources included in the second pair is minimized. A prediction device as described in any one of the appendices A1 to A3.

[0084] (Note A5) The system further includes output means for outputting information indicating at least one of the following: the second pair and the second epoch number corresponding to the second pair, and the range of the first epoch number. The prediction device described in any one of the appendices A1 to A4.

[0085] [Additional Notes B] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0086] (Note B1) An acquisition process in which at least one processor acquires a first pair including a computational resource amount, which is a constraint on the amount of computational resources used for training a language model for a target language, and a target language resource amount, which is the amount of resources of the target language available in the training process, The at least one processor performs a prediction process that refers to a combination of a second pair in which at least one of the computational resources and target language resources included in the first pair is different, and a second epoch number in which the loss in learning using the computational resources and target language resources included in the second pair becomes smaller, and predicts a range of first epoch numbers in which the loss in learning using the computational resources and target language resources included in the first pair becomes smaller. A prediction method that includes this.

[0087] (Note B2) In the prediction process, the at least one processor predicts the range of the first epoch number by using the fact that the first epoch number changes monotonically with respect to the amount of computational resources and the amount of target language resources, respectively. The prediction method described in Appendix B1.

[0088] (Note B3) The amount of computing resources included in the second pair is less than the amount of computing resources included in the first pair. The prediction method described in Appendix B1.

[0089] (Note B4) The first epoch number is the number of epochs in which the loss in learning using the amount of computational resources and the amount of target language resources included in the first pair is minimized. The second epoch number is the number of epochs in which the loss in learning using the amount of computational resources and the amount of target language resources included in the second pair is minimized. The prediction method described in one of the appendices B1 to B3.

[0090] (Note B5) The at least one processor further includes output processing that outputs information indicating at least one of the second pair and the second epoch number corresponding to the second pair, and the range of the first epoch number. The prediction method described in one of the appendices B1 through B4.

[0091] [Additional Note C] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0092] (Note C1) A program that makes a computer function as a prediction device, The aforementioned computer, An acquisition means for acquiring a first pair including the amount of computational resources, which is a constraint on the amount of computational resources used in the training process of a language model for the target language, and the amount of target language resources, which is the amount of resources of the target language available in the training process; Prediction means predicts a range of first epoch numbers in which the loss in learning using the computational resources and target language resources is smaller, by referring to a combination of a second pair in which at least one of the computational resources and target language resources included in the first pair is different, and a second epoch number in which the loss in learning using the computational resources and target language resources included in the second pair is smaller. A prediction program designed to function as such.

[0093] (Note C2) The prediction means predicts the range of the first epoch number by using the fact that the first epoch number changes monotonically with respect to the amount of computational resources and the amount of target language resources, respectively. The prediction program described in Appendix C1.

[0094] (Note C3) The amount of computing resources included in the second pair is less than the amount of computing resources included in the first pair. The prediction program described in Appendix C1.

[0095] (Note C4) The first epoch number is the number of epochs in which the loss in learning using the amount of computational resources and the amount of target language resources included in the first pair is minimized. The second epoch number is the number of epochs in which the loss in learning using the amount of computational resources and the amount of target language resources included in the second pair is minimized. The prediction program described in any one of the appendices C1 to C3.

[0096] (Note C5) The aforementioned computer, The output means is further configured to output information indicating at least one of the following: the second pair and the second epoch number corresponding to the second pair, and the range of the first epoch number. The prediction program described in any one of the appendices C1 through C4.

[0097] [Additional Note D] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0098] (Note D1) It comprises at least one processor, and the at least one processor is An acquisition process to acquire a first pair including the amount of computational resources, which is a constraint on the amount of computational resources used in the training process of a language model for the target language, and the amount of target language resources, which is the amount of resources of the target language available in the training process; A prediction process that refers to a combination of a second pair in which at least one of the computational resources and target language resources included in the first pair is different, and a second epoch number in which the loss in learning using the computational resources and target language resources included in the second pair becomes smaller, and predicts a range of first epoch numbers in which the loss in learning using the computational resources and target language resources included in the first pair becomes smaller, A prediction device that performs this operation.

[0099] The prediction device may also include a memory. Furthermore, the memory may store a program that causes at least one processor to execute each of the aforementioned processes.

[0100] (Note D2) In the prediction process, the at least one processor predicts the range of the first epoch number by using the fact that the first epoch number changes monotonically with respect to the amount of computational resources and the amount of target language resources, respectively. The prediction device described in Appendix D1.

[0101] (Note D3) The amount of computing resources included in the second pair is less than the amount of computing resources included in the first pair. The prediction device described in Appendix D1.

[0102] (Note D4) The first epoch number is the number of epochs in which the loss in learning using the amount of computational resources and the amount of target language resources included in the first pair is minimized. The second epoch number is the number of epochs in which the loss in learning using the amount of computational resources and the amount of target language resources included in the second pair is minimized. A prediction device as described in any one of the appendices D1 to D3.

[0103] (Note D5) The aforementioned at least one processor, Further output processing is performed to output information indicating at least one of the following: the second pair and the second epoch number corresponding to the second pair, and the range of the first epoch number. A prediction device as described in any one of the appendices D1 to D4.

[0104] [Additional Note E] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0105] (Note E1) A program that makes a computer function as a prediction device, To the aforementioned computer, An acquisition process to acquire a first pair including the amount of computational resources, which is a constraint on the amount of computational resources used in the training process of a language model for the target language, and the amount of target language resources, which is the amount of resources of the target language available in the training process; A prediction process that refers to a combination of a second pair in which at least one of the computational resources and target language resources included in the first pair is different, and a second epoch number in which the loss in learning using the computational resources and target language resources included in the second pair becomes smaller, and predicts a range of first epoch numbers in which the loss in learning using the computational resources and target language resources included in the first pair becomes smaller, A non-temporary recording medium on which a prediction program that executes the prediction is recorded. [Explanation of symbols]

[0106] 1. 1A Prediction device 11 Acquisition Department 12 Prediction Section 13 Output section CR computational resources EN 2nd Epoch Number PA1 1st Pair PA2 2nd pair RA Range of the first epoch T: Amount of target language resources

Claims

1. Acquisition means for acquiring a first pair including the amount of computational resources, which is a constraint on the amount of computational resources used in the training process of a language model for the target language, and the amount of target language resources, which is the amount of resources of the target language available in the training process; Prediction means predicts a range of first epoch numbers in which the loss in learning using the computational resources and target language resources included in the first pair becomes smaller, by referring to a combination of a second pair in which at least one of the computational resources and target language resources included in the first pair is different, and a second epoch number in which the loss in learning using the computational resources and target language resources included in the second pair becomes smaller. A prediction device equipped with the following features.

2. The prediction means predicts the range of the first epoch number by using the fact that the first epoch number changes monotonically with respect to the amount of computational resources and the amount of target language resources, respectively. The prediction device according to claim 1.

3. The amount of computing resources included in the second pair is less than the amount of computing resources included in the first pair. The prediction device according to claim 1.

4. The first epoch number is the number of epochs in which the loss in learning using the amount of computational resources and the amount of target language resources included in the first pair is minimized. The second epoch number is the number of epochs in which the loss in learning using the amount of computational resources and the amount of target language resources included in the second pair is minimized. A prediction device according to any one of claims 1 to 3.

5. The system further includes output means for outputting information indicating at least one of the following: the second pair and the second epoch number corresponding to the second pair, and the range of the first epoch number. A prediction device according to any one of claims 1 to 3.

6. An acquisition process in which at least one processor acquires a first pair including a computational resource amount, which is a constraint on the amount of computational resources used for training a language model for a target language, and a target language resource amount, which is the amount of resources of the target language available in the training process; The at least one processor performs a prediction process that refers to a combination of a second pair in which at least one of the computational resources and target language resources included in the first pair is different, and a second epoch number in which the loss in learning using the computational resources and target language resources included in the second pair becomes smaller, and predicts a range of first epoch numbers in which the loss in learning using the computational resources and target language resources included in the first pair becomes smaller. A prediction method that includes this.

7. A program that makes a computer function as a prediction device, The aforementioned computer, Acquisition means for acquiring a first pair including the amount of computational resources, which is a constraint on the amount of computational resources used in the training process of a language model for the target language, and the amount of target language resources, which is the amount of resources of the target language available in the training process; Prediction means predicts a range of first epoch numbers in which the loss in learning using the computational resources and target language resources included in the first pair becomes smaller, by referring to a combination of a second pair in which at least one of the computational resources and target language resources included in the first pair is different, and a second epoch number in which the loss in learning using the computational resources and target language resources included in the second pair becomes smaller. A prediction program designed to function as such.

Images