Prediction device, prediction method, and prediction program

Abstract

To realize a predictive device that can predict the performance of a language model when it has been trained using multiple languages. [Solution] The prediction device includes an acquisition unit that acquires the model size of a language model for a target language to be trained using multiple languages, the amount of training data used in the learning process, and the target language ratio which indicates the proportion of the amount of data for the target language in the amount of training data; and a prediction unit that predicts the loss of the language model using the product of a function that depends on the model size and the amount of training data and a constant power of the target language ratio.

Description

Technical Field

[0004] ,

[0006] , , , , , ,

[0005] , , ,

[0003] , , , , ,

[0001] The present disclosure relates to a prediction device, a prediction method, and a prediction program.

Background Art

[0002] Techniques for predicting the performance by training a language model are known. For example, Non-Patent Document 1 describes that when training language models of multiple domains, the loss of the language model follows a power law with respect to the ratio of the domains.

Prior Art Documents

Non-Patent Documents

[0003] [[ID=二十ー]]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technique described in Non-Patent Document 1, a configuration for predicting the performance when training a language model using multiple languages has not been studied.

[0005] The present disclosure has been made in view of the above problems, and an exemplary object thereof is to provide a technique for predicting the performance of a language model when training a language model using multiple languages.

Means for Solving the Problems

[0006] An illustrative aspect of the present disclosure includes an acquisition means for acquiring the model size of a language model for a target language to be trained using multiple languages, the amount of training data used in the training process of the language model, and the target language ratio, which represents the proportion of the amount of data for the target language among the amount of training data; and a prediction means for predicting the loss of the language model using the product of a function that depends on the model size and the amount of training data and a constant power of the target language ratio.

[0007] An illustrative aspect of the present disclosure relates to a prediction method which includes an acquisition process in which at least one processor obtains the model size of a language model for a target language to be trained using multiple languages, the amount of training data used in the training process of the language model, and the target language ratio which indicates the proportion of the amount of data for the target language in the amount of training data; and a prediction process in which the at least one processor predicts the loss of the language model using the product of a function that depends on the model size and the amount of training data and the target language ratio raised to a constant power.

[0008] An illustrative aspect of the present disclosure relates to a prediction program, which is a program that causes a computer to function as a prediction device, and the computer is configured to function as an acquisition means for acquiring the model size of a language model for a target language to be learned using multiple languages, the amount of training data used in the learning process of the language model, and the target language ratio which indicates the proportion of the amount of data of the target language in the amount of training data, and a prediction means for predicting the loss of the language model using the product of a function that depends on the model size and the amount of training data and the target language ratio raised to a constant power. [Effects of the Invention]

[0009] One illustrative aspect of this disclosure provides a technique for predicting the performance of a language model when it has been trained using multiple languages. [Brief explanation of the drawing]

[0010] [Figure 1]This is a block diagram showing the configuration of the prediction device related to this disclosure. [Figure 2] This flowchart shows the flow of the prediction method related to this disclosure. [Figure 3] This graph shows the loss of LLM when the model size, training data volume, and target language ratio are changed in multilingual one-stage learning related to this disclosure. [Figure 4] This graph shows the loss of LLM when the model size, training data volume, target language ratio, and number of epochs are varied in the multilingual one-stage learning process described herein. [Figure 5] This is a block diagram showing the configuration of the prediction device related to this disclosure. [Figure 6] This flowchart shows the flow of the prediction method related to this disclosure. [Figure 7] This is a block diagram showing the configuration of a computer that functions as a prediction device related to this disclosure. [Modes for carrying out the invention]

[0011] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining some or all of the technologies (things or methods) employed in each of the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technologies employed in each of the exemplary embodiments shown below may also be included in the scope of the present invention. In addition, the effects mentioned in each of the exemplary embodiments shown below are examples of effects that can be expected in that exemplary embodiment and do not define the scope of the present invention. That is, embodiments that do not produce the effects mentioned in each of the exemplary embodiments shown below may also be included in the scope of the present invention.

[0012] [First Exemplary Embodiment] A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is the basic form for each of the exemplary embodiments described later. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems occur. Furthermore, each technology shown in the drawings referenced to explain this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems occur.

[0013] (Configuration of prediction device 1) The configuration of the prediction device 1 will be described with reference to Figure 1. Figure 1 is a block diagram showing the configuration of the prediction device 1. As shown in Figure 1, the prediction device 1 includes an acquisition unit 11 and a prediction unit 12. In this exemplary embodiment, the acquisition unit 11 and the prediction unit 12 implement the acquisition means and the prediction means, respectively.

[0014] (Acquisition part 11) The acquisition unit 11 acquires the model size of the language model for the target language to be trained using multiple languages, the amount of training data used in the language model training process, and the target language ratio, which indicates the proportion of the data for the target language within the total training data. The acquisition unit 11 supplies the acquired model size, training data amount, and target language ratio to the prediction unit 12.

[0015] (Prediction Unit 12) The prediction unit 12 predicts the loss of the language model using the product of a function that depends on the model size and the amount of training data, and a constant power of the proportion of the target language.

[0016] (Effect of prediction device 1) As described above, in the prediction device 1, an acquisition unit 11 that acquires the model size of a language model for a target language to be learned using a plurality of languages, the amount of training data used for the learning process of the language model, and the target language ratio indicating the ratio of the amount of data of the target language in the amount of training data, a function that depends on the model size and the amount of training data, and a prediction unit 12 that predicts the loss of the language model using the product of the constant power of the target language ratio are adopted.

[0017] Therefore, according to the prediction device 1, an effect that the performance of the language model when learning the language model using a plurality of languages can be predicted is obtained.

[0018] (Flow of prediction method S1) The flow of the prediction method S1 will be described with reference to FIG. 2. FIG. 2 is a flowchart showing the flow of the prediction method S1. As shown in FIG. 2, the prediction method S1 includes an acquisition process S11 and a prediction process S12.

[0019] (Acquisition process S11) In the acquisition process S11, the acquisition unit 11 acquires the model size of a language model for a target language to be learned using a plurality of languages, the amount of training data used for the learning process of the language model, and the target language ratio indicating the ratio of the amount of data of the target language in the amount of training data. The acquisition unit 11 supplies the acquired model size, amount of training data, and target language ratio to the prediction unit 12.

[0020] (Prediction process S12) In the prediction process S12, the prediction unit 12 predicts the loss of the language model using the product of a function that depends on the model size and the amount of training data and the constant power of the target language ratio.

[0021] (Effect of prediction method S1) As described above, the prediction method S1 employs a configuration in which the acquisition unit 11 performs an acquisition process S11 to acquire the model size of a language model for a target language to be trained using multiple languages, the amount of training data used in the language model training process, and the target language ratio, which indicates the proportion of the amount of data for the target language among the amount of training data, and the prediction unit 12 performs a prediction process S12 to predict the loss of the language model using the product of a function that depends on the model size and the amount of training data and a constant power of the target language ratio. For this reason, the prediction method S1 can be used to obtain the same effects as the prediction device 1 described above.

[0022] [Second exemplary embodiment] A second exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same function as those described in the above-described exemplary embodiment are denoted by the same reference numerals, and their descriptions are omitted as appropriate. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise. Furthermore, each technology shown in the drawings referenced to describe this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise.

[0023] Training language models (hereinafter also referred to as "LLMs (Large Language Models)") requires a large amount of text corpus. However, languages ​​other than English have relatively few text corpora. Therefore, the following methods are known for training language models for languages ​​with limited text corpus resources. • A method of learning by repeatedly using the same text corpus multiple times (multi-epoch learning) • A method of learning that uses text corpora of other languages ​​in addition to the target language (multilingual learning). • A two-stage learning method in multilingual learning, where the proportion of the target language and other languages ​​is gradually changed. However, when training a language model using a combination of the methods described above, the number of training settings (hyperparameters) increases, making exhaustive exploration costly.

[0024] Therefore, engineers training language models heuristically narrowed down the search space based on previously obtained analysis results regarding how the performance of language models changes depending on the training settings. However, the analysis of language model training settings that has been conducted in the past has been limited, and a problem has arisen in that the optimal search space cannot be narrowed down when training an LLM by combining the above methods.

[0025] Therefore, the inventors are investigating how to narrow down the search space for learning settings that are expected to yield high performance when training a language model targeting a language with limited resources using some or all of the multi-epoch learning, multilingual learning, and two-stage learning described above.

[0026] As an example, the inventors have found that when an LLM for a target language is trained using multiple languages, the performance (loss) of the resulting language model follows a power law of the target language ratio, which is the proportion of the target language data in the total amount of training data used in the learning process.

[0027] Figures 3 and 4 show graphs that form the basis of the findings obtained by the inventors. Figure 3 is a graph showing the loss of LLM when the model size, training data volume, and target language ratio are changed in multilingual one-stage learning. Figure 4 is a graph showing the loss of LLM when the model size, training data volume, target language ratio, and number of epochs are changed in multilingual one-stage learning.

[0028] The graph shown in Figure 3 is a log-log graph with the horizontal axis representing the proportion of the target language (in this example, the proportion of Japanese) and the vertical axis representing the loss achieved by the LLM for the corresponding target language proportion. The differences in the lines indicate differences in model size and training data volume, and these lines are generally parallel. In the graph shown in Figure 3, the number of epochs is 4 or less, where the effects of overfitting are small.

[0029] From the graph shown in Figure 3, the inventors have found that when the number of epochs is small, the loss of LLM can be predicted by the product of a function that depends on the model size and the amount of data (f(model size, amount of data)) and a constant power of the proportion of target languages ​​(proportion of target languages^constant).

[0030] Furthermore, each of the graphs shown in Figure 4 is a log-log graph, with the horizontal axis representing the percentage of the target language (in this example, the percentage of Japanese) and the vertical axis representing the LLM loss achieved for the corresponding percentage of the target language. The differences between the graphs in Figure 4 indicate differences in the amount of data for each target language, with the graphs arranged in order from smallest to largest data amount: top left, top center, top right, left center, center, right center, and bottom left. In Figure 4, the differences in the lines indicate differences in model size. In Figure 4, the intensity of the line color represents the number of epochs, with darker colors indicating a smaller number of epochs.

[0031] In the multiple graphs shown in Figure 4, the amount of computational resources required to train the LLM is the same. Therefore, in Figure 4, it can be seen that the amount of data consumed increases as the model size decreases, and within the same graph, the number of epochs increases as the model size decreases.

[0032] Furthermore, in Figure 4, in the areas with a small number of epochs (darker color), the lines are generally parallel, similar to Figure 3. On the other hand, in the areas with a large number of epochs (lighter color), the lines shift in the direction of greater loss (worse performance) as the number of epochs increases (the loss is exceeding expectations).

[0033] Furthermore, as shown in the upper left graph of Figure 4, the extent of the upward fluctuation in losses does not depend solely on the number of epochs. For example, points P1 and P2 in the upper left graph of Figure 4 have almost the same number of epochs, but point P1 shows a greater upward fluctuation in losses.

[0034] Thus, from the graph in the upper left of Figure 4, the inventors have found that, if the amount of data for the target language repeated is the same, the larger the model size, the more data the LLM can memorize, making it more prone to overfitting and increasing the degree of upward fluctuation in loss.

[0035] The prediction device 1A and the processes performed by the prediction device 1A, as described below, are based on the knowledge mentioned above and on the inventor's own unique perspective.

[0036] (Overview of Prediction Device 1A) Prediction device 1A is a device that predicts the performance of an LLM after training. As an example, prediction device 1A refers to the model size MS of the LLM when an LLM for the target language is trained using multiple languages, the amount of training data TD used in the LLM training process, and the target language ratio LR which indicates the proportion of the training data amount that is the target language, and predicts the loss as the performance of the LLM.

[0037] The multiple languages ​​used to train the LLM include the target language and one or more languages ​​other than the target language. In this disclosure, "language" refers to natural languages ​​such as Japanese and English. In this disclosure, instead of "multiple languages," "multiple domains (e.g., specific fields such as dialects and medicine)" may be used to predict the loss when training the LLM for the target domain.

[0038] Furthermore, the prediction device 1A corrects the predicted loss by referring to the model size MS, the amount of target language data LD, and the number of epochs EN. For example, the prediction device 1A may correct the predicted loss when the value of the number of epochs EN is greater than or equal to a predetermined value (for example, 4).

[0039] (Configuration of prediction device 1A) The configuration of the prediction device 1A will be described with reference to Figure 5. Figure 5 is a block diagram showing the configuration of the prediction device 1A. As shown in Figure 5, the prediction device 1A includes a control unit 10, a storage unit 20, an input / output unit 21, and a communication unit 22.

[0040] (Storage unit 20) The memory unit 20 stores data that the control unit 10 references. Examples of data stored in the memory unit 20 include the model size MS of the LLM to be trained, the amount of training data TD used in the LLM training process, the target language ratio LR which indicates the proportion of the training data to be the target language, the amount of target language data to be repeated LD, and the number of epochs EN.

[0041] The training data volume TD used in the LLM learning process represents the total amount of training data used in the learning process.

[0042] The target language ratio LR indicates the proportion of data in the target language within the total training data volume for the entire learning process.

[0043] The repeated target language data amount LD (hereinafter also simply referred to as "target language data amount LD") indicates the amount of target language data used in one training session.

[0044] In other words, if D is the value of the training data volume TD, r is the value of the target language proportion LR, D_repeat is the value of the target language data volume LD, and k is the value of the number of epochs EN, then the following equation (1) holds. D × r = D_repeat × k ... (1) (Input / output section 21) The input / output unit 21 is an interface to input devices that accept data input and output devices that output data. Examples of input devices include, but are not limited to, microphones, cameras, eye-tracking devices, keyboards, and touchpads. Examples of output devices include, but are not limited to, speakers and liquid crystal displays.

[0045] (Communications Section 22) The communication unit 22 is an interface for sending and receiving data over a network. Examples of the communication unit 22 include, but are not limited to, communication chips in various communication standards such as Ethernet®, Wi-Fi®, and wireless communication standards for mobile data communication networks, as well as USB-compliant connectors.

[0046] (Control Unit 10) The control unit 10 controls each component of the prediction device 1A. The control unit 10 also includes an acquisition unit 11, a prediction unit 12, an output unit 13, and a correction unit 14, as shown in Figure 5. In this exemplary embodiment, the acquisition unit 11, prediction unit 12, output unit 13, and correction unit 14 implement the acquisition means, prediction means, output means, and correction means, respectively.

[0047] (Acquisition part 11) The acquisition unit 11 acquires data from the input / output unit 21 or the communication unit 22. The acquisition unit 11 stores the acquired data in the storage unit 20. As an example, the acquisition unit 11 acquires the model size MS, the amount of training data TD, and the percentage of target languages ​​LR. As another example, the acquisition unit 11 acquires the amount of target language data LD and the number of epochs EN.

[0048] (Prediction Unit 12) The prediction unit 12 predicts the loss of the LLM. The prediction unit 12 stores the predicted loss in the storage unit 20. For example, the prediction unit 12 predicts the loss of the LLM by referring to the model size MS, the amount of training data TD, and the proportion of target languages ​​LR.

[0049] More specifically, if the model size MS is N, the training data volume TD is D, and the target language ratio LR is r, the prediction unit 12 predicts the LLM loss value L using the following equation (2). L(N,D,r)=f(N,D)×r^d ···(2) Here, considering that f(N,D) is the case where r=1 in equation (2), that is, the case of monolingual learning of only the target language that is already known, we define f(N,D) as shown in equation (3) below. f(N,D)=A / N^a+B / D^b+C ···(3) Here, A, B, and C are parameters, and as an example, an LLM can be trained using only the target language with various model sizes and training data amounts, and estimation can be performed using the method described in Reference 1. Reference 1: Hoffmann, Jordan, et al. "Training compute-optimal large language models." arXiv preprint arXiv:2203.15556 (2022) Furthermore, d in equation (2) is also a parameter, and as an example, the LLM can be trained using various values ​​for training data volume and target language proportions with a fixed model size, and then estimated using the method described in Reference 2. Reference 2: Ge Ce, et al. "Data Mixing Made Efficient: A Bivariate Scaling Law for Language Model Pretraining." arXiv preprint arXiv:2405.14908 (2024) (Output section 13) The output unit 13 outputs data via the input / output unit 21 or the communication unit 22. For example, the output unit 13 outputs information indicating loss. Here, the information indicating loss output by the output unit may be information indicating the loss predicted by the prediction unit 12, or information indicating the loss corrected by the correction unit 14, which will be described later. With this configuration, the output unit 13 can notify the user of the loss when the LLM is trained using the model size MS, training data TD, and target language proportion LR. Alternatively, the output unit 13 can notify the user of the loss when the LLM is trained using the model size MS, training data TD, target language proportion LR, target language data amount LD, and number of epochs EN.

[0050] (Correction section 14) The correction unit 14 corrects the loss predicted by the prediction unit 12. The correction unit 14 stores the corrected loss in the storage unit 20. As an example, the correction unit 14 corrects the loss predicted by the prediction unit 12 by referring to the model size MS, the amount of target language data LD, and the number of epochs EN. As described above, the loss predicted by the prediction unit 12 may exceed the predicted loss depending on the model size MS, the amount of target language data LD, and the number of epochs EN. Therefore, the correction unit 14 calculates the loss after the loss predicted by the prediction unit 12 has exceeded the predicted loss by correcting the loss predicted by the prediction unit 12.

[0051] More specifically, if the model size MS is N, the training data volume TD is D, the target language ratio LR is r, the target language data volume LD is D_repeat, and the number of epochs EN is k, the correction unit 14 calculates the corrected loss value L* using the following equation (4). L*=g(f(N,D)×r^d,N,D_repeat,k) ···(4) For example, if we assume that the loss predicted by the prediction unit 12 is L, and L* is a correction value calculated by adding a value to L, then equation (4) can be rewritten as equation (5) below. L*=L+h(N,D_repeat,k) ···(5) Furthermore, as mentioned above, if D_repeat is the same, the larger the model size, the greater the degree of upward fluctuation in loss. Therefore, equation (5) can be rewritten as equation (6) below. L*=L+h(D_repeat / N, k) ···(6) One example of a method for calculating h(D_repeat / N, k) is for the correction unit 14 to use the difference between the loss L predicted by the prediction unit 12 and the loss L* measured in advance using a set of multiple sets including the model size MS, the amount of target language data LD, and the number of epochs EN, where at least one value is different.

[0052] One example of this configuration is a method in which the correction unit 14 corrects the loss using k-nearest neighbor regression. More specifically, the LLM is pre-trained using multiple sets of values, where at least one value differs from the model size MS value N*, the target language data amount LD value D_repeat*, and the number of epochs EN value k*, and the loss L* for each set is measured. In this case, the difference (L*-L) between the measured loss L* and the corresponding loss predicted in advance by the prediction unit 12 is stored in the form (D_repeat* / N*,k*,L*-L).

[0053] Then, for the model size MS, the amount of target language data LD, and the number of epochs EN used for correction, the correction unit 14 extracts n (D_repeat* / N*,k*,L*) values ​​in order of their proximity to these values, and calculates the average of the extracted n L*-L values ​​as the value of h(D_repeat / N,k). With this configuration, the correction unit 14 can suitably calculate the corrected loss value L*.

[0054] Furthermore, the correction unit 14 may be configured to correct the loss when the number of epochs EN is greater than or equal to a predetermined value. As mentioned above, when the number of epochs is small, the effect of overfitting is small, and upward fluctuations in the loss are less likely to occur. Therefore, the correction unit 14 corrects the loss when the number of epochs EN is greater than or equal to a predetermined value (for example, 4), and does not correct the loss when the number of epochs EN is less than the predetermined value. With this configuration, the correction unit 14 can reduce the processing load.

[0055] (Processing performed by prediction device 1A) The flow of the process (prediction method S1A) executed by the prediction device 1A will be explained with reference to Figure 6. Figure 6 is a flowchart showing the flow of prediction method S1A.

[0056] (Acquisition process S11) In the acquisition process S11, the acquisition unit 11 acquires the model size MS, the amount of training data TD, and the percentage of target languages ​​LR. The acquisition unit 11 stores the acquired model size MS, training data amount TD, and percentage of target languages ​​LR in the storage unit 20.

[0057] (Prediction processing S12) In prediction processing S12, the prediction unit 12 predicts the loss of the LLM by referring to the model size MS, the amount of training data TD, and the target language ratio LR. The prediction unit 12 stores the predicted loss in the storage unit 20. The method by which the prediction unit 12 predicts the loss of the LLM is as described above.

[0058] (Output processing S13) In output processing S13, the output unit 13 outputs information indicating the loss predicted by the prediction unit 12.

[0059] (Step S14) In step S14, the acquisition unit 11 acquires the target language data amount LD and the number of epochs EN. The acquisition unit 11 stores the acquired target language data amount LD and number of epochs EN in the storage unit 20.

[0060] (Step S15) In step S15, the correction unit 14 corrects the loss predicted by the prediction unit 12 by referring to the model size MS, the amount of target language data LD, and the number of epochs EN. The correction unit 14 stores the corrected loss in the storage unit 20. The method by which the correction unit 14 corrects the loss is as described above.

[0061] (Output processing S16) In output processing S16, the output unit 13 outputs information indicating the loss corrected by the correction unit 14.

[0062] (Example of application of prediction device 1A) For example, the prediction device 1A may train an LLM using the model size MS, the amount of training data TD, the proportion of target languages ​​LR, the amount of target language data LD, and the number of epochs EN. Known methods may be used as the method for training the LLM. Furthermore, when the prediction device 1A trains an LLM, it may perform a process to determine various learning settings, such as the amount of computational resources used for the learning process, before training the LLM. This reduces the search space required to change the model size in order to generate a higher-performance LLM.

[0063] Furthermore, the prediction device 1A may instruct an external device, different from the prediction device 1A, to train the LLM. In this case, the prediction device 1A may instruct the external device to narrow the range of various learning settings that can be selected using the predicted loss. With this configuration, the prediction device 1A can reduce the search space in which the loss is varied for the external device.

[0064] (Effect of prediction device 1A) As described above, the prediction device 1A predicts the loss of the LLM by referring to the model size MS, the amount of training data TD, and the proportion of target languages ​​LR. Therefore, the prediction device 1A can predict the performance of the LLM when it is trained using multiple languages.

[0065] Furthermore, the prediction device 1A corrects the predicted loss by referring to the model size MS, the number of epochs EN, and the amount of target language data LD. As described above, the predicted loss will be higher depending on the model size MS, the number of epochs EN, and the amount of target language data LD. Since the prediction device 1A can calculate the predicted value after the upward deviation, it can accurately predict the performance of the LLM when it is trained using multiple languages.

[0066] [Examples of implementation using software] Some or all of the functions of prediction devices 1 and 1A (hereinafter also referred to as "each of the above devices") may be implemented by hardware such as integrated circuits (IC chips) or by software.

[0067] In the latter case, each of the above devices is implemented, for example, by a computer that executes instructions for a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as Computer C) is shown in Figure 7. Figure 7 is a block diagram showing the hardware configuration of Computer C, which functions as each of the above devices.

[0068] Computer C comprises at least one processor C1 and at least one memory C2. Memory C2 stores a program P that causes computer C to operate as each of the above-mentioned devices. In computer C, processor C1 reads program P from memory C2 and executes it, thereby realizing each of the above-mentioned devices.

[0069] For processor C1, for example, a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof can be used. For memory C2, for example, flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof can be used.

[0070] Computer C may also be equipped with RAM (Random Access Memory) for loading program P at runtime and for temporarily storing various data. Furthermore, computer C may be equipped with communication interfaces for sending and receiving data with other devices. Additionally, computer C may be equipped with input / output interfaces for connecting input / output devices such as keyboards, mice, displays, and printers.

[0071] Furthermore, program P can be recorded on a non-temporary, tangible recording medium M that is readable by computer C. Such a recording medium M could be, for example, tape, disk, card, semiconductor memory, or programmable logic circuitry. Computer C can acquire program P via such a recording medium M. Program P can also be transmitted via a transmission medium. Such a transmission medium could be, for example, a communication network or broadcast waves. Computer C can also acquire program P via such a transmission medium.

[0072] Furthermore, each of the above functions of each of the above devices may be implemented by a single processor in a single computer, by multiple processors in a single computer working together, or by multiple processors in each of multiple computers working together. In addition, the programs for implementing each of the above functions in each of the above devices may be stored in a single memory in a single computer, distributed and stored in multiple memories in a single computer, or distributed and stored in multiple memories in each of multiple computers.

[0073] [Additional Note A] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0074] (Note A1) An acquisition means for obtaining the model size of a language model for a target language to be trained using multiple languages, the amount of training data used in the training process of the language model, and the target language ratio, which indicates the proportion of the amount of data for the target language within the amount of training data. A prediction means for predicting the loss of the language model using the product of a function that depends on the model size and the amount of training data and a constant power of the proportion of the target language, A prediction device equipped with the following features.

[0075] (Appendix A2) The acquisition means acquires the number of epochs in the learning process and the amount of data for the target language that is repeated, The prediction device further includes correction means for correcting the loss by referring to the model size, the number of epochs, and the amount of data for the target language to be repeated. The prediction device described in Appendix A1.

[0076] (Note A3) The correction means uses the difference between the loss predicted by the prediction means and the loss measured in advance using a plurality of sets including the model size, the amount of data for the target language, and the number of epochs, where at least one value is different. The prediction device described in Appendix A2.

[0077] (Note A4) The correction means corrects the loss using k-nearest neighbor regression. The prediction device described in Appendix A3.

[0078] (Note A5) The correction means corrects the loss when the number of epochs is greater than or equal to a predetermined value. The prediction device described in any one of the appendices A2 to A4.

[0079] (Note A6) The system further includes an output means for outputting information indicating the aforementioned loss. A prediction device as described in any one of the appendices A1 to A5.

[0080] [Additional Notes B] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0081] (Note B1) At least one processor performs an acquisition process to obtain the model size of a language model for a target language to be trained using multiple languages, the amount of training data used in the training process of the language model, and the target language ratio, which represents the proportion of the amount of data for the target language within the amount of training data. The at least one processor performs a prediction process to predict the loss of the language model using the product of a function that depends on the model size and the amount of training data and a constant power of the proportion of the target language. A prediction method that includes this.

[0082] (Note B2) In the acquisition process, the at least one processor acquires the number of epochs in the learning process and the amount of data for the target language to be repeated. The at least one processor further includes a correction process that corrects the loss by referring to the model size, the number of epochs, and the amount of data in the target language being repeated. The prediction method described in Appendix B1.

[0083] (Note B3) In the correction process, the at least one processor uses the difference between the loss predicted in the prediction process and the loss measured in advance using a plurality of sets including the model size, the amount of data for the target language, and the number of epochs, where at least one value is different. The prediction device described in Appendix B2.

[0084] (Note B4) In the correction process, at least one processor corrects the loss using k-nearest neighbor regression. The prediction method described in Appendix B3.

[0085] (Note B5) In the correction process, the at least one processor corrects the loss if the number of epochs is greater than or equal to a predetermined value. The prediction method described in one of the appendices B2 through B4.

[0086] (Note B6) The at least one processor further includes output processing that outputs information indicating the loss. The prediction method is described in one of the appendices B1 through B5.

[0087] [Additional Note C] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0088] (Note C1) A program that makes a computer function as a prediction device, The aforementioned computer, An acquisition means for obtaining the model size of a language model for a target language to be trained using multiple languages, the amount of training data used in the training process of the language model, and the target language ratio, which indicates the proportion of the amount of data for the target language within the amount of training data. A prediction means for predicting the loss of the language model using the product of a function that depends on the model size and the amount of training data and a constant power of the proportion of the target language, A prediction program designed to function as such.

[0089] (Note C2) The acquisition means acquires the number of epochs in the learning process and the amount of data for the target language that is repeated, The aforementioned computer, The prediction device further functions as a correction means for correcting the loss, with reference to the model size, the number of epochs, and the amount of data for the target language to be repeated. The prediction program described in Appendix C1.

[0090] (Note C3) The correction means uses the difference between the loss predicted by the prediction means and the loss measured in advance using a plurality of sets including the model size, the amount of data for the target language, and the number of epochs, where at least one value is different. The prediction device described in Appendix C2.

[0091] (Note C4) The correction means corrects the loss using k-nearest neighbor regression. The prediction program described in Appendix C3.

[0092] (Note C5) The correction means corrects the loss when the number of epochs is greater than or equal to a predetermined value. The prediction program described in any one of the appendices C2 through C4.

[0093] (Appendix C6) The aforementioned computer, Further, the output means is configured to output information indicating the aforementioned loss. The prediction program described in any one of the appendices C1 through C5.

[0094] [Additional Note D] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0095] (Note D1) It comprises at least one processor, and the at least one processor is An acquisition process to obtain the model size of a language model for a target language that is trained using multiple languages, the amount of training data used in the training process of the language model, and the target language ratio, which represents the proportion of the amount of data for the target language within the amount of training data. A prediction process that predicts the loss of the language model using the product of a function that depends on the model size and the amount of training data and a constant power of the proportion of the target language, A prediction device that performs this operation.

[0096] The prediction device may also include a memory. Furthermore, the memory may store a program that causes at least one processor to execute each of the aforementioned processes.

[0097] (Note D2) In the acquisition process, the at least one processor acquires the number of epochs in the learning process and the amount of data for the target language to be repeated. The at least one processor further performs a correction process to correct the loss, with reference to the model size, the number of epochs, and the amount of data in the target language being repeated. The prediction device described in Appendix D1.

[0098] (Note D3) In the correction process, the at least one processor uses the difference between the loss predicted in the prediction process and the loss measured in advance using a plurality of sets including the model size, the amount of data for the target language, and the number of epochs, where at least one value is different. The prediction device described in Appendix D2.

[0099] (Note D4) In the correction process, at least one processor corrects the loss using k-nearest neighbor regression. The prediction device described in Appendix D3.

[0100] (Note D5) In the correction process, the at least one processor corrects the loss if the number of epochs is greater than or equal to a predetermined value. A prediction device as described in any one of the appendices D2 to D4.

[0101] (Note D6) The aforementioned at least one processor, Further output processing is performed to output information indicating the aforementioned loss. A prediction device as described in any one of the appendices D1 to D5.

[0102] [Additional Note E] This disclosure includes the technologies described in the following appendices. However, the present invention is not limited to the technologies described in the following appendices, and various modifications are possible within the scope of the claims.

[0103] (Note E1) A program that makes a computer function as a prediction device, To the aforementioned computer, An acquisition process to obtain the model size of a language model for a target language that is trained using multiple languages, the amount of training data used in the training process of the language model, and the target language ratio, which represents the proportion of the amount of data for the target language within the amount of training data. A prediction process that predicts the loss of the language model using the product of a function that depends on the model size and the amount of training data and a constant power of the proportion of the target language, A non-temporary recording medium on which a prediction program that executes the prediction is recorded. [Explanation of Symbols]

[0104] 1. 1A Prediction device 11 Acquisition Department 12 Prediction Section 13 Output section 14 Correction section MS Model Size TD training data volume LR Target Language Ratio LD Target language data volume EN Epoch Number

Claims

1. An acquisition means for obtaining the model size of a language model for a target language to be trained using multiple languages, the amount of training data used in the training process of the language model, and the target language ratio, which indicates the proportion of the amount of data for the target language within the amount of training data. A prediction means for predicting the loss of the language model using the product of a function that depends on the model size and the amount of training data and a constant power of the proportion of the target language, A prediction device equipped with the following features.

2. The acquisition means acquires the number of epochs in the learning process and the amount of data for the target language that is repeated, The prediction device further includes correction means for correcting the loss by referring to the model size, the number of epochs, and the amount of data for the target language to be repeated. The prediction device according to claim 1.

3. The correction means uses the difference between the loss predicted by the prediction means and the loss measured in advance using a plurality of sets including the model size, the amount of data for the target language, and the number of epochs, where at least one value is different. The prediction device according to claim 2.

4. The correction means corrects the loss using k-nearest neighbor regression. The prediction device according to claim 3.

5. The correction means corrects the loss when the number of epochs is greater than or equal to a predetermined value. A prediction device according to any one of claims 2 to 4.

6. The system further includes an output means for outputting information indicating the aforementioned loss. A prediction device according to any one of claims 1 to 4.

7. At least one processor performs an acquisition process to obtain the model size of a language model for a target language to be trained using multiple languages, the amount of training data used in the training process of the language model, and the target language ratio, which represents the proportion of the amount of data for the target language within the amount of training data. The at least one processor performs a prediction process to predict the loss of the language model using the product of a function that depends on the model size and the amount of training data and a constant power of the proportion of the target language. A prediction method that includes this.

8. A program that makes a computer function as a prediction device, The aforementioned computer, An acquisition means for obtaining the model size of a language model for a target language to be trained using multiple languages, the amount of training data used in the training process of the language model, and the target language ratio, which indicates the proportion of the amount of data for the target language within the amount of training data. A prediction means for predicting the loss of the language model using the product of a function that depends on the model size and the amount of training data and a constant power of the proportion of the target language, A prediction program designed to function as such.

Images