Information processing device, information processing method, and program

Abstract

This information processing device for training a model of a neural network comprises a parameter processing unit that initializes, on the basis of a first index common to layers, a weight parameter included in the model. The weight parameter is adjusted by using a gate parameter.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to training models of neural networks.

[0002] Large language models (LLMs) have attracted considerable attention. Training an LLM requires a huge amount of data, resulting in a huge cost. A spike phenomenon, in which the loss increases rapidly, is known to occur during pre-training of an LLM. The spike phenomenon is an important issue for LLMs not only in terms of its impact on the final loss function value (performance), but also in terms of training costs, since the loss function value sometimes does not return to normal, causing the training to fail completely.

[0003] Toan Q. Nguyen and Julian Salazar. Transformers without tears: Improving the normalization of self-attention. In Jan Niehues, Rolando Cattoni, Sebastian Stuker, MatteoNegri, Marco Turchi, Thanh-Le Ha, Elizabeth Salesky, Ramon Sanabria, Loic Barrault, Lucia Specia, and Marcello Federico, editors, Proceedings of the 16th International Conference on Spoken Language Translation, Hong Kong, November 2-3 2019. Association for Computational Linguistics.

[0004] As one method for avoiding spikes, Non-Patent Document 1 discloses a model parameter initialization method called small initialization. In small initialization, the initial values ​​of the model to be trained are set smaller than the initial values ​​conventionally used in deep learning models. However, spikes still occur even when small initialization is used. Note that spikes are a phenomenon that can occur in models in general, not just LLMs.

[0005] The present invention has been made in view of the above points, and has an object to provide a technique that makes it possible to suppress the occurrence of spike phenomena in the learning of a neural network model.

[0006] According to the disclosed technology, there is provided an information processing device that performs learning of a neural network model, the information processing device including a parameter processing unit that initializes weight parameters included in the model based on a first index common to each layer, and the weight parameters are adjusted using gate parameters.

[0007] According to the disclosed technology, it is possible to suppress the occurrence of spike phenomena in the learning of a neural network model.

[0008] FIG. 1 is a diagram showing an example of the configuration of a learning device 100. FIG. 2 is a diagram showing an example of application of gate parameters. FIG. 3 is a flowchart for explaining the operation of the learning device 100. FIG. 4 is a diagram showing an example of the configuration of an inference device 200. FIG. 5 is a flowchart for explaining the operation of the inference device 200. FIG. 6 is a diagram showing the standard deviation after passing through the gate and the standard deviation of the parameters in the initialization weekly report. FIG. 7 is a diagram showing hyperparameters of the model used in the experiment. FIG. 8 is a diagram showing hyperparameters of the learning settings in the experiment. FIG. 9 is a diagram showing experimental results. FIG. 10 is a diagram showing experimental results. FIG. 11 is a diagram showing an example of the hardware configuration of the device.

[0009] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment. Below, a device configuration and device operation for solving the above-mentioned problems will be described.

[0010] In the text of the specification below, for convenience of description, a bar intended to be placed at the beginning of a character is written at the upper left of the character. － W ｉ " is an example. Furthermore, normal font characters are used to represent matrices and sets. It is clear from the context that these characters represent matrices and sets. Furthermore, the learning device 100 and the inference device 200 described below may both be called information processing devices.

[0011] (Configuration and Operation of Learning Device 100) Fig. 1 shows an example configuration of a learning device 100 according to this embodiment. As shown in Fig. 1, the learning device 100 includes a conversion unit 110, a loss calculation unit 120, a parameter update unit 130, a parameter DB (database) 140, an input unit 150, and a parameter processing unit 160. The learning device 100 also includes a learning data DB 170, from which the input unit 150 can acquire learning data. The learning data may also be referred to as training data.

[0012] Furthermore, the "loss calculation unit 120, parameter update unit 130, and parameter processing unit 160" may be referred to as a learning unit. Note that the learning unit may not include the parameter processing unit 160. In other words, the "loss calculation unit 120 and parameter update unit 130" may be referred to as a learning unit. Furthermore, in FIG. 1, a model that is the output of the result of the initialization process and a trained model that is the output of the result of the learning process are shown outside the learning device 100.

[0013] The conversion unit 110 may be any model as long as it is a neural network model. The processing target of the model may be text, an image, audio, or other data. If the processing target is text, the conversion unit 110 may be a language model using, for example, Transformer.

[0014] The parameter DB 140 stores the model parameters Θ used by the conversion unit 110. The parameter DB 140 may be considered to be a DB (storage unit) inside the conversion unit 110. Alternatively, the parameter DB 140 may be considered to be a "model."

[0015] <About the Model Parameter Θ> The model parameter Θ is all the parameters of the neural network model serving as the transformation unit 110, and the weight parameter among the model parameters Θ is W. The weight parameter W is W, which is a parameter (matrix) of the layer i in the model. ｉ is a set of.

[0016] In this embodiment, a gate parameter α is introduced to adjust the weight parameter W. Specifically, as shown in the following equation, the temporary weight parameter ( － W ｉ ) and gate parameter (α ｉ ) product of W ｉ Expresses.

[0017] W ｉ = α ｉ * － W ｉ In addition, － W ｉ are called actual parameters, and W ｉ may be called a virtual parameter. － W ｉ The term "weight parameter" may be used in a broader sense to encompass "weight parameter W" and "weight parameter W."

[0018] 1 shows that the model parameters stored in the parameter DB 140 include gate parameters and temporary weight parameters. Although the gate parameters and temporary weight parameters are shown separately in FIG. 1, when used, they are used as weight parameters obtained by multiplying the gate parameters by the temporary weight parameters. Note that the model parameters may include parameters other than the weight parameters that are subject to adjustment by the gate parameters.

[0019] In this embodiment, － W ｉ The spike phenomenon is suppressed by setting the standard deviation of the parameter σ (common to the entire model). － W ｉ Each parameter in is initialized with a value sampled from a normal distribution with a mean of 0 and a standard deviation of σ, where σ is a hyperparameter. Note that, as will be described later, the distribution used for initialization is not limited to the normal distribution.

[0020] Note that since initialization means setting parameters to predetermined values, "initialize" may be rephrased as "set."

[0021] Furthermore, the parameter processing unit 160 ｉ The initial value of σ ｉ / σ, where σ ｉ (parameters used in each layer) are W ｉ is the scale (for example, standard deviation) you want to set to σ ｉ (fixed value specified for each layer) is a hyperparameter. σ ｉ By using [mathematical formula - see original document], we give each layer a different scale, as is commonly done in existing methods.

[0022] W obtained by the initialization process by the parameter processing unit 16 ｉ The standard deviation of is σ*α ｉ =σ ｉ This makes it possible to align it with existing methods.

[0023] An example of the above-mentioned existing method is the He Initialization disclosed in "Kaiming He, Xiangyu Zhang, Shaoqing Ren, and JianSun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015."

[0024] That is, during learning, the conversion unit 110 has gate parameters in addition to temporary weight parameters, and performs conversion processing using model parameters including these parameters. Here, the gate parameters are defined as follows.

[0025] The gate parameter is a parameter for adjusting the weight parameter, and more specifically, a parameter that serves to adjust the variation (which may be called scale) of the weight parameter. The scale may also be expressed as "magnitude" or "norm." In this specification, the average is set to 0, and "scale (magnitude / norm)" and "variation (standard deviation)" may be described as being the same thing.

[0026] <Gate parameter α ｉ Example of application of the gate parameter α ｉ The process shown in FIG. 2 is a diagram illustrating an example of applying the vector x (x １ ,...,x ｍ ) into the weight vector w(W ｉｊ１ , ..., W ｉｊｍ ) to a scalar y, and FIG. 2 shows the process for the j-th row of the model parameters in layer i. Although FIG. 2 shows an example in which a bias is used, it is also possible not to use a bias. W ｉ Is W ｉｊｋ is a set of －W ｉ teeth － W ｉｊｋ is a set of.

[0027] As shown in FIG. 2, each element x of vector x ｋ Weight W ｉｊｋ The weighted sum of the values ​​is multiplied by the weight, and the bias is added to the sum, which is then input to the activation function, and the output from the activation function is obtained.

[0028] As shown in the image in FIG. 2, in layer i, － W ｉｊｋ The gate parameter α ｉ Multiplied by W ｉｊｋ is.

[0029] Gate parameter α ｉ The layer to which the gate parameter α is applied is not limited to the linear layer shown in FIG. 2. For example, even if the embedding layer holds parameters, the gate parameter α ｉ As a specific example, if the model is a Transformer, the self-attention layer and MLP layer of the Transformer are composed of linear layers, so each layer has a gate parameter α ｉ For the embedded layer, just prepare the gate parameters as in the linear layer.

[0030] The learning operation of the learning device 100 will be described with reference to the flowchart in Figure 3. The learning here may be supervised learning assuming an arbitrary task, or may be pre-learning. In S101, the parameter processing unit 160 initializes the model parameters (including temporary weight parameters and gate parameters) in the conversion unit 110. Details of the initialization method will be described later.

[0031] In S102, the input unit 150 samples training data (pairs of input data to the model and correct data) from the training data DB 170, inputs the input data to the conversion unit 110, and inputs the correct data to the loss calculation unit 120.

[0032] For example, if the task assumed in the learning is speech recognition, the input data is speech and the correct answer data is text corresponding to the content of the input speech. Also, if the learning assumed is pre-learning of a language model and the learning is for the task of predicting the words that follow some words in text, the input data is "some words in the text" and the correct answer data is "the words that follow them."

[0033] In S103, the conversion unit 110 converts the input data based on the model parameters and outputs the conversion result. For example, in the case of the task of predicting the next word as described above, the conversion unit 110 outputs the text generation probability as the conversion result.

[0034] In S104, the loss calculation unit 120 calculates a loss function value using the correct answer data and the conversion result. For example, in the case of the task of predicting the next word as described above, the loss function value is calculated from the text generation probability for each word and the correct word.

[0035] In S105, the parameter update unit 130 updates the model parameters using the loss function value. Specifically, the model parameters are updated so that the loss function value becomes smaller.

[0036] If steps S102 to S105 have been repeated the specified number of times, the process proceeds to step S107, otherwise the process returns to step S102 (step S106).

[0037] In S107 , the parameter processing unit 160 integrates the temporary weight parameters and gate parameters stored in the parameter DB 140 .

[0038] (Configuration and Operation of Inference Device 200) Fig. 4 shows an example configuration of inference device 200 in this embodiment. As shown in Fig. 4, inference device 200 includes a conversion unit 210, a shaping unit 220, and a parameter DB 230. Parameter DB 230 stores model parameters including weight parameters learned and integrated by learning device 100. As described above, parameters are integrated at the end of learning, so no gate parameters exist during inference.

[0039] The conversion unit 210 has the same functional unit as the conversion unit 110 in the learning device 100. However, the conversion unit 210 executes the inference process using the trained model parameters stored in the parameter DB 230.

[0040] Inference device 200 may be a device separate from learning device 100, or inference device 200 and learning device 100 may be the same device. For example, by providing learning device 100 with shaping unit 220, learning device 100 can also be used as inference device 200.

[0041] The operation of inference device 200 will be described with reference to the flowchart in Figure 5. In S201, input data is input to conversion unit 210. For example, when it is assumed that a task for converting text is to be executed, the input data is a certain text.

[0042] In S202, the conversion unit 210 converts the input data and outputs the conversion result based on the trained model parameters read from the parameter DB 230. For example, when it is assumed that a task of converting text is to be executed, the conversion result is a text generation probability.

[0043] In S203, the shaping unit 220 outputs output data using the conversion result. When the above tasks are assumed to be executed, the output data is converted text.

[0044] (Regarding the Conversion Unit 110) The conversion unit 110 will be described in more detail below.

[0045] The transform unit 110 in this embodiment is a neural network model, and the model structure is not limited to a specific structure, but in this embodiment, it is assumed that this model is a Transformer similar to the existing LLM.

[0046] The transformer 110 includes an embedding layer, a self-attention layer, and an FFN layer. The embedding layer uses W as a parameter matrix (specifically, a matrix of weight parameters). ｅ The Self-Attention layer has a parameter matrix Wｋ , W ｑ , W ｖ , W ｏ The FFN layer has a parameter matrix W ｕ , W ｄ The pair of a Self-Attention layer and an FFN layer is called a Transformer layer.

[0047] Here, the transformation unit 110 is called a "model," and as described above, the weight parameters (matrix) of the layer i constituting the model are denoted by W ｉ For example, the embedding layer has i = e and its weight parameter is W ｅ The Transformer consists of multiple Transformer layers, and each Transformer layer is ｋ , W ｑ , W ｖ , W ｏ、 W ｕ , W ｄ However, the parameter matrix is ​​abbreviated without distinguishing which Transformer layer it belongs to.

[0048] σ ｉ W ｉ is the scale (standard deviation) that you want to set. The learning device 100 stores σ ｉ Alternatively, σ may be read from outside the learning device 100. ｉ may be acquired or received.

[0049] In the conventional parameter initialization, W ｉ Each parameter that constitutes the ｉ In the prior art, σ ｉ It is common to set different values ​​for different layers i.

[0050] Unlike the above-mentioned conventional technology, in this embodiment, ｉ Instead of W ｉ a matrix of the same size as － W ｉ are set (registered) as temporary weight parameters for the corresponding layer i.－ W ｉ Each parameter in is initialized with a value sampled from a normal distribution with a mean of 0 and a standard deviation of σ. Here, σ is a value that is common to the entire model (i.e., each layer). － W ｉ Each parameter in may be initialized with a value sampled from a distribution other than the "normal distribution with mean 0 and standard deviation σ" (for example, a uniform distribution).

[0051] Furthermore, the learning device 100 may read σ from a storage device such as a memory held by the learning device 100 itself, or may acquire or receive σ from outside the learning device 100.

[0052] In each layer i in the model, as shown in the following equation, － W ｉ Don't just use it, always － W ｉ σ ｉ W multiplied by / σ ｉ That is, the conversion unit 110 adjusts the weight parameters using the gate parameters, and performs conversion using the adjusted weight parameters.

[0053] By the above operation, the scale of the parameters used in each layer is set to σ ｉ While maintaining the above, we can adopt a common value σ for the scale of the actual parameters across the entire model.

[0054] As described above, in this embodiment, σ ｉ / σ is a learnable gate parameter α ｉ Then, the parameter processing unit 160 calculates α ｉ The initial value of σ ｉ / σ.

[0055] In the technology according to the present embodiment, one real value α is used for one parameter matrix as a learning target. ｉ Since it only increases the size of the model to be trained, it has almost no effect on the size of the model to be trained.

[0056] That is, in this embodiment, all parameters have a common scale. － Wｉ are registered in the model as temporary weight parameters (stored in the parameter DB 140). When using the temporary weight parameters in the model, σ ｉ / σ － W ｉ The weight parameter W multiplied by ｉ In addition, σ ｉ / σ － W ｉ The multiplication process is performed by the conversion unit 110, for example.

[0057] By the above operation, we can achieve initialization with a uniform standard deviation without changing the behavior of the model. ｉ / σ is a learnable parameter α ｉ and register it in the model as σ ｉ / σ. The learnable parameter α ｉ is the parameter matrix W ｉ Since the number of parameters is only increased by one for each model, it does not significantly affect the total number of model parameters or the training time. Also, since the parameters can be merged back into the original parameters after training, it does not affect inference.

[0058] Note that σ is an example of the first index. The first index is an index selected for updating parameters. For example, the first index is set to a small value that is common to each layer (using a common value can stabilize learning, and using a small value can speed up learning). σ ｉ is an example of the second index. The second index is an index selected for gradient backpropagation in a neural network. For example, the second index is set to keep the norm of the gradient constant. The second index can be set using, for example, the existing He Initialization technique.

[0059] The gate parameter is σ ｉ Since it is determined based on / σ, σ is the first index, σ ｉ is the second index, the gate parameters are initialized using both the first and second indexes. ｉ If / σ is the second index, the gate parameters are initialized using the second index. ｉWhen the second index is used, and σ ｉ In either case, when / σ is used as the second index, the gate parameters are initialized based on the second index. In either case, the second index is set to σ ｉ The value will vary depending on the

[0060] However, the second index in the above description that "the second index is an index selected for backpropagation of gradients in a neural network" is σ ｉ It refers to σ ｉ When / σ is the second index, σ ｉ may be used as the third index. ｉ can be said to be an index related to the gradient backpropagation of the model, and σ can be said to be an index related to the parameter update of the model.

[0061] In addition, σ is an index common to each layer, and σ ｉ is an index that differs for each layer. However, σ ｉ σ calculated based on σ and ｉ / σ can also be said to be an index that differs for each layer.

[0062] (Initialization) The initialization process executed by the parameter processing unit 160 will be described in more detail. As described above, in this embodiment, the learnable gate parameter α ｉ As the initial value of σ ｉ / σ, so σ and σ ｉ Each of these needs to be determined (set).

[0063] Regarding σ, the parameter processing unit 160 may set σ to any value. σ is tunable as a hyperparameter. The parameter processing unit 160 may use σ read from the storage device of the learning device 100 as described above, or may use σ acquired or received from outside the learning device 100.

[0064] σ ｉ Regarding σ, the parameter processing unit 160 uses an arbitrary initialization algorithm to ｉIn this embodiment, σ is determined using a method called He Initialization, which is disclosed in the aforementioned "Kaiming He, Xiangyu Zhang, Shaoqing Ren, and JianSun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015." ｉ Determine.

[0065] Furthermore, the parameter processing unit 160 uses σ ｉ Alternatively, σ obtained or received from outside the learning device 100 may be used. ｉ may also be used.

[0066] He Initialization is a method for maintaining a constant gradient norm from the top to the bottom layers in a multilayer neural network.

[0067] In this embodiment, the following two modifications (1) and (2) are made to apply He Initialization to the transforming unit 110 (i.e., the Transformer model). Note that these modifications themselves are existing techniques.

[0068] (1) The final layer in each residual block, the down_projection layer (W ｕ ) and out_projection layer (W ｏ ) multiply the standard deviation by 1 / √(2N) (where N is the number of layers).

[0069] (2) For the buried layer, a gate with a standard deviation of 1 is prepared.

[0070] 6 shows the scales of the parameters of He Initialization and Small Initialization, as well as the scales after passing through the gate and the actual scales of the parameters for the proposed method, which is the technology according to this embodiment. As shown in FIG. 6, in the proposed method, the values ​​after passing through the gate are aligned with the values ​​of He Initialization.

[0071] As shown in Figure 6, in the proposed method, all parameters are initialized with a common constant σ. ｕ No.d ｏｕｔ and W ｄ No.d ｉｎ Let 4d be the other d ｏｕｔ and d ｉｎ We set all of them to d, where d is the number of dimensions of the hidden state.

[0072] (Regarding Parameter Integration After Learning) In the parameter integration in S107 of FIG. 3, the parameter processing unit 160 － W ｉ and the gate parameter α ｉ is deleted from the model (i.e., deleted from the parameter DB 140), － W ｉ The gate parameter α ｉ W multiplied by ｉ is registered in the model (i.e., stored in the parameter DB 140). ｉ This is done using

[0073] (Reason (Theoretical Background) Why Learning is Stable by the Proposed Method) Hereinafter, the reason (theoretical background) why learning is stabilized by using the proposed method, which is the technology according to this embodiment, will be described.

[0074] In optimization algorithms such as Adam, the parameter update amount ΔWt at each step is calculated as follows: ｔ As determined by.

[0075] In the above formula, m ｔ is the exponential moving average with slope ∂L / ∂W, and v ｔ is the exponential moving average of the squared gradient.

[0076] Therefore, when the scales of parameters within a model are different, learning progresses slowly with large-scale parameters, while learning becomes unstable with small-scale parameters. However, as mentioned above in (1), learning a multi-layer transformer requires each parameter matrix to have a different scale.

[0077] The difference in scale for each parameter matrix is ​​one of the causes of the spike phenomenon. More specifically, the range in which learning progresses efficiently (μ ｔ ≧1e-4), learning may become unstable for small-scale parameters. Furthermore, the learning rate set to avoid spikes is too small for large-scale parameters, and learning of the entire model may not proceed efficiently.

[0078] The advantage of the proposed method is that it can handle different scale requirements by introducing a new gate parameter, α ｉ By assigning this to the weighting parameter, it is possible to set a common scale σ for all weighting parameters.

[0079] Furthermore, in the proposed method, the influence of the gate parameter (α ｉ Since the gate parameter is offset by the Adam update amount ΔW ｔ does not affect the

[0080] Therefore, all weight parameters within the model have a common scale σ and a common learning rate μ ｔ This update is performed independently of the gate parameter value, which differs for each weight parameter. Therefore, all weight parameters are updated equally, resulting in stable learning.

[0081] In other words, in the training using the proposed method, the scale of the weight parameter σ, which is specified to suppress gradient explosion and vanishing in the Transformer model, is ｉ is the temporary weight parameter － W ｉ The update amount of the temporary weight parameter Δ － W ｉThis is because, from the LLM learning, ｉ This can be interpreted as essentially removing the constraint on . Furthermore, unlike existing techniques that make the scale of the initial value dependent on d, the dependence on d is also ｉ By using this function, the scale of the initial value can be determined independently of the model size.

[0082] (Hyperparameter Setting) An example of setting hyperparameters to rapidly and stably proceed with learning will be described.

[0083] <Initial Values> As mentioned above, the introduction of a gate mechanism allows the initial parameter σ to be set to any value. In the experiments described below, unless otherwise specified, the variance was standardized to 4e-5. This corresponds to √(2 / 5d), the standard for small initialization, which is d=10,000, an even smaller value than small initialization. Therefore, it is expected that learning will progress relatively faster when learning at the same learning rate.

[0084] <Learning rate> Because the proposed method is expected to stabilize learning, in the experiment, a learning rate higher than the learning rate (e-4 order) typically used in pre-training of LLM can be set. In the experiment, a learning rate of 1e-3 was adopted.

[0085] <Regarding Batch Size> The batch size in recent LLM pre-training is set to a large value of 4M tokens to avoid unstable training due to spike phenomena, etc. In contrast, in this embodiment, uniform initialization is performed to stabilize training, so the batch size can be set small.

[0086] Setting a smaller batch size leads to an increase in the number of steps when training with the same amount of data, which lowers the final loss function value and is expected to improve performance. However, pre-training for LLM requires a huge amount of computational cost, so it is common to prepare a large number of GPUs and perform parallel calculations. Reducing the batch size limits the number of parallel GPUs, so in our experiments we set it to 1M.

[0087] (About the experiment) In order to confirm the effectiveness of the proposed method, we conducted an experiment to compare the proposed method with the baseline Small Initialization. We will now explain the settings and results of that experiment.

[0088] <Model Settings> In the experiment, we trained a model with the hyperparameters shown in Figure 7. The structure of the target model is similar to LLaMA disclosed in "Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023." However, we used a three-layer neural network with gelu as the activation function instead of SWiGLU.

[0089] <Dataset> Experiments were conducted using the RefinedWeb dataset disclosed in "Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116, 2023." From the perspective of computational complexity, training was performed using data randomly sampled from RefinedWeb with 30B tokens.

[0090] The paper "Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022." points out that an appropriate number of tokens for training is 20 times the model size, and this experiment used data equivalent to 1 / 10 of the total training data.

[0091] <Learning settings> Learning was performed based on the hyperparameters shown in Figure 8. For the learning rate, batch size, and warmup steps, two types of settings were prepared: a setting that prioritizes the speed of learning (rapid setting), and a conventional setting that prioritizes the stability of learning (stable setting).

[0092] <Experimental Results> The experimental results are shown in Figures 9 and 10. Figure 9 shows the loss function values ​​during model training for the proposed method and the baselines (Rapid setting, Stable setting). Figure 10 shows the loss function values ​​at the beginning of model training.

[0093] As shown in Figures 9 and 10, when initialization is performed using the proposed method, stable learning progresses, but initialization using the baseline small initialization causes spikes. Furthermore, the loss function value is also smaller when initialized using the proposed method. In other words, the proposed method achieves stable and fast learning.

[0094] FIG. 11 shows the results of the baseline and proposed methods. ｄ || / ||W ｄ || and ||ΔW ｕ || / ||W ｕwhere ∥W∥ is the norm of the weight parameter W.

[0095] Weight parameter W ｄ The initial scale of the weight parameter W is set small. ｄ Then, the usual weight parameter W ｕ Compared to W, a relatively large parameter update occurs. ｄ The relative magnitude of the parameter update in decreases with each spike event.

[0096] In contrast, the relative magnitude of parameter updates in the proposed method is constant. In other words, it is considered important for avoiding spike phenomena that the relative magnitude of parameter updates falls within a certain range.

[0097] (Hardware Configuration Example) Any of the devices described in this embodiment (learning device 100, inference device 200, information processing device) can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0098] That is, the device can be realized by executing a program corresponding to the processing performed by the device using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0099] Fig. 12 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 12 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B. The computer may further include one or more GPUs.

[0100] The program that realizes the processing on the computer is provided by a recording medium 1001, such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0101] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0102] (Summary, Effects, etc. of the Embodiments) As described above, the techniques described in the present embodiments make it possible to suppress the occurrence of spike phenomena in learning a neural network model.

[0103] That is, in this embodiment, by introducing a gate parameter into a model (e.g., a large-scale language model), it becomes possible to realize stable and high-speed model training. More specifically, by introducing a gate parameter and setting the standard deviation of the initial values ​​of the weight parameters (original parameter matrix) to a common value for all weight parameters (all parameter matrices), it becomes possible to realize stable and high-speed pre-training of a model.

[0104] The following additional notes are provided regarding the above-described embodiments.

[0105] <Supplementary Notes> (Supplementary Item 1) An information processing device that trains a neural network model, comprising: a memory; and at least one processor connected to the memory, wherein the processor initializes weight parameters included in the model based on a first index common to each layer, and the weight parameters are adjusted using gate parameters. (Supplementary Item 2) The information processing device according to Supplementary Item 1, wherein the processor initializes the gate parameters based on a second index that differs for each layer. (Supplementary Item 3) The information processing device according to Supplementary Item 1, wherein the processor initializes the weight parameters based on a distribution corresponding to the first index. (Supplementary Item 4) The information processing device according to Supplementary Item 1, wherein the processor adjusts the weight parameters by multiplying the weight parameters by the gate parameters, and further comprising a learning unit that trains the model including the adjusted weight parameters. (Supplementary Item 5) The information processing device according to Supplementary Item 4, wherein the processor integrates the weight parameters and the gate parameters when training of the model is completed. (Supplementary Item 6) An information processing device for training a neural network model, comprising: a memory; and at least one processor connected to the memory, wherein the processor initializes weight parameters included in the model based on a first index and initializes gate parameters for adjusting the weight parameters based on a second index, wherein the first index is an index related to parameter update of the model and the second index is an index related to gradient backpropagation of the model. (Supplementary Item 7) A training method executed by an information processing device for training a neural network model, comprising: an initialization step of initializing weight parameters included in the model based on a first index common to each layer, and the weight parameters are adjusted using gate parameters. (Supplementary Item 8) A non-transitory storage medium storing a program for causing a computer to function as each unit in the information processing device described in any one of Supplementary Items 1 to 6.

[0106] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0107] REFERENCE SIGNS LIST 100 Learning device 110 Conversion unit 120 Loss calculation unit 130 Parameter update unit 140 Parameter DB 150 Input unit 160 Parameter processing unit 170 Learning data DB 200 Inference device 210 Conversion unit 220 Text generation unit 230 Parameter DB 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. An information processing device that trains a neural network model, comprising a parameter processing unit that initializes weight parameters included in the model based on a first index common to each layer, and the weight parameters are adjusted using gate parameters.

2. The information processing device according to claim 1, wherein the parameter processing unit initializes the gate parameters based on a second index that differs for each layer.

3. The information processing device according to claim 1, wherein the parameter processing unit initializes the weighting parameters based on a distribution corresponding to the first index.

4. The information processing device according to claim 1, further comprising a learning unit that adjusts the weight parameters by multiplying the weight parameters by the gate parameters, and learns the model including the adjusted weight parameters.

5. The information processing device according to claim 4, wherein the learning section integrates the weight parameters and the gate parameters when learning of the model is completed.

6. An information processing device for training a neural network model, comprising a parameter processing unit that initializes weight parameters included in the model based on a first index and initializes gate parameters for adjusting the weight parameters based on a second index, wherein the first index is an index related to parameter update of the model, and the second index is an index related to gradient backpropagation of the model.

7. A learning method executed by an information processing device that trains a neural network model, comprising an initialization step of initializing weight parameters included in the model based on a first index common to each layer, and the weight parameters are adjusted using gate parameters.

8. A program for causing a computer to function as each unit in the information processing device according to any one of claims 1 to 6.

Images