Machine learning program, machine learning method, and information processing device

Abstract

This invention provides a machine learning program, a machine learning method, and an information processing device that reduce computational complexity while suppressing the degradation of the inference performance of machine learning models. [Solution] The process involves constraining some of the multiple parameters used in the inference of a machine learning model to have the same value, determining a common update value for each of the common parameters (a set of parameters with the same constraint value) based on multiple candidate update values, and then having a computer perform the process of training the machine learning model to maintain the same value for the common parameters using the updated value.

Description

[Technical Field]

[0001] The present invention relates to a machine learning program, a machine learning method, and an information processing device. [Background technology]

[0002] In recent years, generative AI (Artificial Intelligence) utilizing Large Language Models (LLMs) has been actively employed. Larger LLMs tend to perform better, and their operation requires computers with large memory capacities. While LLMs demonstrate high performance in various fields, their operational costs are high, making cost reduction through measures such as reducing memory requirements crucial. For example, one way to reduce memory usage is to reduce the number of parameters used in the LLM's calculations.

[0003] For example, techniques such as quantization, pruning, and distillation have been proposed to create LLMs that operate with less memory from existing machine learning models. Quantization is a technique that reduces memory usage by making the values ​​of each element of the matrix used by the LLM less coarse, thereby reducing the amount of information (number of bits) used to represent each element (each parameter) of the matrix. Pruning is a technique that reduces memory usage by deleting some of the elements of the matrix used by the LLM, thereby reducing the number of parameters used in the matrix. Distillation is a technique that reduces memory usage by approximating the matrix used by the LLM with a smaller matrix, thereby reducing the number of parameters used in the matrix.

[0004] Furthermore, the following techniques have been proposed as machine learning methods. For example, there is a technique that restricts each matrix in the tensor part of the model to a real symmetric matrix, diagonalizes the real symmetric matrix by spectral decomposition, applies the diagonal components to the model equation, and optimizes the model parameters using stochastic gradient descent for learning. There is also a technique that reduces the size of the neural network by removing nodes whose weight coefficient norm is below a threshold, calculates the gradient of the weight coefficients based on the objective function, and updates the weight coefficients based on the gradient. There is also a technique that replaces the weight matrix with the product of the first matrix and the second matrix, learns the model using a loss function associated with at least one term of the first and second matrices, and obtains the learned first and second matrices. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2019-101995 [Patent Document 2] Japanese Patent Publication No. 2020-8993 [Patent Document 3] U.S. Patent Application Publication No. 2019 / 0122108 [Overview of the project] [Problems that the invention aims to solve]

[0006] However, techniques such as quantization, pruning, and distillation generally result in lower inference performance compared to the original machine learning model or a machine learning model that uses fewer parameters than the original model but more parameters than when these techniques are applied. Generally, a model with a larger number of parameters and a larger number of bits used to represent each parameter tends to have higher inference performance, but this comes at the cost of increased memory consumption and computational complexity. Therefore, techniques that reduce memory consumption (and computational complexity) by reducing the number of parameters or the number of bits used to represent parameters, such as quantization, pruning, and distillation, suffer from performance degradation. In the case of quantization, the coarser granularity of the values ​​can lead to calculation errors compared to the original machine learning model, which is considered a factor in the decrease in inference performance. In the case of pruning, it can be difficult to determine whether a parameter is unnecessary or not, and there is a risk of deleting important parameters, which is considered a factor in the decrease in inference performance. In the case of distillation, the decrease in the rank of the matrix is ​​considered a factor in the decrease in inference performance.

[0007] Furthermore, in techniques that restrict each matrix in the tensor part to a real symmetric matrix and diagonalize the real symmetric matrix by spectral decomposition to apply the diagonal components to the model equation, all linear transformations are diagonalized by the same rotation matrix. Therefore, the mutual influence of each dimension in the matrix is ​​not considered, which may lead to a decrease in performance. Also, in techniques that reduce the size of the neural network by removing nodes whose weight coefficient norm is below a threshold and then calculate the gradient of the weight coefficients, reducing the size may lead to a decrease in performance. In addition, in techniques that replace the weight matrix with the product of the first and second matrices, it is difficult to reduce the overall computational complexity in processing the model.

[0008] The disclosed technology was developed in view of the above, and aims to provide a machine learning program, a machine learning method, and an information processing device that reduce computational complexity while suppressing a decline in the inference performance of machine learning models. [Means for solving the problem]

[0009] In one aspect of the machine learning program, machine learning method, and information processing apparatus disclosed in the present application, some of the plurality of parameters used for inference of a machine learning model are constrained to have the same value, and based on a plurality of candidate update values corresponding to each of the common parameters that are a set of parameters having the same value after the constraint, a common update value for the common parameters is determined, and the computer is caused to execute a process of training the machine learning model so as to maintain the values of the common parameters identical using the update value.

Effect of the Invention

[0010] In one aspect, the present invention can reduce the amount of calculation while suppressing a decrease in the inference performance of a machine learning model.

Brief Description of the Drawings

[0011] [Figure 1] FIG. 1 is a block diagram of an information processing apparatus according to an embodiment. [Figure 2] FIG. 2 is a diagram showing a change in the number of parameters when converted into a symmetric matrix. [Figure 3] FIG. 3 is a diagram showing an example of conversion into a square matrix when not a square matrix. [Figure 4] FIG. 4 is a diagram showing the relationship between the stability of the gradient and the change in the absolute value of the update value. [Figure 5] FIG. 5 is a diagram for explaining an overview of training of a machine learning model using a symmetric matrix. [Figure 6] FIG. 6 is a flowchart of training processing by the information processing apparatus according to the embodiment. [Figure 7] FIG. 7 is a diagram showing quantization of a matrix. [Figure 8] FIG. 8 is a diagram showing pruning of a matrix. [Figure 9] FIG. 9 is a diagram showing distillation of a matrix. [Figure 10] FIG. 10 is a diagram showing the implementation result of training for pre-training. [Figure 11]Figure 11 is a hardware configuration diagram of the information processing device. [Modes for carrying out the invention]

[0012] The following describes in detail, with reference to the drawings, embodiments of the machine learning program, machine learning method, and information processing apparatus disclosed in this application. However, the following embodiments do not limit the machine learning program, machine learning method, and information processing apparatus disclosed in this application. [Examples]

[0013] Figure 1 is a block diagram of an information processing device according to an embodiment. The information processing device 1 is a device that performs training of a machine learning model 111. The information processing device 1 has a storage unit 11, a matrix transformation unit 12, and a training execution unit 13.

[0014] The memory unit 11 contains a machine learning model 111 and training data 112. The machine learning model 111 is, for example, an LLM.

[0015] Training data 112 is data used to train the machine learning model 111. If the machine learning model 111 is an LLM, then training data 112 is, for example, text data.

[0016] The matrix transformation unit 12 initializes the machine learning model 111 before performing training. When initializing the machine learning model 111, the matrix transformation unit 12 transforms each matrix used in linear transformations during inference by the machine learning model 111, while imposing constraints so that they become symmetric matrices.

[0017] For example, the matrix transformation unit 12 extracts the original matrix W, which is the matrix before transformation included in the machine learning model 111. Then, the matrix transformation unit 12 converts the symmetric matrix Wsym to Wsym = (W + W t It is generated using the transformation formula ) / 2. Here, W t This is the transpose of the original matrix W.

[0018] Figure 2 shows the change in the number of parameters when the matrix is ​​transformed into a symmetric matrix. Here, we will explain the case of transforming the original matrix 121 into a symmetric matrix 122. Both the original matrix 121 and the symmetric matrix 122 are 6x6 matrices.

[0019] In the original matrix 121, each of the 36 elements represents a distinct parameter. In contrast, the symmetric matrix 122 has the same parameters in positions symmetric with respect to the diagonal elements. That is, the symmetric matrix 122 has 21 distinct parameters in the region indicated by 123, and the remaining 15 parameters outside of region 123 have the same values ​​as the parameters included in region 123. Thus, a symmetric matrix retains approximately half the number of parameters compared to the original matrix. Below, parameters that exhibit symmetry in a symmetric matrix will be referred to as "symmetric parameters."

[0020] The information processing device 1 can effectively reduce the number of parameters and achieve memory savings by making the matrix of the machine learning model 111 a symmetric matrix. In addition, a symmetric matrix can maintain the granularity of the values ​​of the original matrix and the rank of the matrix, so the machine learning model 111 can avoid a decrease in inference performance.

[0021] Here, the matrix transformation unit 12 is an example of a "constraint unit." Also, a pair of parameters positioned symmetrically across the diagonal elements of a symmetric matrix is ​​an example of a "common parameter." In this way, the matrix transformation unit 12 constrains that some of the multiple parameters used in the inference of the machine learning model 111 have the same value. Here, the constraint means initializing the machine learning model 111 by setting the constraint that the original matrix is ​​a symmetric matrix. Also, the original matrix is ​​an example of a "first matrix," and the transformed symmetric matrix is ​​an example of a "second matrix." That is, the matrix transformation unit 12 constrains that the multiple parameters are elements of the first matrix used in inference, and that the positions of elements with common values ​​can be calculated by formula, thereby making them common parameters. The matrix transformation unit 12 also transforms the first matrix into the second matrix, which is a symmetric matrix, and constrains that each pair of positions symmetrical with respect to the diagonal elements becomes a common parameter. If the original matrix already satisfies the constraint, this initialization process may be skipped.

[0022] If the original matrix is ​​not a square matrix, the matrix transformation unit 12 performs the transformation to a symmetric matrix as follows. For example, if the original matrix is ​​not a square matrix, the matrix transformation unit 12 performs padding to increase the number of rows or columns and transforms the matrix to a size that can be divided into square matrices. Then, the matrix transformation unit 12 divides the transformed matrix into square matrices and transforms each of them into a symmetric matrix.

[0023] Figure 3 shows an example of a transformation from a non-square matrix to a square matrix. Here, we will explain the case where the original matrix 151 is an 11x6 matrix. The matrix transformation unit 12 adds row 154 to the original matrix 151 by performing padding. Then, the matrix transformation unit 12 divides the padded matrix into square matrices 152 and 153. For example, the matrix transformation unit 12 may perform padding using row 154 that has the same value as column 155 of the square matrix 153, taking into account the transformed symmetric matrix 157.

[0024] Subsequently, the matrix transformation unit 12 transforms the square matrix 152 into a symmetric matrix 156 using the method described above. The matrix transformation unit 12 also transforms the square matrix 153 into a symmetric matrix 157 using the method described above. In this case, the matrix transformation unit 12 generates two 6x6 symmetric matrices, 156 and 157, for the original matrix 151.

[0025] In this way, if the first matrix is ​​not a square matrix, the matrix transformation unit 12 performs padding on the first matrix by increasing either the number of rows or the number of columns so that one of them is an integer multiple of the other. Then, the matrix transformation unit 12 decomposes the padded first matrix into square matrices whose number of rows and columns are the other number, and transforms each of the decomposed square matrices into a second matrix which is a symmetric matrix.

[0026] In this embodiment, padding was performed so that one of the number of rows or columns is an integer multiple of the other, but the padding method and the method of partitioning into square matrices are not limited to this. For example, the matrix transformation unit 12 may perform padding to increase the number of rows, columns, or both, transform the matrix to a size that can be partitioned into two or more square matrices of appropriate sizes, and then decompose it into square matrices of appropriate sizes.

[0027] Returning to Figure 1, let's continue the explanation. The learning execution unit 13 includes an optimizer 31 and a gradient calculation unit 32. It acquires the training data 112 from the storage unit 11. Then, the learning execution unit 13 trains the machine learning model 111, which has completed initialization, using the training data 112.

[0028] For example, the learning execution unit 13 performs training on the machine learning model 111, such as pre-training and post-training. For example, in pre-training, the learning execution unit 13 inputs a predetermined sentence and trains the machine learning model 111 to infer the next word or sentence following the input sentence. For example, in post-training, the learning execution unit 13 trains the machine learning model 111 to generate an expected output for a given input, depending on the purpose.

[0029] The training method of the machine learning model 111 by the learning execution unit 13 will be described in detail below. The learning execution unit 13 inputs the learning data 112 into the machine learning model 111. The gradient calculation unit 32 obtains the inference result output from the machine learning model 111. Then, the gradient calculation unit 32 calculates the gradient using the inference result and the correct data. After that, the gradient calculation unit 32 outputs the calculated gradient to the optimizer 31.

[0030] The optimizer 31 receives the input of the gradient from the gradient calculation unit 32. Next, the optimizer 31 calculates an update matrix for the symmetric matrix using the obtained gradient and the like.

[0031] For example, the optimizer 31 calculates an update matrix D for the original matrix W which is a symmetric matrix. Here, the symmetric parameter w i,j is the element in the i-th column and j-th row of the original matrix W. Also, d i,j is the element in the i-th column and j-th row of the update matrix D.

[0032] Here, for the symmetric parameters in the symmetric matrix, the optimizer 31 uses both the update value obtained from the gradient of the symmetric parameter and the update value obtained from the gradient of the other symmetric parameter at the symmetric position as candidates for the update value for the symmetric parameter. For example, for each of the symmetric parameters w i,j the optimizer 31 sets the update value d i,j and the update value d j,i as corresponding candidates for the update value.

[0033] Next, the optimizer 31 compares the two candidates for the update value corresponding to the symmetric parameter and selects the one with the larger update value. Then, the optimizer 31 performs the update using the selected update value for the symmetric parameter.

[0034] For example, when the two candidates for the update value corresponding to the symmetric parameter w i,j are the update value d i,j and the update value d j,i then |di,j | and |d j,i | and |d are compared and the larger one is selected. Here, |d i,j | and |d j,i | The larger of the two is d(d i,j d j,i ) is expressed as follows. The optimizer 31 then uses the symmetry parameter w i,j The updated value is w i,j -d(d i,j d j,i ) In this case, optimizer 31 has a symmetry parameter w j,i The updated value is w j,i -d(d i,j d j,i )=w i,j -d(d i,j d j,i ) is calculated as follows.

[0035] Here, we will explain why we choose the update value with the larger absolute value. There are certain optimizers that have a mechanism to reduce the absolute value of the update value for parameters whose gradient is unstable. Here, "unstable" means that the sign of the gradient changes with each parameter update.

[0036] Figure 4 shows the relationship between gradient stability and the change in the absolute value of the update value. Graphs 161 and 163 show the gradient on the vertical axis and time on the horizontal axis. That is, graphs 161 and 163 represent gradient stability. Graphs 162 and 164 show the absolute value of the update value on the vertical axis and time on the horizontal axis. Graph 162 shows the change in the update value by a specific optimizer when the gradient changes as shown in graph 161. Graph 164 also shows the change in the update value by a specific optimizer when the gradient changes as shown in graph 163.

[0037] In the case of Graph 161, the sign of the gradient remains unchanged and stable, so the absolute value of the updated value is less likely to become small, as in Graph 162. Conversely, in the case of Graph 163, the sign of the gradient changes and it is unstable, so the absolute value of the updated value becomes small, as in Graph 162.

[0038] In other words, with such a specific optimizer, a small update value corresponds to an unstable gradient. When using such a specific optimizer, prioritizing the updating of parameters with stable gradients is possible by selecting candidate update values ​​with larger absolute values. This can accelerate convergence and improve the performance of the training process. Therefore, optimizer 31 updates the symmetric parameters by selecting the candidate update value with the larger absolute value, as described above, assuming the use of such a specific optimizer.

[0039] In this embodiment, we assumed that optimizer 31 is a specific optimizer equipped with a mechanism to reduce the absolute value of the update value of parameters whose gradients are unstable, and selected the update value with the larger absolute value. However, the method of selecting the update value is not limited to this. It is preferable to select the update value based on indicators such as increasing the training speed or improving the inference performance, depending on the characteristics of optimizer 31. For example, optimizer 31 may calculate the update value using the average of the gradients for each of the two symmetric parameters.

[0040] The learning execution unit 13 uses the optimizer 31 and gradient calculation unit 32 as described above to update the parameters while maintaining the symmetric matrix, thereby performing pre-training and post-training on the machine learning model 111. This completes the training of the machine learning model 111, which performs inference using the symmetric matrix.

[0041] Figure 5 illustrates the overview of training a machine learning model using symmetric matrices. Here, we will use symmetric matrix 131 and matrix 141, which has a similar number of parameters to symmetric matrix 131, for explanation.

[0042] The gradient calculation unit 32 calculates a gradient group 132 for each parameter of the symmetric matrix 131 during training. The gradient calculation unit 32 also calculates each gradient included in the gradient group 132 separately for the same symmetric parameter located at a symmetric position. In the case of matrix 141, a gradient group 142 for each parameter is calculated during training. Each gradient included in the gradient group 142 is calculated for each parameter included in matrix 141.

[0043] Here, when training matrix 141, an optimizer that does not use symmetric matrices, such as optimizer 31, would use gradient 144 to update parameter 143, for example. In contrast, when optimizer 31 trains symmetric matrix 131, it can use gradient 134 corresponding to symmetric parameter 133 and gradient 135 corresponding to symmetric parameter 135 to update symmetric parameter 133, for example. That is, optimizer 31 can update symmetric parameter 133 using the better update value among the update values ​​calculated using gradient 134 and gradient 135, respectively. Therefore, by using symmetric matrices, optimizer 31 can improve training performance and suppress the degradation of inference performance compared to simply reducing the number of parameters.

[0044] In this way, the learning execution unit 13 determines a common update value for the common parameter based on multiple candidate update values ​​corresponding to each of the common parameters, and trains the machine learning model 111 to maintain the common parameter using the update value. For example, the learning execution unit 13 selects the candidate update value with the largest value among the multiple candidate update values ​​as the common update value.

[0045] Figure 6 is a flowchart of the training process by the information processing device according to the embodiment. Next, the flow of the training process by the information processing device 1 according to the embodiment will be explained with reference to Figure 6.

[0046] The matrix transformation unit 12 transforms the original matrix to satisfy the constraint that it is a symmetric matrix and initializes the machine learning model 111 stored in the memory unit 11 (step S1).

[0047] The learning execution unit 13 retrieves the learning data 112 stored in the memory unit 11 (step S2).

[0048] Next, the learning execution unit 13 inputs the training data 112 into the machine learning model 111, which performs inference using a symmetric matrix, and causes it to perform inference (step S3).

[0049] Next, the gradient calculation unit 32 compares the inference results output from the machine learning model 111 with the ground truth data and calculates the gradient (step S4).

[0050] The optimizer 31 updates the parameters of the machine learning model 111 while maintaining the constraints of the symmetric matrix using the gradient calculated by the gradient calculation unit 32 (step S5).

[0051] The learning execution unit 13 determines whether the training is complete or not (step S6). The learning execution unit 13 separately determines the completion of the pre-training and the subsequent post-training such as fine-tuning, but in the flow chart of Figure 11, the pre-training and the post-training after the completion of the pre-training are shown together as a single training.

[0052] If training is not complete (step S6: negative), the learning execution unit 13 returns to step S3. Conversely, if training is complete (step S6: positive), the learning execution unit 13 terminates the training process, considering the machine learning model 111 to be trained.

[0053] In this embodiment, the information processing device 1 reduces the number of parameters by converting the original matrix into a symmetric matrix. However, the converted matrix can be any other matrix as long as the positions of elements with common values ​​within the matrix can be calculated using a formula. For example, the information processing device 1 may convert the original matrix into a Hermitian matrix and train the converted machine learning model 111 to maintain the Hermitian matrix.

[0054] Here, we will explain the comparison between transforming the original matrix into a symmetric matrix and the cases of quantization, pruning, and distillation. Figure 7 shows the quantization of the matrix. Figure 8 shows the pruning of the matrix. Figure 9 shows the distillation of the matrix. Here, we will explain using the case of transforming the original matrix 121 shown in Figure 2.

[0055] As shown in Figure 7, the original matrix 121 is transformed into matrix 201 through quantization. In this case, each parameter of matrix 201 is generated by reducing the amount of information in each parameter of the original matrix 121, thereby coarsening the granularity. For example, if each parameter of the original matrix 121 is represented by 16 bits, then each parameter of matrix 201 will be represented by 8 bits.

[0056] When quantization is performed, the quantized matrix 201 has a coarser granularity than the original matrix 121, which may lead to calculation errors between the two matrices. Therefore, if the machine learning model 111 is quantized, the inference accuracy may decrease.

[0057] Furthermore, as shown in Figure 8, the original matrix 121 is transformed into matrix 202 through pruning. In this case, matrix 202 is generated by deleting some of the parameters of the original matrix 121. In Figure 8, the white areas in matrix 202 represent the deleted parameters.

[0058] When pruning is performed, parameters to be removed from the parameters of the original matrix 121 are selected in order to generate the quantized matrix 202, but there is a risk of selecting important parameters to be removed. Therefore, if pruning is performed on the machine learning model 111, the inference accuracy may decrease.

[0059] Furthermore, as shown in Figure 9, the original matrix 121 is transformed into matrix 203 by distillation. In this case, matrix 203 is generated by approximating the original matrix 121 with a smaller matrix. For example, if the original matrix 121 is a 6x6 matrix, then matrix 203 will be a 4x4 matrix.

[0060] When distillation is performed, the quantized matrix 201 has a lower rank compared to the original matrix 121. Therefore, if distillation is performed on the machine learning model 111, the inference accuracy may decrease.

[0061] Thus, while quantization, pruning, and distillation can all reduce memory usage by decreasing the number of parameters, they may also reduce inference accuracy. In contrast, the information processing device 1 according to this embodiment converts the original matrix 121 of the machine learning model 111 into a symmetric matrix 122, as shown in Figure 2. The information processing device 1 can maintain the granularity of the values ​​of each parameter, the number of parameters, and the rank of the matrix in the conversion to a symmetric matrix 122, thereby suppressing the decrease in the inference accuracy of the machine learning model 111. Furthermore, compared to pruning, there is no need to search for parameters to be deleted.

[0062] Here, we will explain the results of the pre-training exercise using Japanese data. The conditions for the pre-training exercise used here are as follows: The data used for training is Japanese text data publicly available on the internet. The architecture of the machine learning model 111 is Llama. The learning rate and settings used during training of the machine learning model 111 were 3e-4 to 0 with warmup and using a cosine scheduler. The optimizer is AdamW or the optimizer 31 based on AdamW for this embodiment. The global batch size is 512. The number of epochs is 5 or 10 (approximately 10B tokens). The hyperparameters of the optimizer are (0.95, 0.95).

[0063] Furthermore, the details of the machine learning model 111 using Lama as its architecture are as follows: The number of Transformer-blocks (decoders) is 24. The number of dimensions in the hidden layer is 2048 or 1427. The number of attention heads is 16. The numerical representation used for training is bfloat16. The upper limit of the number of tokens that the model can handle is 2048.

[0064] Under the above conditions, the machine learning model 111 was modified and trained for pre-training. Figure 10 shows the results of the pre-training. In Table 210, the 1 / 1 model is the case when a general transformer is used. The 1 / 2 model is the case when the size of the input vector given to the machine learning model 111 is multiplied by (1 / 2). Quantization is the case when quantization is applied to the machine learning model 111. Symmetric matrix conversion is the case when the matrix of the machine learning model 111 is converted to a symmetric matrix by the information processing device 1 according to this embodiment. Perplexity was used for performance evaluation. A lower value of Perplexity indicates better performance.

[0065] As shown in Table 210, the Perplexity, a performance evaluation value when using the information processing device 1, is closest to that of the 1 / 1 model, and the performance degradation is suppressed compared to other cases. Furthermore, the number of parameters when using the information processing device 1 is reduced compared to the 1 / 1 model and is also smaller than in other cases. In addition, the number of bits per parameter when using the information processing device 1 maintains the same value as the 1 / 1 model. Thus, the information processing device 1 according to this embodiment can reduce the number of parameters and suppress the decrease in inference accuracy compared to when the original matrix is ​​used as is.

[0066] As described above, the information processing device 1 according to this embodiment transforms the matrix of the machine learning model 111 into a symmetric matrix and trains the machine learning model 111 while maintaining the symmetric matrix. As a result, the information processing device 1 according to this embodiment can reduce memory consumption while suppressing a decrease in the inference performance of the machine learning model 111.

[0067] Furthermore, while the above explanation used the case where machine learning model 111 is an LLM as an example, the initialization method and parameter update method for machine learning model 111 described above are applicable even if machine learning model 111 is a neural network. For example, the initialization method and parameter update method for machine learning model 111 described above are also applicable to basic neural networks such as 2-layer or 3-layer networks.

[0068] (Hardware configuration) Figure 11 is a hardware configuration diagram of the information processing device. Next, with reference to Figure 11, an example of a hardware configuration for realizing each function of the information processing device 1 will be described.

[0069] As shown in Figure 11, the information processing device 1 includes, for example, a CPU (Central Processing Unit) 91, memory 92, a hard disk 93, and a network interface 94. The CPU 91 is connected to the memory 92, hard disk 93, and network interface 94 via a bus.

[0070] The network interface 94 is an interface for communication between the information processing device 1 and an external device. For example, the network interface 94 relays communication between the CPU 91 and an external device that receives input such as a machine learning model 111 and training data 112.

[0071] The hard disk 93 is an auxiliary storage device. The hard disk 93 implements the functions of the storage unit 11 as illustrated in Figure 1. The hard disk 93 also stores various programs, including programs for implementing the functions of the matrix transformation unit 12 and the learning execution unit 13 as illustrated in Figure 1.

[0072] Memory 92 is the main memory. Memory 92 can be, for example, DRAM (Dynamic Random Access Memory).

[0073] The CPU 91 reads various programs from the hard disk 93, loads them into memory 92, and executes them. In this way, the CPU 91 realizes the functions of the matrix transformation unit 12 and the learning execution unit 13 as illustrated in Figure 1. [Explanation of symbols]

[0074] 1. Information Processing Device 11 Storage section 12 Matrix Transformation Section 13 Learning Execution Unit 31 Optimizer 32 Gradient Calculation Unit 111 Machine Learning Models 112 training data

Claims

1. We constrain some of the multiple parameters used in the calculations of the machine learning model so that they have the same value. Based on a set of common parameters that have the same value, a common update value is determined for the common parameter, The machine learning model is trained to maintain the same values ​​for the common parameters using the updated values. A machine learning program characterized by having a computer perform the processing.

2. The machine learning program according to claim 1, characterized in that the constraint process includes a process of constraining some of the multiple parameters to have the same value by transforming the first matrix into a second matrix in which the multiple parameters are elements of the first matrix and the positions of elements having common values ​​can be calculated by a calculation formula.

3. The machine learning program according to claim 2, characterized in that the constraint process includes transforming the first matrix into the second matrix which is a symmetric matrix, and constraining each pair of positions symmetric with respect to the diagonal elements to become the common parameter.

4. The machine learning program according to claim 3, characterized in that the constraint process includes, if the first matrix is ​​not a square matrix, padding the first matrix by increasing the number of columns or rows so that either the number of rows or the number of columns is an integer multiple of the other, decomposing the padded first matrix into square matrices whose number of rows and columns are the other number, and converting each of the decomposed square matrices into the second matrix which is a symmetric matrix.

5. The machine learning program according to claim 1, characterized in that the process for determining the update value includes a process for selecting the candidate update value with the largest value among the multiple candidate update values ​​as the common update value.

6. Information processing device, We constrain some of the multiple parameters used in the calculations of the machine learning model so that they have the same value. Based on a set of common parameters that have the same value, a common update value is determined for the common parameter, The machine learning model is trained to maintain the same values ​​for the common parameters using the updated values. Let the computer perform the process. A machine learning method characterized by performing a process.

7. A constraint section that restricts some of the multiple parameters used in the calculations of a machine learning model to have the same value, A learning execution unit determines a common update value for each of the common parameters, which are a set of parameters having the same value under constraints, based on a plurality of candidate update values ​​corresponding to each of the common parameters, and trains the machine learning model to maintain the same value for the common parameters using the update value. An information processing device characterized by having the following features.

Images