Optimization apparatus, optimization method, and optimization program

Abstract

To allow even a small-scale edge computer to optimize a parameter by minimizing a loss function in a neural network, thereby saving memory and increasing a speed in the optimization processing.SOLUTION: In an optimizer OP that optimizes a parameter by minimizing a loss function corresponding to a prediction value and a ground truth value in a neural network, while limiting the number of quantization bits based on fixed-point numbers, logarithmic quantization is applied to a momentum mt-1 (mt is a momentum at a time t, and wt is a weight parameter), and a result obtained by multiplying a learning rate by a gradient of a loss function corresponding to the weight parameter is subtracted from a result of the logarithmic quantization, to output a result of the subtraction as a momentum mt, and, the momentum mt is added to a weight parameter wt-1, to be output as a weight parameter wt.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention belongs to the technical field of optimization devices, optimization methods, and optimization programs. More specifically, it belongs to the technical field of optimization devices and optimization methods that optimize each parameter used in a neural network by minimizing a loss function calculated using predicted and correct values ​​in the neural network, and optimization programs used in the optimization device. [Background technology]

[0002] In recent years, research and development on learning devices including neural networks have been actively pursued. Such learning devices require the consideration of an optimization device (so-called optimizer) that optimizes each parameter used in the neural network by minimizing the loss function calculated using the predicted and correct values ​​of the neural network. At this time, examples of optimizers that have been considered include, for example, the optimizer using the SGD (Stochastic Gradient Descent) method, which is one of the basic optimizers, as well as the optimizer using the Momentum method, the RMSProp (Root Mean Square) method, and the Adam (Adaptive-Moment-Estimation) method. Of these, the basic configuration of the optimizer using the Momentum method is disclosed in Non-Patent Document 1 below.

[0003] In this case, compared to the optimizer using the SGD method described above, the memory cost required (memory capacity and corresponding memory price; hereinafter simply referred to as "memory cost") is approximately twice as high for the optimizer using the momentum method, approximately three times as high for the optimizer using the RMSProp method, and approximately four times as high for the optimizer using the Adam method, according to the inventors' research. [Prior art documents]

Non-Patent Literature

[0004]

Non-Patent Literature 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Here, as a storage device in the hardware constituting the optimizer, a non-volatile memory is often used because it can store each parameter necessary even when the supply of power supply is stopped. And when using a non-volatile memory as a storage device, power reduction by reducing the number of accesses to the non-volatile memory (hereinafter simply referred to as “number of accesses”) is desired. Such power reduction is particularly required when attempting to implement the above optimizer with, for example, a hardware-wise small-scale dedicated computer, that is, a so-called edge computer that aims to reduce power by restricting generality and memory capacity.

[0006] On the other hand, it is known that a faster convergence speed of the optimizer's learning results contributes more to power reduction through the reduction in the number of accesses. Therefore, when comparing the various optimizers in terms of convergence speed, it is known that the optimizer using the momentum method is faster than the optimizer using the SGD method, and that the optimizers using the RMSProp method and the Adam method are even faster. Consequently, if an optimizer using the momentum method or the like is used to reduce the number of accesses by improving the convergence speed, the memory cost will end up being higher. Therefore, especially when aiming for low power for edge computers, it is necessary to achieve both a reduction in the number of accesses by improving the convergence speed (i.e., low power consumption) and a reduction in memory, including the price of non-volatile memory (i.e., low memory cost).

[0007] Therefore, the present invention has been made in view of the above requirements, and one example of the problems it solves is to provide an optimization device and optimization method, as well as an optimization program used in the optimization device, that can enable the optimization of each parameter used in a neural network by minimizing the loss function calculated using the predicted value and the correct value in the neural network, even in the case of the edge computer, and that can simultaneously reduce the memory required for the optimization process and reduce the number of accesses (and thereby reduce power consumption) by speeding up the convergence speed of learning. [Means for solving the problem]

[0008] To solve the above problems, the invention described in claim 1 optimizes the parameters of a neural network by minimizing a loss function calculated using predicted and correct values ​​in a neural network, while limiting the number of quantization bits using a fixed-point method with a predetermined number of quantization bits, wherein time is t and the momentum at time t is m t And the weighting parameter is wt When set as such, the momentum m at time t-1 t-1 Logarithmic quantization means for performing logarithmic quantization processing on the momentum m, multiplication means for multiplying the gradient of the loss function corresponding to the weight parameter in the neural network by the learning rate in the neural network, subtraction means for subtracting the multiplication result by the multiplication means from the result of the logarithmic quantization processing, and the subtraction result by the subtraction means is used as the momentum m at time t t Momentum output means for outputting, and the momentum m t Is added to the weight parameter w at time t-1 t-1 To obtain the weight parameter w at time t t Weight parameter output means for outputting, and is provided with.

[0009] In order to solve the above problems, the invention according to claim 3 uses a loss function calculated using a predicted value and a correct value in a neural network, and minimizes it by restricting the number of quantization bits to a fixed-point format with a preset number of quantization bits, thereby optimizing the parameters of the neural network as the optimization device. When the time is set as t and the momentum at the time t is set as m t And the weight parameter is set as w t When set as such, the computer included in the optimization device is configured to perform logarithmic quantization processing on the momentum m at time t-1 t-1 Logarithmic quantization means, multiplication means for multiplying the gradient of the loss function corresponding to the weight parameter in the neural network by the learning rate in the neural network, subtraction means for subtracting the multiplication result by the multiplication means from the result of the logarithmic quantization processing, and the subtraction result by the subtraction means is used as the momentum m at time t t Momentum output means for outputting, and the momentum m t Is added to the weight parameter w at time t-1 t-1 To obtain the weight parameter w at time t t Weight parameter output means for outputting, and functions as such.

[0010] To solve the above problems, the invention described in claim 4 is an optimization method performed in an optimization device that optimizes the parameters of a neural network by minimizing a loss function calculated using predicted and correct values ​​in a neural network, while limiting the number of quantization bits using a fixed-point method with a preset number of quantization bits, wherein the optimization device comprises a logarithmic quantization means, a multiplication means, a subtraction means, a momentum output means, and a weighted parameter output means, where time is t and the momentum at time t is m t And the weighting parameter is w t In this case, the logarithmic quantization means obtains the moment m at time t-1. t-1 Logarithmic quantization is applied to the result. Process The multiplication step involves multiplying the gradient of the loss function corresponding to the weighting parameters in the neural network by the learning rate in the neural network using the multiplication means, subtracting the result of the multiplication step from the result of the logarithmic quantization process using the subtraction means, and subtracting the result of the subtraction step from the momentum m at time t. t The momentum output output by the momentum output means Process And, the weighting parameter output means, the momentum m t The weighting parameter w at time t-1 t-1 Add to this, and the weighting parameter w at time t t This includes a weighted parameter output process that outputs the parameters as follows:

[0011] According to the invention described in any one of claims 1, 3, or 4, when optimizing the parameters of the neural network, the momentum m is minimized by minimizing the loss function calculated using the predicted value and the ground truth value in the neural network while limiting the number of quantization bits using a fixed-point method. t-1Logarithmic quantization is applied to the result, and the result of multiplying the gradient of the loss function corresponding to the weighting parameters by the learning rate is subtracted from the result of the above logarithmic quantization, and the momentum m at time t is obtained from the result of this subtraction. t It outputs as follows. Also, the output momentum m t The weighting parameter w t-1 Add to this the weighting parameter w at time t t It outputs as follows. Therefore, the limitations of the number of quantization bits in the fixed-point scheme and momentum m t-1 By combining this with logarithmic quantization, a dramatic improvement in the convergence speed of learning is achieved, leading to a reduction in the number of accesses and thus lower power consumption. This also enables memory reduction for the optimization device as a whole. Furthermore, by avoiding operations such as square root and division, which require a lot of memory, time, and processing elements, it becomes possible to implement the optimization device even on edge computers, for example, which have significant limitations regarding element area (number of elements), processing execution time, and driving power.

[0012] To solve the above problems, the invention described in claim 2 is an optimization apparatus described in claim 1, wherein the logarithmic quantization means is the momentum m t-1 The system comprises: a logarithmic transformation means that applies a base-2 logarithmic transformation to the result of the logarithmic transformation; a floor function transformation means that applies a floor function transformation to the result of the logarithmic transformation to truncate the decimal part; and a power calculation means that performs a power calculation with base 2 and the result of the floor function transformation as the exponent, and outputs the result of the logarithmic quantization transformation to the subtraction means.

[0013] According to the invention described in claim 2, in addition to the effects of the invention described in claim 1, momentum m t-1The logarithmic quantization process is performed sequentially by applying logarithmic scaling to the result of the logarithmic scaling, applying a floor function to the result of the logarithmic scaling, and then performing a power calculation using the result of the floor function processing as the exponent. Therefore, by applying logarithmic quantization in addition to the limitations of the number of quantization bits imposed by the fixed-point method, the parameters can be optimized efficiently, thereby improving the convergence speed of learning. Furthermore, this improvement in convergence speed reduces the number of accesses required until learning is complete, allowing for lower power consumption for optimization devices and the like. [Effects of the Invention]

[0014] According to the present invention, when optimizing the parameters of a neural network, the momentum m is minimized by minimizing the loss function calculated using the predicted and correct values ​​in the neural network while limiting the number of quantization bits using a fixed-point method. t-1 Logarithmic quantization is applied to the result, and the result of multiplying the gradient of the loss function corresponding to the weighting parameters by the learning rate is subtracted from the result of the above logarithmic quantization, and the momentum m at time t is obtained from the result of this subtraction. t It outputs as follows. Also, the output momentum m t The weighting parameter w t-1 Add to this the weighting parameter w at time t t Output as follows.

[0015] Therefore, the limitations of the number of quantization bits in the fixed-point scheme and momentum m t-1 By combining this with logarithmic quantization, a dramatic improvement in the convergence speed of learning is achieved, leading to a reduction in the number of accesses and thus lower power consumption. This also enables memory reduction for the optimization device as a whole. Furthermore, by avoiding operations such as square root and division, which require a lot of memory, time, and processing elements, it becomes possible to implement the optimization device even on edge computers, for example, which have significant limitations regarding element area (number of elements), processing execution time, and driving power.

[0016] Therefore, optimizing each parameter by minimizing the loss function calculated using the predicted and correct values ​​in the neural network becomes possible even on edge computers. [Brief explanation of the drawing]

[0017] [Figure 1] This is a block diagram illustrating the schematic configuration of the optimizer in the embodiment. [Figure 2] This is a flowchart showing the operation of the optimizer in the embodiment. [Figure 3] This diagram illustrates the effect of the optimization process performed by the optimizer in this embodiment. [Modes for carrying out the invention]

[0018] Next, embodiments for carrying out the present invention will be described with reference to Figures 1 and 2. The embodiments described below are examples in which the present invention is applied to an optimizer that optimizes each parameter for a neural network by minimizing a loss function calculated using predicted and correct values ​​in a learning device that includes a neural network consisting of an input layer, an intermediate layer (hidden layer), and an output layer, each composed of multiple nodes. In this case, the optimizer in the embodiment corresponds to an example of the "optimization device" of the present invention. Figure 1 is a block diagram showing the schematic configuration of the optimizer in the embodiment, and Figure 2 is a flowchart showing the operation of the optimizer.

[0019] When optimizing each of the above parameters by minimizing the loss function calculated using the above predicted value and the above correct value, the optimizer of the embodiment first calculates the momentum m at the timing (hereinafter simply referred to as "time") t-1 in which each quantization as a sequential process is performed, as shown in Figure 2. t-1 The gradient ▽wL(w) of the loss function L(w) corresponding to the weighting parameter w in the above neural network, the learning rate η corresponding to the neural network, and the weighting parameter w at time t-1.t-1 And are input respectively (Figure 2, step S1). Then the optimizer of the embodiment takes the above input momentum m t-1 , the above gradient ▽wL(w) and weighting parameter w t-1 The momentum m at the next time t corresponds to this. t and weighting parameter w t Outputs.

[0020] In other words, the optimizer OP of the embodiment is implemented by the edge computer, for example, which includes non-volatile memory as a storage device. And as shown in Figure 1, the optimizer OP is the momentum m t-1 A logarithm processing unit 1 receives the input, a floor function processing unit 2, a power calculation unit 3, a multiplication unit 4 receives the gradient ▽wL(w) and the learning rate η respectively, a sign inversion unit 5, an addition unit 6, and the weighting parameter w t-1 The system consists of an addition unit 7 to which the input is received, and a base unit 7. In the above configuration, the logarithmization unit 1 corresponds to an example of the "logarithmization means" of the present invention, the floor function processing unit 2 corresponds to an example of the "floor function processing means" of the present invention, and the power calculation unit 3 corresponds to an example of the "power calculation means" of the present invention. Furthermore, the logarithmization unit 1, the floor function processing unit 2, and the power calculation unit 3 constitute an example of the "logarithmic quantization means" of the present invention. In addition, the multiplication unit 4 corresponds to an example of the "multiplication means" of the present invention, the sign inversion unit 5 and the addition unit 6 correspond to an example of the "subtraction means" of the present invention, the addition unit 6 corresponds to an example of the "momentum output means" of the present invention, and the addition unit 7 corresponds to an example of the "weighted parameter output means" of the present invention.

[0021] Here, the above momentum m t-1 and the above weighting parameter w t-1 At the point when the data is input to the optimizer OP (see step S1 in Figure 2), it is quantized under a fixed-point quantization bit limit corresponding to the optimization process of each parameter by minimizing the loss function L(x) calculated using the above predicted value and the above correct value in the optimizer OP.

[0022] Then, the logarithmic unit 1 of the optimizer OP processes the input momentum m t-1 A logarithmic transformation with base "2" is applied to the result, and the result of this logarithmic transformation is output to the floor function processing unit 2 (Figure 2, step S2). Note that in Figure 1, momentum m t-1 The logarithmic processing in the logarithmic unit 1 for the moment m t-1 This is indicated as "in" in [ ].

[0023] Next, the floor function processing unit 2 of the optimizer OP performs a floor function process (floor function processing) on ​​the result of the logarithmic processing, truncating the decimal part, and outputs the result of this floor function processing to the power calculation unit 3 (Figure 2, step S3). In Figure 1, the processing performed by the floor function processing unit 2 on the result of the logarithmic processing is shown in brackets [ ] with the result of the logarithmic processing as "in".

[0024] Next, the power calculation unit 3 of the optimizer OP takes the result of the floor function processing as the exponent and performs a power calculation with base "2", and outputs the resulting power to the adder unit 6 (Figure 2, step S4). In Figure 1, the processing performed by the power calculation unit 3 on the result of the logarithmic processing is shown in brackets [ ] with the result of the logarithmic processing as "in".

[0025] Meanwhile, the multiplication unit 4 of the optimizer OP receives the gradient ▽wL(w) and the learning rate η as input (see step S1 in Figure 2). The multiplication unit 4 then multiplies the gradient ▽wL(w) by the learning rate η in parallel with the logarithmization process in the logarithmization unit 1 (see step S2 in Figure 2), the floor function processing in the floor function processing unit 2 (see step S3 in Figure 2), and the power calculation in the power calculation unit 3 (see step S4 in Figure 2), and outputs the result of the multiplication to the sign inversion unit 5 (see step S5 in Figure 2).

[0026] Subsequently, the sign inversion unit 5 inverts the sign of the multiplication result and outputs the sign inversion result to the adder unit 6 (Figure 2, step S6). The adder unit 6 then adds the sign inversion result to the exponent output from the exponentiation unit 3 (Figure 2, step S7). As a result of the sign inversion process in the sign inversion unit 5 and the addition process by the adder unit 6, the multiplication result from the multiplication unit 4 is subtracted from the exponent output from the exponentiation unit 3 (see Figure 2, step S4).

[0027] The addition result from the addition unit 6 is the moment m at time t. t It is output as and also output to the adder 7 (Figure 2, step S8). As a result, the adder 7 receives the input weighting parameter w t-1 The above moment m is the result of the addition by the addition unit 6. t Add the weight parameter w at time t. t Output as shown (Figure 2, step S9).

[0028] Subsequently, the optimizer OP determines whether or not to terminate its operation as an optimizer OP by determining, for example, whether or not there has been a termination operation by the operator controlling the optimizer OP (Figure 2, step S10). If the determination in step S10 is to terminate the operation (Figure 2, step S10: YES), the optimizer OP terminates the operation. On the other hand, if the determination in step S10 is to continue the operation (Figure 2, step S10: NO), the optimizer OP returns to step S1 and repeats the operation described above using the next input gradient ▽wL(w), etc. (see Figure 2, step S1).

[0029] The momentum m generated by the above series of processes as an optimizer OP t and weighting parameter w t As a result of optimization by the optimizer OP at time t, the data is re-inputted to the optimizer OP for further optimization at the next time t+1 (see Figure 2, step S10: case NO).

[0030] Here, the above series of processes as an optimizer OP can be expressed mathematically as follows: First, the momentum m t-1 Using the above gradient ▽wL(w) and the above learning rate η, the above momentum m t The process of generating and outputting (see steps S1 to S8 in Figure 2) can be expressed by the following equation (1).

number

[0031] On the other hand, the above momentum m t-1 The gradient ▽wL(w), the learning rate η, and the weighting parameter w are as described above. t-1 The above weighting parameter w is used t The process of generating and outputting (see steps S1 to S9 in Figure 2) can be expressed by the following equation (2).

number

[0032] Thus, the optimization process by the optimizer OP in this embodiment is (A) Input momentum m t-1 and weighting parameter w t-1 The limitations on the number of quantization bits performed by the fixed-point scheme, which are imposed in advance, (B) Logarithmic quantization processing (i.e., logarithmic quantization processing by logarithmization unit 1, floor function processing unit 2, and power calculation unit 3), This is equivalent to performing (A) and (B) together. t-1 Multiplication of the hyperparameter β (β × m t-1 This process replaces the previous one, and as a result, it is possible to achieve a more efficient optimization process than the conventional momentum method, which includes the multiplication of the hyperparameter β.

[0033] Note that the momentum m input to the optimizer OP t-1 and weighting parameter w t-1 According to research conducted by the inventors of this invention, aimed at achieving both memory saving and maintaining quantization accuracy, the limit value for the number of quantization bits in the fixed-point scheme described above is preferably 16 bits.

[0034] Furthermore, to specifically explain the memory reduction in the entire optimizer OP, firstly, compared with the optimization process using the RMSProp method and the optimization process using the Adam method, the optimizer OP of this embodiment does not need to store information regarding the square of the gradient ▽wL(w) in non-volatile memory, thus significantly reducing the memory capacity of the non-volatile memory itself compared to the optimization process using the RMSProp method, etc. Next, compared with the optimization process using the momentum method, the optimizer OP of this embodiment performs quantization (approximation) more times (one more time) than the optimization process using the momentum method, thereby reducing the amount of information that needs to be stored in non-volatile memory. As a result of these measures, the memory reduction in the entire optimizer OP of this embodiment is achieved. [Examples]

[0035] Next, we will explain the experimental results (simulation results) obtained by the inventors of the present invention to confirm the effect of the optimization process by the optimizer OP of the embodiment, using Figure 3. Figure 3 is a diagram illustrating the effect of the optimization process by the optimizer OP.

[0036] Specifically, as an example, the inventors of the present invention conducted experiments (simulations) on the relationship between the number of training iterations and the accuracy rate for three optimization processes: the optimization process in the optimizer OP of the embodiment, the optimization process using the conventional SGD method, and the optimization process using the conventional momentum method, under the conditions shown in Table 1 below. As shown in Figure 3, the results showed that the optimization process in the optimizer OP of the embodiment significantly improved the learning convergence speed, as seen from the accuracy rate, compared to both the optimization process using the conventional SGD method and the optimization process using the conventional momentum method. [Table 1]

[0037] On the other hand, as shown in Table 2 below, the characteristics (features) of each optimization process, including the memory cost, show that the hardware load, including the memory cost, is significantly lower for the optimizer OP of the embodiment compared to the optimizer using the momentum method, the optimizer using the RMSProp method, and the optimizer using the Adam method. [Table 2]

[0038] As shown in Figure 3, the optimizer OP of the embodiment is faster than the optimizer using the SGD method and the optimizer using the momentum method in terms of the learning convergence speed as seen from the accuracy rate. Therefore, due to the high convergence speed of the optimizer OP of the embodiment, the number of accesses to the non-volatile memory constituting the optimizer OP can be significantly reduced. As a result, the possibility of applying the optimizer OP to edge computers and the like is broadly expanded due to a significant reduction in power consumption, in parallel with a reduction in hardware load.

[0039] As explained above, according to the configuration of the optimizer OP of the embodiment, the loss function L(w) calculated using the predicted value and the correct value in the neural network is minimized while limiting the number of quantization bits using a fixed-point method, thereby optimizing the parameters of the neural network, momentum m t-1 The result of applying logarithmic quantization to the above neural network is subtracted from the result of the logarithmic quantization by multiplying the gradient ▽wL(w) of the loss function L(w) corresponding to the weight parameter w in the above neural network by the learning rate η, and the result of this subtraction is obtained as the momentum m at time t. t It outputs as follows. Also, the output momentum m t The weighting parameter w t-1 Add to this the weighting parameter w at time t t It outputs as follows. Therefore, the limitations of the number of quantization bits in the fixed-point scheme and momentum m t-1 By combining this with logarithmic quantization, a dramatic improvement in the convergence speed of learning is achieved, leading to a reduction in the number of accesses and thus lower power consumption. This also enables memory reduction for the optimizer OP as a whole. Furthermore, by avoiding operations such as square root and division, which require a lot of memory, time, and processing elements, it becomes possible to implement the optimizer OP even on edge computers, for example, where there are significant limitations on element area (number of elements), processing execution time, and driving power.

[0040] Therefore, optimization of each parameter by minimizing the loss function L(w) calculated using the above-mentioned predicted and correct values ​​in the neural network becomes possible even on the edge computer.

[0041] Also, Momentum m t-1 The logarithmic quantization process is performed sequentially on the result of the logarithmic quantization, followed by floor function processing on the result of the logarithmic quantization, and then exponentiation calculations using the result of the floor function processing as the exponent, thereby obtaining the result of logarithmic quantization. In addition to the limitation on the number of quantization bits in the fixed-point scheme, logarithmic quantization is performed to obtain the weighting parameter w tThis allows for efficient optimization of various parameters, thereby improving the learning convergence speed. Furthermore, this improved convergence speed reduces the number of accesses required to complete the learning process, leading to lower power consumption for the optimization device.

[0042] It is also possible to configure the optimizer OP of the embodiment in software. In this case, the program corresponding to the processing as the optimizer OP described above can be recorded on a recording medium such as an optical disc and read from it, or the program can be obtained from a network such as the Internet and read and executed by, for example, a general-purpose microcomputer. [Industrial applicability]

[0043] As described above, the present invention can be used in the field of learning devices including neural networks, and particularly remarkable effects can be obtained when applied to the field of optimization devices that optimize the parameters used in the neural network by minimizing the loss function L(w) calculated using the predicted and correct values ​​in the learning device. [Explanation of symbols]

[0044] 1. Logarithmic Unit 2. Floor function processing unit 3. Power calculation unit 4 Multiplication part 5. Sign inversion section 6, 7 Addition section OP Optimizer

Claims

1. In an optimization device that optimizes the parameters of a neural network by minimizing a loss function calculated using the predicted and correct values ​​of the neural network, while limiting the number of quantization bits using a fixed-point method with a predetermined number of quantization bits, Let time be t, and let the momentum at time t be m. ｔ And the weighting parameters are w ｔ In that case, The moment m at time t-1 ｔ－１ A logarithmic quantization means that applies logarithmic quantization processing to, A multiplication means for multiplying the gradient of the loss function corresponding to the weighting parameters in the neural network by the learning rate in the neural network, A subtraction means for subtracting the result of the multiplication by the multiplication means from the result of the logarithmic quantization process, The subtraction result by the subtraction means is the moment m at time t. ｔ A momentum output means that outputs as, The aforementioned momentum m ｔ The weighting parameter w at time t-1 ｔ－１ Add to this, and the weighting parameter w at time t ｔ A weighted parameter output means that outputs as, An optimization device characterized by comprising the following features.

2. In the optimization apparatus according to claim 1, The logarithmic quantization means is The aforementioned momentum m ｔ－１ A logarithmic transformation means that applies a base-2 logarithmic transformation to the given value, A floor function processing means that applies a floor function processing to the result of the logarithmic processing, which truncates the decimal part. A power calculation means that uses 2 as the base and the result of the floor function processing as the exponent, and outputs the result of the logarithmic quantization processing to the subtraction means, An optimization device characterized by comprising the following features.

3. In an optimization device that optimizes parameters of a neural network by minimizing a loss function calculated using predicted values and correct values in the neural network while restricting the number of quantization bits to a preset fixed-point format, with time being t and the momentum at time t being m ｔ and the weighted parameter being w ｔ when, the computer included in the optimization device The moment m at time t-1 ｔ－１ A logarithmic quantization means that applies logarithmic quantization processing to the above, A multiplication means for multiplying the gradient of the loss function corresponding to the weight parameters in the neural network by the learning rate in the neural network, A subtraction means that subtracts the result of the multiplication by the multiplication means from the result of the logarithmic quantization process. The subtraction result by the subtraction means is the moment m at time t. ｔ Momentum output means that outputs as, The aforementioned momentum m ｔ The weighting parameter w at time t-1 ｔ－１ Add to this, and the weighting parameter w at time t ｔ A weighted parameter output means that outputs as follows: An optimization program characterized by functioning as such.

4. An optimization method performed in an optimization device that optimizes the parameters of a neural network by minimizing a loss function calculated using predicted and correct values ​​in a neural network, while limiting the number of quantization bits using a fixed-point method with a predetermined number of quantization bits; The optimization device comprises a logarithmic quantization means, a multiplication means, a subtraction means, a momentum output means, and a weighting parameter output means. Let time be t, and let the momentum at time t be m. ｔ And the weighting parameters are w ｔ In that case, The logarithmic quantization means obtains the moment m at time t-1. ｔ－１ A logarithmic quantization process is performed on the result, The multiplication means provides a multiplication step of multiplying the gradient of the loss function corresponding to the weight parameters in the neural network by the learning rate in the neural network, A subtraction step is performed by subtracting the result of the logarithmic quantization process from the result of the multiplication step using the subtraction means. The subtraction result obtained by the subtraction step is the moment m at time t. ｔ The momentum output step is performed by the momentum output means, The weighting parameter output means outputs the momentum m ｔ The weighting parameter w at time t-1 ｔ－１ Add to this, and the weighting parameter w at time t ｔ A weighted parameter output process that outputs as follows: An optimization method characterized by including the following.

Images