Quantization method, quantization program, quantization neural network code generation method, and machine model training device

Abstract

The present invention fixes the of degradation of the calculation accuracy of machine models due to quantization as compared to conventional technologies. A quantization method (S1) for quantizing the calculation of a machine model using a neural network having a plurality of layers comprises: a setting step (S11) for setting a lower limit value and an upper limit value for limiting output values from at least one layer; a training step (S12) for training a machine model by performing a clamping process for clamping the output values within the range from the lower limit value to the upper limit value; and a quantization step (S13) for quantizing a range of output values within the range from the lower limit value to the upper limit value. (Representative Figure 2)

Description

Quantization method, quantization program, quantized neural network code generation method, and machine model learning device

[0001] The present disclosure relates to a quantization method for quantizing the operation of a machine model, a quantization program, a quantized neural network code generation method, and a machine model learning device.

[0002] Machine models (AI, artificial intelligence) are becoming commonplace. While machine models can be used in a variety of fields, they require numerous complex calculations to obtain more accurate results. These complex calculations require long calculation times using large amounts of memory and high-performance processors. Various solutions have been proposed to overcome these drawbacks, one of which is quantization.

[0003] Quantization is a method of handling parameters such as weights or biases optimized by a machine model, or output values ​​obtained during calculations performed by the machine model, by discretizing or reducing the number of bits under predetermined conditions. For example, Patent Document 1 discloses a quantization technique that analyzes what output values ​​are obtained at a predetermined stage of calculations performed by a machine model in the learning stage, and discretizes the numerical values ​​that the machine model can take within the maximum and minimum values ​​of the output value range. By learning and using a machine model that defines discretized numerical values ​​in this way, the amount of calculation in the analysis stage can be reduced.

[0004] Japanese Patent Application Publication No. 2019-032833

[0005] However, in the conventional techniques described above, the range of possible minimum and maximum values ​​of the calculated values ​​increases as the layer becomes deeper, so even if quantization is performed, the quantization error increases as the layer becomes deeper, and there is a risk of the calculation accuracy decreasing.

[0006] An object of one aspect of the present disclosure is to provide a quantization technique that can improve the reduction in the computational accuracy of a machine model due to quantization compared to conventional techniques.

[0007] In order to solve the above problems, a quantization method according to one aspect of the present disclosure is a quantization method for quantizing the calculations of a machine model using a neural network having multiple layers, and includes a setting step of setting lower and upper limits to limit the output value from at least one of the layers, a learning step of performing a clamping process on the output value to keep it within a range from the lower limit to the upper limit, thereby training the machine model, and a quantization step of quantizing the range of the output value within the range from the lower limit to the upper limit.

[0008] A quantization program according to one aspect of the present disclosure is a quantization program that quantizes the calculations of a machine model using a neural network having multiple layers, and causes a computer to execute a setting process that sets lower and upper limits to limit the output value from at least one of the layers, a learning process that performs a clamping process on the output value to keep it within a range from the lower limit to the upper limit, thereby training the machine model, and a quantization process that quantizes the range of the output value within the range from the lower limit to the upper limit.

[0009] A quantized neural network code generation method according to one aspect of the present disclosure includes the steps of: detecting a function capable of quantizing an output value from a code of a machine model using a neural network having multiple layers; and incorporating, after the detected function, a clamp function that performs a clamping process to keep the output value of the function within a range from a preset lower limit value to an upper limit value.

[0010] A machine model learning device according to one aspect of the present disclosure includes an upper and lower limit value acquisition unit that acquires set lower and upper limit values, a setting unit that sets the lower and upper limit values ​​acquired by the upper and lower limit value acquisition unit, multiple quantization regions in the range from the lower limit value to the upper limit value, and a representative value for each quantization region in a machine model, a learning unit that trains the machine model, and a quantization library acquisition unit that acquires the quantization library acquired by learning.

[0011] According to one aspect of the present disclosure, it is possible to improve the reduction in computational accuracy of a machine model due to quantization compared to conventional techniques.

[0012] 1 is a schematic diagram showing the configuration of a machine model using a neural network according to embodiment 1 of the present disclosure. FIG. 2 is a flowchart of a quantization method S1 executed by the machine model according to embodiment 1. FIG. 3 is a schematic diagram comparing the difference between the calculation process of the convolutional neural network according to embodiment 1 and the calculation process of a convolutional neural network of the prior art. FIG. 4 is a schematic diagram comparing the difference between the calculation result of the convolutional neural network of embodiment 1 and the calculation result of a convolutional neural network of the prior art in the learning stage. FIG. 5 is a schematic diagram showing the magnitude of an error due to quantization. FIG. 6 is a schematic diagram showing an example of quantizing a numerical value expressed as a floating-point number. FIG. 7 is a block diagram showing the configuration of a machine model learning device 2 according to embodiment 1. FIG. 8 is a flowchart showing the flow of a machine model learning method S2 according to embodiment 1. FIG. 9 is a schematic diagram showing the configuration of a machine model using a neural network according to embodiment 2 of the present disclosure. FIG. 10 is a schematic diagram showing the configuration of a convolutional neural network according to embodiment 3 of the present disclosure. FIG. 11 is a flowchart showing the flow of a quantization code generation method S3 according to embodiment 4 of the present disclosure. FIG. 12 is a schematic diagram showing the configuration of a machine model using a neural network according to the prior art.

[0013] [Embodiment 1] An embodiment of the present disclosure will be described in detail below. A quantization method according to embodiment 1 is a quantization method for quantizing the computation of a machine model using a neural network having multiple layers. Neural networks include convolutional neural networks and recurrent neural networks. A neural network is one configuration of a machine model and includes an input layer, multiple hidden layers, and an output layer. Each of these input layer, hidden layer, and output layer is also simply referred to as a "layer." Each layer has multiple channels (operating units), and the results of computations performed on the multiple channels are output to channels in the next layer. In this embodiment, a method for quantizing the results of computations performed on each layer will be described. Before describing the machine model using a neural network according to embodiment 1, the configuration of a machine model using a neural network in the prior art will be described with reference to FIG. 12. Hereinafter, a "machine model using a neural network" will also be simply referred to as a "machine model."

[0014] FIG. 12 is a schematic diagram showing the configuration of a machine model according to the prior art. The machine model is a neural network model having multiple layers. A neural network model generally has an input layer, a hidden layer, and an output layer, and further has multiple hidden layers. One hidden layer has multiple channels. Each channel is a computing unit that performs a predetermined calculation, and the calculation differs for each channel.

[0015] 12 shows the flow of computational processing in which the results of computations in layer A, which is one of the hidden layers, are passed to layer B, which is the next hidden layer. As shown in FIG. 12, the results of computations in each channel A0, A1..., An (n is an integer of 2 or greater) in layer A are multiplied by the respective weights WA0, WA1..., WAn, the sum is added, a bias is added, and the result is sent to channel B0 of the next layer B. Such computations are performed for each of channels B0, B1..., Bm (m is an integer of 2 or greater) in layer B. The weights and biases may be different for each computation directed to each channel B0, B1..., Bm.

[0016] Next, the flow of calculation processing in the machine model according to embodiment 1 will be described. Fig. 1 is a schematic diagram showing the configuration of the machine model according to embodiment 1 of the present disclosure. As shown in Fig. 1, the results calculated for each channel A0, A1 ..., An (n is an integer of 2 or more) of layer A are multiplied by the respective weights WA0, WA1 ..., WAn, the results are summed, and a bias is further added, similar to the prior art.

[0017] The machine model according to the first embodiment differs from the prior art in that the biased result is clamped and quantized, and the output is sent to channel B0. As with the prior art, the calculation of multiplying by a weight and adding a bias is performed for each of the channels B0, B1, ... Bm (m is an integer of 2 or greater) of layer B, but clamping and quantization do not necessarily have to be performed before the input of all of the channels B0, B1, ... Bm.

[0018] The clamping process will now be described. The clamping process is performed using a clamping function. The clamping function is a function that limits an input numerical value to within a predetermined numerical range. For example, assume that the predetermined numerical range is "greater than or equal to a and less than or equal to b." If the input numerical value c is smaller than the lower limit value a, the clamping process outputs a. If the input numerical value c is greater than the upper limit value b, the clamping process outputs b. If the input numerical value c is greater than the lower limit value a and less than or equal to the upper limit value b, the clamping process outputs c as is.

[0019] Next, quantization in the first embodiment will be described. In the first embodiment, quantization is a process of converting an input numerical value into a discretized numerical value within a certain numerical range. In the first embodiment, quantization is performed within the range of the lower limit value and the upper limit value used in the clamping process described above. Any quantization method can be used. As an example, quantization may be performed by dividing a predetermined numerical range (in the first embodiment, the numerical range from the lower limit value to the upper limit value) by, for example, 2 ｎThe value may be divided into eight regions (quantization regions), and the numerical value in each region may be converted into a representative value for that region. For example, when n=3, the predetermined numerical range may be divided into eight regions, and a representative value for each region may be determined in advance. The representative value may be determined in any manner. For example, the representative value may be the median, lower limit, or upper limit of the region. Furthermore, the width of each quantization region may not be equal. By this type of quantization, any numerical value in the predetermined numerical range is discretized into one of eight representative values.

[0020] 2 is a flowchart of a quantization method S1 executed by the machine model according to the first embodiment. The quantization method S1 includes a setting step S11, a learning step S12, and a quantization step S13. In step S11, a lower limit and an upper limit are set to limit the output value from at least one hidden layer of the neural network. These upper and lower limit values ​​are used in the clamping process described above. These upper and lower limit values ​​may be set in advance by a user or may be set by at least one processor based on predetermined conditions.

[0021] The upper and lower limits may be set based on, for example, the minimum and maximum values ​​of the input values ​​input to the hidden layer of the neural network or the output values ​​output from the hidden layer. For example, if an activation function is incorporated into the neural network, the upper and lower limits may be set based on at least one of the minimum and maximum values ​​of the output of the activation function. For example, if a sigmoid function is incorporated as the activation function, the range of values ​​that the sigmoid function can take is from -1 to +1, so the upper limit may be set to +1 and the lower limit to -1. This allows for more effective setting of the upper and lower limits.

[0022] Next, in step S12, the output values ​​from at least one layer are clamped to fit within a range from a lower limit to an upper limit, and the machine model is trained. Note that in this training stage, the output values ​​from the layers are clamped but not quantized, and the machine model is trained. The training in step S12 is also referred to as pre-training. This step S12 may be performed using a machine model training device including at least one processor and at least one memory. This training step may include an accumulation step of recording the output values ​​from the layers or the results of clamping the output values. Specifically, the output values ​​output from each layer or the results of the clamping process may be accumulated over multiple training stages. This is for determining in the next step S13 what numerical range the output values ​​or the results of the clamping process fall within.

[0023] Next, in step S13, the range of output values ​​is quantized within the range from the lower limit to the upper limit set in the setting step S11. That is, the range from the lower limit to the upper limit set in the setting step S11 is divided into a predetermined number of regions (quantization regions), and a representative value for each region is determined. Then, in the learning step S12, the output values ​​of the trained machine model are quantized. As described above, a machine model using a neural network that performs quantized numerical processing can improve the reduction in the computational accuracy of the machine model due to quantization compared to conventional techniques. A learning method for training a neural network machine model that performs quantized numerical processing will be described later. Note that, in the quantization step S13, the range of output values ​​accumulated in the learning step S12 may be quantized within the range from the lower limit to the upper limit. That is, an accumulation step for acquiring output values ​​may be provided in the learning step S12, and in the quantization step S13, the range from the minimum value to the maximum value of the accumulated output values ​​may be divided into a predetermined number of regions (quantization regions) within the set range from the lower limit to the upper limit, and a representative value for each region may be determined.

[0024] FIG. 3 is a schematic diagram comparing the difference in the computational processing of the convolutional neural network of embodiment 1 with that of a convolutional neural network of the prior art. As shown in 301 of FIG. 3, the convolutional neural network of embodiment 1 incorporates clamp functions 3A, 3B, and 3C indicated by "Clamp" after convolutional layers 1A, 1B, and 1C indicated by "Conv," respectively. On the other hand, the convolutional neural network of the prior art shown in FIG. 3B does not incorporate any clamp functions. Other than that, the configurations of the two are the same. Hereinafter, the processing performed by clamp function 3A will also be referred to as clamp processing 3A. The convolutional processing performed in the convolutional layer is the same as that of the prior art.

[0025] 4 is a schematic diagram comparing the difference in output values ​​(computation results) of the convolutional neural network of embodiment 1 and the output values ​​of a convolutional neural network of the prior art during the learning stage. More specifically, FIG. 4 compares the input values, output values ​​of convolutional channels 1A, 1B, and 1C, and the range of output values ​​from clamp functions 3A, 3B, and 3C between embodiment 1 and the prior art. FIG. 4 shows the possible ranges of the many output values ​​obtained during the learning stage.

[0026] In Figure 4, the numerical ranges of output values ​​at the positions indicated by the dotted double-headed arrows in Figure 3 are compared between the right side (conventional technology) and the left side (embodiment 1) of Figure 4. The input value range for embodiment 1 is indicated by reference numeral 10, and the input value range for the conventional technology is indicated by reference numeral 20. As shown in the figure, the numerical ranges of the input values ​​are the same. First, let us look at the output 13A after the clamping process 3A of the first layer. As shown in Figure 4, the output 11A from the convolutional layer 1A of embodiment 1 is limited to the output 13A by the clamping process 3A. Specifically, the minimum value 11Amin of the output 11A remains unchanged in the output 13A, but the maximum value 11Amax of the output 11A exceeds a preset upper limit, and therefore is limited to the upper limit value by the clamping process 3A in the output 13A. On the other hand, the output 21A of the conventional technology is not subjected to the clamping process, and therefore remains in the numerical range output from the convolutional layer 1A.

[0027] Next, let us look at the output 13B after the clamping process 3B in the second layer in embodiment 1. Because the lower limit 11Bmin and upper limit 11Bmax of the output value range 11B from the convolutional layer 1B exceed both the upper and lower limits, the range of the output 13B after the clamping process is limited to the range from the set lower limit to the upper limit. On the other hand, the output 21B of the prior art is not subjected to the clamping process, so it remains in the numerical range output from the convolutional layer 1B.

[0028] Note that there is an addition operation indicated by "Add" at the end of the second layer. Since no clamping process is performed after this, even in the first embodiment, the upper and lower limits are exceeded. However, since clamping process 3C is performed after the third convolutional layer 1C, the range of its output 13C is limited to within the set upper and lower limits.

[0029] Next, the quantization of output values ​​will be described. As mentioned above, quantization is performed within the range of set upper and lower limit values. On the left side of FIG. 4 (Embodiment 1), the range from the lower limit value to the upper limit value is divided into eight equal regions, and the boundaries between them are indicated by vertical lines. Quantization is performed by converting output values ​​that fall within each range into a representative value for that range. As mentioned above, the method for determining the representative value and the method for dividing the regions are arbitrary. The regions may be divided at uneven intervals corresponding to a logarithm or a histogram. Alternatively, for example, if the distribution is normal, the center may be divided finely and the areas near the upper and lower limits may be divided coarsely. In this way, quantization within the range of preset upper and lower limit values ​​can reduce quantization error.

[0030] FIG. 5 is a schematic diagram showing the magnitude of error due to quantization. When output values ​​from a computing unit are quantized, a range of values ​​is converted to a representative value of that range. FIG. 5 shows the case where output values ​​are quantized between the minimum and maximum values. The upper diagram in FIG. 5 shows the case where the range from the minimum to the maximum value is narrow, while the lower diagram shows the case where the range from the minimum to the maximum value is wide. As shown in the diagram, even when divided into the same number of regions, if the width of the range between the minimum and maximum values ​​increases, the range of values ​​represented by the representative value also becomes wider. This means that the quantization error increases.

[0031] In the prior art, there is a method of quantizing based on the maximum and minimum values ​​of output values ​​from the arithmetic units integrated during the learning stage. However, in the method of quantizing based on the maximum and minimum values ​​of output values ​​from the arithmetic units integrated during the learning stage, as shown in FIG. 5, as the range between the maximum and minimum values ​​increases, the width of one quantization region increases, and the quantization error upon quantization also increases. In contrast, in the first embodiment, quantization is performed between a predetermined lower limit and upper limit value, rather than within the range between the maximum and minimum values, thereby preventing an increase in quantization error. This is because, as described in the learning step S12 of FIG. 2, by performing a clamping process to fit within the range between the upper and lower limit values ​​and training the machine model, it is guaranteed that the error will not increase even if quantization is performed between the lower and upper limit values, rather than within the range between the maximum and minimum values.

[0032] In the example shown in FIG. 4 , the clamping process 3A limits the output 13A from 11Amin to a set upper limit. In this case, the minimum value of the output values ​​accumulated in the accumulation step included in the learning step is greater than the lower limit. In other words, the range from the minimum value to the maximum value is narrower than the range from the lower limit to the upper limit. In this case, if a method is used in which the range from the lower limit to the upper limit is divided into a predetermined number of parts and quantized to their representative values, the width of each quantization region may become large, resulting in a deterioration in accuracy. Therefore, in such a case, quantization may be performed using the minimum value of the output value instead of the lower limit. In other words, if the minimum value of the output values ​​accumulated during the learning step is greater than the lower limit, the quantization step may quantize the output values ​​within the range from the minimum value to the upper limit.

[0033] Conversely, there may be cases where the maximum value of the output values ​​accumulated in the accumulation step included in the learning step is smaller than the set upper limit value. In such cases, quantization may be performed using the maximum value of the output values ​​instead of the upper limit value. In other words, if the maximum value of the output values ​​accumulated at the stage of executing the learning step is smaller than the upper limit value, quantization may be performed within the range from the lower limit value to the maximum value of the output value in the quantization step. By adjusting the range of the upper and lower limits as described above, it is possible to reduce the possibility of an increase in quantization error.

[0034] The representation format of the output value is arbitrary. For example, it may be represented as a floating-point number consisting of an exponent and a mantissa, or as a fixed-point number. When represented as a floating-point number, the exponent may be quantized. Furthermore, if the exponent is shared by each variable, the calculation becomes more efficient. Furthermore, if the mantissa is an integer, the calculation becomes even more efficient.

[0035] FIG. 6 shows an example of quantizing an output value represented by a floating-point number. Reference numeral 601 in FIG. 6 is an example of a representation of a floating-point number. This format is a representation method consisting of a binary sign part (positive or negative), a mantissa part, and an exponent part. In this representation method, at least one of the exponent part and the mantissa part of the floating-point number may be quantized in the quantization step. Reference numeral 602 in FIG. 6 shows a case in which the mantissa part is quantized to an integer and the exponent part is quantized, and the exponent part is shared and used by multiple variables (output values) 1 to 3. This allows the parts to be quantized and the parts not to be quantized to be appropriately selected in consideration of computational resources.

[0036] An example of a method for quantizing the exponent will be described. For example, the exponent can be quantized by the following formula (1): Exponent = INT (log ２ (MAX(|Maximum value|, |Minimum value|))) + 1 ... (1)

[0037] For example, if -3.5≦data≦+2.5, the maximum value is +2.5 and the minimum value is -3.5, so the maximum absolute value is 3.5, and log ２ Since (3.5) = 1.807..., it is converted to an integer and the exponent is quantized to 2. If the number of quantization bits is 8, the integer part can represent -128 to +127, so it can be expressed as the following values. For example, if the exponent part is expressed as (exp), then (exp) = 2, so +2.5 is 2.5 x 128 / 2 (exp) = 2.5 x 128 / 2 ２ = 80, and -3.5 is 3.5 x 128 / 2 (exp) = 3.5 x 128 / 2 ２=-112. In other words, +2.5 can be expressed as an integer part of +80 and an exponent part of 2, and -3.5 can be expressed as an integer part of -112 and an exponent part of 2.

[0038] The above-described quantization reduces the amount of data and also eliminates or collectively calculates the exponent, thereby reducing the amount of calculation. Note that the present disclosure is not limited to the above-described representation format, and any representation format may be used as long as it can represent values ​​from the maximum to the minimum. Furthermore, while the above example describes a method for quantizing output values, at least one of the weights and biases may also be quantized. The above description describes an example in which clamping and quantization are performed in the hidden layer. However, clamping and quantization may also be performed in at least one of the input layer and the output layer.

[0039] The quantization program that causes a computer to execute the above-described quantization method is a quantization program that quantizes the calculations of a machine model using a neural network having multiple layers. The quantization program causes the computer to execute a setting process that sets lower and upper limits for limiting output values ​​from at least one layer, a learning process that performs a clamping process on the output values ​​to fit within a range from the lower limit to the upper limit, thereby training the machine model, and a quantization process that quantizes the range of output values ​​within the range from the lower limit to the upper limit. This embodiment also includes a computer-readable non-transitory recording medium that stores this quantization program. Note that, in the quantization process, the range of output values ​​accumulated during the learning process may be quantized within a range from the lower limit to the upper limit. That is, an accumulation process that acquires output values ​​may also be executed during the learning process, and in the quantization process, the range from the minimum to maximum of the accumulated output values ​​may be divided into a predetermined number of regions (quantization regions) within the set range from the lower limit to the upper limit, and a representative value for each region may be determined.

[0040] (Machine model learning method and learning device) Next, a machine model learning method S2 and a machine model learning device 2 will be described with reference to the drawings. Fig. 7 is a block diagram showing the configuration of the machine model learning device 2 according to the first embodiment.

[0041] As shown in FIG. 7 , the machine model learning device 2 includes a control device 20 and a communication unit 30. The control device 20 controls the entire machine model learning device 2. The control device 20 includes a main control unit 210 and a memory 220. The main control unit 210 includes an upper / lower limit value acquisition unit 211, a setting unit 212, a learning unit 213, and a quantization library acquisition unit 214. The main control unit 210 can be configured using, for example, at least one general-purpose processor such as an MPU (Micro Processing Unit) or a CPU (Central Processing Unit). The main control unit 210 may also include a dedicated processor configured, for example, by an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a PLD (Programmable Logic Device). The memory 220 may include multiple types of memory such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The memory 220 may also include built-in or external memory such as a hard disk drive (HDD), a solid state drive (SSD), etc. As an example, the main control unit 210 implements the functions of an upper and lower limit value acquisition unit 211, a setting unit 212, a learning unit 213, and a quantization library acquisition unit 214 by loading various control programs recorded in the ROM of the memory 220 into the RAM and executing them.

[0042] The machine model learning device 2 is connected wirelessly or wired to an external storage unit 50 via the communication unit 30 so as to be able to communicate information. A machine model 51 using a neural network is stored in the storage unit 50. The machine model 51 may be stored in the memory 220 of the machine model learning device 2, but here, a case where the machine model 51 is stored in the storage unit 50 will be described.

[0043] The upper and lower limit value acquisition unit 211 acquires the set lower limit value and upper limit value. The lower limit value and upper limit value may be set by a user. In this case, the upper and lower limit value acquisition unit 211 may acquire the lower limit value and upper limit value recorded in the memory 220 by the user. Alternatively, the upper and lower limit value acquisition unit 211 may acquire the lower limit value and upper limit value directly input by the user using an input / output interface (not shown) of the machine model learning device 2.

[0044] Furthermore, the upper and lower limit value acquiring unit 211 may acquire the upper and lower limit values ​​set by an upper and lower limit value setting unit (not shown). The upper and lower limit value setting unit may set the upper and lower limit values ​​based on, for example, the type of activation function included in the machine model 51, and record the set upper and lower limit values ​​in the memory 220. The upper and lower limit value acquiring unit 211 may acquire the upper and lower limit values ​​recorded in the memory 220. The upper and lower limit value acquiring unit 211 transmits the acquired upper and lower limit values ​​to the setting unit 212.

[0045] The setting unit 212 sets the lower limit value and upper limit value acquired by the upper / lower limit value acquisition unit 211 in the machine model 51. Specifically, the setting unit 212 writes the lower limit value and upper limit value to the clamp function program included in the machine model 51 via the communication unit 30. If the machine model 51 includes multiple clamp functions, the setting unit 212 sets the lower limit value and upper limit value for each of them.

[0046] The learning unit 213 trains the machine model 51 using the lower limit and upper limit set by the setting unit 212 via the communication unit 30. This stage is a preliminary learning for determining the minimum and maximum output values ​​of each layer of the machine model. In this preliminary learning, the minimum and maximum output values ​​of each layer of the machine model 51 are acquired by the quantization library acquisition unit 214.

[0047] The quantization library acquisition unit 214 acquires a quantization library. The quantization library acquisition unit 214 acquires the accumulated results, performs quantization, and performs calculations using the quantized values. A quantization library may be used to perform this quantization calculation. The quantization library acquisition unit 214 acquires a quantization library from the accumulated output, etc. Specifically, the quantization library acquisition unit 214 performs quantization, for example, by performing the calculation described in paragraph 0037, or conversely, returns the quantized value to a value that can be calculated. The quantization library may include at least some of the minimum and maximum values ​​of the output values ​​accumulated in the preliminary learning, the lower and upper limits for clamping, multiple quantization ranges ranging from the lower limit to the upper limit and their representative values, and numerical representation formats and their quantized values. The quantization library may further include quantized values ​​of quantized learned parameters such as weights and biases.

[0048] The setting unit 212 sets, in the machine model 51, a plurality of quantization regions in the range from a lower limit value to an upper limit value, and a representative value for each of the quantization regions, from the quantization library.

[0049] The learning unit 213 quantizes the output values ​​from each layer using a plurality of quantization regions in the range from the lower limit to the upper limit and their representative values, and re-learns the quantized values, thereby generating a trained quantized machine model.

[0050] (Machine Model Training Method) Next, a method for training the machine model 51 using a neural network will be described with reference to the drawings. Fig. 8 is a flowchart showing the flow of the machine model training method (machine model construction method) S2 according to the first embodiment. As shown in the figure, the training method S2 includes a setting step S21, a pre-learning step S22, a quantization step S23, and a re-learning step S24. The setting step S21, pre-learning step S22, and quantization step S23 are the same as the setting step S11, learning step S12, and quantization step S13 of the quantization method S1 described above, and therefore will not be described again.

[0051] The re-learning step S24 is a step in which the machine model is re-trained using the quantized library acquired by learning, specifically, a plurality of quantization regions in the range from the lower limit value to the upper limit value, and a representative value for each of the quantization regions. This step allows the machine model 51 to learn while undergoing quantization processing with a smaller quantization range, and a machine model 51 with less quantization error is constructed.

[0052] According to the above-described quantization method and machine model learning device, by setting lower and upper limits that restrict the output values ​​from the layers of the machine model, it is possible to improve the reduction in the computational accuracy of the machine model due to quantization more than with conventional techniques.

[0053] [Embodiment 2] Another embodiment of the present disclosure will be described below. For ease of explanation, the same reference numerals will be used to designate components having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0054] FIG. 9 is a schematic diagram showing the configuration of a machine model using a neural network according to the second embodiment. The diagram 901 in FIG. 9 shows a PixelShuffle function 5 after convolution functions 1A and 1B, followed by a clamp function 6. However, the PixelShuffle function 5 simply rearranges data and does not involve any numerical transformation. Therefore, performing clamping after such a function is not expected to improve computational accuracy. Therefore, clamping may be performed before functions included in the neural network that do not involve numerical transformation. In other words, clamping may be performed after functions included in the neural network that involve numerical transformation. Examples of functions that do not involve numerical transformation include, in addition to PixelShuffle, split (code that divides data) and cat (code that combines multiple data). Examples of functions that involve numerical transformation include, in addition to the Conv function (a function that performs convolution), the ConvTranspose function (a function that performs transpose convolution). Furthermore, calculations involving conversion of numerical values ​​include inner product calculations, matrix multiplication, nonlinear calculations (such as logarithm calculations), etc. When a calculation involving conversion of numerical values ​​is performed in a function, clamping may be performed after the function.

[0055] If there is a function whose calculation accuracy cannot be improved even by performing clamping (a function that does not involve numerical conversion), clamping 7 may be performed before that function, as shown in 902 in Fig. 9. In 902 in Fig. 9, clamping 7 and clamping 6 are performed before and after PixelShuffle 5, respectively. However, as shown in 903 in Fig. 9, if there is a function whose calculation accuracy cannot be improved, clamping 7 may be performed only before that function, and clamping may not be performed after that function.

[0056] According to the configurations shown in 902 and 903 above, by performing clamping processing at least before the calculation processing that cannot improve the calculation accuracy, it is possible to improve the decrease in the calculation accuracy of the machine model due to quantization more than in the prior art.

[0057] [Embodiment 3] Another embodiment of the present disclosure will be described below. For ease of explanation, the same reference numerals will be used to designate components having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0058] FIG. 10 is a schematic diagram showing the configuration of a convolutional neural network according to a third embodiment of the present disclosure. The convolutional neural network performs a process of convolutionally processing data from multiple channels and passing the data to one channel in the next layer. This process is performed for all channels in the next layer. In this case, the lower and upper limits for limiting output values ​​may be common to some or all of the multiple convolution operators in one layer. That is, in the step of setting the quantization method, common lower and upper limits may be set for some or all of the output values ​​in one layer. This reduces the amount of data and improves the reduction in computational accuracy of the machine model due to quantization compared to conventional techniques.

[0059] Furthermore, by setting common lower and upper limits for some or all of the output values ​​in one layer, the exponents of the values ​​quantized within that range may be made common for some or all of the output values ​​in one layer. By configuring in this way, when processing a large amount of data, only one shift amount is required when performing addition or multiplication, thereby reducing the amount of calculation.

[0060] [Embodiment 4] Another embodiment of the present disclosure will be described below. For convenience of explanation, the same reference numerals will be used to designate components having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0061] 11 is a flowchart showing the flow of a quantized neural network code generation method S3 (hereinafter referred to as "quantized code generation method S3") according to a fourth embodiment of the present disclosure. The quantized code generation method S3 is a method for incorporating a clamp function into the code of a machine model using an existing neural network. The quantized code generation method S3 may be executed using a quantized code generation device including at least one processor and at least one memory. A quantized code generation program is recorded in this memory.

[0062] 11, the quantization code generation method S3 includes steps S31 to S36. Step S31 is a step in which at least one processor reads the neural network code from the memory of the machine model. Note that if the step is before loading into the machine model, the code is read from the memory in which it is recorded, rather than from the memory of the machine model.

[0063] In step S32, at least one processor extracts function names from the read neural network code. Functions used in the neural network code are given predetermined code names and recorded in memory, so the code names are extracted.

[0064] Step S33 is a step in which at least one processor determines whether the extracted function is a function whose calculation accuracy will be improved by clamping when quantized. Whether or not a function's calculation accuracy will be improved by clamping is known in advance and recorded in memory, so the determination can be made based on this information. If it is determined in step S33 that the extracted function is a function whose calculation accuracy will be improved by clamping (step S33: YES), the process proceeds to step S34. If it is determined in step S33 that the extracted function is not a function whose calculation accuracy will be improved by clamping (step S33: NO), the process proceeds to step S35.

[0065] Step S34 is a step in which at least one processor adds a clamp function after the function in the neural network code. Note that upper and lower limit values ​​for the clamp function may be written when the clamp function is added. In this case, the upper and lower limit values ​​to be written may be recorded in memory in advance for each type of clamp function. The type of clamp function is determined, for example, by which function the clamp function is added after. The flow then proceeds to step S35.

[0066] In step S35, at least one processor determines whether all functions have been extracted. If it is determined in step S35 that all functions have been extracted (step S35: YES), the process proceeds to step S36. If it is not determined that all functions have been extracted (step S35: NO), the process returns to step S32.

[0067] In step S36, at least one processor writes the neural network code that has been processed in steps S32 to S35 into the memory of the machine model, and the flow ends. If the code has not yet been loaded into the machine model, the code is written into the memory where it was recorded, and the flow ends.

[0068] The quantization code generation method S3 described above can be described as follows. That is, the quantization neural network code generation method includes the steps of: detecting a function capable of quantizing output values ​​from machine model code using a neural network having multiple layers; and incorporating, after the detected function, a clamp function that performs clamping to keep the output value of the function within a range from a preset lower limit to an upper limit. A function capable of quantizing output values ​​is, for example, a function for which clamping may improve the accuracy of quantization. A function for which clamping may improve the accuracy of quantization is, for example, a function involving numerical transformation. As described in the second embodiment, examples of functions involving numerical transformation include the Conv function and the ConvTranspose function. Furthermore, functions that perform operations involving numerical transformation, such as inner product operations, matrix multiplication, and nonlinear operations (e.g., logarithm calculations), can also be considered functions involving numerical transformation.

[0069] The above method makes it easy to incorporate clamp functions into neural networks, reducing the amount of work required by the user. Furthermore, clamp functions can be added only where they will have the effect of improving calculation accuracy.

[0070] [Example of implementation by software] The functions of the machine model learning device 2 (hereinafter referred to as the "device") can be realized by a program that causes a computer to function as the device, and a program that causes a computer to function as each control block of the device (particularly each part included in the main control unit 210).

[0071] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The functions described in each of the above embodiments are realized by executing the program using the control device and storage device.

[0072] The program may be non-transitory and may be recorded on one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be supplied to the device via any wired or wireless transmission medium.

[0073] In addition, some or all of the functions of each of the control blocks can be realized by logic circuits. For example, integrated circuits in which logic circuits that function as each of the control blocks are formed are also included in the scope of the present disclosure. In addition, the functions of each of the control blocks can also be realized by, for example, a quantum computer.

[0074] The machine model learning device 2 according to each aspect of the present disclosure may be realized by a computer. In this case, the control program for the machine model learning device 2 that causes the computer to operate as each part (software element) of the machine model learning device 2 to realize the machine model learning device 2 on the computer, and the computer-readable recording medium on which the control program is recorded, also fall within the scope of the present disclosure.

[0075] (Summary) A quantization method according to aspect 1 of the present disclosure is a quantization method for quantizing the calculations of a machine model using a neural network having multiple layers, and includes a setting step of setting lower and upper limit values ​​that limit the output value from at least one of the layers, a learning step of performing a clamping process on the output value to keep it within a range from the lower limit value to the upper limit value, and training the machine model, and a quantization step of quantizing the range of the output value within the range from the lower limit value to the upper limit value.

[0076] According to the above configuration, the decrease in the accuracy of calculation of the machine model due to quantization can be improved more than in the prior art.

[0077] A quantization method according to a second aspect of the present disclosure is the quantization method according to the first aspect, wherein the quantization step quantizes at least one of an exponent part and a mantissa part of the floating-point number.

[0078] According to the above configuration, it is possible to reduce the amount of calculations for floating-point numbers while improving the decrease in the calculation accuracy of the machine model compared to the prior art.

[0079] A quantization method according to aspect 3 of the present disclosure is the same as that according to aspect 1 or 2, in which the learning step includes an accumulation step of recording the output values, and the quantization step quantizes the range of the output values ​​accumulated in the accumulation step within a range from the lower limit value to the upper limit value.

[0080] According to the above configuration, it is possible to reduce the amount of calculation and improve the decrease in calculation accuracy of the machine model compared to the prior art.

[0081] A quantization method according to aspect 4 of the present disclosure is, in aspect 3, such that if the minimum value of the output values ​​accumulated in the stage of executing the learning step is greater than the lower limit value, the output values ​​are quantized within a range from the minimum value to the upper limit value in the quantization step.

[0082] According to the above configuration, it is possible to improve the reduction in the calculation accuracy of the machine model more effectively than by quantizing within the range from the lower limit value to the upper limit value.

[0083] A quantization method according to aspect 5 of the present disclosure is, in aspect 3, such that if the maximum value of the output values ​​accumulated in the stage of executing the learning step is smaller than the upper limit value, the output values ​​are quantized within a range from the lower limit value to the maximum value of the output value in the quantization step.

[0084] According to the above configuration, it is possible to improve the reduction in the calculation accuracy of the machine model more effectively than by quantizing within the range from the lower limit value to the upper limit value.

[0085] A quantization method according to aspect 6 of the present disclosure is any one of aspects 1 to 5, wherein in the setting step, the lower limit value and the upper limit value are set based on the minimum and maximum values ​​of the input values ​​input to the layer or the output values ​​output from the layer.

[0086] According to the above configuration, the lower limit and the upper limit can be set efficiently.

[0087] A quantization method according to aspect 7 of the present disclosure is any one of aspects 1 to 6, wherein in the setting step, the lower limit value and the upper limit value are set to be common to some or all of the output values ​​in the one layer.

[0088] According to the above configuration, lower and upper limits can be set efficiently for a plurality of outputs.

[0089] A quantization method according to an eighth aspect of the present disclosure is any one of the first to seventh aspects, wherein the clamping process is performed before a function included in the neural network that does not involve conversion of numerical values.

[0090] According to the above configuration, it is possible to reduce the number of locations where clamping processing is performed as much as possible, while improving the decrease in the accuracy of calculation of the machine model compared to the prior art.

[0091] A quantized neural network code generation method according to a ninth aspect of the present disclosure includes the steps of: detecting a function capable of quantizing an output value from a code of a machine model using a neural network having multiple layers; and incorporating, after the detected function, a clamp function that performs a clamping process to keep the output value of the function within a range from a preset lower limit value to an upper limit value.

[0092] According to the above configuration, it is possible to mechanically generate a quantized neural network code that can improve the reduction in the calculation accuracy of the machine model due to quantization compared to the prior art.

[0093] A quantization program according to aspect 10 of the present disclosure is a quantization program that quantizes the calculations of a machine model using a neural network having multiple layers, and causes a computer to execute a setting process that sets lower and upper limits to limit the output value from at least one of the layers, a learning process that performs a clamping process on the output value to keep it within a range from the lower limit to the upper limit, thereby training the machine model, and a quantization process that quantizes within the range from the lower limit to the upper limit.

[0094] According to the above configuration, the decrease in the accuracy of calculation of the machine model due to quantization can be improved more than in the prior art.

[0095] A non-transitory recording medium according to aspect 11 of the present disclosure is a computer-readable non-transitory recording medium having the quantization program according to aspect 10 recorded thereon.

[0096] A machine model learning device according to aspect 12 of the present disclosure includes an upper and lower limit value acquisition unit that acquires set lower and upper limit values, a setting unit that sets the lower and upper limit values ​​acquired by the upper and lower limit value acquisition unit, a plurality of quantization regions in the range from the lower limit value to the upper limit value, and a representative value for each of the quantization regions in a machine model, a learning unit that trains the machine model, and a quantization library acquisition unit that acquires the quantization library acquired by learning.

[0097] According to the above configuration, it is possible to train a machine model that improves the reduction in the calculation accuracy of the machine model due to quantization more than in the prior art.

[0098] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present disclosure. Furthermore, new technical features can be formed by combining the technical means disclosed in each embodiment. (Cross-reference to related applications) This application is incorporated herein by reference to Japanese Patent Application No. 2023-131793, the entire contents of which are incorporated herein by reference.

[0099] 2...Machine model learning device 20...Control device 210...Main control unit 211...Upper and lower limit value acquisition unit 212...Setting unit 213...Learning unit 214...Quantization library acquisition unit 220...Memory 30...Communication unit 50...Storage unit 51...Machine model

Claims

1. A quantization method for quantizing the calculations of a machine model using a neural network having multiple layers, comprising: a setting step for setting a lower limit value and an upper limit value that limit an output value from at least one of the layers; a learning step for performing a clamping process on the output value to keep it within the range from the lower limit value to the upper limit value, thereby learning the machine model; and a quantization step for quantizing the range of the output value within the range from the lower limit value to the upper limit value.

2. The quantization method according to claim 1, wherein in said quantization step, at least one of an exponent part and a mantissa part of a floating-point number is quantized.

3. The quantization method according to claim 1 or 2, wherein the learning step includes an accumulation step of recording the output values, and the quantization step quantizes the range of the output values ​​accumulated in the accumulation step within a range from the lower limit value to the upper limit value.

4. A quantization method as described in claim 3, wherein, if the minimum value of the output values ​​accumulated in the stage of executing the learning step is greater than the lower limit value, the output values ​​are quantized within a range from the minimum value to the upper limit value in the quantization step.

5. A quantization method as described in claim 3, wherein, if the maximum value of the output values ​​accumulated in the stage of executing the learning step is smaller than the upper limit value, the quantization step quantizes within the range from the lower limit value to the maximum value of the output value.

6. A quantization method as described in claim 1 or 2, wherein in the setting step, the lower limit value and the upper limit value are set based on the minimum and maximum values ​​of the input values ​​input to the layer or the output values ​​output from the layer.

7. A quantization method according to claim 1 or 2, wherein in said setting step, a common lower limit value and upper limit value are set for a part or all of the output values ​​in said one layer.

8. The quantization method according to claim 1 or 2, wherein the clamping process is performed before any function included in the neural network that does not involve conversion of numerical values.

9. A method for generating quantized neural network code, comprising: a step of detecting a function capable of quantizing an output value from a code of a machine model using a neural network having multiple layers; and a step of incorporating, after the detected function, a clamp function that performs a clamp process to keep the output value of the function within a range from a preset lower limit value to an upper limit value.

10. A quantization program for quantizing the calculations of a machine model using a neural network having multiple layers, the quantization program causing a computer to execute the following: a setting process for setting a lower limit value and an upper limit value that limit an output value from at least one of the layers; a learning process for performing a clamping process on the output value to keep it within the range from the lower limit value to the upper limit value, thereby training the machine model; and a quantization process for quantizing within the range from the lower limit value to the upper limit value.

11. A non-transitory computer-readable recording medium having the quantization program according to claim 10 recorded thereon.

12. A machine model learning device comprising: an upper and lower limit value acquisition unit that acquires set lower and upper limits; a setting unit that sets in a machine model the lower and upper limits acquired by the upper and lower limit value acquisition unit, a plurality of quantization regions in the range from the lower limit value to the upper limit value, and a representative value for each quantization region; a learning unit that trains the machine model; and a quantization library acquisition unit that acquires a quantization library acquired by learning.

Images