Learning device, learning method, and program

Abstract

To provide a learning device capable of suppressing a deterioration in inference accuracy of a neural network even if Per-Tensor quantization is performed.SOLUTION: A learning device 10 for executing learning of a neural network 111 having one or more intermediate layers for respectively outputting a plurality of output values includes a deviation calculation part 20 for calculating a deviation of a plurality of output values in each of one or more intermediate layers obtained by applying input data included in learning data prepared in advance to the neural network 111, and a learning execution part 21 for learning a parameter of each layer of the neural network 111 so as to reduce the deviation.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to a learning device, a learning method, and a program for executing learning of a neural network. [Background technology]

[0002] Conventionally, when a neural network is incorporated into an AI accelerator of a system on a chip (SoC), various parameters of the neural network expressed in floating point are quantized to fixed point in order to reduce the computational load of the neural network.

[0003] For example, in a convolutional neural network (CNN) with a convolutional layer, each convolutional layer outputs the numerical values ​​of several channels as output values. The number of channels is also called the number of filters in the convolutional layer. In CNN quantization, a method that uses a common scaling coefficient for all channels (called per-tensor quantization) is often used. The scaling coefficient is a number that is multiplied to achieve a certain scale, i.e., to keep the scale within a specific range.

[0004] Incidentally, when performing per-tensor quantization, if there is a bias in the possible values ​​(i.e., output values) for each channel in multiple channels, the difference in values ​​after quantization disappears, and it is known that the recognition accuracy of the per-tensor quantized CNN deteriorates.

[0005] For this reason, for example, Patent Document 1 proposes a method for offsetting bias in output values ​​for each channel by performing scaling conversion in adjacent convolution layers of a CNN. Also, for example, Patent Document 2 proposes a method for eliminating bias in output values ​​for each channel by determining a scaling coefficient so as to minimize quantization error. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2020-119518 A [Patent Document 1] JP 2020-119528 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, the technology disclosed in Patent Document 1 has limitations on the configuration of applicable neural networks, such as inability to apply to neural networks including an activation layer that cannot be permuted with scaling transformations such as ReLU6 or Swish. In addition, the technology disclosed in Patent Document 2 does not change the output value of the convolution layer, so it is not possible to essentially reduce the bias in the output value for each channel, and there is a limit to suppressing the deterioration of the recognition accuracy of CNN.

[0008] In this way, when per-tensor quantization is performed on a neural network, the amount of calculation required for inference using the neural network can be reduced, but there is a problem in that the inference accuracy may deteriorate.

[0009] The present disclosure has been made in consideration of the above-mentioned circumstances, and aims to provide a learning device and the like that can suppress deterioration of the inference accuracy of a neural network even when per-tensor quantization is performed. [Means for solving the problem]

[0010] In order to achieve the above-mentioned object, a learning device according to one embodiment of the present disclosure is a learning device that performs learning of a neural network having one or more intermediate layers each outputting a plurality of output values, and includes a bias calculation unit that calculates a bias in a plurality of output values ​​in each of the one or more intermediate layers obtained by applying input data included in pre-prepared learning data to the neural network, and a learning execution unit that trains parameters of each layer of the neural network so as to reduce the bias.

[0011] This makes it possible to suppress degradation of the inference accuracy of the neural network even when per-tensor quantization is performed.

[0012] Furthermore, these general or specific aspects may be realized by an apparatus, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be realized by any combination of a system, a method, an integrated circuit, a computer program, and a recording medium. Effect of the Invention

[0013] The present disclosure makes it possible to provide a learning device or the like that can suppress deterioration of the inference accuracy of a neural network even when per-tensor quantization is performed. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram conceptually illustrating an example of a neural network of the present disclosure. [Figure 2A] FIG. 2A is an example of the output values ​​of channel 1 of the convolutional layer shown in FIG. 1. [Figure 2B] FIG. 2B is an example of an output value of channel 2 of the convolutional layer shown in FIG. 1. [Figure 2C] FIG. 2C is a diagram conceptually illustrating that there is no bias in the possible values ​​of the N channels of the convolutional layer shown in FIG. 1. [Figure 3A] FIG. 3A is a diagram conceptually illustrating per-tensor quantization of multiple output values ​​of the convolution layer shown in FIG. 1. [Figure 3B] FIG. 3B is a diagram showing output values ​​before and after quantization in the case where a plurality of output values ​​of channel 1 shown in FIG. 2A are uniformly quantized. [Figure 3C] FIG. 3C is a diagram showing output values ​​before and after quantization in the case where a plurality of output values ​​of channel 2 shown in FIG. 2B are uniformly quantized. [Figure 4A] FIG. 4A is another example of output values ​​of channel 2 of the convolutional layer shown in FIG. 1. [Figure 4B] FIG. 4B is a diagram conceptually illustrating that there is a bias in the possible values ​​of the N channels of the convolutional layer shown in FIG. [Figure 4C] FIG. 4C is a diagram showing output values ​​before and after quantization in the case where a plurality of output values ​​of channel 2 shown in FIG. 4A are uniformly quantized. [Diagram 5] FIG. 5 is a block diagram illustrating a functional configuration of the learning device according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a hardware configuration of a computer that realizes the functions of the learning device according to the embodiment by software. [Figure 7] FIG. 7 is a diagram conceptually showing the bias calculated by the bias calculation unit according to the embodiment. [Figure 8] FIG. 8 is a diagram for explaining an example of a method for specifically calculating the bias Li of the i-th convolutional layer according to the embodiment. [Figure 9] FIG. 9 is a diagram for explaining another example of a method for specifically calculating the bias Li of the i-th convolutional layer according to the embodiment. [Figure 10A] FIG. 10A is a diagram conceptually illustrating an example of a function that is added as a bias according to the embodiment. [Figure 10B] FIG. 10B is a diagram conceptually illustrating an example of a function that is added as a bias according to the embodiment. [Figure 10C] FIG. 10C is a diagram conceptually illustrating an example of a function that is added as a bias according to the embodiment. [Figure 11] FIG. 11 is a flowchart showing an outline of the operation of the learning device in the embodiment. [Figure 12] FIG. 12 is a flowchart showing an example of detailed operation of the learning device according to the embodiment. [Figure 13] FIG. 13 is a flowchart showing an example of a specific operation of step S12 in FIG. [Figure 14A] FIG. 14A is a diagram conceptually illustrating an example of the configuration of an intermediate layer according to an embodiment. [Figure 14B]FIG. 14B is a diagram conceptually illustrating a configuration example of the intermediate layer according to the embodiment. [Figure 14C] FIG. 14C is a diagram conceptually illustrating a configuration example of the intermediate layer according to the embodiment. [Figure 14D] FIG. 14D is a diagram conceptually illustrating an example of the configuration of the intermediate layer according to the embodiment. [Figure 14E] FIG. 14E is a diagram conceptually illustrating an example of the configuration of the intermediate layer according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the drawings. Note that each of the embodiments described below shows a specific example of the present disclosure. The numerical values, shapes, materials, standards, components, the arrangement and connection forms of the components, steps, and the order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure. In addition, among the components in the following embodiments, components that are not described in the independent claims showing the highest concept of the present disclosure are described as optional components. In addition, each figure is not necessarily a strict illustration. In each figure, substantially the same configuration is given the same symbol, and duplicated explanations may be omitted or simplified.

[0016] (Embodiment) First, we explain that when per-tensor quantizing a neural network having an intermediate layer that outputs multiple output values, if there is a bias in the multiple output values ​​output by the intermediate layer, the inference accuracy of the neural network after quantization will decrease.

[0017] FIG. 1 is a diagram conceptually illustrating an example of a neural network of the present disclosure.

[0018] FIG. 1 shows a CNN including a convolution layer as an example of a neural network of the present disclosure. The convolution layer of the CNN shown in FIG. 1 is an example of an intermediate layer of the neural network of the present disclosure. In the convolution layer shown in FIG. 1, an example is shown in which multiple output values ​​of the previous convolution layer are input and N channel output values ​​are output as multiple output values. As described above, the number of filters in the convolution layer is also called the number of channels, so the number of filters in the convolution layer shown in FIG. 1 is N.

[0019] Fig. 2A is an example of an output value of channel 1 of the convolutional layer shown in Fig. 1. Fig. 2B is an example of an output value of channel 2 of the convolutional layer shown in Fig. 1.

[0020] Fig. 2A shows an example of a plurality of output values ​​of channel 1, which is one of the N channels. Similarly, Fig. 2B shows an example of a plurality of output values ​​of channel 2, which is one of the N channels. Since an image is input to the neural network shown in Fig. 1, the plurality of output values ​​of channel 2 are shown as two-dimensional values.

[0021] Fig. 2C is a diagram conceptually showing that there is no bias in the possible values ​​of the N channels of the convolutional layer shown in Fig. 1. The horizontal axis indicates the channel, and the vertical axis indicates the possible values ​​of the channel. In Fig. 2C, the N channels are shown as channel 1 to channel N, and the maximum value of the multiple output values ​​of each of channel 1 to channel N is shown as the possible values ​​of channel 1 to channel N.

[0022] FIG. 3A is a diagram conceptually illustrating per-tensor quantization of multiple output values ​​of the convolution layer shown in FIG. 1.

[0023] FIG. 3A shows that the possible values ​​of channel N of the convolutional layer shown in FIG. 1, which are expressed as floating point values ​​from -2.5 to 2.5, are quantized using the same formula (with a uniform quantization width) to fit within a specific range expressed as a fixed point, such as an 8-bit integer (0 to 254).

[0024] Fig. 3B is a diagram showing output values ​​before and after quantization when a plurality of output values ​​of channel 1 shown in Fig. 2A are uniformly quantized. Fig. 3C is a diagram showing output values ​​before and after quantization when a plurality of output values ​​of channel 2 shown in Fig. 2B are uniformly quantized. More specifically, by quantizing each of the plurality of output values ​​before quantization shown in (a) of Fig. 3B and Fig. 3C using the same equation, a plurality of output values ​​after quantization shown in (b) of Fig. 3B and Fig. 3C can be obtained.

[0025] As shown in Figure 2C, if there is no bias in the possible values ​​of the N channels of the convolutional layer, the differences between the multiple output values ​​after quantization will not be lost, as shown in Figures 3B and 3C (b). Therefore, the recognition accuracy of the CNN after quantization is less likely to deteriorate.

[0026] Next, we explain the case where there is a bias in the possible values ​​of the N channels in the convolutional layer.

[0027] FIG. 4A is another example of the output value of channel 2 of the convolutional layer shown in FIG. 1. FIG. 4B is a diagram conceptually showing that there is a bias in the possible values ​​of the N channels of the convolutional layer shown in FIG. 1. FIG. 4A is another example of the output value of channel 2 shown in FIG. 2B, and it can be seen that the maximum value is 0.025, which is smaller than the maximum value of the output value of channel 1 shown in FIG. 2A. In this case, as shown in FIG. 4B, the possible values ​​of channel 2 are smaller than the other channels 1, 3 to N, and it can be seen that there is a bias in the possible values ​​of the N channels of the convolutional layer shown in FIG. 1.

[0028] Fig. 4C is a diagram showing output values ​​before and after quantization when the multiple output values ​​of channel 2 shown in Fig. 4A are uniformly quantized. Fig. 4C shows an example in which the multiple output values ​​before quantization shown in Fig. 4C (a) are quantized using the same formula as Fig. 3B and Fig. 3C to obtain multiple output values ​​after quantization shown in Fig. 4C (b). Note that the multiple output values ​​shown in Fig. 4C (a) correspond to the multiple output values ​​shown in Fig. 4A.

[0029] In the multiple output values ​​after quantization shown in (b) of Fig. 4C, the differences are crushed and no longer visible, so if the output values ​​of such channel 2 are quantized, the recognition accuracy of the quantized CNN is likely to deteriorate. In other words, if there is a bias in the possible values ​​of the N channels of the convolutional layer as shown in (b) of Fig. 4C, the differences between the multiple output values ​​after quantization are crushed as shown in (b) of Fig. 4C, so the recognition accuracy of the quantized CNN is likely to deteriorate.

[0030] Next, the learning device and the like according to the present embodiment will be described.

[0031] [1. Learning device 10] The learning device 10 according to this embodiment will be described below.

[0032] The learning device 10 is realized by a computer or the like, and is a device for executing learning of a neural network having one or more intermediate layers, each of which outputs a plurality of output values.

[0033] FIG. 5 is a block diagram showing a functional configuration of learning device 10 according to the present embodiment.

[0034] 5, the learning device 10 includes an acquisition unit 1 and a learning unit 2. Note that, in the learning device 10, the acquisition unit 1 is not essential.

[0035] [1-1. Hardware configuration] Before describing the functional configuration of learning device 10 according to the present embodiment, an example of the hardware configuration of learning device 10 according to the present embodiment will be described with reference to FIG.

[0036] FIG. 6 is a diagram showing an example of the hardware configuration of a computer 1000 that realizes the functions of the learning device 10 according to the present embodiment by software.

[0037] 6, the computer 1000 is a computer including an input device 1001, an output device 1002, a CPU 1003, an internal storage 1004, a RAM 1005, a reading device 1007, a transmission / reception device 1008, and a bus 1009. The input device 1001, the output device 1002, the CPU 1003, the internal storage 1004, the RAM 1005, the reading device 1007, and the transmission / reception device 1008 are connected by the bus 1009.

[0038] The input device 1001 is a user interface device such as an input button, a touch pad, a touch panel display, etc., and accepts user operations. The input device 1001 may be configured to accept voice operations and remote operations using a remote control or the like in addition to accepting touch operations by the user.

[0039] The output device 1002 is used in combination with the input device 1001, and is configured with a touch pad or a touch panel display, etc., to notify the user of information that should be made known to the user.

[0040] The built-in storage 1004 is a flash memory or the like. The built-in storage 1004 may previously store at least one of a program for implementing the functions of the learning device 10 and an application that utilizes the functional configuration of the learning device 10. The built-in storage 1004 may also store the acquired neural network 111, the acquired learning data, a first loss function used in the learning unit 2, a calculation formula or procedure for calculating bias, parameters updated by learning, and the like.

[0041] The RAM 1005 is a random access memory, and is used to store data and the like when a program or application is executed.

[0042] The reading device 1007 reads information from a recording medium such as a USB (Universal Serial Bus) memory. The reading device 1007 reads the above-mentioned programs and applications from a recording medium on which the programs and applications are recorded, and stores the programs and applications in the built-in storage 1004.

[0043] The transmitting / receiving device 1008 is a communication circuit for performing wireless or wired communication. The transmitting / receiving device 1008 may communicate with, for example, a server device connected to a network, download the above-mentioned programs and applications from the server device, and store them in the built-in storage 1004.

[0044] The CPU 1003 is a central processing unit that copies programs and applications stored in the internal storage 1004 to the RAM 1005, and sequentially reads out instructions included in the programs and applications from the RAM 1005 and executes them.

[0045] Next, each functional configuration of the learning device 10 according to the present embodiment will be described with reference to FIG.

[0046] [1-2. Acquisition part 1] The acquisition unit 1 acquires the neural network 111 to be learned by the learning unit 2, and stores it in a storage unit such as the internal storage 1004, for example.

[0047] Here, the neural network 111 is a neural network having one or more intermediate layers each outputting a plurality of output values. The neural network 111 is, for example, a convolutional neural network, but is not limited thereto. The neural network may be a neural network including a convolutional layer, a depthwise convolution layer, or a fully connected layer as one or more intermediate layers each outputting a plurality of output values. Note that, when the intermediate layer is a fully connected layer, it can be treated in the same way as a convolutional layer, etc., by replacing the convolutional layer with a fully connected layer, the channel with a node, and the output value of the channel with the output value of the node.

[0048] Moreover, the acquisition unit 1 acquires learning data prepared in advance, and stores the data in a storage unit such as the built-in storage 1004.

[0049] [1-3. Learning section 2] [1-3-1. Bias calculation unit 20] The bias calculation unit 20 calculates bias in a plurality of output values ​​in each of one or more intermediate layers obtained by applying input data included in the learning data prepared in advance to the neural network 111.

[0050] For example, when the neural network 111 is a CNN, the bias calculation unit 20 calculates the bias of a plurality of output values ​​output by a convolution layer serving as one or more intermediate layers, that is, the output values ​​of several channels.

[0051] 7 is a diagram conceptually illustrating the bias calculated by bias calculation unit 20 according to the present embodiment. In FIG. 7, when the number of channels in a convolutional layer serving as an i-th hidden layer is eight, the distribution of possible values ​​(actual output values) of the eight channels of the convolutional layer and the calculated bias L i Here, the distribution of possible values ​​of the eight channels of the convolutional layer is the distribution of the maximum value among the multiple output values ​​actually output by each of the eight channels, and hereinafter the distribution of possible values ​​is also referred to as the distribution of actual output values.

[0052] As shown in the example of Fig. 7, the distribution of actual output values ​​is biased. In the example of Fig. 7, an ideal distribution without bias is a distribution (uniform distribution) in which the output values ​​of channel 4 are evenly arranged among channels 1 to 8.

[0053] In this embodiment, the bias calculation unit 20 calculates the bias L as the area of ​​the difference between the distribution of the actual output values ​​and the ideal distribution, as shown in FIG. i Calculate.

[0054] More specifically, the bias calculation unit 20 calculates the bias in the intermediate layer by calculating the difference between the distribution of multiple output values ​​of the intermediate layer and a uniform distribution in which the maximum values ​​of the multiple output values ​​are uniformly arranged.

[0055] In addition, bias calculation unit 20 may calculate the bias of multiple output values ​​in each of all intermediate layers that neural network 111 has, or may calculate the bias of multiple output values ​​in each of some of all intermediate layers that neural network 111 has.

[0056] That is, when the bias calculation unit 20 targets all of one or more intermediate layers in the neural network 111 for calculation, it only has to calculate the area of ​​the difference between the distribution of actual output values ​​in each of the one or more intermediate layers and the ideal distribution as the bias for each intermediate layer. Also, when the bias calculation unit 20 targets a portion of the one or more intermediate layers for calculation, it only has to calculate the area of ​​the difference between the distribution of actual output values ​​in each of the portion of intermediate layers and the ideal distribution as the bias for each intermediate layer.

[0057] Below, using Figure 8, we consider the bias L of the i-th convolutional layer. i An example of a specific method for calculating the above will be described.

[0058] FIG. 8 shows the bias L of the i-th convolutional layer according to this embodiment. i 8 is a diagram for explaining an example of a method for specifically calculating the bias of the convolution layer from the actual output values ​​shown in FIG.

[0059] That is, first, bias calculation unit 20 takes the absolute value of the actual output value of each of channels 1 to 8 in the i-th convolutional layer, and calculates the maximum value for each channel (step 1).

[0060] Next, the bias calculation unit 20 finds the largest value among the maximum values ​​across all channels calculated in step 1 (step 2). In the example shown in Fig. 8, the bias calculation unit 20 finds the maximum value of channel 4 as the largest value among channels 1 to 8.

[0061] Next, the bias calculation unit 20 divides the maximum value of each channel calculated in step 1 by the maximum value found in step 2 plus a small value to prevent division by zero, and normalizes the maximum value among channels 1 to 8 to 1 (step 3). In the example shown in FIG. 8, the bias calculation unit 20 normalizes the maximum values ​​of channels 1 to 8 calculated in step 1 by, for example, 1×10 -7 (also written as 1E-7) and normalized so that the maximum value of channel 4 is 1. Note that small values ​​to prevent division by 0 are 1×10 -7 However, any small value that can prevent division by zero can be used.

[0062] Next, bias calculation unit 20 calculates the sum of the maximum values ​​of each channel normalized in step 3, and divides the calculated sum by the number of channels (step 4). In the example shown in Fig. 8, bias calculation unit 20 calculates the sum of the heights of the bar graphs of channels 1 to 8 as the sum of the maximum values ​​of each channel normalized in step 3, and divides the calculated sum by 8.

[0063] Next, the bias calculation unit 20 subtracts the value calculated in step 4 from 1 and sets the result as the bias L of the i-th convolution layer. i In the example shown in FIG. 8, the bias calculation unit 20 calculates the bias L by subtracting (1-(the sum of the heights of the normalized bar graphs of channels 1 to 8 / 8)). i Here, (1-(sum of heights of normalized bar graphs of channels 1 to 8 / 8)) is the area of ​​the difference between the distribution of actual output values ​​(maximum values) of channels 1 to 8 and the ideal distribution when the output values ​​of channel 4 are uniformly arranged.

[0064] Note that the bias of the i-th convolutional layer, L iThe specific method of calculating is not limited to the one described with reference to Fig. 8. For example, the maximum value of a channel whose actual output value (maximum value) is small may not be included in the calculations of steps 4 and 5. This case will be described below with reference to Fig. 9.

[0065] FIG. 9 shows the bias L of the i-th convolutional layer according to this embodiment. i 9 is a diagram for explaining another example of a method for specifically calculating. FIG. 9 also explains a method for calculating the bias of the convolutional layer from the actual output values ​​shown in FIG. 7. The example shown in FIG. 9 is different from the example shown in FIG. 8 in step 4a. The calculation method of steps 1 to 3 shown in FIG. 9 is the same as the calculation method explained in steps 1 to 3 shown in FIG. 8, so the explanation will be omitted.

[0066] In step 4a, the bias calculation unit 20 calculates the sum of the maximum values ​​of channels that are equal to or greater than the threshold among the maximum values ​​of each channel normalized in step 3, and divides the calculated sum by the number of channels. In the example shown in Fig. 9, the bias calculation unit 20 calculates the sum of the maximum values ​​of channels 1-2, 4-5, and 7-8 that are equal to or greater than the threshold among the maximum values ​​(heights of the bar graphs) of channels 1-8 normalized in step 3. Then, the bias calculation unit 20 divides the calculated sum by 6, which is the number of channels that are equal to or greater than the threshold.

[0067] Next, the bias calculation unit 20 subtracts the value calculated in step 4a from 1 and sets the result as the bias L of the i-th convolution layer. i In the example shown in FIG. 9, the bias calculation unit 20 calculates the bias L by subtracting (1-(the sum of the heights of the six normalized bar graphs of channels 1-2, 4-5, and 7-8 / 6)). i Here, (1-(the sum of the heights of the six normalized bar graphs of channels 1-2, 4-5, and 7-8 / 6)) is the area of ​​the difference between the distribution of the actual output values ​​(maximum values) of channels 1-2, 4-5, and 7-8 and the ideal distribution when the output values ​​of channel 4 are uniformly arranged in six bars.

[0068] In this way, the bias calculation unit 20 may calculate, as the bias, the bias of output values ​​equal to or greater than a threshold value corresponding to the maximum value of the multiple output values ​​among the multiple output values ​​in each of one or more intermediate layers. In other words, the bias calculation unit 20 may exclude a normalized channel from participating in the bias calculation according to its output value (maximum value). This allows the weight of a channel that has a small contribution to the inference by the neural network 111 to approach 0.

[0069] The threshold applied in step 4a may be, for example, 1 / 256, or may be up to about several percent of the largest value among channels 1 to 8, but is not limited to this. The threshold applied in step 4a may be a value that can prevent a channel output value (maximum value) from being included in the bias calculation when the output value is close to 0.

[0070] [1-3-2. Learning Execution Unit 21] The learning execution unit 21 learns the parameters of each layer of the neural network 111 so as to reduce the bias calculated by the bias calculation unit 20. In this embodiment, the parameters of each layer are the weights of each layer of the neural network 111 including the intermediate layers.

[0071] In this embodiment, the learning execution unit 21 includes a loss function calculation unit 211 and an optimization unit 212, as shown in FIG.

[0072] <Loss function calculation unit 211> The loss function calculation unit 211 calculates a first loss function that calculates an error between an inference result of the neural network 111 obtained by applying input data to the neural network 111 and a correct label included in the learning data. In this embodiment, a cross-entropy loss is used as the first loss function, but the first loss function is not limited to this. A mean squared error may be used as the first loss function. When the cross-entropy error is used, the sum of the products of the natural logarithm (ln value) of the "inference value" of the neural network 111 and the correct data value is calculated. When the mean squared error is used, the sum of the squares of the differences between the "inference value" of the neural network 111 and the "correct data value" is calculated.

[0073] Furthermore, the loss function calculation unit 211 calculates a second loss function by adding the bias calculated by the bias calculation unit 20 to the calculated first loss function.

[0074] In this embodiment, the loss function calculation unit 211 adds the bias L of the target convolution layer as the bias calculated by the bias calculation unit 20 to the first loss function. i More specifically, the loss function calculation unit 211 calculates the second loss function by adding the sum of

number

[0075] The bias added to the first loss function is the bias L of all convolution layers of the neural network 111. i However, this is not limited to the above. i That is, in the above sigma calculation, α i= 0, only the bias in the output values ​​of some of the convolution layers of the neural network 111 may be added to the first loss function. This makes it possible to exclude the bias in the output values ​​of the convolution layers that have a small contribution to the quantization error from the optimization target.

[0076] In addition, the loss function calculation unit 211 adds the bias L of the target convolution layer as the bias calculated by the bias calculation unit 20 to the first loss function. i However, the present invention is not limited to this. The loss function calculation unit 211 may calculate the second loss function by adding to the first loss function a function that uses bias as an argument and that contributes more to the second loss function as the bias increases. More specifically, the loss function calculation unit 211 calculates the second loss function by adding to the first loss function the sum of

number

[0077] Here, using FIGS. 10A to 10C, the function f i An example of (x) will be described.

[0078] 10A, 10B, and 10C show the function f i FIG. 13 is a diagram conceptually illustrating an example of (x).

[0079] That is, the function f i (x) is as shown in FIG. 10A ,

number

number

number

[0080] Such a function f i By adding (x) as the bias, the bias L i The larger the value of , the greater the contribution to the second loss function. As a result, a relatively small bias L i The second loss function is optimized while allowing values ​​of

[0081] <Optimization unit 212> The optimization unit 212 optimizes the second loss function calculated by the loss function calculation unit 211, thereby causing the parameters of each layer of the neural network 111 to learn so as to reduce the bias calculated by the bias calculation unit 20.

[0082] In other words, the optimization unit 212 performs learning (optimization) to minimize the second loss function, thereby making it possible to learn parameters such as the weights of the neural network while reducing bias in multiple output values ​​in each of one or more intermediate layers.

[0083] In this embodiment, the optimization unit 212 optimizes the second loss function by Adam, and updates the weights of the neural network 111 so that the second loss function is minimized, thereby training the neural network 111. Here, Adam is an optimization method that combines momentum and an adaptive learning rate with Stochastic Gradient Descent (SGD). Note that the optimization method for the second loss function is not limited to Adam. For example, the optimization method for the second loss function may be only Stochastic Gradient Descent, or may be RMSprop or Layer-wise Adaptive Rate Scaling (LARS), and is not limited to these.

[0084] [2. Operation of the learning device 10] An example of the operation of the learning device 10 configured as above will be described below.

[0085] FIG. 11 is a flowchart showing an outline of the operation of the learning device 10 in this embodiment.

[0086] First, the learning device 10 calculates the bias of multiple output values ​​in an intermediate layer obtained by applying input data included in previously prepared learning data to the neural network 111 (S1). For example, when the neural network 111 is a CNN, the bias calculation unit 20 calculates the bias of output values ​​of several channels, which are multiple output values ​​output by a convolution layer of the neural network 111. Then, the learning device 10 calculates the bias of the output values ​​for each channel by calculating the difference between the distribution of the multiple output values ​​and a uniform distribution in which the maximum values ​​of the multiple output values ​​are uniformly arranged.

[0087] Next, the learning device 10 trains the parameters of each layer of the neural network 111 so as to reduce the bias calculated in step S1 (S2).

[0088] FIG. 12 is a flowchart showing an example of detailed operation of the learning device 10 in this embodiment.

[0089] As shown in FIG. 12, first, the learning device 10 reads a neural network 111 having one or more intermediate layers each outputting a plurality of output values ​​(S10). For example, the learning device 10 reads the neural network 111 that solves an image classification task. Note that the learning device 10 is not limited to reading the neural network 111 that solves an image classification task. The learning device 10 may read a neural network 111 that solves a recognition task in computer vision, a detection task in computer vision, or a segmentation task. The learning device 10 may also read a neural network 111 that solves a natural language processing task or a reinforcement learning task.

[0090] Next, the learning device 10 calculates a first loss function (S11). More specifically, the learning device 10 calculates the first loss function that calculates the error between an inference result of the neural network 111 obtained by applying the input data to the neural network 111 and a correct label included in the learning data. Here, for example, a cross-entropy error is used as the first loss function.

[0091] Next, the learning device 10 calculates the bias (S12). More specifically, the learning device 10 calculates the bias of multiple output values ​​in each of one or more intermediate layers obtained by applying input data included in the learning data prepared in advance to the neural network 111. Note that the order of steps S11 and S12 is not limited to this order and may be reversed.

[0092] Here, a specific example of the operation in step S12 will be described with reference to FIG.

[0093] FIG. 13 is a flowchart showing an example of a specific operation of step S12 in FIG.

[0094] In the following description, it is assumed that the intermediate layer of the neural network 111 is a convolutional layer.

[0095] 13, first, the learning device 10 takes absolute values ​​for multiple output values ​​of the i-th convolutional layer, and calculates the maximum value for each channel (S121). More specifically, the learning device 10 takes absolute values ​​for actual output values ​​of each channel in the i-th convolutional layer, and calculates the maximum value for each channel.

[0096] Next, the learning device 10 calculates the largest value among the maximum values ​​across all channels calculated in step S121 (S122).

[0097] Next, the learning device 10 normalizes the largest value calculated in step S122 so that it becomes 1 (S123). More specifically, the learning device 10 divides the maximum value of each channel calculated in step S121 by a value obtained by adding a small value to prevent division by zero to the maximum value calculated in step S122, and normalizes the largest value calculated in step S122 so that it becomes 1.

[0098] Next, the learning device 10 calculates a value by dividing the sum of the maximum values ​​of each channel normalized in step S123 by the number of channels (S124).

[0099] Next, the learning device 10 subtracts the value calculated in step S124 from 1 and sets the value as the bias L of the i-th convolutional layer. i (S125).

[0100] Finally, if the learning device 10 does not calculate bias in other convolutional layers (No in S126), it ends the process here and proceeds to step S13 shown in Fig. 12. Note that, if the learning device 10 calculates bias in other convolutional layers as well (Yes in S126), it repeats the process from step S121.

[0101] Returning to FIG. 12, the explanation will be continued below.

[0102] Next, the learning device 10 calculates a second loss function by adding the bias calculated in step S12 to the first loss function calculated in step S11 (S13). For example, the learning device 10 calculates a second loss function by adding the bias L of the convolution layer calculated in step S12 to the first loss function calculated in step S11 (S13). i The learning device 10 may calculate the second loss function by adding, to the first loss function calculated in step S11, a function that uses the bias calculated in step S12 as an argument and that contributes more to the second loss function as the bias increases.

[0103] Next, the learning device 10 optimizes using Adam and updates the weights of the neural network 111 so as to minimize the second loss function (S14). More specifically, the learning device 10 optimizes the second loss function calculated in step S13 using Adam, thereby learning parameters of each layer of the neural network 111 so as to reduce the bias calculated in step S12. As described above, the optimization method of the second loss function is not limited to Adam. For example, the optimization method of the second loss function may be only stochastic gradient descent, or may be RMSprop or LARS.

[0104] In this manner, in this embodiment, before per-tensor quantization, the weights of the neural network 111 are learned so as to reduce bias in one or more intermediate layers of the neural network 111 expressed in floating point.

[0105] [3. Effects, etc.] In this embodiment, before per-tensor quantization, the weights of the neural network 111 are trained so as to reduce the bias of possible values ​​of one or more intermediate layers of the neural network 111 expressed in floating point.

[0106] More specifically, the present invention provides a learning method for performing learning of a neural network having one or more intermediate layers, each of which outputs a plurality of output values, the learning method including a bias calculation step for calculating the bias of a plurality of output values ​​in each of the one or more intermediate layers obtained by applying input data contained in pre-prepared learning data to the neural network, and a learning execution step for learning parameters of each layer of the neural network so as to reduce the bias.

[0107] This makes it possible to perform per-tensor quantization on a neural network in which the bias in the multiple output values ​​of each of one or more intermediate layers is reduced, thereby suppressing deterioration in the inference accuracy of the neural network even when per-tensor quantization is performed.

[0108] Therefore, even when performing inference using a per-tensor quantized neural network with limited computing resources such as an SoC, it is possible to achieve both inference accuracy and a low computational effort.

[0109] In addition, the inference accuracy is not limited to the precision and recall where the output value of the model after quantization indicates whether or not the correct answer has been inferred, but may be at least one combination of the precision, recall, F-value calculated by the harmonic mean of the precision and recall, and accuracy.

[0110] In the above embodiment, two-dimensional data such as an image is input to the neural network, and two-dimensional data output values ​​are output as channel outputs, but this is not limited to the above. Three-dimensional data may be input to the neural network, and the intermediate layer may perform convolution on a three-dimensional feature map to output three-dimensional data output values. The same can be said for this embodiment.

[0111] In the above embodiment, the intermediate layer of the neural network is a convolutional layer, but the present invention is not limited to this.

[0112] 14A to 14E are diagrams conceptually showing examples of the configuration of the intermediate layer according to the present embodiment.

[0113] In the above embodiment, the intermediate layer of the neural network 111 is described as a convolution layer as shown by Conv in Fig. 14A, but is not limited thereto. The convolution layer may be a depthwise convolution layer or a group convolution layer.

[0114] In addition, the intermediate layer of the neural network 111 may be composed of a convolution layer and a subsequent normalization layer, as shown by Conv and Norm in FIG. 14B. That is, one or more intermediate layers of the neural network 111 may include a layer composed of a convolution layer and a normalization layer subsequent to the convolution layer. In this case, the bias of the multiple output values ​​in the layer becomes the bias of the multiple output values ​​obtained from the normalization layer. Batch Normalization, Layer Normalization, or the like can be used for the normalization layer.

[0115] The intermediate layer of the neural network 111 may further include an activation layer between the convolution and the subsequent normalization layer, as shown by Conv, Activation, and Norm in Fig. 14C. Here, the activation layer is a layer characterized by a function called an activation function, and the activation function may be ReLU, ReLU6, Swish, GELU, Tanh, Sigmoid, or the like.

[0116] In addition, the intermediate layer of the neural network 111 may be composed of a convolution layer followed by an activation layer, as shown by Conv and Activation in Fig. 14D. That is, one or more intermediate layers of the neural network 111 may include a layer composed of a convolution layer and an activation layer following the convolution layer. In this case, the bias of the multiple output values ​​in the layer is the bias of the multiple output values ​​obtained from the activation layer.

[0117] Note that the intermediate layer of the neural network 111 may further include a normalization layer between the convolution layer and the subsequent activation layer, as shown by Conv, Norm, and Activation in FIG. 14E.

[0118] (Other embodiments) Although the learning device and learning method according to the present disclosure have been described above based on the embodiments, the present disclosure is not limited to these embodiments. As long as they do not deviate from the gist of the present disclosure, various modifications conceived by a person skilled in the art to the embodiments and other forms constructed by combining some of the components in the embodiments are also included within the scope of the present disclosure.

[0119] The following forms may also be included within the scope of one or more aspects of the present disclosure.

[0120] (1) Some of the components constituting the above learning device may be a computer system consisting of a microprocessor, ROM, RAM, a hard disk unit, a display unit, a keyboard, a mouse, etc. A computer program is stored in the RAM or hard disk unit. The microprocessor achieves its functions by operating in accordance with the computer program. Here, the computer program is composed of a combination of multiple instruction codes that indicate commands for a computer to achieve a specified function.

[0121] (2) Some of the components constituting the learning device may be composed of one system LSI (Large Scale Integration). The system LSI is an ultra-multifunctional LSI manufactured by integrating multiple components on a single chip, and specifically, is a computer system including a microprocessor, ROM, RAM, etc. A computer program is stored in the RAM. The system LSI achieves its functions by the microprocessor operating in accordance with the computer program.

[0122] (3) Some of the components constituting the above learning device may be composed of an IC card or a standalone module that can be attached to each device. The IC card or the module is a computer system composed of a microprocessor, ROM, RAM, etc. The IC card or the module may include the above-mentioned ultra-multifunction LSI. The microprocessor operates according to a computer program, causing the IC card or the module to achieve its functions. The IC card or the module may be tamper-resistant.

[0123] (4) Furthermore, some of the components constituting the learning device may be the computer program or the digital signal recorded on a computer-readable recording medium, such as a flexible disk, a hard disk, a CD-ROM, an MO, a DVD, a DVD-ROM, a DVD-RAM, a BD (Blu-ray (registered trademark) Disc), a semiconductor memory, etc. Also, some of the components may be the digital signal recorded on such a recording medium.

[0124] In addition, some of the components constituting the above-mentioned learning device may transmit the computer program or the digital signal via a telecommunications line, a wireless or wired communication line, a network such as the Internet, data broadcasting, etc.

[0125] (5) The present disclosure may be embodied as any of the methods described above, a computer program for implementing these methods by a computer, or a digital signal comprising the computer program.

[0126] (6) The present disclosure may also provide a computer system having a microprocessor and a memory, the memory storing the computer program, and the microprocessor operating in accordance with the computer program.

[0127] (7) The program or the digital signal may also be implemented by another independent computer system by recording it on the recording medium and transferring it, or by transferring the program or the digital signal via the network, etc.

[0128] (8) In addition, some of the components that make up the above-mentioned learning device may be performed on a cloud or server device.

[0129] (9) The above-described embodiments and modifications may be combined with each other. [Industrial Applicability]

[0130] The present disclosure can be used in a learning device, a learning method, a program, and the like that executes learning of a neural network that performs quantization processing for incorporation into a SoC or the like. [Explanation of symbols]

[0131] 1 Acquisition part 2. Learning Department 10 Learning Device 20 Bias calculation section 21 Learning Execution Department 111 Neural Networks 211 Loss function calculation part 212 Optimization Department 1000 Computers 1001 Input Device 1002 Output device 1003 CPU 1004 Internal Storage 1005 RAM 1007 Reading device 1008 Transmitting and receiving device 1009 Bus

Claims

1. A learning device that executes learning of a neural network having one or more intermediate layers that each output a plurality of output values, comprising: a bias calculation unit that calculates biases of a plurality of output values ​​in each of the one or more intermediate layers obtained by applying input data included in previously prepared learning data to the neural network; a learning execution unit that learns parameters of each layer of the neural network so as to reduce the bias; the bias calculation unit calculates the bias by calculating a difference between a distribution of the plurality of output values ​​and a uniform distribution in which maximum values ​​of the plurality of output values ​​are uniformly arranged. Learning device.

2. The learning execution unit, a loss function calculation unit that calculates a first loss function that calculates an error between an inference result of the neural network obtained by applying the input data to the neural network and a correct label included in the learning data, and calculates a second loss function by adding the bias calculated by the bias calculation unit to the first loss function; and an optimization unit that optimizes the second loss function to train parameters of each layer of the neural network so as to reduce the bias. The learning device according to claim 1 .

3. the bias calculation unit calculates, as the bias, a bias of a plurality of output values ​​in each of all the intermediate layers included in the neural network; The learning device according to claim 2 .

4. the bias calculation unit calculates, as the bias, a bias of a plurality of output values ​​in each of a part of the intermediate layers among all the intermediate layers included in the neural network; The learning device according to claim 2 .

5. the bias calculation unit calculates, as the bias, a bias of output values ​​that are equal to or greater than a threshold corresponding to a maximum value of the plurality of output values ​​in each of the one or more intermediate layers; The learning device according to any one of claims 2 to 4.

6. the loss function calculation unit calculates the second loss function by adding to the first loss function a function that uses the bias as an argument, the function having a larger contribution to the second loss function as the bias increases; The learning device according to any one of claims 2 to 5.

7. Each of the one or more intermediate layers is a convolution layer, a depthwise convolution layer, or a fully connected layer. The learning device according to any one of claims 1 to 6.

8. The one or more intermediate layers include a layer composed of a convolutional layer and a normalization layer following the convolutional layer, The bias of the multiple output values ​​in the layer is the bias of the multiple output values ​​obtained from the normalization layer. The learning device according to any one of claims 1 to 6.

9. The one or more intermediate layers include a layer composed of a convolutional layer and an activation layer following the convolutional layer, The bias of the output values ​​in the layer is the bias of the output values ​​obtained from the activation layer. The learning device according to any one of claims 1 to 6.

10. A learning method in which a computer executes learning of a neural network having one or more intermediate layers each outputting a plurality of output values, comprising: a bias calculation step of calculating bias of a plurality of output values ​​in each of the one or more intermediate layers obtained by applying input data included in training data prepared in advance to the neural network; and a learning execution step of learning parameters of each layer of the neural network so as to reduce the bias; In the bias calculation step, the bias is calculated by calculating a difference between a distribution of the plurality of output values ​​and a uniform distribution in which maximum values ​​of the plurality of output values ​​are uniformly arranged. How to learn.

11. A program for causing a computer to execute a learning method for executing learning of a neural network having one or more intermediate layers each outputting a plurality of output values, comprising: a bias calculation step of calculating bias of a plurality of output values ​​in each of the one or more intermediate layers obtained by applying input data included in training data prepared in advance to the neural network; A learning execution step of learning parameters of each layer of the neural network so as to reduce the bias. Let the computer run In the bias calculation step, the bias is calculated by calculating a difference between a distribution of the plurality of output values ​​and a uniform distribution in which maximum values ​​of the plurality of output values ​​are uniformly arranged. program.

Images