Arithmetic and control device, control method for arithmetic and control device, and program

Abstract

To provide an arithmetic control apparatus, a control method of the arithmetic control apparatus, and a program that control a drive frequency related to neuron arithmetic of a neural network in more suitable mode.SOLUTION: An arithmetic control apparatus 100 including an input layer, a plurality of intermediate layers that extract feature quantity included in an input value from the input layer, and an output layer, and based on the neural network includes an arithmetic unit 106 and a frequency control unit 110. The arithmetic unit 106 executes neural arithmetic for each layer. The frequency control unit 110 controls a drive frequency relating to the neuron arithmetic according to the number of factors of the arithmetic unit relating to the neuron arithmetic, and an arithmetic amount for each layer of the neural network.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an arithmetic and control device, a control method for an arithmetic and control device, and a program. [Background technology]

[0002] In an environment where so-called consumer products such as digital cameras are made to execute inference in machine learning, it is not realistic to install a computing unit (multiply-and-accumulate unit) related to neuron operations in a quantity sufficient to satisfy the amount of calculations of a trained model, since the circuit scale would be enormous. Also, in neuron operations, when the results of the operations of a series of nodes for one layer of a neural network are not all available, it is difficult to perform the calculation of the next layer.

[0003] In addition, there may be a difference in the amount of calculation for each layer of a neural network, and in such a situation, for example, in order to ensure that the calculation is completed within a specified time, the driving frequency of the arithmetic unit may be controlled to a frequency that matches the layer with a larger amount of calculation. However, when such control is applied, when calculations are performed for a layer with a smaller amount of calculations, there may be a time period in which the arithmetic unit operates even though no calculation is required, which may result in inefficient operation of the circuit and excessive power consumption. In view of this situation, various techniques for optimizing power by controlling the drive frequency have been studied. For example, Patent Document 1 discloses an example of a technique for controlling the drive frequency supplied from a clock generator to a computing unit so that the power consumption for processing a desired unit of data amount falls within a predetermined range. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2004-303206 A Summary of the Invention [Problem to be solved by the invention]

[0005] On the other hand, when neuron operations of a neural network are performed under circumstances in which the operation results are required within a specified time, even if the drive frequency is controlled by focusing only on the amount of data and power consumption, the processing may not be completed within the time limit or excessive power consumption may occur.

[0006] In view of the above problems, an object of the present invention is to control the drive frequency related to neuron operations of a neural network in a more suitable manner. [Means for solving the problem]

[0007] The arithmetic and control device according to the present invention is an arithmetic and control device based on a neural network including an input layer, a plurality of intermediate layers for extracting features included in input values ​​from a previous layer, and an output layer, and includes a calculation means for executing neuron operations for each layer, and a frequency control means for controlling a drive frequency for the neuron operations, wherein the frequency control means is The number of elements of the computing unit related to the neuron operation; The drive frequency is controlled according to the amount of calculation for each layer of the neural network. Effect of the Invention

[0008] According to the present invention, it is possible to control the drive frequency related to the neuron calculations of a neural network in a more suitable manner. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram illustrating an example of a hardware configuration of an arithmetic and control device. [Diagram 2] FIG. 1 illustrates an example of a neural network. [Diagram 3] FIG. 11 is a diagram showing an example of the relationship between the number of calculations and the drive frequency. [Figure 4] 11 is a graph showing an example of the relationship between the number of calculations and the drive frequency. [Diagram 5] 4 is a flowchart showing an example of processing by an arithmetic and control device. [Figure 6] FIG. 13 is a diagram showing another example of a neural network. [Figure 7] FIG. 11 is a diagram showing another example of the relationship between the number of calculations and the drive frequency. [Figure 8] 13 is a graph showing another example of the relationship between the number of calculations and the drive frequency. [Figure 9A] 10 is a flowchart showing another example of the processing of the arithmetic and control device. [Figure 9B] 10 is a flowchart showing another example of the processing of the arithmetic and control device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0011] <Hardware configuration> An example of a hardware configuration of the arithmetic and control device according to the present embodiment will be described with reference to Fig. 1. The arithmetic and control device 100 includes a central processing unit (CPU) 102, a read only memory (ROM) 103, a random access memory (RAM) 104, and an auxiliary storage device 105. The arithmetic and control device 100 also includes a calculator 106 and a frequency control unit 110. The arithmetic and control device 100 may also include at least one of an input unit 108 and an output unit 109. The CPU 102, the ROM 103, the RAM 104, the auxiliary storage device 105, the calculator 106, the input unit 108, the output unit 109, and the frequency control unit 110 are each connected to one another via a system bus 101.

[0012] The CPU 102 is a central processing unit that controls various operations of the arithmetic and control device 100. For example, the CPU 102 may control the operation of the arithmetic and control device 100 as a whole. The RAM 104 is a main storage memory for the CPU 102, and is used as a work area or a temporary storage area for developing various programs. The ROM 103 stores a program (eg, BIOS) for the CPU 102 to control the operation of the arithmetic and control device 100.

[0013] The auxiliary storage device 105 stores an OS (Operating System) which is basic software, programs such as various applications, and various data. Examples of the various data stored in the auxiliary storage device 105 include data of trained models used for various inferences. The auxiliary storage device 105 can be realized by, for example, a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SDD).

[0014] The calculator 106 is a schematic diagram of a calculator that executes neuron calculations in a neural network. The calculator 106 does not necessarily need to be provided as an independent configuration, and may be realized by a configuration capable of executing calculation processing, such as the CPU 102. As another example, the calculator 106 may be realized by a calculation unit such as a GPU (Graphics Processing Unit) that can execute calculation processing of a two-dimensional array in a more suitable manner.

[0015] The input unit 108 is a schematic diagram showing a configuration for receiving instructions from a user. The input unit 108 can be realized by, for example, a pointing device such as a mouse, a keyboard, and an operation device such as a touch panel. The output unit 109 is a schematic diagram showing a configuration for presenting various information to a user. The output unit 109 can be realized by a display device, such as a display, that presents information to a user by displaying various display information, a screen, or the like.

[0016] The frequency control unit 110 controls the drive frequency of the process related to various inferences in machine learning executed by at least the arithmetic unit 106. The frequency control unit 110 may be configured to be capable of controlling the drive frequency of the process of each component in the arithmetic control device 100, not limited to the process related to various inferences in machine learning. The frequency control unit 110 does not necessarily need to be provided as an independent configuration, and may be realized by a configuration that plays a role of various controls, such as the CPU 102.

[0017] The above-mentioned configuration is merely an example, and does not necessarily limit the hardware configuration of the arithmetic and control device 100. As a specific example, some of the components of the arithmetic and control device 100 described above (for example, at least one of the input unit 108 and the output unit 109) may be externally attached to the outside of the arithmetic and control device 100. In addition, the processing described below with reference to Figures 5, 9A, and 9B is realized in a desired arithmetic and control device 100 by the CPU 102 expanding a program stored in the ROM 103 or the auxiliary storage device 105 into the RAM 104 and executing it.

[0018] <Neural Network> An example of a neural network in machine learning will be outlined with reference to FIG. 2. In the example shown in FIG. 2, the neural network includes an input layer, hidden layers A to D, and an output layer. Specifically, each of nodes 201 to 204 indicates a node of the input layer. Also, nodes 211 to 215, nodes 221 to 223, nodes 231 to 232, and nodes 241 to 244 indicate nodes of hidden layers A, B, C, and D, respectively. Also, nodes 251 to 252 indicate nodes of the output layer. Each of hidden layers A to D and the output layer may correspond to a layer that extracts a feature amount included in an input value from a previous layer. Also, the hidden layer is also called a "middle layer."

[0019] L20 shows a schematic diagram of an operation (neuron operation) executed in hidden layer A in response to an input from the input layer to the hidden layer A. Hereinafter, the operation L20 is also referred to as an operation (neuron operation) corresponding to hidden layer A for convenience. The operation L20 corresponding to hidden layer A is an operation executed in response to an input from four nodes in the input layer to five nodes in the hidden layer A, and corresponds to a 4×5 matrix operation. Similarly, operations L21, L22, L23, and L24 correspond to operations corresponding to hidden layer B, hidden layer C, hidden layer D, and output layer, respectively. That is, operation L21 corresponds to a 5×3 matrix operation executed in association with input from five nodes of hidden layer A to three nodes of hidden layer B. Furthermore, operation L22 corresponds to a 3×2 matrix operation executed in association with input from three nodes of hidden layer B to two nodes of hidden layer C. Furthermore, operation L23 corresponds to a 2×4 matrix operation executed in association with input from two nodes of hidden layer C to four nodes of hidden layer D. Furthermore, operation L24 corresponds to a 4×2 matrix operation executed in association with input from four nodes of hidden layer D to two nodes of the output layer.

[0020] <Processing> An example of the processing of the arithmetic and control device 100 according to this embodiment will be described with reference to Figs. 3 to 5, focusing in particular on the control of the drive frequency related to the neuron arithmetic by the frequency control section 110.

[0021] First, refer to Fig. 5. In S501, the CPU 102 causes the frequency control unit 110 to calculate a frequency for driving the arithmetic unit 106 used in various inferences in machine learning. Here, an example of a specific method for determining the frequency will be described below with reference to Figs. 3 and 4. Hereinafter, the frequency will also be referred to as a "driving frequency".

[0022] First, Fig. 3 will be described. Fig. 3 shows an example of the relationship between the number of calculations (in other words, the amount of calculation) and the drive frequency in each of the calculations L20 to L24 corresponding to each layer in the neural network shown in Fig. 2. The number of calculations 301 indicates the number of multiply-add calculations executed in the matrix calculation for each layer (in other words, the number of multiply-add calculations required to be executed in the matrix calculation for each layer). The maximum number of calculations 302 indicates the maximum number of multiply-add calculations that can be executed in the calculation time allocated to the calculator 106 to execute the calculation for each layer (neuron calculation). The maximum drive frequency 303 indicates the maximum drive frequency of the calculator 106. 3, the maximum number of calculations 302 when the calculator 106 is driven at the maximum drive frequency 303 is constant between calculations L21 to L24. Therefore, in the example shown in FIG. 3, the maximum drive frequency 303 is set to 1000 KHz, and the maximum number of calculations 302 is set to 20.

[0023] The drive frequency 304 is calculated based on the maximum number of calculations 302, the number of calculations 301, and the maximum drive frequency 303. A specific calculation method is as shown in the relational expression shown below as (Equation 1).

[0024]

number

[0025] By calculating the driving frequency 304 as described above, it is possible to determine the driving frequency of the calculator 106 required to complete the neuron operation within a predetermined operation time in each layer. As a specific example, in the case of the operation L20 corresponding to the hidden layer A, it is 1000KHz×20 times÷20 times=1000KHz. Similarly, in the case of the operation L21 corresponding to the hidden layer B, it is 1000KHz×15 times÷20 times=750KHz. In addition, in the case of the operation L22 corresponding to the hidden layer C, it is 1000KHz×6 times÷20 times=300KHz. In addition, in the case of the operation L23 corresponding to the hidden layer D, it is 1000KHz×8 times÷20 times=400KHz. In addition, in the case of the operation L24 corresponding to the output layer, it is 1000KHz×8 times÷20 times=400KHz. Furthermore, the number of elements of the multiply-and-accumulate calculator constituting the calculator 106 may be taken into consideration when determining the drive frequency 304. As a specific example, when the calculator 106 is composed of a plurality of elements (multiply-and-accumulate calculators) each capable of executing a plurality of calculations in parallel, the number of calculations executed by each element is reduced, and therefore it is possible to keep the drive frequency 304 of each element lower.

[0026] Next, Fig. 4 will be described. Fig. 4 is a graph showing an example of the relationship between the number of calculations 301 and the drive frequency 304 for each of the calculations L21 to L24 corresponding to each layer of the neural network described with reference to Figs. 2 and 3. In Fig. 4, the graph shown with a thick line shows the number of calculations 301 for each of the calculations L20 to L24. Moreover, the graph shown with a thin line shows a schematic transition along a time series of the drive frequency 304 of the calculator 106 when the calculator 106 sequentially executes each of the calculations L20 to L24. That is, in the example shown in Fig. 4, it can be seen that the drive frequency of the calculator 106 changes in proportion to the number of calculations 301 required for each of the calculations L20 to L24.

[0027] Here, reference is again made to Fig. 5. In S502, the CPU 102 drives the calculator 106 based on the drive frequency calculated in S501. In S503, the CPU 102 causes the calculator 106 to execute one layer's worth of neuron operations. For example, in the example shown in FIG. 2, the calculator 106 first executes operation L20 corresponding to the hidden layer A.

[0028] In S504, the CPU 102 determines whether or not the neuron operations of a series of layers are completed. When the CPU 102 determines in S504 that the neuron operations of the series of layers are not completed, the process proceeds to S501. In this case, the CPU 102 executes the processes from S501 onward again for the other layers in which the neuron operations are not completed. Then, when CPU 102 determines in S504 that the neuron operations for a series of layers are completed, it ends the series of processes shown in FIG.

[0029] By applying the above-mentioned control, it becomes possible to control the operating frequency of the computing unit according to the amount of calculation for each layer of the neural network when executing inference in machine learning. This makes it possible to complete neural calculations for each layer within a specified calculation time, and further makes it possible to suppress excessive power consumption by preventing the occurrence of situations in which the computing unit is overdriven.

[0030] <Modification> A modified example of the arithmetic control device according to this embodiment will be described below. In this modified example, an example of a technology that can control the drive frequency of the arithmetic unit in consideration of the amount of calculation across multiple layers of a neural network in machine learning inference will be described. Specifically, in this modified example, an example of a technology that can control the drive frequency of the arithmetic unit 106 in a more suitable manner when inference is performed using a trained model that requires an amount of calculation that exceeds the processing capacity of the arithmetic unit 106 will be described. Note that in this modified example, the description will be given with a particular focus on the parts that are different from the embodiment described above, and detailed descriptions of parts that are substantially similar to the embodiment described above will be omitted.

[0031] (Neural Networks) First, an example of a neural network used to explain the arithmetic and control device according to this modification will be outlined with reference to Fig. 6. In the example shown in Fig. 6, the neural network includes an input layer, hidden layers A to D, and an output layer. Specifically, each of nodes 201 to 204 indicates a node in the input layer. Also, nodes 211 to 215 and node 601 indicate a node in hidden layer A. Also, nodes 221 to 223, nodes 231 to 232, and nodes 241 to 244 indicate nodes in hidden layer B, hidden layer C, and hidden layer D, respectively. Also, nodes 251 to 252 indicate nodes in the output layer.

[0032] Moreover, the operations L60, L61, L62, L63, and L64 correspond to operations corresponding to hidden layer A, hidden layer B, hidden layer C, hidden layer D, and output layer, respectively. Specifically, the operations L60, L61, L62, L63, and L64 correspond to matrix operations that are neuron operations for each layer, similar to the operations L20, L21, L22, L23, and L24 shown in FIG. That is, the example shown in FIG. 6 differs from the example shown in FIG.

[0033] (process) Next, referring to Figures 7, 8, 9A, and 9B, an example of the processing of the arithmetic and control device 100 according to this modification will be described, focusing in particular on the control of the drive frequency for neuron operations by the frequency control unit 110. Note that, as a premise, the arithmetic and control device 100 according to this modification is assumed to execute neuron operations of each of a plurality of layers of a neural network at a predetermined calculation time.

[0034] First, Fig. 7 will be described. Fig. 7 shows an example of the relationship between the number of calculations and the drive frequency in each of the calculations L60 to L64 corresponding to each layer in the neural network shown in Fig. 6. The number of calculations 301, the maximum number of calculations 302, and the maximum drive frequency 303 are substantially the same as the example shown in Fig. 3.

[0035] The number of carryover operations 701 indicates the number of operations that were carried over because the multiply-and-accumulate operation in the previous layer was not completed within the specified calculation time due to insufficient calculation capacity of the calculator 106. For example, the number of carryover operations 701 for calculation L60 is 0 because there is no previous layer.

[0036] The number of operations 702 indicates the number of operations executed in the target layer. Specifically, if the sum of the number of operations 701 and the number of operations 301 is equal to or greater than the maximum number of operations 302, the maximum number of operations 302 is set as the number of operations 702. On the other hand, if the sum of the number of operations 701 and the number of operations 301 is less than the maximum number of operations 302, the sum of the number of operations 701 and the number of operations 301 is set as the number of operations 702. As a more specific example, in the case of operation L60, the number of operations carried over 701 (0) plus the number of operations requested to be performed in the matrix operation for each layer (301) (24) is 24, which is equal to or greater than the maximum number of operations 302 (20), so 20 is set as the number of operations 702. In addition, in the case of operation L62, the number of operations carried over 701 (2) plus the number of operations requested to be performed in the matrix operation for each layer (301) (6) is 8, which is less than the maximum number of operations 302 (20), so 8 is set as the number of operations 702.

[0037] The remaining number of calculations 703 indicates the number of calculations remaining after the calculator 106 executes the calculation for each layer. The remaining number of calculations 703 is calculated according to the relational expression shown below as (Equation 2).

[0038]

number

[0039] For example, in the case of the operation L60 shown in FIG. 6, the remaining number of operations 703 is 24+0-20=4. The remaining number of operations 703 in this operation L60, which is 4, corresponds to the carryover number of operations 701 in the operation L61 executed next to the operation L60. If the calculated value of the remaining number of operations 703 is less than 0, the remaining number of operations 703 is set to 0. For example, in the case of the operation L60, the calculated value of the remaining number of operations 703 is 6+2-20=-12, which is less than 0. Therefore, in this case, the remaining number of operations 703 is set to 0.

[0040] The driving frequency 704 indicates the driving frequency of the calculator 106 in each layer. The method of calculating the driving frequency 704 is as shown in the relational expression shown below as (Equation 3).

[0041]

number

[0042] Next, Fig. 8 will be described. Fig. 8 is a graph showing an example of the relationship between the number of calculations 702 and the drive frequency 704 when the processing described later with reference to Figs. 9A and 9B is applied to each of the calculations L60 to L64 corresponding to each layer of the neural network described with reference to Figs. 6 and 7. In Fig. 8, the graph shown with a thick line shows the number of calculations 702 for each of the calculations L60 to L64. Moreover, the graph shown with a thin line shows a schematic transition along the time series of the drive frequency 704 of the calculation unit 106 when the calculation unit 106 sequentially executes each of the calculations L60 to L64. 7 and 8, in the case of the neural network shown in Fig. 6, when the processing of the operations L60 and L61 is completed, the remaining number of operations 703 is generated, but when the processing of the operation L62 is completed, the remaining number of operations 703 becomes 0. Therefore, when the operations L63 and L64 are executed, the operator 106 is driven at the drive frequency 704 that can complete the neuron operations of the corresponding layer.

[0043] Next, the processing of the arithmetic and control device 100 according to this modified example will be specifically described with reference to Figures 9A and 9B. First, Figure 9A will be described. In S901, the CPU 102 calculates the number of operations remaining for execution of operations in the target layer based on the sum of the number of operations carried over 701 from the previous layer and the number of operations 301 in the target layer. In S902, the CPU 102 calculates the number of operations 702 to be executed in the target layer based on the number of operations calculated in S901 and the maximum number of operations 302 that the calculator 106 can process. In S903, the CPU 102 causes the frequency control unit 110 to calculate the drive frequency 704 of the calculator 106 based on the number of calculations 702 calculated in S902. In S904, the CPU 102 drives the calculator 106 based on the drive frequency 704 calculated in S903.

[0044] In S905, the CPU 102 causes the calculator 106 to execute the calculations for the number of calculations 702 calculated in S903. As described above, it is difficult to perform the calculations for the next layer in the neuron calculations when the calculation results of a series of nodes for the previous layer are not available. In light of this situation, the process of S905 will be described in detail with reference to FIG. 9B.

[0045] In S911, the calculator 106 executes the neuron operations remaining in the previous layer for the number of carryover operations 701. When the process of S911 is completed, the operation results of a series of nodes in the previous layer are all available. In S912, the calculator 106 prepares data necessary for the calculation of the target layer. Specifically, the calculator 106 sets the output values ​​(results of neuron calculations) obtained in S911 as input values ​​of the target layer, and further sets weight coefficients of each node for the neuron calculations in the layer. In S913, the calculator 106 performs neuron calculations on the target layer. The number of calculations of the neuron calculations performed in S913 is the difference between the maximum number of calculations 302 that the calculator 106 can process and the number of neuron calculations remaining in the previous layer that were performed in S911 (the number of calculations carried over 701).

[0046] Now, reference is again made to FIG. 9A. In S906, the CPU 102 calculates the remaining number of calculations 703 in the target layer. The remaining number of calculations 703 calculated in S906 is applied as the carryover number of calculations 701 in the neuron calculation of the next layer.

[0047] In S907, the CPU 102 determines whether or not the neuron operations of a series of layers are completed. When the CPU 102 determines in S907 that the neuron operations of the series of layers are not completed, the process proceeds to S901. In this case, the CPU 102 executes the processes from S901 onward again for the other layers in which the neuron operations are not completed. Then, when the CPU 102 determines in S907 that the neuron operations for the series of layers are completed, it ends the series of processes shown in FIG. 9A.

[0048] By applying the above-mentioned control, even if the amount of calculations in one layer exceeds the maximum number of calculations that can be performed by the calculator 106, the calculations of a series of layers can be completed in a desired time by processing across the calculation times of multiple layers. Also, as in the above-mentioned embodiment, it is possible to control the driving frequency of the calculator 106 according to the amount of calculations for each layer of the neural network. This makes it possible to drive the calculator 106 at the minimum necessary frequency, and is expected to have the effect of suppressing excessive power consumption.

[0049] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a recording medium, and one or more processors in a computer of the system or device read and execute the program. Also, the present invention can be realized by a circuit (e.g., ASIC) for implementing one or more of the functions. [Explanation of symbols]

[0050] 100 Arithmetic and control device 106 Arithmetic unit 110 Frequency control section

Claims

1. An arithmetic and control device based on a neural network including an input layer, a plurality of intermediate layers that extract features included in an input value from a previous layer, and an output layer, A computing means for performing neuron operations for each layer; A frequency control means for controlling a drive frequency related to the neuron operation; having The frequency control means controls the driving frequency in accordance with the number of elements of the computing unit related to the neuron operation and the amount of operation for each layer of the neural network. Arithmetic and control device.

2. The frequency control means determines the drive frequency according to the amount of calculation for each layer and the calculation time allocated to the neuron calculation in the layer. The arithmetic and control device according to claim 1 .

3. The frequency control means determines the drive frequency according to an amount of calculation that the calculation means can perform during the calculation time. The arithmetic and control device according to claim 2.

4. When a neuron operation for a first layer is not completed within the allocated operation time as a result of the neuron operation performed at the drive frequency for the first layer, the frequency control means determines an amount of calculation for determining the drive frequency when a neuron operation is performed on a second layer following the first layer as a sum of an amount of calculation on the second layer and an amount of calculation on the remaining neuron operation on the first layer; the computing means executes a remaining neuron operation for the first layer during the computing time allocated for the second layer, and then executes a neuron operation for the second layer; The arithmetic and control device according to claim 2 or 3.

5. An input layer, and multiple intermediate layers that extract features contained in the input values ​​from the previous layer; An arithmetic and control device based on a neural network including an output layer, the arithmetic and control device executes the arithmetic and control device, A computation step of performing neuron computation for each layer; A frequency control step of controlling a drive frequency related to the neuron operation; Including, In the frequency control step, the driving frequency is controlled according to the number of elements of the computing unit related to the neuron operation and the amount of operation for each layer of the neural network. A method for controlling an arithmetic and control device.

6. A program for causing a computer to function as each of the means of the arithmetic and control device according to any one of claims 1 to 4.

Images