Processing Unit

Abstract

Provided is a computation processing device that can further reduce power consumption. A computation processing device 10 performs computation processing based on a convolutional neural network. A memory unit 15 includes a first storage circuit 21 and a non-volatile second storage circuit 22. A computation circuit unit 16 receives element data and load data read from the second storage circuit 22 as input and performs convolution computation. When the convolution computation is performed, a power gate control unit 17 blocks power supply to memory cells in the second storage circuit 22 that retain bits outside of an effective data length set in the computation.

Description

[Technical Field]

[0001] The present invention relates to a processing unit. [Background technology]

[0002] Processing devices that perform image recognition using convolutional neural networks, i.e., neural networks with convolutional layers, are known, and are expected to be applied to robot control, vehicle driving control, etc. Convolutional computations in convolutional neural networks, including image recognition, require a huge number of product-sum operations in which data from the input layer and intermediate layer is weighted and added using the weight data of the convolution filter. The data used in these operations is stored in memory, and is read out sequentially during the product-sum operation and sent to the computation circuit.

[0003] Patent Document 1 describes a convolutional neural network processing device that aims to improve memory access speed and reduce power consumption by incorporating different types of memory, such as SRAM, MRAM (magnetoresistive RAM), and ReRAM (resistive RAM), on a semiconductor substrate.

[0004] Patent Document 2 describes a processing device configured to read multiple pieces of data for one channel (feature plane) of an input layer or the like and weight data for a convolution filter from a memory in a convolutional neural network, and calculate multiple pieces of data for one channel of the next layer in parallel using multiple product-sum operators. Also, Non-Patent Document 1 describes an example of using a memory optimized for each layer in a convolutional neural network.

[0005] On the other hand, power gating, which cuts off the power supply to an arithmetic circuit such as a processor core to suppress leakage current, is known as a technology for reducing power consumption. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] US Patent Application Publication No. 2019 / 0363131 [Patent Document 2] U.S. Patent No. 10,078,620 [Non-patent literature]

[0007] [Non-Patent Document 1] M. Peemen, AAA Setio, B. Mesman, and H. Corporaal, “Memory-centric accelerator design for Convolutional Neural Networks,” in 2013 IEEE 31st ICCD, 2013, pp. 13-19. Summary of the Invention [Problem to be solved by the invention]

[0008] However, in a convolutional neural network processing device, the convolutional calculation process requires a huge number of product-sum operations, which has led to a problem of high power consumption. It is particularly important to reduce power consumption in end-use devices such as robots, cars, and mobile terminals. Therefore, it is desirable to further reduce power consumption in processing devices that perform calculations such as convolutional neural networks.

[0009] The present invention has been made in view of the above circumstances, and has as its object to provide a processing unit that can further reduce power consumption. [Means for solving the problem]

[0010] In order to achieve the above object, the arithmetic processing device of the present invention comprises a memory unit that holds arithmetic data related to weighting calculations and at least a portion of which is a non-volatile storage area, an arithmetic circuit unit that performs arithmetic processing including the weighting calculations using some or all of the arithmetic data input from the memory unit, and a power gate unit that, during arithmetic processing, cuts off the supply of power to some or all memory cells within the non-volatile storage area other than memory cells that store some or all of the arithmetic data that are input to the arithmetic circuit unit during the arithmetic processing. [Effects of the Invention]

[0011] According to the present invention, for memory cells in a non-volatile memory area provided in a memory unit that holds calculation data, during calculation processing, power supply is cut off to some or all of the memory cells other than those that store some or all of the calculation data to be input to the calculation circuit unit during the calculation processing, thereby making it possible to reduce power consumption. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 2 is a block diagram showing an outline of a processing unit; [Figure 2] FIG. 1 is an explanatory diagram illustrating an example of connected layers of a convolutional neural network. [Figure 3] FIG. 4 is a block diagram showing a configuration of a second memory circuit. [Figure 4] 10 is a flowchart showing control of an effective data length. [Figure 5] FIG. 10 is an explanatory diagram showing a power gating state of the second memory circuit. [Figure 6] FIG. 1 is a block diagram showing a configuration of a product-sum operation circuit having a power-gated register; [Figure 7] 10 is a block diagram showing a product-sum operation circuit configured with cell macros to which a signal specifying a valid data length is input; FIG. [Figure 8] FIG. 10 is a block diagram showing a product-sum operation circuit configured with four product-sum operation units with short input data length. [Figure 9] FIG. 10 is a block diagram showing an outline of a processing device according to a second embodiment. [Figure 10] FIG. 10 is a block diagram showing a second memory circuit according to a second embodiment. [Figure 11] FIG. 10 is an explanatory diagram showing the input state of data to a product-sum operation circuit; [Figure 12] FIG. 2 is an explanatory diagram showing a state of pixel-parallel convolution processing by the arithmetic processing device; [Figure 13] FIG. 10 is an explanatory diagram showing the state of channel-parallel convolution calculation processing; [Figure 14] FIG. 11 is an explanatory diagram showing a state of convolution calculation processing in a first mode according to the third embodiment. [Figure 15] FIG. 11 is an explanatory diagram showing a state of convolution calculation processing in a second mode according to the third embodiment. [Figure 16] FIG. 10 is an explanatory diagram showing the state of a convolution operation process in a first channel-parallel mode. DETAILED DESCRIPTION OF THE INVENTION

[0013] [First embodiment] 1, a processing device 10 performs arithmetic processing based on a convolutional neural network. The processing device 10 includes a convolution processing unit 12 that performs convolutional arithmetic processing using a convolution filter on a channel (also referred to as a feature plane), a pooling unit 13 that performs pooling processing, and the like. The processing device 10 also has a configuration that performs various processes of a convolutional neural network, such as normalization processing and activation processing, but only the convolution processing unit 12 and the pooling unit 13 are depicted in FIG.

[0014] The convolution processing unit 12 corresponds to a neural network in which multiple layers are connected. Each layer has one or more channels. The first layer is an input layer, which is, for example, an image made up of RGB channels. In the neural network shown in FIG. 2 as an example, first to third layers are connected. The first layer has three channels ch1-1 to ch1-3, the second layer has four channels ch2-1 to ch2-4, and the third layer has two channels ch3-1 to ch3-2. There may be two layers or four or more layers. Each layer may have one or more channels. The number of channels may increase or decrease between previous and next layers, or the number of channels may remain the same.

[0015] A channel is composed of a plurality of element data arranged two-dimensionally. The size of each channel, i.e., the number of element data in the row and column directions, is arbitrary and is not particularly limited. Generally, the lower hierarchical levels on the input side have a smaller number of channels and larger channel sizes, while the higher hierarchical levels where processing is more advanced have a larger number of channels and smaller channel sizes. Note that in this example, a two-dimensional channel will be described, but one-dimensional or three- or more-dimensional channels may also be used.

[0016] The convolution processing unit 12 generates the (n+1)th layer by performing a convolution operation process using a convolution filter on the channel of the nth layer, where n is an integer equal to or greater than 1. Generating a layer involves generating each channel that constitutes the layer, and generating a channel involves calculating each element data that constitutes the channel through a convolution operation. A convolution filter is a two-dimensional array of weight data that serves as a weight for the element data, and in this example, one convolution filter is composed of 3 x 3 (3 rows and 3 columns) pieces of weight data. Each piece of weight data in the convolution filter is set to a value depending on the purpose of the convolution filter, etc. In the following description, the nth layer will be referred to as the previous layer relative to the (n+1)th layer, and the (n+1)th layer will be referred to as the next layer relative to the nth layer. Therefore, a channel of the next layer is generated by performing a convolution operation on a channel of the previous layer.

[0017] The convolution processing unit 12 generates each channel of the next layer by performing a convolution operation on each channel of the previous layer, and uses all channels of the previous layer for each channel of the next layer. The convolution processing unit 12 also performs the convolution operation using a convolution filter corresponding to the combination of a channel of the previous layer and a channel of the next layer.

[0018] 2, for example, when generating channel ch3-1 in the third layer, if a convolution operation is applied to channel ch2-1, a convolution filter corresponding to the combination of channel ch2-1 and channel ch3-1 is used, and when a convolution operation is performed on channel ch2-2, a convolution filter corresponding to the combination of channel ch2-2 and channel ch3-1 is used. In this way, when generating channel ch3-1, the convolution operation is performed using four convolution filters corresponding to the four combinations of channel ch3-1 and channels ch2-1 to ch2-4. When generating channel ch3-2, the convolution operation is performed using four convolution filters corresponding to the four combinations of channel ch3-2 and channels ch2-1 to ch2-4.

[0019] Note that any number of channels in the previous layer can be used to generate one channel in the next layer, and one channel in the previous layer can be used to generate one channel in the next layer. Also, all or some of the multiple convolution filters used in one layer may have a common weight array. Furthermore, when the weight arrays of the convolution filters are common, one convolution filter with that common weight array may be prepared and used to calculate multiple channels.

[0020] The pooling unit 13 performs pooling on each channel of the final layer obtained by the convolution operation processing of the convolution processing unit 12, thereby generating channels with reduced sizes in the row and column directions. In this example, the pooling unit 13 is configured to perform maximum value pooling. In maximum value pooling, for each channel, the channel is divided into multiple pooling sub-regions of 2 rows and 2 columns so that the sub-regions do not overlap with each other, and element data of the maximum value within the sub-region is output for each of these pooling sub-regions. Note that the size of the pooling sub-regions is not limited to 2 rows and 2 columns. Furthermore, the multiple pooling sub-regions can be divided so that they overlap with each other. Furthermore, instead of maximum value pooling, average value pooling may be performed, in which the average value of the element data of the pooling sub-regions is output. The convolution processing unit 12 can further perform convolution operation processing on the layer consisting of channels reduced by the pooling unit 13.

[0021] The convolution processing unit 12 includes a memory unit 15, an arithmetic circuit unit 16, a power gate control unit 17, and a controller 18 that controls these units in an integrated manner. The memory unit 15 includes a first storage circuit 21 and a second storage circuit 22. The first storage circuit 21 is configured, for example, with a DRAM. This first storage circuit 21 stores element data of each channel of the hierarchical level to which the convolution operation is applied, i.e., the previous hierarchical level, and also stores element data of each channel of the next hierarchical level calculated by the convolution operation process.

[0022] The second memory circuit 22 is a nonvolatile memory that retains data (bits) even when power is cut off and allows data to be read and written by powering on. It is provided as a nonvolatile memory area of ​​the memory unit 15. The second memory circuit 22 has a faster access speed than the first memory circuit 21. It is preferable to use a second memory circuit 22 that is physically closer to the arithmetic circuit unit 16 than the first memory circuit 21 (i.e., has a shorter wiring length (distance)). The second memory circuit 22 is preferably a memory in which each memory cell is composed of an MRAM (magnetoresistive random access memory). Compared to other nonvolatile memories, MRAM has a smaller cell size, which is advantageous for convolutional neural networks that require high-density integration of memory cells. Furthermore, its low operating voltage is advantageous for reducing power consumption. The entire memory unit 15 may be a nonvolatile memory area.

[0023] For example, the second storage circuit 22 holds all the weight data of each convolution filter used in the convolution calculation process of each layer. Furthermore, when calculating the channels of one layer through the convolution calculation process, the element data used in the convolution calculation process of that layer (hereinafter referred to as the convolution calculation process of the target layer) is read from the first storage circuit 21 prior to the convolution calculation process of the target layer and written to and held in the second storage circuit 22. The convolution processing unit 12 performs the convolution calculation process using the element data and weight data read from the second storage circuit 22. In this example, the element data and weight data are each calculation data.

[0024] The weight data used in the convolution calculation process of the target layer may be read from the first storage circuitry 21 and written to the second storage circuitry 22. If the size of each storage area allocated to the element data and weight data used in the convolution calculation process of the target layer is small compared to the total number of pieces of element data and weight data, the element data and weight data stored in the second storage circuitry 22 may be updated to different element data and weight data read from the first storage circuitry 21 depending on the progress of the calculation, etc.

[0025] Where N is an integer of 2 or greater, the element data and load data have a fixed data length of N bits, but in this example, the effective data length (number of upper bits) for the element data and load data is determined in advance for each layer depending on, for example, the required calculation precision. If M is an integer of 2 or greater and the effective data length is M bits, then "N≧M" holds, and the effective data length of the element data and load data is the same as or shorter than the original data length.

[0026] The effective data length can be changed based on a pre-set calculation accuracy depending on, for example, an application executed using the arithmetic processing device 10 or an operation mode of a device equipped with the arithmetic processing device 10 (for example, an operation mode according to the remaining battery level, an operation mode requiring different recognition accuracy in processing of a convolutional neural network, etc.). The effective data length may be different or the same between layers.

[0027] In this example, the element data and weight data multiplied in the convolution operation when calculating the channel of one layer are described as having the same effective data length, but they may be different. Also, this example describes a case where the effective data lengths of the element data and weight data are the same, but there are layers where their effective data lengths are different.

[0028] In this example, the arithmetic circuit unit 16 has a plurality of product-sum calculation circuits 24 arranged in parallel, and the plurality of product-sum calculation circuits 24 perform convolution calculations, which are product-sum calculations, in parallel. The product-sum calculations performed by the product-sum calculation circuits 24 include a weighting calculation in which corresponding data is multiplied between element data and weight data, and an operation in which the results of the weighting calculations are added together. One product-sum calculation circuit 24 calculates one piece of element data for one channel by a convolution calculation.

[0029] The arithmetic circuit unit 16 is provided with, for example, 64 product-sum calculation circuits 24, which calculate up to 64 pieces of element data in parallel. The product-sum calculation circuits 24 are set with an effective data length determined in advance by the controller 18, change the data length to be set as the calculation target, and perform calculation processing on the element data and weight data portions of the set effective data length. Specifically, the calculation is performed by regarding the M bits of upper bits (part of the data) indicated by the effective data length as the data to be calculated.

[0030] There is no particular limitation on the configuration of the arithmetic circuit unit 16 and the product-sum arithmetic circuit 24. For example, the product-sum arithmetic circuit 24 may be provided with multipliers in the same number as the weight data of one convolution filter and an adder that adds up the multiplication results of these multipliers, and each product-sum arithmetic circuit 24 may multiply each weight data of one convolution filter by the corresponding element data in parallel, and the multiplication results may be added up by the adder.

[0031] If the effective data lengths of the element data and the load data are different, the effective data lengths of these data may be set in the sum-of-products operation circuit 24. The operation circuit unit 16 is also provided with a bit number adjustment circuit (not shown), which converts the data length of the operation results (element data) output from each sum-of-products operation circuit 24 into a predetermined data length before writing the data into the first storage circuit 21.

[0032] As will be described in detail later, the power gate control unit 17 controls the power supply to each memory block constituting the second memory circuit 22, i.e., controls power gating, under the control of the controller 18. The effective data length is set by the controller 18, and the power gate control unit 17 controls power gating based on the set effective data length.

[0033] The controller 18 acquires the effective data length for each layer. The effective data length acquired by the controller 18 is determined, for example, by the operation mode of the device incorporating the arithmetic processing device 10 as described above. When performing a convolution operation, the controller 18 sets the acquired effective data length in the power gate control unit 17 and also in each product-sum operation circuit 24. In this example, the controller 18 is a data length setting unit.

[0034] 3, the second memory circuit 22 is provided with a memory array in which a plurality of nonvolatile memory cells 31 are arranged in a matrix, and this memory array is divided into a plurality of memory blocks MB. In this example, one memory block MB is composed of N memory cells 31 arranged in a single row in the horizontal direction in the figure, and each memory block MB holds one piece of data of up to N bits. The number of memory cells 31 constituting a memory block MB may be determined according to the maximum data length of the data to be held.

[0035] Each memory cell 31 of the same bit in each memory block MB is connected to a common power supply line, and a drive voltage (VDD) is applied via the power supply line. That is, for example, each memory cell 31 of the first bit in each memory block MB is connected to the power supply line corresponding to the first bit, and each memory cell 31 of the second bit is connected to the power supply line corresponding to the second bit.

[0036] The second memory circuit 22 is provided with a PG switch 33 which, together with the power gate control unit 17, constitutes a power gate unit. The PG switch 33 is made up of a plurality of PG switches 33a provided on each power supply line connected to the memory cells 31. That is, a PG switch 33a is provided corresponding to the memory cell 31 of each bit in each memory block MB. The PG switch 33a is made up of a MOS transistor or the like, and is controlled to be turned on and off by the power gate control unit 17. The power gate control unit 17 can independently control the on and off of each PG switch 33a.

[0037] When one PG switch 33a is turned on, a drive voltage is applied to each memory cell 31 connected to that PG switch 33a via a power line, and those memory cells 31 receive power, enabling bit writing and reading. When one PG switch 33a is turned off, the drive voltage is no longer applied to each memory cell 31 connected to that PG switch 33a via a power line, and power supply to those memory cells 31 is cut off. This allows power gating to be performed on memory cells 31 that store one or more specific bits in each memory block MB. Writing and reading of bits in memory cells 31 to which power supply has been cut off becomes impossible.

[0038] When performing convolutional operation processing, if the effective data length is set by the controller 18, the power gate control unit 17 turns on the PG switches 33a corresponding to the most significant M bits indicated in the effective data length of the memory block MB and turns off the other PG switches 33a. This cuts off the power supply to the memory cells 31 that hold bits other than the most significant bits for the number of bits used in the convolutional operation processing of the target layer, thereby reducing power consumption. Note that when writing data to the second memory circuit 22, the power gate control unit 17 turns on all the PG switches 33a, enabling element data and weight data to be written to each memory block MB at their original data lengths.

[0039] When the arithmetic processing device 10 performs arithmetic processing based on a convolutional neural network, the controller 18 determines whether the arithmetic processing is a convolutional arithmetic processing, as shown in Fig. 4. If the arithmetic processing is a convolutional arithmetic processing, the controller 18 acquires an effective data length corresponding to the layer to be calculated, and sets the effective data length in the power gate control unit 17. Prior to this setting, the element data to be used in the current convolutional arithmetic processing is read from the first storage circuit 21 and written to and held in the second storage circuit 22.

[0040] When the effective data length is set, the power gate control unit 17 turns on the PG switches 33 for the most significant bits indicated by the set effective data length and turns off the other PG switches 33. As a result, in each memory block MB of the second storage circuit 22, power is supplied to the memory cells 31 for the effective data length, and power supply to the other memory cells 31 is cut off. Therefore, power supply is cut off to the memory cells 31 that hold data portions (lower bits) of the element data and load data stored in the second storage circuit 22 that are not used in the convolution operation, including the element data and load data used in the convolution operation processing of the target layer. This reduces power consumption while enabling reading of the data portions of the effective data length of the element data and load data used in the operation.

[0041] 5 shows an example in which the original data length of the element data of the previous layer and the weight data of the convolution filter applied thereto is 8 bits (N=8), when the effective data length is 8 bits (M=8) and the element data of the next layer is calculated by convolution operation processing, power is supplied (ON) to all memory cells 31 for 8 bits in each memory block MB of the second storage circuit 22, as shown in FIG. 5(A). In this case, 8-bit element data and weight data are each read out from the second storage circuit 22 and sent to the operation circuit unit 16.

[0042] 5B, when calculating element data of the next layer with an effective data length of 4 bits (M=4), power is supplied (ON) to the memory cells 31 of the most significant 4 bits and power is cut off (OFF) to the memory cells 31 of the least significant 4 bits for each memory block MB of the second storage circuit 22. In this case, the most significant 4 bits of the element data and weight data corresponding to the effective data length are read from the second storage circuit 22 and input to the arithmetic circuit unit 16 as 4-bit element data and weight data for use in the calculation.

[0043] 5(C), when element data of the next layer is calculated with an effective data length of 6 bits (M=6), each memory block MB supplies power to the memory cells 31 of the most significant 6 bits (ON) and cuts off power supply to the memory cells 31 of the least significant 2 bits (OFF). In this case, the most significant 6 bits of the element data and weight data corresponding to the effective data length are read from the second memory circuit 22 and input to the arithmetic circuit unit 16 as 6-bit element data and weight data for use in arithmetic. The memory cells 31 corresponding to the least significant 2 bits of the most significant 6 bits of the weight data to be read this time have their power supply cut off when the effective data length was 4 bits. However, since the memory cells 31 are nonvolatile, the 2 bits of data are not lost. Therefore, the most significant 6 bits of the 8-bit weight data are read from the second memory circuit 22, and the arithmetic circuit unit 16 correctly performs arithmetic with the effective data length.

[0044] It is also possible to provide a PG switch that controls the application of drive voltage on a memory block basis, and cut off the power supply to each memory cell 31 of a memory block MB that does not store load data or element data, or that stores load data or element data that is unrelated to the hierarchical level that performs the convolution operation. In other words, during the convolution operation in the arithmetic circuit unit 16, it is possible to cut off the power supply to each memory cell 31 of memory blocks MB other than the memory block MB that stores the element data or load data that is provided to the arithmetic circuit unit 16 in the convolution operation.

[0045] 6, power gating may be performed on a register 34 that constitutes the product-sum operation circuit 24. In this example, the product-sum operation circuit 24 has a register 34, a multiplier 35, and an adder 36. The multiplier 35 multiplies element data by weight data, and the adder 36 adds the data held in the register 34 to the multiplication result of the multiplier 35. The register 34 holds the addition result of the adder 36. The register 34 has a plurality of bit cells 34a, each storing one bit, and the number of bit cells 34a corresponds to the maximum data length of the operation result by the product-sum operation circuit 24; for example, 24 bit cells 34a are provided corresponding to 24 bits.

[0046] The product-sum operation circuit 24 is provided with a switch unit 38. Like the PG switch 33 of the second memory circuit 22, the switch unit 38 forms a power gate unit together with the power gate control unit 17. The switch unit 38 is made up of a plurality of PG switches 38a provided corresponding to each of the bit cells 34a, and each PG switch 38a is provided on a power line connected to the corresponding bit cell 34a. Each PG switch 38a is made up of a MOS transistor or the like, and is independently turned on and off by the power gate control unit 17.

[0047] When a PG switch 38a is turned on, a drive voltage is applied to the bit cell 34a corresponding to that PG switch 38a, and that bit cell 34a receives a power supply, enabling writing and reading of bits. When a PG switch 38a is turned off, the drive voltage is no longer applied to the bit cell 34a connected to that PG switch 38a via the power line, cutting off the power supply to the bit cell 34a and disabling writing and reading of bits in that bit cell 34a. This allows power gating to be performed on one or more specific bit cells 34a in each register 34.

[0048] During convolution processing, the controller 18 sets the data length of the calculation result calculated by the product-sum calculation circuit 24 in the convolution processing as the effective data length in the power gate control unit 17. The effective data length of this calculation result is determined by the number of channels in the previous layer, the filter size of the convolution filter, and the effective data lengths of the element data and weight data, and can be known before the calculation. The power gate control unit 17 turns on the PG switches 38a corresponding to the bit width of the number of bits indicated by the effective data length of the calculation result, and turns off the other PG switches 38a. As a result, during the convolution processing, power supply is cut off to bit cells 34a in an area exceeding the data length of the calculation result calculated by the product-sum calculation circuit 24 in the convolution processing (unused bit cells 34a other than those with the effective data length), thereby reducing power consumption.

[0049] It should be noted that power gating can be similarly performed on bit cells of registers provided in the sum-of-products operation circuit 24 for holding multiplication results and addition results, and registers for holding data input to the sum-of-products operation circuit 24.

[0050] When the sum-of-products operation circuit 24 of the arithmetic circuit unit 16 is configured with an ASIC cell macro, as shown in FIG. 7, a signal specifying the effective data length may be input to the sum-of-products operation circuit 24, and the length of the data to be operated on may be changed according to the effective data length. Alternatively, as shown in FIG. 8, the sum-of-products operation circuit 24 may be divided into multiple sum-of-products operation units 39 each having an input data length shorter than the maximum input data length of the data input thereto, and the output data of each unit may be appropriately combined (added) with data obtained by bit-shifting the output data according to the effective data length to obtain the operation result. In the example shown in FIG. 8, the 16×16-bit sum-of-products operation circuit 24 is configured with four sum-of-products operation units 39 each having an input data length of 8 bits. This configuration also enables operations with a higher degree of parallelism in the sum-of-products operation when the data length to be operated on is shorter. That is, the sum-of-products operation circuit 24 can be operated as one 16×16-bit sum-of-products processor, or can be operated in parallel as two 16×8-bit sum-of-products processors or four 8×8-bit sum-of-products processors.

[0051] In the above, a part of the memory unit (the second memory circuit) is a nonvolatile memory, but the entire memory unit may be a nonvolatile memory.

[0052] [Second embodiment] In the second embodiment, power consumption is effectively reduced by efficiently using a convolution filter stored in a power-gated non-volatile memory area to perform convolution calculations. The basic calculations based on a convolutional neural network performed by a calculation processing device in the second embodiment are the same as those in the first embodiment, and the names and terms of data, etc., are the same as those in the first embodiment, and detailed descriptions thereof are omitted. Components that are substantially the same as those in the first embodiment are denoted by the same reference numerals, and detailed descriptions thereof are omitted.

[0053] 9, the arithmetic processing device 50 includes a convolution processing unit 52, a pooling unit 13, etc. In addition, the arithmetic processing device 50 includes normalization processing, activation processing (not shown), etc. The convolution processing unit 52 includes a memory unit 15, an arithmetic circuit unit 56, a power gate control unit 57, and a controller 58 that controls these units in an integrated manner.

[0054] The memory unit 15 has a first memory circuit 21 and a second memory circuit 62. The first memory circuit 21 is a large-capacity memory configured, for example, with a DRAM or the like. This first memory circuit 21 stores element data of each channel of the hierarchical layer to which the convolution operation is applied, i.e., the previous hierarchical layer, and also stores element data of each channel of the next hierarchical layer calculated by the convolution operation. Furthermore, this first memory circuit 21 stores weight data of each convolution filter used in the convolution operation processing of each hierarchical layer.

[0055] The second storage circuit 62 of the memory unit 15 is similar to the second storage circuit 22 (see FIG. 1) of the first embodiment, except for the unit of power gating, as will be described later. The weight data of the convolution filters read from the first storage circuit 21 is written to this second storage circuit 62 so as to hold the weight data of the multiple convolution filters used in the convolution operation processing of the target layer. When the memory size of the second storage circuit 62 is small compared to the total number of weight data used in the convolution operation of one layer, the weight data of the used convolution filter stored in the second storage circuit 62 is updated to the weight data of another convolution filter read from the first storage circuit 21, for example, depending on the progress of the operation. In this example, the convolution filter has 3 rows and 3 columns, but this is not limited to this.

[0056] The arithmetic circuit unit 56 performs convolution calculation processing using the element data read out from the first memory circuit 21 and the weight data read out from the second memory circuit 62. This convolution processing unit 52 is composed of a plurality of, for example, 64 product-sum calculation circuits 61, and each product-sum calculation circuit 61 is composed of nine multipliers 61a arranged in parallel, an adder 61b, a register 61c that holds the addition results of the adders, etc. The adder 61b adds each multiplication result of the multiplier 61a to the data held in the register 61c.

[0057] As in the first embodiment, the element data may be read from the first memory circuit 21 and written to the second memory circuit 62, and the element data read from the second memory circuit 62 and the load data may be used to perform a convolution operation.

[0058] Focusing on one product-sum operation circuit 61, the product-sum operation circuit 61 uses nine multipliers 61a to multiply in parallel nine pieces of element data in a small region corresponding to a convolution filter in one channel of the previous layer by nine pieces of weight data of the convolution filter. The multiplication results of these multipliers 61a are added by an adder 61b to calculate data obtained by convolving the element data in the small region corresponding to the convolution filter in one channel of the previous layer with the convolution filter. While sequentially switching channels in the previous layer, the adder 61b sequentially adds data obtained by convolving element data in small regions at the same positions (coordinates) of each channel with the convolution filter and convolved data of other channels held in register 61c to calculate one piece of element data in the next layer. As described above, a convolution filter corresponding to the combination of a channel in the previous layer and a channel in the next layer is used, so when generating one channel in the next layer, the convolution filter is also switched each time the channel in the previous layer is switched.

[0059] The product-sum operation circuit 61 is provided with, for example, a register (not shown) for holding input data, and the input element data and weight data are held in the register, and the multiplier 61a multiplies the element data and weight data held in the register. The element data held in the register is held until the next element data is input, and the weight data held in the register is held until the next weight data is input.

[0060] The arithmetic circuit unit 56 operates multiple product-sum calculation circuits 61 in parallel to calculate element data as described above. Each product-sum calculation circuit 61 is controlled so that element data of the same channel in the previous layer is input to it, and so that weight data of the same convolution filter is input to it. Through this control, the arithmetic circuit unit 56 calculates multiple (64 in this example) element data in one channel in the next layer in parallel.

[0061] Each channel in the previous layer is divided into multiple small regions of the same size as the convolution filter, with the center of each region being the position of the element data to be calculated. Each small region partially overlaps with the other small regions. The number of small regions in the channel in the previous layer is set to the same number of element data in the next layer. Each element data of one small region is input to one product-sum operation circuit 61, and element data of up to 64 small regions in one channel are input in parallel to each product-sum operation circuit 61 in the operation circuit unit 56.

[0062] The above-described manner in which a plurality of element (pixel) data of one channel in the next layer is calculated in parallel using a common convolution filter is called pixel parallelism.

[0063] The power gate control unit 57 controls the power gating of each memory block MB constituting the second memory circuit 62 under the control of the controller 58, as will be described in detail later.

[0064] 10, the second memory circuit 62 has a plurality of memory blocks MB, each consisting of N memory cells 31. Each memory block MB stores load data. Each memory block MB is connected to a corresponding power supply line, and a drive voltage (VDD) is applied via the power supply line. That is, each memory cell 31 in each memory block MB is connected to the power supply line corresponding to that memory block MB.

[0065] The second memory circuit 62 is provided with a switch section 63 that, together with the power gate control section 57, constitutes a power gate section. The switch section 63 is made up of a plurality of PG switches 63a provided on each power line. That is, a PG switch 63a is provided corresponding to each memory block MB. The PG switches 63a are made up of MOS transistors or the like, and are independently controlled to be turned on or off by the power gate control section 57.

[0066] With the above configuration, when one PG switch 63a is turned on, the corresponding memory block MB receives power, enabling data to be written to and read from it. When one PG switch 63a is turned off, power to the memory block MB corresponding to that PG switch 63a is cut off. This allows power gating to be performed for each memory block MB. Data cannot be written to or read from a memory block MB whose power supply has been cut off.

[0067] When each product-sum operation circuit 61 performs a convolution operation, a block designation signal is input from the controller 58 to the power gate control unit 57. The block designation signal designates a memory block MB holding weight data to be used in the convolution operation, i.e., a memory block MB holding weight data to be read from the second memory circuit 62 for input to the operation circuit unit 56. The power gate control unit 57 turns on the PG switch 63a corresponding to the memory block MB designated by the block designation signal. This enables reading of the memory block MB to be used in the convolution operation. Each product-sum operation circuit 61 multiplies corresponding weight data and element data in parallel and adds the multiplication results. In this example, one convolution filter is composed of nine weight data, so nine memory block MBs can be read for one convolution filter.

[0068] In this example, when each product-sum operation circuit 61 performs a convolution operation, one common convolution filter is used. Therefore, as shown in FIG. 11, nine pieces of element data are input to each product-sum operation circuit 61 of the operation circuit unit 56, and nine pieces of weight data of one convolution filter read from nine memory blocks MB to which power is supplied from the second storage circuit 62 are input.

[0069] On the other hand, when performing a convolution operation, the power gate control unit 57 turns off the PG switch 63a corresponding to the memory block MB that holds the load data not used in the convolution operation, i.e., the memory block MB from which the load data is not read. This cuts off the power supply to the memory block MB not used in the convolution operation, thereby reducing power consumption. Note that when writing data to the second memory circuit 62, the power gate control unit 57 turns on the PG switch 63a corresponding to the memory block MB into which the data is to be written, for example.

[0070] A case will now be described in which, with the above configuration, each channel of the next layer is generated from each channel of the previous layer, as shown in an example in Fig. 12. First, weight data for a plurality of convolution filters for generating each channel of the next layer to be used in convolution calculation processing is read from the first storage circuit 21 and written to the second storage circuit 62. After this, the channels of the next layer are generated one by one.

[0071] For example, when generating the first channel Ch2-1 of the next hierarchy, first, each weight data of the convolution filter F11-21 corresponding to the combination of the first channel Ch1-1 and channel Ch2-1 of the previous hierarchy is read from the second storage circuit 62. Therefore, prior to this reading, the PG switch 63a corresponding to each memory block MB holding each weight data of the convolution filter F11-21 is turned on, and the other PG switches 63a are kept off. After this, nine weight data are read from each memory block MB corresponding to the turned-on PG switch 63a, and are set in each product-sum calculation circuit 61, i.e., held in the register of each multiplier 61a.

[0072] After the weight data is set, 64 small regions, the same number as the number of product-sum calculation circuits 61, are selected from the multiple small regions of the first channel Ch1-1 of the previous layer, and element data of each selected small region is sent to the calculation circuit unit 56. As described above, each element data of one small region is input to one product-sum calculation circuit 61, and the element data of each selected small region is input to each product-sum calculation circuit 61 and held in a register. Thereafter, each product-sum calculation circuit 61 calculates data by convolving the element data of the small region with a convolution filter F11-21 using the nine element data of channel Ch1-1 and the nine weight data held in the register (FIG. 12(A)). This convolved data is held in register 61c.

[0073] When the convolution calculation for the 64 small regions of channel Ch1-1 is completed as described above, the target of the convolution calculation in the previous layer is shifted to channel Ch1-2 (FIG. 12(B)). Further, the PG switches 63a corresponding to each memory block MB holding each weight data of the convolution filter F12-21 corresponding to the combination of channel Ch1-2 and channel Ch2-1 are turned on, and the other PG switches 63a are turned off. Then, the nine weight data of the convolution filter F12-21 are read from the second memory circuit 62 and set in each product-sum calculation circuit 61.

[0074] Thereafter, 64 small areas are selected from the plurality of small areas in the second channel Ch1-2 in the previous layer, and element data for each selected small area is sent to the arithmetic circuit unit 56. As described above, element data for each small area is input to each product-sum operation circuit 61 and held in a register. Note that the small area selected in channel Ch1-2 is in the same position as the small area selected in channel Ch1-1, and element data for the small areas in the same position are input to the same product-sum operation circuit 61.

[0075] Then, the element data of the corresponding small region is convolved in the convolution filter F12-21 using the nine element data of channel Ch1-2 held in the register and the nine weight data of the convolution filter F12-21 by each product-sum operation circuit 61. The sum of the convolved data for channel Ch1-2 and the convolved data for channel Ch1-1 is held in the register 61c.

[0076] Thereafter, the convolution operation is performed in the same manner while switching the channels of the previous layer one by one. At this time, each time the channel of the previous layer that is the subject of the convolution operation is switched, the convolution filter is switched, and in order to switch the convolution filter, the memory block MB that is turned on and off in the second storage circuit 62 is switched.

[0077] When the convolution operation for the last channel of the previous layer (channel Ch1-64 in FIG. 12) is completed, the element data for channel Ch2-1 is held in the register 61c of each product-sum operation circuit 61. In this way, 64 element data for channel Ch2-1 are calculated in parallel, and each calculated element data is written to the first storage circuit 21.

[0078] If there is uncalculated element data for channel Ch2-1, i.e., if there are unselected small areas in each channel of the previous layer, the element data for channel Ch2-1 is calculated in parallel using the same procedure as above until there are no more unselected small areas.

[0079] After all element data for channel Ch2-1 are calculated and written to the first memory circuit 21 as described above, channel Ch2-2 is generated using a similar procedure. When generating channel Ch2-2, the PG switches 63a corresponding to each memory block MB holding each weight data of the convolution filter corresponding to the combination of channel Ch2-2 and the channel that is the target of the convolution operation on the previous layer are turned on, and the other PG switches 63a are turned off. Then, nine weight data are read from each memory block MB corresponding to the turned-on PG switches 63a and set in each product-sum operation circuit 61. In this way, a convolution operation is performed using a convolution filter corresponding to the combination of channel Ch2-2 and the channel that is the target of the convolution operation on the previous layer to generate channel Ch2-2. Thereafter, a similar procedure is used to generate channels up to the final channel on the next layer (channel Ch2-64 in FIG. 12).

[0080] When generating each channel of the next layer in pixel-parallel as described above, each product-sum operation circuit 61 performs convolution operation using a single common convolution filter. Therefore, in the second storage circuit 62, power is supplied only to the memory block MB that stores the weight data of that single convolution filter, and power supply to memory blocks MB that store the weight data of the other convolution filters can be cut off. This allows power gating to be performed on the second storage circuit 62, effectively reducing power consumption.

[0081] Furthermore, by performing the processing as described above, if the second memory circuit 62 has a storage size capable of holding the weight data of all convolution filters required to generate at least one channel, the data held in the second memory circuit 62 will not be rewritten with the weight data read from the first memory circuit 21 during the generation of one channel, and the weight data will only be rewritten when the channel of the next layer to be generated is repeatedly changed. This reduces the number of accesses to the first memory circuit 21 and the second memory circuit 62, which is advantageous for reducing power consumption and improving processing speed. This is particularly advantageous in a configuration in which the capacity of the second memory circuit 62 is small compared to the number of pieces of weight data.

[0082] In a mode in which one element data is calculated in parallel for each of a plurality of channels in the next layer (channel parallelism, described later), convolution calculations are performed simultaneously using a plurality of convolution filters corresponding to a plurality of channels in the next layer and one channel in the previous layer. Therefore, if the number of channels in the next layer increases, rewriting will occur unless the second storage circuit 62 has a storage size that can hold the weight data of all the convolution filters used to generate the next layer.

[0083] For example, data rewriting occurs in which the weight data of one set of convolution filters held in the second storage circuit 62 is rewritten with the weight data of another set of convolution filters read from the first storage circuit 21, with the same number of convolution filters as the product-sum operation circuits 61, and then the weight data of the set of convolution filters before rewriting is read from the first storage circuit 21 and written to the second storage circuit 62. However, in the arithmetic processing device 50 of this example, by performing the convolution operation process in pixel parallel as described above, such rewriting does not occur if the second storage circuit 62 has a storage size capable of holding the weight data of all the convolution filters required to generate at least one channel.

[0084] As can be seen from the above explanation, the number of element data calculated in parallel by the arithmetic circuit unit 56 is the same as the number of product-sum operation circuits 61 provided in the arithmetic circuit unit 56. Therefore, it is preferable to set the number of product-sum operation circuits 61 provided in the arithmetic circuit unit 56 to a divisor of the number of element data of the channel at the next layer. This makes it possible to eliminate idle product-sum operation circuits 61 and to use resources effectively.

[0085] Furthermore, when the pooling unit 13 performs maximum value pooling or average value pooling on a layer (next layer) generated by the convolution operation, it is preferable that the size of each channel of the layer, i.e., the number of rows and the number of columns, be a multiple of 2. This makes it possible to efficiently apply maximum value pooling or average value pooling, which processes each non-overlapping region of 2 rows and 2 columns. Note that, when p and q are each integers equal to or greater than 1, and the pooling unit 13 performs pooling on a region of p rows and q columns, it is preferable that the number of rows of each channel of the layer generated by the convolution operation be a multiple of the value p and the number of columns be a multiple of q. Note that the size of each channel of the layer generated by the convolution operation when the next pooling operation is performed is similar in other embodiments.

[0086] Table 1 shows examples of the number of times weight data is read from the first memory circuit 21 and whether power gating of the second memory circuit 62 is possible when the convolution calculation process is performed pixel-parallel as described above using the calculation circuit unit 56 and when the convolution calculation process is performed channel-parallel.

[0087] [Table 1]

[0088] In Table 1, "Layer" is the layer number of the previous layer. "Number of filters" is the number of convolution filters used in the convolution operation when generating the next layer. In the example of Table 1, a convolution filter corresponding to the channel to be generated is used regardless of the channel of the previous layer, so the number of convolution filters shown in the "Number of filters" column is the same as the number of channels of the next layer to be generated (number of output channels). "Output data size" indicates the number of element data in the row and column directions of the generated channel. For example, if the output data size is 416 x 416, each generated channel will have 416 rows and 416 columns. "Number of weight data" is the total number of weight data.

[0089] "Number of load data reads" is the number of times load data is read from the first memory circuit 21. "Gating possible" indicates whether or not the power supply can be cut off in the second memory circuit 62. The value in parentheses in the "Gating possible" column indicates the ratio of memory blocks MBs that can be power gated (power supply can be cut off) to all memory blocks MBs in the second memory circuit 62. In the example of Table 1, the second memory circuit 62 is made up of 576 memory blocks MBs, and for example, "(63 / 64)" indicates that the power supply to 567 memory blocks MBs out of the 576 memory blocks MBs can be cut off.

[0090] Channel parallelism is a mode in which one element data is calculated in parallel for each of multiple channels in the next layer, as shown in Fig. 13. Therefore, when performing convolution calculation processing in channel parallelism, element data of one small area is read from one channel in the previous layer (e.g., channel Ch1-1) and input to each product-sum calculation circuit, and weight data of different convolution filters (e.g., F11-21 to F11-264) determined by the combination of the channel in the previous layer and each channel in the next layer is input to each product-sum calculation circuit to perform convolution calculation. Then, the channel from which element data is read is switched for each convolution calculation.

[0091] As can be seen from Table 1, in the arithmetic processing device 50 employing pixel parallelism, even if the total number of weight data of the convolution filter used in the convolution calculation increases, the weight data to be read is only that of one convolution filter, so power gating is possible. On the other hand, in the case of channel parallelism, power gating becomes impossible when the total number of weight data becomes large. It can also be seen that the number of times weight data is read from the first memory circuit 21 to the second memory circuit 62 can be significantly reduced (to about 1 / 5 in the example of Table 1) by using pixel parallelism compared to channel parallelism.

[0092] In the above example, the next channel is generated each time generation of one channel in the next layer is completed, but it is also possible to calculate in parallel the same number of element data as the product-sum operation circuit 61 for the next channel in the next layer each time the same number of element data as the product-sum operation circuit 61 for one channel in the next layer is calculated in parallel. In this case, after calculating in parallel the same number of element data as the product-sum operation circuit 61 for the final channel of the next layer, the position of each small region in the channel of the previous layer is changed and element data is calculated according to the above procedure, and this is repeated until there are no unselected small regions left, to generate all channels in the next layer.

[0093] [Third embodiment] In the third embodiment, the convolution operation sequence is switched between a lower layer on the input layer side and an upper layer where processing has progressed. Note that the arithmetic processing device of the third embodiment is similar to the arithmetic processing device 50 shown in FIG. 9 of the second embodiment, except that the convolution operation sequence is switched, as will be described in detail later. For this reason, components that are substantially the same as those of the second embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0094] In the third embodiment, the arithmetic processing device 50 switches the convolution operation sequence between a lower hierarchical layer on the input layer side and an upper hierarchical layer where processing has progressed, using a controller 58 as a priority control unit. The arithmetic processing device 50 has a first mode and a second mode which have different convolution operation sequences, and the controller 58 controls each unit to achieve the sequence of each mode. Both the first mode and the second mode are pixel-parallel convolution operation processes, but differ in the type of data that is preferentially stored in the second memory circuit 62, the order in which element data are calculated, etc.

[0095] The first mode of convolution calculation processing is advantageous when the channel size is large, the number of channels is small, and the total number of weight data of each convolution filter used in the convolution calculation processing of the target layer is smaller than the total number of element data of each channel of the previous layer. The convolution processing unit 52 performs the convolution calculation processing in this first mode for the lower layer on the input layer side.

[0096] In the first mode, the second memory circuit 62 holds the weight data of each convolution filter used in the convolution operation process of the target layer in preference to the element data, and holds a predetermined number of element data of the previous layer. Therefore, the storage area of ​​the second memory circuit 62 is adjusted so that it holds more weight data than element data.

[0097] In the first mode, the arithmetic circuit unit 56 performs a convolution operation using the weight data and element data read from the second memory circuit 62. If the amount of element data in the previous layer is large compared to the size of the storage area secured in the second memory circuit 62, the arithmetic circuit unit 56 updates the used element data stored in the second memory circuit 62 with other element data read from the first memory circuit 21, depending on the progress of the operation, etc. Note that if the number of weight data used in the convolution operation process of the target layer is large compared to the size of the storage area secured in the second memory circuit 62, the weight data held in the second memory circuit 62 is updated in the same way as the element data; however, since the total number of weight data is relatively small in the lower layers, such updates rarely occur.

[0098] In the first mode, the channel for calculating element data is switched every time the same number of element data as the number of product-sum calculation circuits 61 in the calculation circuit unit 56 is calculated for one channel. In this first mode, as shown in Fig. 14(A), weight data of the convolution filter Fa1-b1 corresponding to the combination of the first channel Cha-1 on the previous layer and the first channel Chb-1 on the next layer is set in each product-sum calculation circuit 61, and then element data of the same number of small regions as the product-sum calculation circuits 61 are input to the corresponding product-sum calculation circuits 61.

[0099] Next, weight data of convolution filter Fa2-b1 corresponding to the combination of the second channel Cha-2 on the previous layer and the first channel Chb-1 on the next layer is set in each product-sum operation circuit 61, and then element data of the same number of small regions as those of the product-sum operation circuits 61 are input to the corresponding product-sum operation circuits 61. Thereafter, in the same manner, the convolution filter is switched while the channel on the previous layer is changed, and element data of the same number of small regions as those of the product-sum operation circuits 61 for each channel are sequentially input to the corresponding product-sum operation circuits 61. In this way, the same number of element data as those of the product-sum operation circuit 61, for example 64, are calculated in parallel for the first channel Chb-1 on the next layer.

[0100] 14(B), element data for the second channel Chb-2 on the next layer is calculated. First, weight data for convolution filter Fa1-b2 corresponding to the combination of the first channel Cha-1 on the previous layer and the second channel Chb-2 on the next layer is set in each product-sum operation circuit 61, and then element data for the same number of small regions as those of the product-sum operation circuits 61 are input to the corresponding product-sum operation circuits 61. Next, weight data for convolution filter Fa2-b2 corresponding to the combination of the second channel Cha-2 on the previous layer and the second channel Chb-2 on the next layer is set in each product-sum operation circuit 61, and then element data for the same number of small regions as those of the product-sum operation circuits 61 are input to the corresponding product-sum operation circuits 61. Thereafter, in a similar manner, the convolution filter is switched while the channel on the previous layer is changed, and the element data for the same number of small regions as those of the product-sum operation circuits 61 for each channel are sequentially input to the corresponding product-sum operation circuits 61. As a result, 64 element data are calculated in parallel for the second channel Chb-2 in the next layer.

[0101] By repeating the same procedure as above, 64 element data are calculated for each channel up to the final channel of the next layer.

[0102] After calculating the 64 element data for the final channel of the next layer, the positions of the multiple small regions in the channel of the previous layer are changed and the above procedure is repeated to calculate all element data for all channels of the next layer.

[0103] As described above, in the first mode, the position of the calculated element data is changed each time a number of element data corresponding to each product-sum calculation circuit 61 is calculated for all channels in the next hierarchy, and finally all element data are calculated for all channels in the next hierarchy.

[0104] On the other hand, the second mode of convolution calculation is advantageous when the channel size is small, the number of channels is large, and the total number of weight data of each convolution filter used in the convolution calculation of the target layer is greater than the total number of element data of the previous layer used in the convolution calculation of the target layer. The convolution processing unit 52 performs the convolution calculation of the upper layer in this second mode.

[0105] In the second mode, the second memory circuit 62 holds element data of the previous layer used in the convolution operation processing of the target layer in preference to weight data of the convolution filter, and holds a predetermined number of weight data of the convolution filter used in the convolution operation processing of the target layer. Therefore, the storage area of ​​the second memory circuit 62 is adjusted so that it holds more element data than weight data.

[0106] In this second mode, too, the arithmetic circuit unit 56 performs convolution operations using the weight data and element data read from the second memory circuit 62. As in the first mode, when there is a large amount of element data and weight data in the previous layer compared to the size of the storage area secured in the second memory circuit 62, the data is updated to the data read from the first memory circuit 21. In the upper layers, the total number of element data is relatively small, so updates are not frequent.

[0107] The second mode differs from the first mode in that the next channel is generated pixel-parallel each time generation of one channel is completed. That is, in the second mode, as shown in FIG. 15A, weight data of a convolution filter Fc1-d1 corresponding to a combination of the first channel Chc-1 on the previous layer and the first channel Chd-1 on the next layer is set in each product-sum operation circuit 61, and then element data of the same number of small regions as the product-sum operation circuit 61 is input to the corresponding product-sum operation circuit 61. Next, weight data of a convolution filter Fc2-d1 corresponding to a combination of the second channel Chc-2 on the previous layer and the first channel Chd-1 on the next layer is set in each product-sum operation circuit 61, and then element data of the same number of small regions as the product-sum operation circuit 61 is input to the corresponding product-sum operation circuit 61. Similarly, the convolution filter is switched while the channel on the previous layer is changed, and the element data of the same number of small regions as the product-sum operation circuit 61 for each channel is input sequentially to the corresponding product-sum operation circuit 61. As a result, for the first channel Chd-1 on the next layer, the same number of element data as in the product-sum calculation circuit 61 is calculated in parallel. Therefore, this initial calculation operation is the same as in the first mode.

[0108] 15(B), each small region in the channel of the previous layer is changed to a different one from the previous one. Weight data of convolution filter Fc1-d1 corresponding to the combination of the first channel Chc-1 on the previous layer and the first channel Chd-1 on the next layer is set in each product-sum operation circuit 61, and then element data of each small region in a position different from the previous one is input to the corresponding product-sum operation circuit 61. Next, weight data of convolution filter Fc2-d1 corresponding to the combination of the second channel Chc-2 on the previous layer and the first channel Chd-1 on the next layer is set in each product-sum operation circuit 61, and then element data of each small region is input to the corresponding product-sum operation circuit 61.

[0109] Thereafter, in a similar manner, the convolution filter is switched while the channel of the previous layer is changed, and the element data of each small region for each channel of the previous layer is input sequentially to the corresponding product-sum operation circuit 61, and the same number of element data as for the first channel Chd-1 of the next layer is further calculated in parallel. By repeating the above procedure, all of the element data of the first channel Chd-1 of the next layer is calculated.

[0110] After the generation of the first channel Chd-1 on the next layer is completed in the above manner, each element data of the second channel Chd-2 on the next layer is calculated in a similar manner. At this time, weight data of the convolution filter corresponding to the combination of each channel on the previous layer that is the subject of the convolution calculation and the channel Chd-2 on the next layer to be generated is set in each product-sum calculation circuit 61. After the generation of each channel on the second channel Chd-2 on the next layer is completed, each channel on the third layer and beyond is generated in a similar manner.

[0111] The controller 58 sets the first mode during convolution calculation processing in which the previous layer is before a predetermined switching layer, and sets the second mode during convolution calculation processing in which the previous layer is after the switching layer. The switching layer that switches between the first mode and the second mode should preferably be the layer in which, when the convolution calculation processing is repeated, the total number of element data used in the convolution calculation processing of that layer first becomes larger than the total number of load data used in the convolution calculation processing to generate one layer.

[0112] In both the first mode and the second mode, when a convolution operation is performed, power gating is performed to cut off the power supply to memory blocks MB of the second storage circuit 62 that store data that is not used in the convolution operation. The power supply is cut off not only to memory blocks MB that store load data that is not used in the convolution operation, but also to memory blocks MB that store element data that is not used in the convolution operation. This reduces the power consumption of the arithmetic processing device 50.

[0113] Furthermore, by performing the convolution operation processing in the first mode in the lower hierarchical layers and in the second mode in the upper hierarchical layers as described above and switching the data to be preferentially stored in the second storage circuit 62, repeated reads from the second storage circuit 62 are increased, reads from the first storage circuit 21 to the second storage circuit 62 are effectively reduced, and the convolution operation processing is performed efficiently. In addition, delays due to simultaneous reads of both load data and element data to the first storage circuit 21 are avoided.

[0114] In the above example, the convolution calculation is performed in pixel parallelism in both the lower and upper layers, but it is also preferable that the convolution calculation in the first mode in the lower layer be performed in channel parallelism. In this case, in the first mode, as in the pixel-parallel first mode, the second storage circuit 62 stores the weight data of each convolution filter used in the convolution calculation in the target layer in preference to the element data, and stores a predetermined number of element data in the previous layer.

[0115] 16A, in the first channel parallel mode, first, the weight data of one convolution filter is set in one product-sum operation circuit 61, and the weight data of convolution filters Fa1-b1, Fa1-b2, ... corresponding to the first channel Cha-1 on the previous layer and the 1st to 64th channels on the next layer are set in the product-sum operation circuit 61. Thereafter, each element data of one small region in the first channel Cha-1 on the previous layer is input to each product-sum operation circuit 61, and one piece of data is calculated for each of the 1st to 64th channels on the next layer and stored in a register 61c.

[0116] 16(B), convolution filters Fa2-b1, Fa2-b2, ... corresponding to the second channel Cha-2 on the previous layer and the first to 64th channels on the next layer are set in each product-sum operation circuit 61. After setting, element data of the small region at the same position as the first channel Cha-1 in the second channel Cha-2 on the previous layer is input to each product-sum operation circuit 61, and one piece of data is calculated for each of the first to 64th channels on the next layer. These calculated pieces of data are added by adder 61b to the data held in each register 61c, and the corresponding pieces of data are held in each register 61c as new data.

[0117] Similarly, one piece of convolved data is calculated for each of the 1st to 64th channels of the next layer from each element data of one small region for each of the remaining channels of the previous layer while changing the convolution filter set in each product-sum calculation circuit 61. Then, the calculated data and the corresponding data held in register 61c are added together. In this way, one piece of element data is calculated in parallel for each of the 1st to 64th channels of the next layer.

[0118] After calculating one element data for each of the first to 64th channels in the next layer in the above manner, the same procedure as above is repeated while changing the position of the small area in each channel in the previous layer to calculate all element data for each of the 65th and subsequent channels in the next layer.

[0119] Even when the first channel parallel mode is used, power gating is performed to cut off power supply to memory blocks MB that hold data not used in the convolution operation, thereby reducing the power consumption of the arithmetic processing device 50.

[0120] It is preferable to switch between the first channel-parallel mode and the second pixel-parallel mode so that, when convolution calculation processing is repeatedly performed, the second pixel-parallel mode is selected from the processing in which the number of channels in the next layer first exceeds the number of product-accumulation circuits 61. This is because, in channel parallelism, when the number of channels in the next layer is equal to or less than the number of product-accumulation circuits 61, the channel parallelism means that the element data needs to be read from the first storage circuit 21 to the second storage circuit 62 only once, thereby reducing the number of accesses to the first storage circuit 21.

[0121] As shown in an example in Table 2, when convolution calculation processing is repeatedly performed and there are 128 product-sum calculation circuits 61, the first channel-parallel mode is used up to the convolution calculation processing in which the fourth layer is calculated as the next layer, and the second pixel-parallel mode is used from the convolution calculation processing in which the fifth layer is calculated as the next layer.

[0122] [Table 2] [Explanation of symbols]

[0123] 10, 50 Processing unit 13 Pooling Department 15 Memory section 16, 56 Arithmetic circuit section 17, 57 Power gate control unit 18, 58 Controller 21 1st memory circuit 22, 62 2nd memory circuit 24, 61 Multiply-and-accumulate circuit 31 memory cells 39 Multiply-and-accumulate unit

Claims

1. a memory unit that stores calculation data related to weighting calculation, at least a part of which is a nonvolatile storage area; an arithmetic circuit unit that performs arithmetic processing including the weighting operation using a part or all of the arithmetic data input from the memory unit; a power gate unit that cuts off power supply to some or all memory cells other than memory cells storing some or all of the operation data to be input to the arithmetic circuit unit during an arithmetic process in the nonvolatile storage area; Equipped with a data length setting unit that sets a predetermined effective data length for the operation data for each operation process by the operation circuit unit; the arithmetic circuit unit performs arithmetic processing on a portion of the arithmetic data of the set effective data length, The power gate unit cuts off the power supply to memory cells that store a data portion of the operation data other than the effective data length among the memory cells in the nonvolatile memory area as memory cells other than the memory cells that store a portion of the operation data. A processing device characterized by:

2. the memory unit holds, as the calculation data, load data that become weights for the weighting calculation and element data to be weighted; the arithmetic circuit unit includes a product-sum arithmetic circuit having a multiplier that multiplies the load data by the element data, a register, and an adder that adds the multiplication result of the multiplier and the data held in the register and holds the result in the register; The power gate unit cuts off the power supply to unused bit cells of the register based on the effective data length. The processor according to claim 1 .

3. the memory unit stores, as the element data, the element data of a channel in a previous layer in a convolutional neural network and the weight data which are weights of a convolutional filter; 3. The arithmetic processing device according to claim 2, wherein the product-sum operation circuit performs convolution operation processing using the element data and the weight data of the previous layer to calculate the element data of the channel of the next layer.

4. 4. The arithmetic processing device according to claim 3, wherein the data length setting unit sets the effective data length determined in advance for each of the convolutional arithmetic processes of each layer of a convolutional neural network.

5. 5. The processor according to claim 1, wherein the data length setting unit sets the effective data length according to an operation mode.

6. 5. The arithmetic processing device according to claim 1, wherein the arithmetic circuit unit has a plurality of multiply-accumulate operators that operate on data with a number of bits shorter than the maximum value of the effective data length of the data related to the arithmetic processing, and performs arithmetic processing by combining some or all of the plurality of multiply-accumulate operators depending on the set effective data length.

7. A memory unit that stores calculation data related to weighting calculation, at least a portion of which is a non-volatile storage area; an arithmetic circuit unit that performs arithmetic processing including the weighting operation using a part or all of the arithmetic data input from the memory unit; a power gate unit that cuts off power supply to some or all memory cells other than memory cells storing some or all of the operation data to be input to the arithmetic circuit unit during an arithmetic process in the nonvolatile storage area; Equipped with the memory unit stores, as the calculation data, element data of a plurality of channels in a previous layer and weight data of a plurality of convolution filters, and stores the weight data in the non-volatile storage area; the arithmetic circuit unit has a plurality of product-sum calculation circuits that perform a convolution operation on the element data of a channel of a previous layer to calculate the element data of a channel of a next layer in parallel, the weight data of one of the convolution filters selected according to a channel of a previous layer or a channel of a next layer and read from the nonvolatile storage area is input to the plurality of product-sum operation circuits, and the plurality of product-sum operation circuits perform convolution operations in parallel using the input weight data and the element data of one channel of the previous layer to calculate the element data of one channel of the next layer, respectively; The power gate unit cuts off power supply to memory cells in the nonvolatile storage area that store the load data other than the load data input to the plurality of product-sum operation circuits during the convolution operation processing of the plurality of product-sum operation circuits. A processing device characterized by:

8. 8. The arithmetic processing device according to claim 7, wherein the number of said plurality of product-sum operation circuits is the same as a divisor of the number of data in said next-level channel.

9. 9. The arithmetic processing device according to claim 7, wherein the arithmetic circuit unit generates the next-level channels in which the number of data in the row direction and the number of data in the column direction are each a multiple of two.

10. 10. The arithmetic processing device according to claim 9, further comprising a pooling unit that calculates a maximum value or an average value for each of a plurality of small regions obtained by dividing the next-level channel into two rows and two columns.

11. A memory unit that stores calculation data related to weighting calculation, at least a portion of which is a non-volatile storage area; an arithmetic circuit unit that performs arithmetic processing including the weighting operation using a part or all of the arithmetic data input from the memory unit; a power gate unit that cuts off power supply to some or all memory cells other than memory cells storing some or all of the operation data to be input to the arithmetic circuit unit during an arithmetic process in the nonvolatile storage area; Equipped with the memory unit holds channel element data and convolution filter weight data as the calculation data; the arithmetic circuit unit repeatedly performs a convolution operation process to generate data for a channel of a next layer by applying a convolution filter to the element data for the channel of a previous layer; a priority control unit that, when the previous hierarchical layer is a layer prior to a predetermined switching layer, causes the weight data used in the convolution operation to be preferentially stored in the nonvolatile storage area, and, when the previous hierarchical layer is a layer subsequent to the switching layer, causes the element data of the channel of the previous hierarchical layer used in the convolution operation to be preferentially stored in the nonvolatile storage area; A processing device characterized by:

12. 12. The arithmetic processing device according to claim 11, wherein the nonvolatile storage area stores all of the weight data used in the convolution operation in which the previous hierarchical level is the level before the switching hierarchical level.

13. 13. The processing device according to claim 11, wherein the non-volatile storage area holds all of the element data used in the convolution operation process when the previous hierarchical level is the switching hierarchical level or later.

14. 13. The arithmetic processing device according to claim 11, wherein the arithmetic circuit unit calculates the element data of one channel in the next hierarchy in parallel.

15. 14. The arithmetic processing device according to claim 13, wherein the switching layer is a layer in which, when the convolution operation is repeatedly performed, the total number of the weight data used in the convolution operation of one layer first becomes larger than the total number of the element data of the previous layer used in the convolution operation of the layer.

16. The arithmetic circuit unit When the previous hierarchy is a hierarchy before the switching hierarchy, one element data is calculated in parallel for each of the plurality of channels in the next hierarchy; When the previous layer is the switching layer or later, the element data of one channel of the next layer is calculated in parallel.

13. The processing unit according to claim 11 or 12.

17. 17. The arithmetic processing device according to claim 16, wherein the switching hierarchical level is a hierarchical level in which the number of channels in the next hierarchical level first exceeds the number of multiply-accumulate circuits constituting the plurality of multiply-accumulate circuits when the convolution arithmetic processing is repeatedly performed.

18. 13. The arithmetic processing device according to claim 1, wherein the memory unit has a first memory circuit and a second memory circuit provided as the non-volatile memory area, the second memory circuit being physically closer to the arithmetic circuit unit than the first memory circuit or having a faster access speed.

19. 13. The processor according to claim 1, wherein the memory cells of the nonvolatile storage area are configured by MRAM.

Images