Arithmetic processing unit

Abstract

Provided is a computation processing device that can further reduce power consumption. Element data from within a pooling region is sequentially inputted into a pooling processing unit 22 from a convolution computation unit. The pooling processing unit 22 has a nonvolatile resistor 32, a comparator 33, and a multiplexer 34. The comparator 33 compares the element data from the convolution computation unit with element data stored at the resistor 32. On the basis of the comparison results, the multiplexer 34 selects element data that has a large value, and the selected element data is stored at the resistor 32. After the element data from within the pooling region has been inputted, the element data stored at the resistor 32 is outputted as pooling data. When the pooling processing unit 32 is waiting for input of element data, a PG switch 35 is turned off, and power supply to the resistor 32 is disconnected.

Description

Technical Field

[0001] The present invention relates to an arithmetic processing unit.

Background Art

[0002] An arithmetic processing unit that performs image recognition and the like using a convolutional neural network, that is, a neural network having a convolutional layer, is known, and its application to robot control, vehicle driving control, and the like is expected. In such a convolutional neural network including image recognition, convolutional arithmetic processing and pooling processing are performed. In the convolutional arithmetic processing, a huge product-sum operation is performed in which data of an input layer or an intermediate layer is weighted and added using the load data of a convolutional filter. In the pooling processing, for example, the maximum value is extracted or the average value is calculated from a plurality of convolutional operation results obtained by the convolutional arithmetic processing.

[0003] In Patent Document 1, it is proposed to reduce the circuit scale for performing the convolutional operation by obtaining, in an arithmetic processing unit for the arithmetic processing of a convolutional neural network, a part of all the convolutional operation results required for one pooling processing every one operation cycle.

[0004] On the other hand, as a technique for suppressing power consumption, power gating that cuts off power supply to an arithmetic circuit such as a processor core and suppresses leakage current is known.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] Incidentally, the processing units used in convolutional neural networks require a vast number of operations, such as multiply-accumulate calculations, as described above, and power consumption has become a problem. In particular, it is important to minimize power consumption in end devices such as robots, cars, and mobile terminals. For this reason, there is a desire to further reduce the power consumption of processing units that perform calculations like those in convolutional neural networks.

[0007] This invention has been made in view of the above circumstances, and aims to provide a processing unit that can reduce power consumption. [Means for solving the problem]

[0008] To achieve the above objective, the arithmetic processing unit of the present invention comprises a convolution calculation unit that sequentially outputs convolution calculation result data, a pooling calculation unit having a pooling calculation circuit and a non-volatile pooling memory circuit, wherein the pooling memory circuit holds the convolution calculation result data or the calculation result of the pooling calculation circuit as retained data, and each time the convolution calculation result data is input from the convolution calculation unit, the pooling calculation circuit calculates and outputs pooling data in the pooling area using the retained data, and a power gating unit that cuts off the power supply to the pooling memory circuit while waiting for the input of the convolution calculation result data from the convolution calculation unit.

[0009] The present invention provides a processing unit comprising: a convolution unit that sequentially outputs convolution result data in the row direction of a channel for each row of a channel in which a plurality of convolution result data are arranged in a two-dimensional array; and a pooling processing unit that has a pooling calculation circuit and a non-volatile pooling storage circuit, and outputs the convolution result data that is the maximum value for each pooling region obtained by dividing the plurality of convolution result data into 2 rows and 2 columns of the channel as pooling data, wherein the pooling storage circuit has a buffer connected in Y+2 stages, with Y being the number of columns of the channel (Y being an even number of 2 or more), and each time the convolution result data from the convolution calculation unit is input to the first stage buffer, the first stage buffer processes the convolution result data that has been input to it. The pooling operation circuit receives a data group consisting of the convolution result data from the first, second, Y+1, and Y+2 buffers, and includes a comparator that compares each of the convolution result data in the data group, and a selector that selects and outputs the convolution result data that has the maximum value among the data group based on the comparison result of the comparator, and the pooling processing unit outputs the convolution result data output from the selector as pooling data when each of the convolution result data in the data group becomes a combination of the convolution result data in one pooling region. [Effects of the Invention]

[0010] According to the present invention, when the pooling processing unit is waiting for input of convolution calculation result data from the convolution calculation unit, the power gating unit cuts off the power supply to the non-volatile pooling memory circuit, thereby reducing the power consumption of the calculation processing unit.

[0011] According to the present invention, the number of buffers that hold the convolution operation result data can be set to less than the number of element data in the channel (number of columns × number of rows) (number of columns + 2), thereby reducing power consumption. [Brief explanation of the drawing]

[0012] [Figure 1] This is a block diagram illustrating the schematic of a processing unit. [Figure 2] This is an explanatory diagram illustrating an example of a connected hierarchy in a convolutional neural network. [Figure 3] This is an explanatory diagram showing the relationship between the movement of the convolutional region and the pooling region. [Figure 4] Block diagram showing the configuration of the processing unit. [Figure 5] This is a block diagram showing the configuration of a convolution circuit. [Figure 6] This is a block diagram showing the configuration of the pooling processing unit. [Figure 7] This is an explanatory diagram showing the state of the convolution operation when performing operations on the first element data of the next layer using channel parallelism. [Figure 8] This is an explanatory diagram showing the state of the convolution operation when performing operations on the second element data of the next layer using channel parallelism. [Figure 9] This is an explanatory diagram showing the period during which the PG switch is turned off. [Figure 10] This block diagram shows an example configuration of a pooling processing unit that performs mean pooling. [Figure 11] This block diagram shows an example configuration of a pooling processing unit that performs weighted mean pooling. [Figure 12] This is a block diagram showing a pooling processing unit composed of buffers with multiple register sections connected in stages. [Modes for carrying out the invention]

[0013] In FIG. 1, the arithmetic processing unit 10 performs arithmetic processing based on a convolutional neural network. The arithmetic processing unit 10 includes an arithmetic unit 11 that performs convolutional arithmetic processing and pooling processing using a convolutional filter on channels (also referred to as feature planes), a memory unit 12, a power gating control unit 14, and a controller 15 that comprehensively controls these components. As will be described in detail later, the arithmetic unit 11 is provided with k (k is an integer greater than or equal to 2) arithmetic units 17 that perform convolutional arithmetic processing and pooling processing in parallel.

[0014] The convolutional neural network based on the above-described arithmetic processing unit 10 has multiple connected layers. Each layer has one or more channels. The first layer is an input layer, such as an image composed of RGB channels. In the convolutional neural network shown as an example in FIG. 2, the first to fourth layers are connected. The first layer has three channels ch1-1 to ch1-3, the second layer has four channels ch2-1 to ch2-4, the third layer has three channels ch3-1 to ch3-3, and the fourth layer has three channels ch4-1 to ch4-3.

[0015] Among the first to fourth layers, the first and second layers are the layers to be subjected to convolutional arithmetic processing. Through convolutional arithmetic processing, channels ch2-1 to ch2-4 of the second layer are generated from channels ch1-1 to ch1-3 of the first layer, and channels ch3-1 to ch3-3 of the third layer are generated from channels ch2-1 to ch2-4 of the second layer. The third layer is the layer to be subjected to pooling processing. Through pooling processing, channels ch4-1 to ch4-3 of the fourth layer are generated from channels ch3-1 to ch3-3 of the third layer.

[0016] Note that the number of channels in each layer can be one or more. Also, in convolutional arithmetic processing, the number of channels may increase or decrease between the previous and subsequent layers, or may remain unchanged. In pooling processing, the number of channels in the previous and subsequent layers is the same. The number of layers may be three or five or more.

[0017] The arithmetic unit 11 generates the (n + 1)-th layer by performing a convolution operation process or a pooling operation process on the channels of the n-th layer, where n is an integer greater than or equal to 1. The generation of a layer means generating each channel that constitutes the layer, and the generation of a channel means calculating each element data that constitutes the channel. In the following description, the n-th layer may be referred to as the previous layer with respect to the (n + 1)-th layer, and the (n + 1)-th layer may be referred to as the next layer with respect to the n-th layer. Therefore, the channels of the next layer are generated by the convolution operation process and the pooling operation process on the channels of the previous layer.

[0018] A channel is composed of a plurality of element data arranged in a two-dimensional array. The two-dimensional array of element data is an arrangement in terms of data structure, meaning that the position of each element data is specified in two variables (rows and columns in this description), and position information is given so that the positional relationship between the element data is specified. The same applies to the load data described later. The size of each channel, that is, the number of element data in the row direction and the column direction, is arbitrary and not particularly limited. In this example, two-dimensional channels are described, but one-dimensional or three-dimensional or higher-dimensional channels may also be used.

[0019] In the convolution operation process, element data is calculated by the convolution operation. The element data calculated by the convolution operation is a value obtained by adding the results of applying the convolution filter to each element data within the convolution region for each channel of the previous layer, among the elements at the same position in each convolution region of each channel. The application of the convolution filter is to obtain the sum-of-products operation result of the element data within the convolution region and the load data of the convolution filter.

[0020] A convolutional filter is a two-dimensional array of load data that acts as weights for element data. In this example, one convolutional filter consists of 3x3 (3 rows, 3 columns) load data. Each load data in a convolutional filter is set to a value appropriate to the purpose of that convolutional filter. In this example, a convolutional filter is used that corresponds to the combination of channels in the previous and next layers.

[0021] The convolution region defines the range on the channel to which the convolution filter is applied, and has the same array size as the convolution filter (3 rows and 3 columns in this example). In the convolution operation, the load data of the convolution filter and the element data of the convolution region are multiplied at their corresponding positions. During the convolution operation, the convolution region is moved to scan the entire channel by moving its position one element at a time, and the element data calculation process is performed each time the convolution region is moved.

[0022] In this example, a convolution operation is performed on each channel of the next level using all the channels of the previous level. Additionally, a convolution operation is performed using a convolution filter that corresponds to the combination of channels from the previous and next levels.

[0023] Therefore, in the example shown in Figure 2, for example, when generating the third-level channel ch3-1, when applying a convolution filter to channel ch2-1, the convolution filter associated with the combination of channel ch2-1 and channel ch3-1 is used, and when applying a convolution filter to channel ch2-2, the convolution filter associated with the combination of channel ch2-2 and channel ch3-1 is used. In this way, when generating channel ch3-1, the convolution operation is performed using four convolution filters corresponding to each of the four possible combinations of channel ch3-1 and channels ch2-1 to ch2-4. Similarly, when generating channel ch3-2, the convolution operation is performed using four convolution filters corresponding to each of the four possible combinations of channel ch3-2 and channels ch2-1 to ch2-4, and when generating channel ch3-3, the convolution operation is performed using four convolution filters corresponding to each of the four possible combinations of channel ch3-3 and channels ch2-1 to ch2-4.

[0024] In the convolution operation, any number of channels from the previous level can be used to generate one channel in the next level, and one channel from the previous level can be used to generate one channel in the next level. Furthermore, all or part of the multiple convolution filters used in a single level may share a common array of weights. In addition, if the array of weights of the convolution filters is common, one convolution filter with that common array of weights may be prepared and used when calculating multiple channels.

[0025] As an example, pooling generates channels in the next tier by reducing the size in the row and column directions from each channel in the previous tier. In this example, the pooling process is a maximum value pooling process, which extracts the maximum value from a 2x2 pooling area. For this purpose, each channel is divided into multiple 2x2 pooling areas that do not overlap with each other, and the element data of the maximum value within each of these pooling areas is output as the result of the pooling process. Note that the size of the pooling area is not limited to 2x2. One of p and q may be an integer of 1 or more, and the other an integer of 2 or more, so that the pooling area is p x q. Furthermore, instead of the maximum value pooling process, an average value pooling process may be used, which outputs the average value of the element data in the pooling area, as will be described later. A convolution operation can also be performed on the tier consisting of channels reduced by the pooling process. The pooling areas can also be divided so that they partially overlap with each other, in which case the pooling process can be performed to generate channels in the next tier that have the same size in the row and column directions as the channels in the previous tier.

[0026] As shown in Figure 3(A), the convolution region Ra in this example is 3 rows and 3 columns. When the calculation unit 11 obtains element data for channel ChB of the next level (the third level in the example of Figure 2) that is the target of pooling, the convolution region Ra in channel ChA of the previous level moves sequentially to the position shown in Figure 3(A), then to the position shown in Figure 3(B) which is moved one element data unit in the row direction from the position shown in Figure 3(A), then to the position shown in Figure 3(C) which is moved one element data unit in the column direction from the position shown in Figure 3(A), and finally to the position shown in Figure 3(D) which is moved one element data unit in the row direction from the position shown in Figure 3(C). This allows for the continuous calculation of element data within one pooling region Rb in channel ChB of the next level. Note that the order in which the convolution region Ra moves is not limited to the above order, as long as the element data within one pooling region Rb is calculated continuously.

[0027] In Figure 1, the memory unit 12 stores the load data of the convolution filter and the element data of each channel in the layer to which the convolution operation is applied, i.e., the previous layer. It also writes the convolution operation result data and pooling data, i.e., the element data of each channel in the next layer. For layers targeted by pooling, the element data obtained from the convolution operation is passed to the pooling operation within the calculation unit 17 and is therefore not written to the memory unit 12.

[0028] The power gating control unit 14 controls the power supply to each computing unit 17, i.e., power gating, under the control of the controller 15, as will be described in detail later.

[0029] As shown in Figure 4, the arithmetic unit 17 includes a convolution unit 21 that performs convolution operations, a pooling unit 22 that performs pooling (extraction of the maximum value in this example), and an activation function unit 23. In addition to these, the arithmetic unit 17 is also provided with a bit length adjustment circuit (not shown) that converts the data length of the element data output from the convolution unit 21 to a predetermined data length.

[0030] The convolution unit 21 performs a convolution operation to obtain element data. The convolution unit 21 calculates one element data in a single convolution operation. When calculating one element data, the convolution unit 21 is input with nine element data within the convolution region and nine load data of the convolution filter for each channel of the previous layer, while sequentially switching between channels of the previous layer.

[0031] Element data from the convolution unit 21 is input to the activation function processing unit 23 and transformed using an activation function. Examples of activation functions include the step function, sigmoid function, normalized linear unit (ReLU), leaky normalized linear unit (Leaky ReLU), and hyperbolic tangent function. The element data that has passed through the activation function processing unit 23 is sent to the memory unit 12 and the pooling processing unit 22 as element data for the next level.

[0032] The pooling processing unit 22 performs the pooling process described above and outputs the element data that represents the maximum value within the pooling region. Power supply to this pooling processing unit 22 is controlled by the power gating control unit 14.

[0033] In the following explanation, element data obtained by the convolution operation performed by the convolution operation unit 21 (including data that has passed through the activation function processing unit 23) will be specifically referred to as convolution operation result data, and element data obtained by the pooling process performed by the pooling processing unit 22 will be specifically referred to as pooled data.

[0034] Figure 5 shows an example of the convolution unit 21. This convolution unit 21 consists of the same number of multipliers 24 (9 in this example) as the load data of the convolution filter, a multiplexer 25, an adder 26, and a register 27. Each multiplier 24 receives element data and load data as input and outputs the result of multiplying them. The multiplexer 25 selects and outputs one multiplication result from each multiplier 24. The register 27 holds the addition result of the adder 26. Each time a multiplication result is output from the multiplexer 25, the adder 26 adds that multiplication result from the multiplexer 25 to the data held in the register 27 and stores the added result in the register 27. The element data of each channel in the previous layer and the load data of the convolution filter are input to the convolution unit 21, and finally the addition result held in the register 27 is output as the convolution result data (element data). However, the configuration of the convolution unit 21 is not limited to this.

[0035] In Figure 6, the pooling processing unit 22 includes a pooling arithmetic circuit 31 and a register 32 as a non-volatile pooling memory circuit. The pooling arithmetic circuit 31 works in cooperation with the register 32 to extract element data that is the maximum value in the pooling region. This pooling arithmetic circuit 31 is composed of a comparator 33 and a multiplexer 34. The register 32 is composed of, for example, multiple non-volatile flip-flops (NV-FFs) using magnetic tunnel junction (MTJ) elements. Non-volatile flip-flops using magnetic tunnel junction elements are advantageous in convolutional neural networks where high-density integration is required because they are smaller in size on the substrate compared to other non-volatile flip-flops, and they are also advantageous in reducing power consumption because they have a low operating voltage.

[0036] Since register 32 is non-volatile, it retains data even when the power supply is cut off, and by supplying power, it is possible to read the data that was retained when the power was cut off and to output data. This register 32 holds the element data selected by the multiplexer 34 as retained data. Register 32 is reset each time the output of the maximum value of the pooling area is completed, and the contents it holds are set to the initial value (value "0"). Note that the configuration of the pooling memory circuit is not limited to the above.

[0037] The comparator 33 and multiplexer 34, which constitute the pooling operation circuit 31, receive element data from the convolution operation unit 21 and element data held in the register 32 via the activation function processing unit 23. The comparator 33 compares the two input element data and outputs a selection signal to the multiplexer 34 to select the element data with the larger value. The multiplexer 34 functions as a selector and outputs one of the input element data based on the selection signal. As a result, the element data with the larger value between the element data from the convolution operation unit 21 and the element data held in the register 32 is output from the multiplexer 34, and this output element data is held in the register 32 as new retained data.

[0038] By sequentially inputting the element data of the pooling region calculated by the convolution unit 21 into the pooling calculation circuit 31, the element data that represents the maximum value in the pooling region is held in the register 32, and the held element data is output as pooling data for one pooling region.

[0039] A drive voltage (VDD) is applied to register 32 via the PG switch 35. The PG switch 35, together with the power gating control unit 14, constitutes the power gating section. The PG switch 35 is made up of a MOS transistor or the like, and its on / off state is controlled by the power gating control unit 14. When the PG switch 35 is turned on, register 32 receives power and becomes capable of writing and outputting (reading) data. When the PG switch 35 is turned off, the drive voltage is no longer applied to register 32, i.e., the power supply is cut off, and it becomes impossible to write or output data. This enables power gating to register 32. In this example, a PG switch 35 is provided for each pooling processing unit 22, but a single PG switch 35 common to all pooling processing units 22 may also be provided.

[0040] The power gating control unit 14 reduces power consumption by turning on the PG switch 35 during the pooling process, at least while writing and outputting element data to register 32, and turning off the PG switch 35 at other times. In this example, during the pooling process, the PG switch 35 is turned off while the pooling processing unit 22 is waiting for input of element data from the convolution processing unit 21, i.e., while the pooling processing unit 22 is not performing any processing, and turned on at other times. Specifically, the period during which the PG switch 35 is turned on is from the timing of the output of element data from the convolution processing unit 21, i.e., the input of element data to the pooling processing unit 22, until new element data is held in register 32 by the processing of the pooling processing circuit 31, and until the output of pooling data is completed, if applicable. The PG switch 35 is turned off outside of the pooling process.

[0041] In this example, as described above, each element data of the pooling region calculated sequentially by the convolution unit 21 is input to the pooling processing unit 22 each time element data is calculated. Therefore, the pooling processing period begins at the start of the convolution operation to generate the channel of the hierarchy to be pooled, or at the time the first element data of the hierarchy to be pooled is input to the pooling processing unit 22. The pooling processing period ends when the output of the final element data of the channel generated by the pooling process from the pooling processing unit 22 is completed.

[0042] When focusing on a single pooling region, the pooling processing period begins when the convolution operation for calculating the element data of that pooling region starts or when the first element data of that pooling region is input to the pooling processing unit 22, and ends when the output of the element data that represents the maximum value of that pooling region held in register 32 is completed. The completion of the output of the element data held in register 32 is when the element data output from register 27 is acquired by the circuit that is supposed to acquire it from register 27. In this example, the completion of the output of the element data is when the memory unit 12 latches the element data.

[0043] The arithmetic processing unit 10 performs convolution processing in a manner called channel parallelism, where one element data is calculated in parallel for each of the k channels of the next hierarchical level, using K arithmetic units 17 provided in the arithmetic unit 11. Furthermore, if the hierarchical level generated by the convolution processing is subject to pooling processing, the arithmetic processing unit 10 moves the convolution region within the k channels so that multiple element data within the pooling region are calculated sequentially, as described above, each time one element data is calculated for each of the k channels of the next hierarchical level. If the next hierarchical level generated by the convolution processing is not subject to pooling processing, the element data may be calculated in a manner other than that described above.

[0044] Next, we will explain the operation of the above configuration in the case where a convolution operation is performed on the nth layer to generate the (n+1)th layer, and a pooling operation is performed on this (n+1)th layer to generate the (n+2)th layer. In the convolution operation, as shown in Figures 7 and 8, the nth layer consists of channels ChA1, ChA2, ..., and the (n+1)th layer consisting of channels ChB1, ChB2, ... is generated from this nth layer.

[0045] First, each calculation unit 17 of the calculation unit 11 performs an operation to apply a convolution filter to the convolution region Ra of the first channel ChA1 of the nth layer. The nine element data of the convolution region Ra of channel ChA1 are read from the memory unit 12 and input to the convolution calculation unit 21 of each calculation unit 17. In addition, the load data of nine convolution filter loads are input to one convolution calculation unit 21, so that the load data of the first channel ChA1 and the convolution filters FA1B1, FA1B2, ... corresponding to the first to kth channels ChB1, ChB2, ... of the next layer are read from the memory unit 12 and input to the convolution calculation unit 21 of each calculation unit 17. As a result, each convolution unit 21 multiplies the element data of the convolution region Ra of the channel ChA1 input to it with the load data of the convolution filter by the corresponding data, and stores the sum of the product results, which are the sum of these multiplication results, in the register 27 (Figure 7(A)).

[0046] Next, each calculation unit 17 performs an operation to apply a convolution filter to the convolution region Ra in the second channel ChA2 of the nth tier, which is at the same position as the first channel ChA1. The nine element data of the convolution region Ra in channel ChA2 are input to the convolution calculation unit 21 of each calculation unit 17, and the load data of the convolution filters FA2B1, FA2B2, ... corresponding to the second channel ChA2 and the first to kth channels ChB1, ChB2, ... of the next tier are input to each calculation unit 17.

[0047] Each calculation unit 17 corresponds to one channel of the next level until, for example, the calculation of all element data for k channels is completed, and the corresponding channel of the next level does not change. For this reason, for example, when calculating the first channel ChA1 of the previous level, the calculation unit 17 that received the load data of the convolution filter FA1B1 corresponding to the second channel ChB1 will also receive the load data of the convolution filter FA2B2 corresponding to the second channel ChB2 when calculating the second channel ChA2.

[0048] As described above, when element data and load data are input to each calculation unit 17, the register 27 of each convolution calculation unit 21 stores a value obtained by adding the sum-of-products result obtained by applying a convolution filter to the convolution region Ra of channel ChA1 with the sum-of-products result obtained by applying a convolution filter to the convolution region Ra of channel ChA2 (Figure 7(B)).

[0049] Similarly, the convolution unit 21 of each arithmetic unit 17 sequentially performs the operation of applying a convolution filter to the convolution region Ra at the same position as the first channel ChA1 for each channel from the third channel onward in the nth layer. Once the operation of applying the convolution filter to the convolution region Ra of the last channel in the nth layer is completed, the register 27 of each convolution unit 21 stores the sum of the sum-of-products results obtained by applying the convolution filter to the convolution region Ra of each channel in the previous layer, that is, the first element data for each of the 1st to kth channels in the next layer. The first element data obtained in this way (convolution operation result data) is output from the convolution unit 21.

[0050] After calculating the first element data as described above, the convolution unit 21 of each calculation unit 17 shifts the convolution region Ra by one element data unit in the row direction, as shown in Figure 8(A), and performs the operation to apply the convolution filters FA1B1, FA1B2, ... to the convolution region Ra of the first channel ChA1 of the previous layer using the same procedure as described above. After this, the convolution unit 21 performs the operation to apply the convolution filters FA2B1, FA2B2, ... to the convolution region Ra of the second channel ChA2 of the previous layer, again using the same procedure, as shown in Figure 8(B). Similarly thereafter, the operation to apply the convolution filters to the convolution region Ra of each channel from the third onward of the previous layer is performed sequentially, and the second element data for each of the 1st to kth channels is calculated and output from the convolution unit 21.

[0051] After calculating the second element data, the convolution region Ra is shifted by one element data unit in the column direction from its initial position, and the third element data is calculated and output by the convolution unit 21 using the same procedure as above. After calculating the third element data, the convolution region Ra is shifted by one element data unit in the row direction, and the fourth element data is calculated and output by the convolution unit 21 using the same procedure as above. In this way, the four element data of the pooling region that is subject to pooling processing are calculated sequentially.

[0052] On the other hand, as shown in Figure 9, the pooling processing unit 22 is in a waiting state for element data input during the period T1 in which the convolution calculation unit 21 is performing the convolution calculation. During this waiting state T1 for element data input, the power gating control unit 14 turns off the PG switch 35, and the power supply to the registers 32 of each pooling processing unit 22 is cut off.

[0053] At the moment the first element data (convolution operation result data) is output from the convolution operation unit 21, the power gating control unit 14 turns on the PG switch 35. This supplies power to the registers 32 of each pooling processing unit 22, enabling data writing. When the element data output from the convolution operation unit 21 is input to the pooling processing unit 22 via the activation function processing unit 23, the input element data and the data held in the registers 32 are compared by the comparator 33, and the multiplexer 34 is controlled based on the comparison result. Since the registers 32 are reset when the convolution operation starts and hold an initial value (value "0"), the input element data is selected by the multiplexer 34 and written to the registers 32.

[0054] As described above, once the first element data is written to register 32, the pooling processing unit 22 waits for input of the second element data, which the convolution calculation unit 21 calculates over period T2. The power gating control unit 14 turns off the PG switch 35, cutting off the power supply to the register 32 of each pooling processing unit 22.

[0055] At the moment the convolution unit 21 outputs the second element data, the power gating control unit 14 turns on the PG switch 35, supplying power to the register 32. The second element data output from the convolution unit 21 is compared with the element data held in the register 32 by the comparator 33, and the multiplexer 34 is controlled based on the comparison result. Since the register 32 is non-volatile, when the power supply is restored, it outputs the data it held before the power was cut off. Therefore, the second element data output from the convolution unit 21 is compared with the element data held in the register 32 by the comparator 33, and the multiplexer 34 is controlled based on the comparison result.

[0056] Since register 32 holds the first element data, the first element data and the second element data are compared by the comparator 33. Through this comparison, the element data with the larger value is selected by the multiplexer 34, and the selected element data is written to register 32. When new element data is written to register 32 in this way, the convolution unit 21 waits for the third element data to be calculated during period T3, the PG switch 35 is turned off, and the power supply to register 32 of each pooling processing unit 22 is cut off.

[0057] At the moment the convolution unit 21 outputs the third element data, the power gating control unit 14 turns on the PG switch 35, supplying power to the register 32. The third element data and the element data held in the register 32 are then compared by the comparator 33. The element data with the larger value is selected by the multiplexer 34, and the selected element data is written to the register 32. Once new element data is written to the register 32, the convolution unit 21 waits for the fourth element data to be calculated during period T4, and the PG switch 35 is turned off, cutting off the power supply to the register 32 of each pooling processing unit 22.

[0058] At the moment the convolution unit 21 outputs the fourth element data, the PG switch 35 is turned on, supplying power to register 32. Then, the fourth element data is compared with the element data held in register 32, and the element data with the larger value is selected and written to register 32.

[0059] As a result, register 32 holds the element data with the largest value among the 1st to 4th element data in the (n+1)th layer pooling area, and the element data held in register 32 is output from the pooling processing unit 22 as element data (pooling data) of the (n+2)th layer. When this element data from the pooling processing unit 22 is latched in, for example, the memory unit 12, that is, when the output is complete, it waits for input of the 1st element data of the next pooling area, and the PG switch 35 is turned off, cutting off the power supply to register 32.

[0060] After outputting the fourth element data, the convolution unit 21 moves further into the convolution region Ra and calculates element data for the next pooling region using the same procedure as above. The pooling processing unit 22 compares the element data with the data held in register 32 each time the first to fourth element data is output, and holds the element data with the largest value in the new pooling region in register 32, outputting that element data as the (n+2)th layer element data. While the pooling processing unit 22 is waiting for element data input as described above, the PG switch 35 is turned off and power supply to register 32 is cut off.

[0061] By repeating the above procedure, all element data is calculated for each of the 1st to kth channels of the (n+2)th layer. If there are channels beyond the (k+1)th in the (n+2)th layer, all element data is calculated for each of all channels by repeating the same procedure as above. Note that if the number of channels for which pooling data is calculated is less than the number of arithmetic units 17, some arithmetic units 17 will not perform calculations. In such cases, power supply to the registers 32 in the arithmetic units 17 that are not performing calculations may be cut off.

[0062] As described above, the arithmetic processing unit 10 performs pooling processing using the pooling processing unit 22. However, while the pooling processing unit 22 is waiting for input of element data, power gating is implemented to cut off the power supply to the register 32. As a result, the leakage current of the register 32 is suppressed while waiting for input of element data, and the power consumption of the arithmetic processing unit 10 is low.

[0063] When register 32 is configured with a non-volatile flip-flop using a magnetic tunnel junction element (hereinafter referred to as the non-volatile configuration), the calculated ratio of the power consumption (operating power consumption + standby power consumption) when it is configured with a normal flip-flop that is not non-volatile (hereinafter referred to as the normal configuration) can be, for example, 0.22. In calculating this ratio, the operating power consumption of the non-volatile configuration is assumed to be 10 times that of the normal configuration, the "standby power consumption:operating power consumption" ratio of the normal configuration is assumed to be "30:110", and the ratio of the number of operating cycles of register 32 in the arithmetic processing unit 10 to the number of standby cycles is assumed to be "0.006".

[0064] Figure 10 shows an example of a pooling processing unit 22 configured to perform average value pooling processing that outputs the average value of element data in the pooling region. The pooling processing unit 22 consists of a pooling arithmetic circuit 31 and a register 42. The pooling arithmetic circuit 31 consists of an adder 43 and a 2-bit shifter 44, and works in cooperation with the register 42 to calculate the average value of element data in the pooling region. The adder 43 adds the data held in the register 42 to the element data, which is the input convolution operation result data. The register 42 is a non-volatile pooling memory circuit and holds the addition result of the adder 43. The 2-bit shifter 44 is a bit shift circuit and is provided as a divider. The 2-bit shifter 44 shifts the addition result obtained by adding up to the last element data in the pooling region in the adder 43 by 2 bits, thereby calculating the quotient when divided by the number of element data in the pooling region (4).

[0065] With the above configuration, the pooling processing unit 22 outputs element data (pooling data) which is the average value of the element data in the pooling region, each time four element data of the pooling region are input to the pooling region, based on the result of a shift operation from the 2-bit shifter 44.

[0066] Register 42, like register 32 (see Figure 6), is non-volatile and is power-gated by the on / off switching of the PG switch 35. Therefore, during the pooling process, while the pooling processing unit 22 is waiting for input of element data from the convolution processing unit 21, i.e., while the pooling processing unit 22 is not performing processing, the PG switch 35 is turned off and the power supply is cut off. This reduces the power consumption of the arithmetic processing unit 10.

[0067] As shown in the example in Figure 11, a multiplier 45 that multiplies by a predetermined weight may be provided before the adder 43, thereby weighting the element data input as a result of the convolution operation according to the position of the element data. The weight can be, for example, a weight following a two-dimensional Gaussian.

[0068] Figure 12 shows an example of a pooling processing unit 22 configured with a buffer 51a having multiple stages of register sections 51 as pooling memory circuits. In this example, the pooling processing unit 22 includes a comparator 52 and a multiplexer 53 as a selector, in addition to the register section 51. Element data (convolution operation result data) from the convolution operation unit 21 is input to the register section 51.

[0069] In this example, the pooling area is set to 2 rows and 2 columns. Furthermore, for the channels of the hierarchy targeted for pooling, the convolution unit 21 calculates element data for each row sequentially, starting from the first row, and for each row, calculates element data sequentially from one end to the other. Additionally, the number of columns in the channels of the hierarchy targeted for pooling is assumed to be even.

[0070] Each buffer 51a receives each bit of data in parallel, holds data that is input synchronously with the clock, for example, and outputs the held data in parallel. For example, a parallel-input, parallel-out (PIPO) shift register can be used as such a buffer 51a. When the number of columns (number of element data in one row) of each channel in the hierarchy targeted by pooling is Y (where Y is an even number of 2 or more), the register section 51 has buffers 51a connected in (Y+2) stages.

[0071] Each buffer 51a is connected in a multi-stage configuration such that the output of the preceding buffer 51a is input to the subsequent buffer 51a. Specifically, element data from the convolution unit 21 is input to the first buffer 51a via the activation function processing unit 23, and the second and subsequent buffers 51a are connected so that the output from the preceding buffer 51a is input to the subsequent buffer 51a. A clock signal is input to each buffer 51a in synchronization with the input of element data to the first buffer 51a. As a result, each time element data is input to the register unit 51, the first buffer 51a holds the element data from the convolution unit 21, and the second and subsequent buffers 51a hold the element data output from the preceding buffer 51a.

[0072] The comparator 52 and multiplexer 53, which constitute the pooling operation circuit 31, receive element data from the first, second, Y+1, and Y+2 stage buffers 51a as data sets. The comparator 52 compares the four input element data and outputs a selection signal to select and output the element data with the largest value. Based on the selection signal, the multiplexer 53 selects and outputs one element data from the four input element data.

[0073] The pooling arithmetic circuit 31, under the control of the controller 15, performs a comparison by the comparator 52 and a selection by the multiplexer 53 when the four element data output (held) by each buffer 51a of the 1st stage, 2nd stage, Y+1st stage, and Y+2nd stage become a combination of element data within a single pooling area. Specifically, with m being an integer greater than or equal to 1, after the input of the (2m-1)·Yth element data and before the input of the 2m·Yth element data, a comparison by the comparator 52 and a selection by the multiplexer 53 are performed every time two element data are input.

[0074] By configuring the pooling processing unit 22 as described above, the multiplexer 53 selects the element data that represents the maximum value of each pooling area, which is divided into multiple 2x2 sections without overlapping the channels targeted for pooling, and outputs it as pooling data.

[0075] The register section 51 is configured as a non-volatile memory circuit. That is, each buffer 51a is non-volatile. As in other examples, it is preferable that the buffer 51a be composed of a plurality of non-volatile flip-flops (NV-FFs) using magnetic tunnel junction elements. The register section 51 has a drive voltage (VDD) applied to it via the PG switch 35, and during the pooling process, while the pooling processing unit 22 is waiting for input of element data from the convolution operation unit 21, the PG switch 35 is turned off to reduce power consumption. In other words, when the first buffer 51a receives and stores element data from the convolution unit 21, and simultaneously when each buffer 51a from the second stage onward receives and stores element data from the previous buffer 51a, or when outputting pooling data, the PG switch 35 is turned on until the comparison of element data by the comparator 52 and selection by the multiplexer 53 are completed and the output from the multiplexer 53 is finished. The PG switch 35 is turned off and the power supply to the register unit 51 is cut off when the pooling processing unit 22 does not need to operate. [Explanation of symbols]

[0076] 10 Arithmetic Processing Unit 14 Power Gating Control Unit 21 Convolution Unit 22 Pooling Processing Unit 32, 42 registers 33 Comparator 34 Multiplexer 51 Register section 51a Buffer 52 Comparator 53 Multiplexer

Claims

1. A convolution unit that sequentially outputs convolution operation result data, A pooling processing unit having a pooling calculation circuit and a non-volatile pooling memory circuit, wherein the pooling memory circuit holds the convolution calculation result data or the calculation result of the pooling calculation circuit as retained data, and each time the convolution calculation result data is input from the convolution calculation unit, the pooling calculation circuit calculates and outputs pooling data in the pooling area using the retained data, A power gating unit that cuts off the power supply to the pooling memory circuit while waiting for input of the convolution calculation result data from the convolution calculation unit. A processing unit characterized by comprising:

2. The pooling operation circuit includes a comparator that compares the convolution operation result data from the convolution operation unit with the retained data, and a selector that receives the convolution operation result data from the convolution operation unit and the retained data as input, and selects and outputs the data with the larger value from the input data based on the comparison result of the comparator. The pooling memory circuit stores the data output by the pooling calculation circuit as new retained data. The pooling processing unit outputs the data held by the pooling storage circuit as pooling data upon input of each of the convolution calculation result data in the pooling region to the pooling calculation circuit. The arithmetic processing device according to feature 1.

3. The pooling operation circuit includes an adder that adds the convolution operation result data from the convolution operation unit and the held data, and a divider that divides the addition result of the adder by the number of convolution operation result data in the pooling area. The pooling memory circuit stores the addition result of the adder as new retained data. The pooling processing unit outputs the data obtained by dividing the sum of the adder results obtained by inputting each of the convolution result data of the pooling region into the pooling calculation circuit by the divider as pooling data. The arithmetic processing device according to feature 1.

4. The pooling operation circuit includes a multiplier that weights the convolution operation result data from the convolution operation unit by a predetermined weight, an adder that adds the multiplication result from the multiplier to the held data, and a divider that divides the addition result of the adder by the number of convolution operation result data in the pooling area. The pooling memory circuit stores the addition result of the adder as new retained data. The pooling processing unit outputs the data obtained by dividing the sum of the adder results obtained by inputting each of the convolution result data of the pooling region into the pooling calculation circuit by the divider as pooling data. The arithmetic processing device according to feature 1.

5. The arithmetic processing device according to claim 3 or 4, characterized in that the divider is a bit shift circuit that shifts data by a number of bits corresponding to the number of convolution result data in the pooling region.

6. The arithmetic processing device according to any one of claims 1 to 5, characterized in that the convolution result data in the pooling region of p rows and q columns on a channel arranged in a two-dimensional array of multiple convolution result data is input to the pooling processing unit.

7. The arithmetic processing device according to claim 6, characterized in that the pooling area is 2 rows and 2 columns.

8. The arithmetic processing apparatus according to any one of claims 1 to 7, characterized in that the pooling memory circuit is composed of non-volatile registers.

9. The arithmetic processing unit according to claim 8, characterized in that the register is composed of a non-volatile flip-flop.

10. The convolution calculation unit sequentially outputs the convolution calculation result data in the row direction of the channel for each row of the channel in which the multiple convolution calculation result data are arranged in a two-dimensional array. The pooling processing unit outputs the convolution result data that is the maximum value for each pooling region obtained by dividing the multiple convolution result data into 2x2 groups of the channel as pooling data. The pooling memory circuit has a non-volatile buffer connected in Y+2 stages, where Y is the number of channels (Y is an even number of 2 or more), and each time the convolution result data from the convolution unit is input to the first stage buffer, the first stage buffer holds and outputs the input convolution result data, and each buffer from the second stage onward holds and outputs the convolution result data output from the previous buffer. The pooling operation circuit has a data group consisting of the convolution operation result data from each of the first stage, second stage, Y+1 stage, and Y+2 stage buffers as input, a comparator that compares each of the convolution operation result data in the data group, and a selector that selects and outputs the convolution operation result data that has the maximum value among the data group based on the comparison result of the comparator. The pooling processing unit outputs the convolution result data output from the selector as pooling data when each of the convolution result data in the data group becomes a combination of the convolution result data in one pooling area. The arithmetic processing device according to feature 1.

11. The arithmetic processing apparatus according to claim 10, characterized in that the buffer is a non-volatile parallel input, parallel output type shift register.

12. The arithmetic processing unit according to claim 11, characterized in that the shift register is composed of non-volatile flip-flops.

13. The arithmetic processing apparatus according to claim 12, characterized in that the non-volatile flip-flop is a circuit including a magnetic tunnel junction element.

14. A convolution unit that sequentially outputs the convolution result data in the row direction of the channel for each row of a channel in which multiple convolution result data are arranged in a two-dimensional array, A pooling processing unit having a pooling calculation circuit and a non-volatile pooling memory circuit, outputs the convolution calculation result data that is the maximum value for each pooling region obtained by dividing the multiple convolution calculation result data into 2x2 groups of the channel as pooling data. Equipped with, The pooling memory circuit has a buffer connected in Y+2 stages, where Y is the number of columns in the channel (Y is an even number of 2 or more), and each time the convolution result data from the convolution unit is input to the first stage buffer, the first stage buffer holds and outputs the input convolution result data, and each buffer from the second stage onward holds and outputs the convolution result data output from the preceding buffer. The pooling operation circuit has a data group consisting of the convolution operation result data from each of the first stage, second stage, Y+1 stage, and Y+2 stage buffers as input, a comparator that compares each of the convolution operation result data in the data group, and a selector that selects and outputs the convolution operation result data that has the maximum value among the data group based on the comparison result of the comparator. The pooling processing unit outputs the convolution result data output from the selector as pooling data when each of the convolution result data in the data group becomes a combination of the convolution result data in one pooling area. A processing unit characterized by the following features.

Images