Arithmetic processing device

Abstract

Provided is an arithmetic processing device capable of efficiently accessing a feature surface storage part. In an arithmetic processing device 10 of a neural network, an arithmetic part 11 applies a filter to each channel of a feature surface to perform convolution arithmetic operation. A feature surface storage part 12 is composed of a plurality of memory banks B that can be accessed simultaneously, and stores therein feature data of the feature surface of each layer. A bank address calculation part 14 calculates and designates a memory bank B for holding the feature data and an address of the memory bank B when inputting / outputting the feature data to / from the feature surface storage part 12. The bank address calculation part 14 calculates a bank number and an address of the memory bank so that a plurality of feature data items in a window that is of at least the same size as a filter and is located at an arbitrarily defined position on a channel are held in mutually different memory banks B.

Description

Processing Unit

[0001] The present invention relates to a processing unit.

[0002] A processing unit for image recognition using a neural network is known. The convolution calculations performed by such a processing unit require a huge number of product-sum operations, in which data from input layers and intermediate layers are weighted and added using filter weight data. To speed up the neural network calculation process, a processing unit is known in which a feature plane storage unit for storing feature planes is configured with multiple memories, the feature planes are interleaved across the multiple memories on a line-by-line basis, and four pixels of data are stored per address (see Patent Document 1).

[0003] JP 2021-9566 A JP 2018-55570 A

[0004] In a convolutional neural network processing device, a huge number of product-sum operations are required for convolutional processing, resulting in a large number of accesses to the feature plane storage unit. For this reason, efficient data input / output to / from the feature plane storage unit is desired in order to reduce power consumption and speed up processing.

[0005] The present invention has been made in view of the above circumstances, and has as its object to provide a processing device that can efficiently access a feature plane storage unit.

[0006] In order to achieve the above object, a processing device of the present invention is a neural network processing device having a processing unit that performs convolutional operations by applying a filter to each channel of a feature plane, and is equipped with: a feature plane storage unit composed of multiple memory banks that can be accessed simultaneously, for storing feature data of the feature plane of a convolutional layer; and a bank address calculation unit that calculates and specifies a memory bank that holds the feature data and an address of the memory bank when inputting or outputting feature data to the feature plane storage unit, and the bank address calculation unit calculates the bank number of the memory bank for feature data and the address of the memory bank based on the channel size of the feature plane and the position of the feature data on the feature plane, in accordance with a decision rule that causes multiple pieces of feature data within a window of at least the same size as the filter at any position on the channel to be stored in different memory banks.

[0007] Furthermore, in a processing device of the present invention, a neural network processing device having a processing unit that performs a convolution operation by applying a filter to each channel of a feature plane includes: a feature plane storage unit that is composed of (M × N) memory banks that can be accessed simultaneously, where M and N are integers of 2 or greater, and that stores feature data of an input-side feature plane or an output-side feature plane of a convolution layer, and from which a maximum of M rows and N columns of feature data on a channel of the input-side feature plane are read; and a bank address calculation unit that calculates the memory bank for writing and reading feature data to and from the feature plane storage unit, the position of which on the feature plane is specified by equation (A) where SIZE_H is the channel size in the row direction of the feature plane, SIZE_W is the channel size in the column direction of the feature plane, START is the start address of the feature plane storage unit corresponding to the feature plane, X is the bank number that specifies the memory bank, and Y is the address in the memory bank, with D being the channel number, H being the row position, and W being the column position, using the following formula (A): ...(A)

[0008] According to the processing device of the present invention, the feature plane storage unit for storing feature data is composed of multiple memory banks that can be accessed simultaneously, and multiple pieces of feature data within a window the same size as a filter are held in different memory banks, and the memory banks and addresses for storing feature data are determined. This makes it possible to simultaneously read out multiple pieces of feature data to which a filter is applied, and to efficiently access the feature plane storage unit.

[0009] 10 is a block diagram showing the configuration of a processing unit; FIG. 11 is an explanatory diagram showing the configuration of a feature plane; FIG. 12 is an explanatory diagram showing the storage area of ​​a memory bank; FIG. 13 is a block diagram showing the configuration of a selector unit; FIG. 14 is a block diagram showing the configuration of a processing unit; FIG. 15 is an explanatory diagram showing the relationship between the position of feature data on the feature plane and the bank number; FIG. 16 is an explanatory diagram showing the processing procedure when a 3-row, 3-column filter is applied by the processing unit; FIG. 17 is an explanatory diagram showing the processing at the first clock when a 1-row, 1-column filter is applied by the processing unit; FIG. 18 is an explanatory diagram showing the processing at the second clock when a 1-row, 1-column filter is applied by the processing unit; FIG. 19 is a block diagram showing the configuration of a processing unit of a second embodiment; FIG. 19 is an explanatory diagram showing the processing at the first clock when the processing unit of FIG. 10 is used; FIG. 19 is an explanatory diagram showing the processing at the second clock when the processing unit of FIG. 10 is used; FIG. 20 is a block diagram showing an example of reading out 4-row, 4-column feature data and storing it in a register unit; FIG. 21 is an explanatory diagram showing an example of applying filters in parallel to four 3-row, 3-column feature data extracted from 4-row, 4-column feature data. FIG. 1 is an explanatory diagram showing an example in which a calculation unit is provided with a plurality of registers and filters are applied in parallel to four pieces of 3-by-3 feature data extracted from 4-by-4 feature data; FIG. 2 is a block diagram showing an example in which a pooling circuit is provided in a calculation unit; FIG. 3 is a block diagram showing an example in which a feature plane storage unit is power-gated; and FIG. 4 is a block diagram showing an example in which a weight storage unit is power-gated.

[0010] 1, a processing device 10 performs convolution operations in a convolution layer of a neural network for image recognition, for example. The neural network on which the processing device 10 is based has multiple layers connected together. The first layer is an input layer, which represents an image composed of, for example, RGB channels. Multiple convolution layers are connected following this input layer.

[0011] In one layer (convolutional layer), a feature surface of the previous layer (referred to as an input feature surface) is generated by performing a convolution operation using a filter (kernel) on each channel of the feature surface of the previous layer. That is, where i is an integer of 1 or greater, in the convolution operation of the (i+1)th layer, a feature surface of the (i+1)th layer is generated by performing a convolution using a filter (kernel) on each channel of the feature surface of the i-th layer.

[0012] The feature surface of each layer is composed of one or more channels. One channel is composed of multiple feature data arranged two-dimensionally. A two-dimensional array of feature data is a data structure arrangement, meaning that the position of each feature data on the channel is specified by two variables (row and column in this explanation) and that position information is assigned so that the positional relationship between feature data is specified. The position of feature data on the feature surface is specified by the position on the channel and the channel number of the channel. Note that when the input layer is the previous layer, the feature surface of the previous layer is, for example, an input image consisting of RGB channels, and the feature data is each RGB image data.

[0013] 2, in this example, when indicating the position of feature data on a channel, the column direction (height direction) position (row position) of the feature data on the channel is indicated as H, and the row direction (width direction) position (column position) is indicated as W, and is expressed as position (H, W). When distinguishing channels on the feature plane, the channel number is indicated as D, and when indicating the position of feature data on the feature plane, it is expressed as position (H, W, D). H, W, and D are all integers equal to or greater than 0, and for example, the upper left corner of the channel is expressed as position (0, 0).

[0014] 1, a processing device 10 includes a calculation unit 11, a feature plane storage unit 12, a bank address calculation unit 14, a weight storage unit 15, a selector unit 16, a network information holding unit 17, and a control unit 18 that controls each unit of the processing device 10. The functions of the processing device 10 are implemented, for example, as an ASIC. In designing an ASIC, for example, the functions of the processing device 10 can be described as source code in the C language, and RTL generated by high-level synthesis of the source code can be used. In this example, the feature plane storage unit 12 and the weight storage unit 15 are configured as on-chip memories.

[0015] The calculation unit 11 receives feature data from the feature plane storage unit 12 via the selector unit 16, and also receives filter weight data from the weight storage unit 15. The calculation unit 11 includes a plurality of calculation units 21, each of which performs 64 convolution calculations in parallel in this example. ０ ~21 ６３ These calculation units 21 ０ ~21 ６３ In this example, the calculation unit 21 ０ ~21 ６３ The calculation unit 21 performs a convolution calculation using a filter with a filter size of 3 rows and 3 columns, and a convolution calculation using a filter with a size of 1 row and 1 column. ０ ~21 ６３ When no particular distinction is made, they are referred to as the arithmetic unit 21.

[0016] One arithmetic unit 21 corresponds to one channel of the output feature plane and calculates feature data for the corresponding channel. The arithmetic unit 11 can calculate feature data for up to 64 channels in parallel. If the number of channels of the output feature plane is greater than the number of arithmetic units 21 of the arithmetic unit 11, the channels of the output feature plane are divided and calculations are performed such that calculations for the remaining channels are performed after calculations for the maximum number of channels are completed. The arithmetic unit 11 generates the output feature plane by performing a convolution calculation on each channel of the input feature plane. Note that the number of arithmetic units 21 is not limited to 64. The arithmetic unit 11 may also perform calculations other than product-sum calculations, such as pooling, activation functions, bit shifting, adding or subtracting a predetermined value to or from feature data, and copying feature data.

[0017] The feature plane storage unit 12 is a memory that holds feature planes and stores feature data. An image of the input layer is input to and written into the feature plane storage unit 12 via an interface (I / F) 22. Feature data of the feature planes of each convolution layer is input from the calculation unit 11 and written into the feature plane storage unit 12. The feature data read from the feature plane storage unit 12 is sent to the calculation unit 11 via a selector unit 16.

[0018] The feature surface storage unit 12 has 16 memory banks B ０ ~B １５ The bank number that specifies the memory bank is memory bank B. ０ , B １ , B ２ ..., B １５ The order is 0, 1, 2, ..., 15. Memory bank B ０ ~B １５ is composed of, for example, an SRAM. In the following description, the memory bank B ０ ~B １５ When there is no particular distinction between them, they will be referred to as memory bank B.

[0019] The feature surface storage unit 12 includes a memory bank B ０ ~B １５ That is, the feature plane storage unit 12 has memory banks B ０ ~B １５The same or different addresses can be simultaneously specified for each of the above, and feature data can be read and written.

[0020] In this example, 16 memory banks B are provided, and the total number of memory banks is 16, but the number of memory banks is not limited to this. The number of memory banks may be determined so that feature data within a window of at least the same size as the filter on the same channel can be simultaneously read out from the feature plane storage unit 12. In other words, the number of memory banks may be set to be such that a window that defines the range from which feature data is read out, corresponding to a filter used in the arithmetic processing device 10, is the same size as the filter, and at least a plurality of feature data within the window at any position on the channel can be stored in different memory banks B.

[0021] Here, M and N are integers equal to or greater than 2, P is a positive integer equal to or less than M, and Q is a positive integer equal to or less than N. If the maximum filter size in the arithmetic processing device 10 is P rows and Q columns (P×Q), the number of memory banks should be set to "M×N". This makes it possible to simultaneously read out each piece of feature data within a window with a window size of M rows and N columns set at any position on the same channel. Furthermore, from the viewpoint of facilitating arithmetic processing using bit shifts or the like in calculating the bank numbers described below, it is preferable that the number of memory banks is a power of 2, i.e., M and N are both powers of 2. In this example, since the maximum filter size is 3 rows and 3 columns, M and N are each set to 4, and the number of memory banks is set to 16 (=2 ４ )

[0022] Input and output of feature data to and from the feature plane storage unit 12, i.e., writing of feature data to and reading of feature data from the feature plane storage unit 12, is controlled by the control unit 18. The bank number of memory bank B into which feature data is written and the address in that memory bank B are calculated and specified by the bank address calculation unit 14. Similarly, the bank number of memory bank B from which feature data is read and the address in that memory bank B are calculated and specified by the bank address calculation unit 14. The control unit 18 writes and reads feature data to and from the addresses in memory bank B calculated by the bank address calculation unit 14.

[0023] In this example, when a convolution operation is performed using a filter with a filter size of 3 rows and 3 columns, nine pieces of feature data on the same channel corresponding to this filter are read out simultaneously. Also, when a convolution operation is performed using a filter with a filter size of 1 row and 1 column (multiplication of each piece of feature data by weight data), one piece of feature data is read out simultaneously from each of eight consecutive channels, for a total of eight pieces of feature data.

[0024] As will be described in detail later, the bank address calculation unit 14 calculates the bank number of memory bank B and the address in memory bank B when inputting or outputting feature data to or from the feature plane storage unit 12. As described above, feature data is input or output to or from the feature plane storage unit 12 using the memory bank and address of the bank number calculated by this bank address calculation unit 14.

[0025] The weight storage unit 15 holds weight data for each filter used in the convolution calculation process for each layer, and sends the read filter weight data to the calculation unit 11. For each layer of the convolution calculation, a filter is prepared in advance for each combination of a channel for that layer and a channel for the previous layer, and is held in the weight storage unit 15. The weight storage unit 15 stores nine consecutive addresses corresponding to nine pieces of weight data as sets. The weight storage unit 15 is configured to be able to simultaneously read and output the same number of sets of weight data as the calculation units 21.

[0026] In the weight storage unit 15, for a layer where a convolution operation is performed using a 3-row, 3-column filter, nine weight data constituting the filter are stored in nine consecutive addresses and organized as a set. Each data line that transmits weight data from the weight storage unit 15 to the arithmetic unit 21 corresponds one-to-one to nine positions (hereinafter referred to as local positions) such as the center, upper right, and lower left within the 3-row, 3-column filter. In this way, the weight data is input to the arithmetic unit 21 using a specific data line corresponding to the local position.

[0027] For a layer where a convolution operation is performed using a 1-row, 1-column filter, eight weight data sets for eight consecutive channels of the input feature surface are stored in eight of nine consecutive addresses, for example, in the order of the channels of the input feature surface. In this case, for example, the ninth address of the nine consecutive addresses is left blank. One set of eight weight data sets corresponds to one common channel of the output feature surface. For each channel of the output feature surface, a number of weight data sets corresponding to the channels of the input feature surface are prepared in advance. One set of weight data for the 1-row, 1-column filter is output via data lines corresponding to eight local positions (e.g., excluding the bottom right) of the 3-row, 3-column filter, according to the stored addresses. This allows the arithmetic processing device 10 to efficiently perform a convolution operation using a 1-row, 1-column filter by utilizing a 3-row, 3-column arithmetic unit.

[0028] The weight storage unit 15 is configured as, for example, an MRAM using a magnetic tunnel junction (MTJ) element, or a nonvolatile memory such as FeRAM, PRAM, ReRAM, etc. The weight storage unit 15 can also be configured as an SRAM or the like, but if the weight storage unit 15 is configured as a nonvolatile memory, it is advantageous in terms of reducing power consumption because there is no need to load a filter when the arithmetic processing device 10 starts operating.

[0029] The selector unit 16 is provided between the feature plane storage unit 12 and the calculation unit 11. The selector unit 16 associates a plurality of pieces of feature data read out from the feature plane storage unit 12 with local positions within the window, and outputs them to the calculation unit 11. In this way, when a 3-row, 3-column filter is used, the selector unit 16 ensures that the input of the plurality of pieces of feature data to the calculation unit 11 is based on the two-dimensional array on the channels, and when a 1-row, 1-column filter is used, the selector unit 16 ensures that the input of the plurality of pieces of feature data to the calculation unit 11 is based on the channel order of the feature data.

[0030] The network information storage unit 17 stores network information such as the hierarchical structure of the neural network processed by the arithmetic processing unit 10, the number of channels in each layer, the channel size, the filter size, etc. When a convolution operation is performed, this network information is read out by the control unit 18 and sent to the bank address calculation unit 14, where it is used to calculate the bank number and address.

[0031] As shown in Fig. 3, memory bank B stores one piece of feature data at one address (Y). Memory bank B stores feature data divided by hierarchy. When inputting or outputting feature data to or from feature plane storage unit 12, each memory bank B is assigned one address, and reads or writes feature data from or to the designated address.

[0032] The feature data for one layer is stored in an area (hereinafter referred to as a storage area) with consecutive addresses in memory bank B. The start address (top address) of each storage area is predetermined. The i-th storage area that stores the feature data for the i-th layer has a start address START. ｉ In the example shown in FIG. 3, for example, address 0 (=START １ ) to address START ２ The first storage area, address START, stores feature data of the first layer (input layer) up to -1. ２In the feature plane storage unit 12, an area of ​​memory B where feature data of the second layer is stored may be released and allocated as a storage area of ​​another layer. For example, in the convolution operation process of the fourth layer, if a continuous area necessary to store the feature plane (feature data) of the fourth layer can be secured in the first storage area and the second storage area, a fourth storage area where feature data of the fourth layer is stored is set in the area that was the first storage area and the second storage area. In this case, the start address (START ４ ) is the start address of the first storage area (START １ ) is the same as

[0033] 4, the selector section 16 is made up of nine data selectors 24. Each data selector 24 has 16 inputs and one output, and is connected to a memory bank B on the input side. ０ ~B １５ Each data selector 24 is independently controlled by a selection signal from the control unit 18, and the memory bank B ０ ~B １５ and outputs the feature data read from the selected memory bank B. The control of this data selector 24 is performed based on the position of the feature data on the feature plane to be read, the position of the window on the channel, etc. The number of data selectors 24 in the selector unit 16 only needs to be the same as the number of feature data items to be output from the selector unit 16. For example, if 16 feature data items are to be output simultaneously to the calculation unit 11, the selector unit 16 may be configured with 16 data selectors 24.

[0034] In order to ensure that the input of multiple feature data to the calculation unit 11 is based on a two-dimensional array on the channel, for example, each data line transmitting feature data from the selector unit 16 to each calculation unit 21 corresponds one-to-one to a local position within the 3-row, 3-column window, similar to the data line between the weight storage unit 15 and the calculation unit 21. That is, the selector unit 16 inputs feature data to the calculation unit 21 using a specific data line corresponding to the local position. Each data selector 24 is associated with one of the local positions within the window and connected to the data line for that corresponding local position. As a result, under the control of the control unit 18, the selector unit 16 can input the nine feature data within the window read from the nine memory banks B to each calculation unit 21 in association with their respective local positions, regardless of the memory bank B from which they were read.

[0035] When performing a convolution operation using a 3-row, 3-column filter, the control unit 18 controls the selector unit 16 so that the data selector 24 whose local position matches that of the feature data is connected to the memory bank B from which feature data is read. When performing a convolution operation using a 1-row, 1-column filter, each data selector 24 is associated with a channel number, and the control unit 18 controls each data selector 24 of the selector unit 16 so that the data selector 24 corresponding to the channel number of the feature data is connected to the memory bank B from which feature data is read. This ensures that multiple feature data are input to the calculation unit 11 in the order of the channels of the feature data. In the following description, the control of the selector unit 16 to input to the calculation unit 11 is referred to as feature data rearrangement.

[0036] 5, the arithmetic unit 21 in this example is provided with a convolution circuit 29 including a product-sum calculator 26 that applies a filter to feature data, an adder 27, and a register 28. One arithmetic unit 21 corresponds to one channel of the output feature plane, and as described above, one arithmetic unit 21 calculates each piece of feature data for one channel of the output feature plane.

[0037] When a convolution operation is performed using a 3-row, 3-column filter, the product-sum calculator 26 receives as input nine pieces of feature data read from the feature plane storage unit 12 and nine pieces of filter weight data read from the weight storage unit 15. The product-sum calculator 26 multiplies the feature data and weight data input from the same data line having the same associated local position, and calculates the sum of the multiplication results of the nine pieces of feature data and weight data.

[0038] When a convolution operation is performed using a 1-row, 1-column filter, eight pieces of feature data for eight channels, each consisting of one piece of feature data for each channel, are input to the product-sum calculator 26 from the feature plane storage unit 12 via the selector unit 16. Also, eight sets of weight data for each filter are input to the product-sum calculator 26 from the weight storage unit 15. As described above, the eight sets of weight data and the eight pieces of feature data input via the selector unit 16 are input to the product-sum calculator 26 so as to correspond to each other at local positions of the 3-row, 3-column filter (window). Thus, the product-sum calculator 26 multiplies the feature data and weight data for corresponding channels of the input-side feature plane, and calculates a product-sum operation result, which is the sum of the multiplication results.

[0039] The adder 27 adds the data held in the register 28 to the result of the sum-of-products operation from the sum-of-products operator 26. The register 28 holds the result of the addition by the adder 27. This register 28 is initialized when the operation of the operation unit 21 starts, and the value held in the register 28 is reset to "0". As a result, the register 28 newly holds the result of the sum-of-products operation output from the sum-of-products operator 26 multiplied by the value held in the register 28.

[0040] As described above, the bank address calculation unit 14 calculates the bank number that specifies the memory bank B when feature data is written to the feature plane storage unit 12, and the address of the memory bank B specified by that bank number. It also calculates the bank number that specifies the memory bank B when feature data is read from the feature plane storage unit 12, and the address of the memory bank B specified by that bank number.

[0041] The bank address calculation unit 14 calculates a bank number for writing feature data and an address of the memory bank B for that bank number so that multiple pieces of feature data within a window at least as large as the filter at any position on the channel are stored in different memory banks B, and calculates a bank number for reading feature data written at the bank number and address determined in this way and an address of the memory bank B for that bank number. The bank numbers and addresses for writing and reading are both calculated based on the channel size of the feature plane and the position of the feature data on the feature plane. That is, the bank address calculation unit 14 calculates the bank number of memory bank B for feature data and its address in that memory bank based on the channel size of the feature plane and the position of the feature data on the feature plane, in accordance with a decision rule that causes multiple pieces of feature data within a window at least as large as the filter at any position on the channel to be stored in different memory banks B.

[0042] This allows multiple feature data within a window of the same size as a filter on one channel at any filter position to be read simultaneously from different memory banks B, thereby efficiently reading from the feature plane storage unit 12.

[0043] In this example, feature data at the same position on a channel is stored in different memory banks for multiple consecutive channels, with the maximum number of channels being the same as the number of memory banks. That is, the bank address calculation unit 14 calculates the bank number of memory bank B for feature data and the address on that memory bank according to a decision rule that also includes this content. This improves the efficiency of access to the feature plane storage unit 12 when writing feature data and performing convolution operations using a 1-row, 1-column filter. In this example, since there are 16 memory banks, feature data at the same position on 16 consecutive channels can be read and written simultaneously.

[0044] The bank number X of memory bank B for feature data at position (H, W, D) on the feature plane and the address Y of memory bank B are calculated by equation (1). As shown in Figure 2, "SIZE_H" in equation (1) is the channel size in the column direction (number of rows), and "SIZE_W" is the channel size in the row direction (number of columns). Note that "%" in equation (1) is a modulo operator.

[0045] ...(1)

[0046] The above formula (1) assumes that the number of memory banks is 16, and allows each piece of feature data within a 3-row, 3-column window to be stored in a different memory bank B, while also allowing each piece of feature data in 4 rows and 4 columns on a channel to be stored in a different memory bank B. That is, each piece of feature data in 4 rows and 4 columns on one channel can be simultaneously written to and read from the feature plane storage unit 12. Furthermore, formula (1) determines the bank numbers and addresses so that 16 pieces of feature data that are at the same row position but consecutive in the column direction on one channel are stored in different memory banks B. This method of determining the bank numbers and addresses is useful for efficient writing, since it allows 16 pieces of image data that are consecutive in the column direction of the input image to be simultaneously written to the feature plane storage unit 12 in the input layer.

[0047] When reading out each piece of feature data within a window on a channel in the input feature plane of the i-th layer for a convolution operation on the (i+1)-th layer, bank address calculation unit 14 uses the position (H, W, D) on the input feature plane of each piece of feature data within the window and the column width SIZE_H and row width SIZE_W of the channel of the input feature plane to determine the bank number X of memory bank B for each piece of feature data and the address Y of memory bank B. Control unit 18 controls data selector 24 based on the position (H, W) on the channel of the input feature plane and the bank number X determined for that position (H, W).

[0048] For reference, Fig. 6 shows the relationship between the position (H, W, D) of feature data on the feature plane and the bank number X calculated by equation (1). Fig. 6 shows the channel numbers 0 and 1, with the channel size SIZE_H of each channel set to "4" and the channel size SIZE_W set to "28". Note that Fig. 6 shows two shaded areas of 4 rows and 4 columns as an example.

[0049] If the window size is M rows and N columns as described above, the number of memory banks should be M x N. In this case, the bank number X of memory bank B and the address Y of memory bank B can be found using equation (2). Note that equation (1) is equation (2) with M = N = 4.

[0050] ... (2)

[0051] Instead of equation (2), the bank number X of memory bank B and the address Y of memory bank B may be calculated using equation (3).

[0052] ...(3)

[0053] Next, regarding the operation of the above configuration, the procedure for convolution calculation processing by the processor 10 using a 3-by-3 filter on the (i+1)th layer will be described. As schematically shown in FIG. 7 , the input feature surface on the i-th layer is composed of channels ChA0, ChA1, ..., with channel numbers 0, 1, .... From this input feature surface, an output feature surface is generated, composed of channels ChB0, ChB1, ..., with channel numbers 0, 1, ..., on the (i+1)th layer. For simplicity's sake, it is assumed that zero-padded channels of the input feature surface are used to generate zero-padded channels of the output feature surface. The position of the feature data on the channels of the output feature surface calculated by the convolution calculation processing is assumed to be the same as the center position of the channels on the input feature surface of the 3-by-3 window. The feature data of each channel of the input feature surface is calculated by the bank address calculator 14 based on Equation (1) and written to address Y in memory bank B with the specified bank number.

[0054] In response to the input of the first clock supplied to the arithmetic processing unit 10, a convolution operation is performed on nine pieces of feature data within a window Ra centered on the position (1,1) of the first channel ChA0 in order to calculate the feature data at the position (1,1) of the channels ChB0, ChB1, ... (step ST1). First, in order to read the nine pieces of feature data from the feature plane storage unit 12, the start address START of the ith storage area common to each channel of the input-side feature plane is ｉ The channel sizes SIZE_H and SIZE_W and the positions (=(H, W, D)) of each of the nine pieces of feature data on the feature plane are input from the control unit 18 to the bank address calculation unit 14. At this time, the positions of the nine pieces of feature data on the input side feature plane are (0,0,0) to (0,2,0), (1,0,0) to (1,2,0), and (2,0,0) to (2,2,0).

[0055] Based on the information input from the control unit 18, the bank address calculation unit 14 calculates the bank number and address Y of memory bank B holding nine pieces of feature data according to equation (1), and sends the result to the control unit 18. Since each piece of feature data of the input-side feature surface is written to an address in memory bank B with a bank number determined based on equation (1), the bank numbers of the memory banks holding the nine pieces of feature data will be different from one another.

[0056] When the bank number and address are input, the control unit 18 sends a selection signal to the selector unit 16 to switch each data selector 24 so that the memory bank B specified by the bank number corresponding to each local position is selected by the data selector 24 associated with that local position.

[0057] For example, based on equation (1), the control unit 18 can identify the bank number of memory bank B from which each piece of feature data in 3 rows and 3 columns is to be read, based on its position on the feature plane (row position, column position, channel number). Meanwhile, for each piece of feature data in 3 rows and 3 columns to be read, the control unit 18 can identify a local position within the 3 rows and 3 columns window based on its position on the feature plane. This allows each local position in 3 rows and 3 columns to correspond to the bank number of each memory bank B, and enables switching of each data selector 24 as described above.

[0058] After the selector section 16 is switched, feature data is read from the feature plane storage section 12 under the control of the control section 18 in accordance with the bank number and address from the bank address calculation section 14. As a result, one piece of feature data is read from each of the ith storage areas of the nine memory banks B, and nine pieces of feature data are simultaneously read from the feature plane storage section 12 and output to the selector section 16. The nine pieces of feature data are rearranged by the selector section 16 and input to each arithmetic unit 21 of the arithmetic section 11.

[0059] In parallel with the reading of the nine pieces of feature data, nine pieces of weight data for each of the filters FA0B0, FA0B1, ... corresponding to the channels ChB0, ChB1, ... for the channel ChA0 are read from the weight storage unit 15 and input to each arithmetic unit 21. At this time, each piece of weight data for the filter FA0B0 corresponding to the channel ChB0 is input to the arithmetic unit 21. ０ The weight data of the filter FA0B1 corresponding to the channel ChB1 is calculated by the calculation unit 21. １ , and one arithmetic unit 21 is associated with one channel of the output feature plane, and each weight data of the corresponding filter is input to each arithmetic unit 21.

[0060] As a result, each calculation unit 21 receives the feature data read within the window and the weight data of the filter, multiplies those having the same local position, and outputs the sum of the multiplication results as the product-sum calculation result. Since the register 28 is initialized and reset to "0" at the start of calculation, the product-sum calculation result from the product-sum calculator 26 is input to the register 28 via the adder 27. As a result, the register 28 holds the product-sum calculation result from the product-sum calculator 26 as an integrated value.

[0061] After the first product-sum operation result is stored in register 28b as described above, in response to the input of the second clock, a convolution operation is performed on nine feature data within window Ra centered on position (1, 1) of second channel ChA1 of the input feature plane (step ST2). In order to read out the nine feature data from channel ChA1 from feature plane storage unit 12, the start address START of the ith storage area common to each channel of the input feature plane is set to START. ｉ The control unit 18 inputs the channel sizes SIZE_H and SIZE_W and the positions on the feature plane for each of the nine feature data to the bank address calculation unit 14. At this time, the positions on channel ChA1 of the nine feature data within window Ra are the same as those for channel ChA0, i.e., positions (0,0,1) to (0,2,1), (1,0,1) to (1,2,1), and (2,0,1) to (2,2,1).

[0062] In response to the input from the control unit 18, the bank address calculation unit 14 calculates, for each of the nine feature data positions, the bank number and address of the memory bank B in which the nine feature data are stored, according to equation (1).Then, the bank number and address calculated by the bank address calculation unit 14 are input to the control unit 18.

[0063] The control unit 18 sends a selection signal to the selector unit 16 to switch each data selector 24 so that the memory bank B specified by the bank number corresponding to each local position is selected by the data selector 24 associated with that local position. Thereafter, one piece of feature data is read from each of the ith storage areas of the nine memory banks B according to the bank numbers and addresses from the bank address calculation unit 14, and nine pieces of feature data are simultaneously read from the feature plane storage unit 12 and output to the selector unit 16. The nine pieces of feature data are rearranged by the selector unit 16 and input to each arithmetic unit 21 of the arithmetic unit 11.

[0064] In parallel with the reading of the nine pieces of feature data, nine pieces of weight data for each of the filters FA1B0, FA1B1, ... corresponding to channels ChB0, ChB1, ... for channel ChA1 are read from the weight storage unit 15 and input to each arithmetic unit 21. As a result, in each arithmetic unit 21, the product-sum calculator 26 performs a product-sum operation on each piece of feature data read within the window and each piece of filter weight data, and outputs the result as a second product-sum operation result to the adder 27. The adder 27 adds the integrated value (first product-sum result) held in the register 28 and the result of the second product-sum operation, and the addition result is held in the register 28 as a new integrated value.

[0065] In response to the input of the third clock, the convolution operation is performed in the same procedure as above on nine pieces of feature data within window Ra whose center is the position (1,1) of the third channel of the input feature plane. Similarly, for each subsequent clock input, convolution operations are sequentially performed on nine pieces of feature data within window Ra whose center is the position (1,1) of the remaining channels of the input feature plane. By performing the convolution operation on nine pieces of feature data within window Ra whose center is the position (1,1) of the final channel of the input feature plane in the above manner, the feature data at position (1,1) in channels ChB0, ChB1, ... of the output feature plane is calculated. At this stage, arithmetic unit 21 ０ , 21 １ The integrated value held in the register 28 is the feature data of the position (1,1) on the channels ChB0, ChB1, . . .

[0066] The control unit 18 sets the start address START of the (i+1)th storage area common to each channel of the input feature plane. （ｉ＋１） The control unit 18 sends the position of the obtained feature data on the feature plane, i.e., the channel numbers 0 to 15 and the position (1, 1) of the feature data on the channel, along with the channel sizes SIZE_H and SIZE_W. In response to this, the bank address calculation unit 14 calculates the bank number and address corresponding to the position (1, 1) on each channel of the channel numbers 0 to 15 based on equation (1), and sends them to the control unit 18.

[0067] After this, the control unit 18 executes the calculation unit 21 ０ ~21 １５The 16 integrated values ​​are extracted from each register 28 as feature data for the position (1, 1) on the channel with channel numbers 0 to 15 on the output-side feature plane, and sent to the feature plane storage unit 12. Then, under the control of the control unit 18, each of the 16 feature data is input to a memory bank B specified by the bank number for the position (1, 1) of the corresponding channel number 0 to 15, and written to address Y of the memory bank B to which it was input. In this way, 16 feature data are simultaneously written to the (i+1)th storage area of ​​the 16 memory banks B. In this way, feature data can be efficiently written into the feature plane storage unit 12 in groups of 16.

[0068] If the number of channels of the input feature surface is greater than 16, then for each channel from channel number 16 onwards, the same procedure is used to write 16 sets of feature data at the position (1, 1) into feature surface storage unit 12. In this way, 16 sets of feature data at the position (1, 1) for a maximum of 64 channels of the input feature surface are written into feature surface storage unit 12.

[0069] If the number of channels in the input feature plane is greater than 64, the feature data for the position (1, 1) of each channel from channel number 64 onwards is calculated by performing a convolution operation for each of the 64 channels in the same manner, and 16 sets of calculated feature data are written to the feature plane storage unit 12.

[0070] As described above, after the feature data for position (1, 1) of each channel on the input feature plane is written to feature plane storage unit 12, feature data for position (1, 2) of channels ChB0, ChB1, ... on the input feature plane are calculated by convolution calculation and written to feature plane storage unit 12. In this case, nine feature data items are read out within window Ra centered at position (1, 2) on each channel on the input feature plane.

[0071] Thereafter, while shifting the window Ra in the column and row directions, the same procedure as above is used to calculate feature data at each position of each channel of the output feature surface and write the data to the feature surface storage unit 12. In this way, the feature surface of the output feature surface is generated.

[0072] As described above, each piece of feature data for each channel of the feature plane of the (i+1)th layer is written to an address in memory bank B determined based on equation (1). That is, the feature data in four rows and four columns is stored in different memory banks B. Therefore, in the subsequent (i+2)th layer, when the feature plane of the (i+1)th layer is used as the input feature plane and a convolution operation is performed on each channel of the input feature plane using a three-row, three-column filter, the nine feature data within the window can be simultaneously read from different memory banks B, and efficiently read from the feature plane storage unit 12. Of course, the feature data in four rows and four columns can also be simultaneously read.

[0073] Next, the procedure for convolution calculation processing using a 1-row, 1-column filter will be described. As in the example using the above 3-row, 3-column filter, the feature surface of the i-th layer, which serves as the input feature surface, is composed of channels ChA0, ChA1, ..., and from this input feature surface, an output feature surface composed of channels ChB0, ChB1, ... of the i+1-th layer is generated. Note that in this description, it is assumed that the channels of the i-th layer and the i+1-th layer are not zero-padded.

[0074] In processing using a 1-row, 1-column filter, one piece of feature data is read for each channel of the input feature plane, so that feature data for eight channels is read at a time. Also, one arithmetic unit 21 performs convolution calculations for one channel, and the arithmetic units 21 perform the convolution calculations in parallel.

[0075] In response to the input of the first clock, the control unit 18 reads out the feature data at the position (0,0) of each channel of channel numbers 0 to 7 in the input feature plane, as shown in FIG. 8, in order to calculate the feature data at the position (0,0) of channels ChB0, ChB1, .... To do this reading, the control unit 18 reads out the feature data at the position (0,0) of each channel of channel numbers 0 to 7 in the input feature plane, as shown in FIG. ｉ The channel sizes SIZE_H and SIZE_W and the positions (0,0,0) to (0,0,7) of the eight pieces of feature data to be read on the feature plane are sent to the bank address calculation unit 14 .

[0076] Based on the information input from the control unit 18, the bank address calculation unit 14 calculates the bank number and address of memory bank B for each of the eight feature data to be read using equation (1), and outputs the results to the control unit 18. Each feature data of the input-side feature surface is written to memory bank B with a bank number determined based on equation (1), and the eight feature data to be read are located at the same channel position and belong to eight consecutive channels. Therefore, the bank numbers calculated for the same eight feature data will be different from one another.

[0077] The control unit 18 receives the bank number and address associated with each local position, and sends a selection switching signal to the selector unit 16 to switch each data selector 24 so that the data selector 24 associated with each local position selects the memory bank B specified by the bank number associated with that local position. In this way, the eight data selectors 24 are connected to the memory bank B from which feature data is read.

[0078] After switching the selector section 16, the control section 18 reads out feature data from the feature plane storage section 12 in accordance with the bank number and address from the bank address calculation section 14. As a result, one piece of feature data is read out from each of the ith storage areas of the eight memory banks B, and eight pieces of feature data are simultaneously read out from the feature plane storage section 12 and output to the selector section 16. The eight pieces of feature data are rearranged via the selector section 16 and input to each arithmetic unit 21 of the arithmetic section 11.

[0079] Meanwhile, in parallel with the reading of the feature data, weight data is read from the weight storage unit 15 and input to each arithmetic unit 21. As a result, the weight data input to each arithmetic unit 21 corresponds to the channels ChB0, ChB1, ... of the output feature plane to which the arithmetic unit 21 corresponds, and is a weight data set consisting of eight weight data corresponding to the channels ChA0, ChA1, ..., ChA7 of the input feature plane.

[0080] As described above, eight pieces of feature data and eight pieces of weight data are input to each arithmetic unit 21. In the arithmetic unit 21, the eight pieces of feature data and eight pieces of weight data input for the same channel are multiplied together, and the sum of the multiplication results is output as the product-sum calculation result from the product-sum calculator 26. Since the register 28 is initialized and reset to "0" at the start of calculation, the first product-sum calculation result from the product-sum calculator 26 is stored in the register 28 via the adder 27 as an integrated value.

[0081] For example, the arithmetic unit 21 corresponding to the channel ChB0 ０ A set of weight data consisting of eight pieces of feature data at positions (0,0) of channels ChA0, ChA1, ..., ChA7 and eight pieces of weight data for channels ChA0, ChA1, ..., ChA7 for channel ChB0 is input to the arithmetic unit 10. Then, the feature data at position (0,0) of channel ChA0 is multiplied by the weight data for channel ChA0 for channel ChB0, and the feature data at position (0,0) of channel ChA1 is multiplied by the weight data for channel ChA1 for channel ChB0, and a product-sum operation is performed on the eight pieces of feature data and the eight weight data.

[0082] In response to the input of the second clock, the control unit 18 reads out the feature data at the position (0,0) of each channel of channel numbers 8 to 15 in the input feature plane, as shown in Figure 9. To do this, the control unit 18 reads out the start address START of the i-th storage area common to the input feature plane. ｉ The channel sizes SIZE_H and SIZE_W and the positions (0,0,8) to (0,0,15) on the feature plane of the eight pieces of feature data to be read are sent to the bank address calculation unit 14. Thereafter, using the same procedure, the feature data at the position (0,0) of each channel of channel numbers 8 to 15 is read from the feature plane storage unit 12.

[0083] Furthermore, a set of weight data is read out from weight storage unit 15 and input to each arithmetic unit 21. At this time, the weight data read out from weight storage unit 15 and input to each arithmetic unit 21 corresponds to channels ChB0, ChB1, ... of the output feature plane to which arithmetic unit 21 corresponds, and is a set of eight weight data corresponding to channels ChA8, ChA9, ..., ChA15 of the input feature plane.

[0084] As a result, in each arithmetic unit 21, a multiply-and-accumulate operation is performed on the eight feature data at position (0,0) of channels ChA8 to ChA15 and the eight weight data of channels ChA8 to ChA15 for the channels of the input feature plane, and the result of the second multiply-and-accumulate operation is added to the result of the first multiply-and-accumulate operation held in register 28, and the result is held in register 28 as new content.

[0085] In response to the input of the third clock, each feature data at position (0,0) in the next 8 channels on the input side feature plane is read from the feature plane storage unit 12, and the weight data is also read, and a product-sum operation is performed in the same manner as above. Thereafter, each feature data at position (0,0) on the input side feature plane is read in the same manner for every 8 channels up to the final channel, and the weight data is also read, and a product-sum operation is performed. When the convolution operation of the feature data at position (0,0) on the final channel is completed, each operation unit 21 ０ , 21 １ . . , the feature data at the position (0, 0) in the corresponding channels ChB0, ChB1, . . . of the (i+1)th layer are held in the register 28.

[0086] The control unit 18 sets the start address START of the (i+1)th storage area common to each channel of the output side feature plane. （ｉ＋１）The control unit 18 sends the channel sizes SIZE_H and SIZE_W, as well as the position of the obtained feature data on the feature plane, i.e., the channel numbers 0 to 15 and the position (0, 0) of the feature data on the channel. When this information is input from the control unit 18, the bank address calculation unit 14 calculates the bank number and address Y corresponding to the position (0, 0) on each channel of the channel numbers 0 to 15 based on equation (1).

[0087] Thereafter, under the control of the control unit 18, 16 convolution operation results are extracted from the register 28 of each arithmetic unit 21 and written to the feature plane storage unit 12 as feature data at position (0,0) of the (i+1)th layer. The procedure at this time is the same as when a 3-row, 3-column filter is used. The 16 feature data are simultaneously written to the (i+1)th storage areas of 16 memory banks B, and feature data at position (0,0) for up to 64 channels of the output feature plane are written to the feature plane storage unit 12 in groups of 16. In this way, the feature data generated by the convolution operation can be efficiently written to the feature plane storage unit 12 in groups of 16.

[0088] If the number of channels in the output feature plane is greater than 64, feature data for the position (0, 0) of each channel from channel number 63 onwards is calculated by convolution operation in the same procedure for each of the 64 channels, and 16 sets of calculated feature data are written to the feature plane storage unit 12.

[0089] As described above, the feature data for position (0, 0) of each channel in the (i+1)th layer is written to feature plane storage unit 12. Then, while sequentially changing the channel of the input feature plane to be read, feature data for each channel of the output feature plane is calculated in a similar procedure, and written to feature plane storage unit 12, thereby generating the feature plane in the (i+1)th layer.

[0090] As described above, the arithmetic processing device 10 can efficiently read feature data from and write feature data to the feature plane storage unit 12 not only in convolution calculation processing using a 3×3 filter but also in convolution calculation processing using a 1×1 filter. Furthermore, efficient access can reduce power consumption.

[0091] Second Embodiment The second embodiment is designed to reduce the number of times weight data is read from the feature surface storage unit and the weight storage unit. Note that the second embodiment is similar to the first embodiment except for the following description, and the same reference numerals are used to designate substantially the same components, and detailed descriptions thereof will be omitted.

[0092] 10 shows a configuration in which feature data of P rows and Q columns in a window is sequentially input to the calculation unit 11 while the window is moved to multiple positions in one channel and then to the next channel, and the filter weight data corresponding to that channel is read out and input to the calculation unit 11 each time the window moves to the next channel. The convolution circuit 29 of the calculation unit 21 in this example includes a product-sum calculator 26, an adder 27, four registers 28a to 28d, and selectors 41a to 41d and 42. A plurality of calculation units 21 are provided in the calculation unit 11. The four registers 28a to 28d form a register unit 31.

[0093] The selectors 41a to 41d have two inputs and one output, and are provided corresponding to the registers 28a to 28d. The input side of the selector 41a is connected to the output of the adder 27 and the output of the corresponding register 28a, and the output side is connected to the input of the corresponding register 28a. As a result, when the selector 41a selects the adder 27, the value held by the register 28a is updated to the value output from the adder 27, and when the selector 41a selects the register 28a, the content is maintained unchanged. The input sides of the selectors 41b to 41d are connected to the output of the adder 27 and the output of the corresponding registers 28b to 28d, and the output sides are connected to the inputs of the corresponding registers 28b to 28d, and the registers 28b to 28d operate in the same manner as the register 28a.

[0094] The selector 42 has four inputs and one output, and its inputs are connected to the outputs of the registers 28a to 28d, and its output is connected to the adder 27. The selector 42 selects one of the registers 28a to 28d and inputs the data held by the selected register to the adder 27.

[0095] The selectors 41a to 41d are switched in sequence by the control unit 18, with one selector selecting the adder 27 and the other selectors selecting the corresponding register. The selector 42 is switched by the control unit 18 in synchronization with the selectors 41a to 41d so as to select the register corresponding to one of the selectors 41a to 41d that has selected the adder 27.

[0096] When a convolution operation is performed using a 3-row, 3-column filter, the product-sum calculator 26 receives, for each clock, nine pieces of feature data read from the feature plane storage unit 12 and, for each four clocks, nine pieces of filter weight data read from the weight storage unit 15. The product-sum calculator 26 holds, for example, the feature data and weight data input to a plurality of registers provided therein, and these are updated when new data is input.

[0097] In the following description, it is assumed that an output feature surface consisting of zero-padded channels of the (i+1)th layer is generated from each channel of the zero-padded input feature surface of the i-th layer, and that the position of feature data on the channels of the output feature surface calculated by the convolution operation is the same as the center position of a 3-by-3 window on the channels of the input feature surface at that time.

[0098] For example, in response to the input of the first clock, selector 41a is switched to select adder 27, and selector 42 is switched to select register 28a. Furthermore, the values ​​held in registers 28a to 28d are reset to "0." After this, as shown in step ST21 of FIG. 11 , nine feature data within window Ra centered on position (1, 1) of channel ChA0 on the input-side feature plane are read out and input to each arithmetic unit 21. In parallel with this, weight data for filters FA0B0, FA0B1, ... corresponding to channels ChB0, ChB1, ... for channel ChA0 are read out from weight storage unit 15, and input to the arithmetic units 21 of the corresponding channels.

[0099] In each arithmetic unit 21, the product-sum calculator 26 performs a product-sum calculation on the nine input feature data and nine weight data, and the result is sent to the adder 27. Since the value held in the register 28a is "0", the product-sum calculation result output from the product-sum calculator 26 is held in the register 28a. As a result, in each arithmetic unit 21, the product-sum calculation result obtained by applying the filter of the output feature plane corresponding to each arithmetic unit 21 to the feature data within the window centered on the position (1, 1) of channel ChA0 is held in the register 28a. For example, ０ In this example, the filter FA0B0 is applied to the feature data within a window centered at position (1,1) of channel ChA0, and the result is written to register 28a.

[0100] In response to the input of the second clock, selector 41b is switched to select adder 27, and selector 42 is switched to select register 28b. Thereafter, as shown in step ST22, nine feature data within window Ra centered on position (1, 2) of channel ChA0 of the input-side feature plane are read out and input to each arithmetic unit 21. At this time, weight data for filters FA0B0, FA0B1, ... corresponding to channels ChB0, ChB1, ... for channel ChA0 have already been input and held in each arithmetic unit 21, so no weight data is read out.

[0101] In each arithmetic unit 21, the product-sum calculator 26 performs a product-sum operation on the nine weight data input in response to the second clock and the weight data input in response to the first clock, and the product-sum operation result is sent to the adder 27. As in the case of register 28a, the product-sum operation result from the product-sum calculator 26 is held in register 28b. As a result, in each arithmetic unit 21, the product-sum operation result obtained by applying the filter of the output feature plane corresponding to that arithmetic unit 21 to the feature data within a window centered on position (1, 2) of channel ChA0 is held in register 28b.

[0102] In response to the input of the third clock, selector 41c is switched to select adder 27, and selector 42 is switched to select register 28c. Thereafter, as shown in step ST23 of Fig. 12, nine pieces of feature data within window Ra centered on position (2,1) of channel ChA0 of the i-th layer are read and input to each arithmetic unit 21. As a result, each arithmetic unit 21 applies its corresponding (i+1)-th layer filter to the feature data within the window centered on position (2,1) of channel ChA0, and the result of the product-sum operation is stored in register 28c.

[0103] In response to the input of the fourth clock, selector 41d is switched to select adder 27, and selector 42 is switched to select register 28d. Thereafter, as shown in step ST24, nine pieces of feature data within window Ra centered on position (2,2) of channel ChA0 of the i-th layer are read and input to each arithmetic unit 21. As a result, each arithmetic unit 21 applies its corresponding (i+1)-th layer filter to the feature data within the window centered on position (2,2) of channel ChA0, and the product-sum operation result is stored in register 28a. Note that the order in which the center of window Ra of the i-th layer channel is moved to four positions is merely an example, and the four positions may be moved in any order.

[0104] After applying the filters to the feature data within window Ra centered on the four positions of channel ChA0 as described above, at the fifth clock, the read target on the input feature plane is shifted from channel ChA0 to channel ChA1. Again, selector 41a selects adder 27, and selector 42 selects register 28a. After this, nine feature data within window Ra centered on position (1,1) of channel ChA1 are read and input to each arithmetic unit 21. In parallel with this, weight data for filters FA1B0, FA1B1, ... corresponding to channels ChB0, ChB1, ... for channel ChA1 are read from weight storage unit 15 and input to the arithmetic units 21 of the corresponding channels.

[0105] In response to the sixth clock, selector 41b selects adder 27 and selector 42 selects register 28b. Thereafter, nine pieces of feature data within a window Ra centered on the position (1,2) of channel ChA1 are read and input to each arithmetic unit 21. In response to the seventh clock, selector 41c and selector 42 are switched over in the same manner as when the third clock was input, and nine pieces of feature data within a window Ra centered on the position (2,1) of channel ChA1 are read and input to each arithmetic unit 21. In response to the eighth clock, selector 41d and selector 42 are switched over in the same manner as when the fourth clock was input, and nine pieces of feature data within a window Ra centered on the position (2,1) of channel ChA1 are read and input to each arithmetic unit 21.

[0106] After applying the filter to each piece of feature data within window Ra centered on the four positions of channel ChA1 in the above manner, the readout target on the input feature plane is shifted from channel ChA1 to channel ChA2, and the filter is applied to each piece of feature data within window Ra centered on the four positions of channel ChA2 in the same manner. Thereafter, the filter is applied while successively shifting the readout target channels on the input feature plane to the final channel in the same manner, with window Ra being moved to the same position on each channel in the same manner.

[0107] By reading out the feature data and weight data as described above, the registers 28a to 28d of each arithmetic unit 21 hold the results of the convolution operations at the positions (1, 1), (1, 2), (2, 1), and (2, 2) of the output feature plane corresponding to the arithmetic unit 21.

[0108] Under the control of the control unit 18, the convolution operation results are extracted from the registers 28a to 28d of each arithmetic unit 21 as feature data of the (i+1)th layer and written to the feature plane storage unit 12. At this time, 16 convolution operation results (feature data) at the same position of consecutive channels are extracted from each arithmetic unit 21 and written to the feature plane storage unit 12, thereby enabling efficient writing.

[0109] If the number of channels in the output feature plane is greater than 64, the feature data at the four positions (1, 1), (1, 2), (2, 1), and (2, 2) of the channels with channel number 63 and onward are calculated by performing convolution operations for each of the 64 channels in the same manner, and 16 sets of calculated feature data are written to the feature plane storage unit 12.

[0110] The feature data for the four positions (1,1), (1,2), (2,1), and (2,2) of each channel on the output feature surface are written to feature surface storage unit 12. Then, while changing the four positions on the channels of the input feature surface, the feature data for each channel of the output feature surface is calculated in a similar procedure and written to feature surface storage unit 12, thereby generating a feature surface of the (i+1)th layer.

[0111] In this example, as described above, weight data is read from the weight storage unit 15 every four times nine pieces of feature data are input to the arithmetic unit 21, i.e., every four clocks, so the number of times the weight storage unit 15 is accessed can be reduced.

[0112] In this example, a case where a filter with 3 rows and 3 columns is applied is described, but the same can be applied to a case where a filter with 1 row and 1 column is applied.

[0113] In the above example, the feature plane storage unit 12 is accessed four times corresponding to four positions, and nine feature data are simultaneously read in each access. However, a configuration in which the window size is set to four rows and four columns and 16 feature data corresponding to four positions of three rows and three columns is simultaneously read in one access may also be used. In this case, as shown in FIG. 13 , a register unit 45 is provided between the calculation unit 11 and the feature plane storage unit 12. The window size is set to four rows and four columns, and 16 feature data within the window are simultaneously read from one channel of the i-th layer from the feature plane storage unit 12. Note that in this example, the filter size is small compared to the window size of M rows and N columns. If P is a positive integer smaller than M and Q is a positive integer smaller than N, the filter size is P rows and Q columns.

[0114] The register unit 45 is composed of 16 (=M×N) registers 45a corresponding to a window of 4 rows and 4 columns (=M rows and N columns), and each register 45a holds feature data. There is a one-to-one correspondence between the registers 45a and local positions within the window. A first selector unit 46 is provided between the feature plane storage unit 12 and the register unit 45, and a second selector unit 47 is provided between the register unit 45 and the calculation unit 11.

[0115] The first selector unit 46 associates each of the 16 feature data read from the feature plane storage unit 12 with a local position within a 4-row, 4-column window. The first selector unit 46 is composed of 16 data selectors (not shown), each with 16 inputs and 1 output.

[0116] Each data selector in the first selector section 46 has a memory bank B ０ ~B １５ The data selectors are independently controlled by a selection signal from the control unit 18, and the registers 45a having the same local positions are connected to the output side. ０ ~B １５and outputs the feature data read from the selected memory bank B to the register 45a. As a result, the 16 feature data within the window read from the 16 memory banks B are held in the register 45a corresponding to the local position of the feature data, regardless of which memory bank B they are read from, and the feature data are rearranged into the arrangement within the window.

[0117] The second selector unit 47 extracts feature data of 3 rows and 3 columns from the feature data of 4 rows and 4 columns held in the register unit 45, and outputs the extracted feature data of 3 rows and 3 columns to the calculation unit 11, corresponding to each local position in the filter. This second selector unit 47 is composed of nine data selectors (not shown), each with 16 inputs and one output.

[0118] Each data selector of the second selector unit 47 is connected to a register 45 a on the input side. Similar to the selector unit 16, each data selector of the second selector unit 47 is associated with one of the local positions within the window. Each data selector of the second selector unit 47 is independently controlled by the control unit 18 to select one of the 16 registers 45 a and output the feature data read from the selected register 45 a. Under the control of the control unit 18, the data selector is switched at each clock. The second selector unit 47 sequentially changes the center position, for example, to position (1, 1) on the channel of the input feature plane at the first clock, position (1, 2) at the second clock, position (2, 1) at the third clock, and position (2, 2) at the fourth clock. The second selector unit 47 extracts feature data of three rows and three columns at four different center positions and outputs the extracted data to the calculation unit 11.

[0119] According to this example, by reading out the feature data of 4 rows and 4 columns from the feature plane storage unit 12 once, the feature data of 3 rows and 3 columns at the four center positions is read out. In addition, for each reading out of the feature data of 4 rows and 4 columns, it is sufficient to read out the weight data once from the weight storage unit 15, resulting in efficient access.

[0120] FIG. 14 shows an example in which four convolution circuits 29 are provided in each arithmetic unit 21, and four sets of feature data of three rows and three columns (P rows and Q columns) with different center positions are extracted from feature data of four rows and four columns (M rows and N columns), and input into the four convolution circuits 29, thereby applying filters in parallel to the feature data of three rows and three columns with different center positions.

[0121] Each arithmetic unit 21 is provided with four convolution circuits 29. Feature data of 4 rows and 4 columns is read from the feature plane storage unit 12 using a 4 row and 4 column window, and the 4 row and 4 column feature data is extracted so as to be distributed into four sets of feature data of 3 rows and 3 columns with different center positions. The extracted four sets of feature data of 3 rows and 3 columns with different center positions are input to the four convolution circuits 29 in each arithmetic unit 21. That is, the feature data of 3 rows and 3 columns input to each convolution circuit 29 in one arithmetic unit 21 have different center positions. The same filter weight data is input to each convolution circuit 29 in one arithmetic unit 21. As a result, filters are applied in parallel to the feature data of 3 rows and 3 columns with different center positions.

[0122] 15 shows a configuration in which, as in the example shown in FIG. 14 , four sets of feature data of three rows and three columns (P rows and Q columns) with different center positions are extracted from the feature data of four rows and four columns (M rows and N columns), and parallel filters are applied to each set, while, as in the example shown in FIG. 10 , the feature data of P rows and Q columns in the window are input sequentially to the calculation unit 11 while the window is moved to multiple positions in one channel and then moved to the next channel.

[0123] The arithmetic unit 21 in this example, like the example in FIG. 14, has four convolution circuits 29, and each convolution circuit 29 receives input of feature data in three rows and three columns with different center positions extracted from feature data read out in four rows and four columns.

[0124] The convolution circuit 29 includes a product-sum calculator 26, an adder 27, a plurality of registers 52 constituting a register unit 50, and a selector 53 and a selector 54 for each register 52. The number of registers 52 is the same as the number of times that filter weight data read from the weight storage unit 15 is reused. The number of times of reuse is the number of times that feature data of 3 rows and 3 columns is input consecutively to the product-sum calculator 26 for one channel of the input-side feature plane, i.e., the number of times that inputs are made until the window moves to the next channel.

[0125] The selector 53 has two inputs and one output, and the input side is connected to the output of the adder 27 and the output of the corresponding register 52. The output side of the selector 53 is connected to the input of the corresponding register 28a. The control unit 18 switches the selectors 53 in order so that only one selector 53 selects the adder 27, and each selector 53 selects the adder 27 in turn.

[0126] Each selector 53 is switched every time feature data of four rows and four columns is read from the feature plane storage unit 12, that is, every time the feature data input to the arithmetic unit 21 is updated. As the number of reuses is switched, the selectors 53 that select the adder 27 go through one cycle. That is, every time the window moves to the next channel, the leading selector 53 selects the adder 27. The input side of the selector 54 is connected to the output of each register 52, and the output side is connected to the adder 27. The selector 54 is controlled to select the register 52 corresponding to the selector 53 that has selected the adder 27. As a result, when the corresponding selector 53 selects the adder 27, the value held in the register 52 is updated to the value output from the adder 27, and otherwise the register 52 retains its contents unchanged.

[0127] Feature data is read from the feature plane storage unit 12 with a window size of 4 rows and 4 columns on one channel as described above. For one channel, the stride is set to "2," the position of the window on the channel is moved, and feature data of 4 rows and 4 columns within the window is read with each movement. Four sets of feature data of 3 rows and 3 columns with different center positions are extracted from the read 4 rows and 4 columns feature data and input to four convolution circuits 29 in each arithmetic unit 21. Each time the channel of feature data read from the feature plane storage unit 12 changes, weight data for the filters corresponding to that channel of feature data is read from the weight storage unit 15 and input to the four convolution circuits 29 in each arithmetic unit 21.

[0128] Each convolution circuit 29 in the arithmetic unit 21 performs a convolution operation on one of four 3-row, 3-column feature data sets with different center positions extracted from the 4-row, 4-column feature data set. The register 52 whose contents are updated is switched sequentially in synchronization with the movement of the 4-row, 4-column window. When the channel of the feature data being read changes, the register 52 whose contents are updated returns to the first one. Therefore, each register 52 holds the result of the convolution operation for a specific position of each channel of the output feature plane. The convolution results held in these registers 52 are then written as feature data in groups of 16 to the feature plane storage unit 12.

[0129] According to the above configuration, the weight data of each filter can be read from the weight storage unit 15 and input to each arithmetic unit 21 at the timing when the channel for reading out the feature data changes, so the number of times that the weight storage unit 15 is read can be reduced.

[0130] In the above examples, the window size (M rows and N columns) is set to 4 rows and 4 columns, and the filter size (P rows and Q columns) is set to 3 rows and 3 columns, and four sets of feature data with different center positions are extracted. However, the window size and filter size are not limited to these. For example, the filter size can be set to 4 rows and 4 columns, 5 rows and 5 columns, etc. The window size can also be set to 5 rows and 5 columns, 6 rows and 6 columns, etc. The window size can be determined based on the number of regions with different center positions to be extracted from the window and the region size (corresponding to the filter size). For example, M rows and N columns can be set to 5 rows and 5 columns, and six sets of feature data with different center positions can be extracted, or M rows and N columns can be set to 6 rows and 6 columns, and four sets of feature data with different center positions can be extracted. The number of rows (P, M) and the number of columns (Q, N) of the filter and window can be different.

[0131] When the arithmetic unit 21 is configured with four convolution circuits 29 as in the above examples, a pooling circuit 55 may be connected as shown in FIG. 16 , which calculates the average or maximum value of the four convolution circuits 29 and sets this as feature data of the output feature plane.

[0132] In each of the above embodiments, the memory bank B of the feature surface storage unit 12 ０ ~B １５ The memory bank B is a volatile memory configured with SRAM, etc. ０ ~B １５ Alternatively, the memory bank B may be configured as a nonvolatile memory such as an MRAM using a magnetic tunnel junction (MTJ) element, or an FeRAM, PRAM, or ReRAM. In this case, as shown in FIG. 17, it is preferable to provide a power gating unit 60 and perform power gating to reduce power consumption. The power gating unit 60 is ０ ~B １５and a power gate control unit 63 that controls the on / off of the switch unit 62. In this case, it is preferable to control the switch unit 62 so that power is supplied to the storage area that stores the input feature surface and the storage area that stores the output feature surface, and that the power supply to the other storage areas is turned off.

[0133] 18, when the weight storage unit 15 is configured as a nonvolatile memory, it is preferable to perform power gating by providing a power gating unit 67 including a power gate control unit 65 and a switch unit 66 that turns on / off the power supply from a power supply 61 to the weight storage unit 15 under the control of the power gate control unit 65. In this case, the switch unit 66 is controlled so that power is supplied to a storage area that stores weight data for filters in a hierarchical layer that performs convolution calculation processing, and power supply to other storage areas is turned off.

[0134] 10 Arithmetic processing device 11 Arithmetic section 12 Feature plane storage section 14 Bank address calculation section 15 Weight storage section 21 Arithmetic unit 26 Product-sum calculator 28, 28a to 28d, 41a to 41d, 52 Register 29 Convolution circuit 45 Register section 31, 50 Register section 60, 67 Power gating section B Memory bank

Claims

1. A neural network processing device having a processing unit that performs convolution operations by applying a filter to each channel of a feature plane, comprising: a feature plane storage unit that is composed of multiple memory banks that can be accessed simultaneously and that stores feature data of the feature planes of a convolution layer; and a bank address calculation unit that calculates and specifies the memory bank that holds the feature data and the address of the memory bank when inputting or outputting feature data to the feature plane storage unit, wherein the bank address calculation unit calculates the bank number of the memory bank for feature data and the address of the memory bank based on the channel size of the feature plane and the position of the feature data on the feature plane, in accordance with a decision rule that causes multiple feature data within a window of at least the same size as the filter at any position on the channel to be stored in different memory banks.

2. The processing unit according to claim 1, characterized in that the decision rule further causes feature data of multiple consecutive channels with the same number of channels as the number of memory banks in the memory bank to be stored in different memory banks for feature data at the same position on a channel.

3. The arithmetic processing device according to claim 1 or 2, characterized in that M and N are integers equal to or greater than 2, P is a positive integer equal to or less than M, and Q is a positive integer equal to or less than N, the size of the window is at most M rows and N columns, the filter has a filter size of at most P rows and Q columns, and the feature surface storage unit is composed of (M x N) memory banks.

4. A neural network arithmetic processing device having an arithmetic unit that performs convolutional operations by applying a filter to each channel of a feature plane, comprising: a feature plane storage unit that is composed of (M x N) memory banks that can be accessed simultaneously, where M and N are integers of 2 or greater, and that stores feature data of the input feature plane or the output feature plane of a convolutional layer, and from which a maximum of M rows and N columns of feature data on the channels of the input feature plane are read; and a bank address calculation unit that calculates the memory bank and an address on the memory bank for writing and reading feature data to and from the feature plane storage unit, the position of which on the feature plane is specified by the channel number D, row position H, and column position W using equation (A), where SIZE_H is the channel size in the row direction of the feature plane, SIZE_W is the channel size in the column direction of the feature plane, START is the start address of the feature plane storage unit corresponding to the feature plane, X is the bank number that specifies the memory bank, and Y is the address in the memory bank. ...(A) 5. The processing unit according to claim 4, wherein feature data within a window of up to M rows and N columns is simultaneously read out from said feature plane storage unit.

6. The arithmetic processing device according to claim 5, characterized in that for feature data at the same position on a channel calculated by the calculation unit, feature data of multiple consecutive channels, up to the same number of channels as the number of memory banks, is simultaneously written to the feature surface memory unit.

7. The arithmetic processing device according to claim 5, wherein P is a positive integer equal to or less than M, Q is a positive integer equal to or less than N, and the filter has at most P rows and Q columns in size.

8. The processing unit according to claim 5, wherein M and N are four.

9. The processing unit according to claim 4, wherein M and N are powers of two.

10. A method for manufacturing a filter comprising: a weight storage unit storing weight data of the filter; the calculation unit having a plurality of calculation units corresponding to channels of an output feature plane, where P is a positive integer between 2 and M, Q is a positive integer between 2 and N, and k is a positive integer less than P×Q; the calculation unit has a convolution circuit including a convolution product-accumulator that performs a product-accumulation operation on the input weight data and feature data, and a register that stores a new integrated value obtained by integrating a stored integrated value with the result of the product-accumulation operation output from the product-accumulator; when the filter size is P rows and Q columns, the P rows and Q columns window is moved so that reading from the P rows and Q columns window is performed continuously for the same positions on a plurality of channels, and the P rows and Q columns feature data read from the feature plane storage unit for each channel is input to the convolution circuit, and each time the window moves to the next channel, the filter weight data corresponding to that channel read from the weight storage unit is input to the convolution circuit; when the filter size is 1 row and 1 column, 5. The arithmetic processing device according to claim 4, wherein after simultaneously reading out data from the same position on each of k consecutive channels, each window in one row and one column is moved so as to move onto each of the next k channels, and each piece of feature data in one row and one column for the k channels read out from the feature plane storage unit is input to the convolution circuit, and each time each window in one row and one column moves to each of the next k channels, weight data of each filter corresponding to that k channel read out from the weight storage unit is input to the convolution circuit.

11. The arithmetic processing device according to claim 4, further comprising: a weight storage unit that stores weight data of the filter; the arithmetic unit has a plurality of arithmetic units corresponding to channels of a feature plane on the output side; a window size of P rows and Q columns, where P is a positive integer not greater than M and Q is a positive integer not greater than N; each time the window moves from moving to a plurality of positions in one channel to moving to the next channel, feature data of P rows and Q columns in the window is read out from the feature plane storage unit and input; and each time the window moves to the next channel, weight data of the filter corresponding to that channel is read out from the weight storage unit and input; the arithmetic processing device includes a multiply-accumulate operator that performs a multiply-accumulate operation on the input feature data of P rows and Q columns and the weight data, and a plurality of registers that hold the accumulated values, and a convolution circuit having a register unit that switches the register that holds a new accumulated value obtained by accumulating the accumulated value held by the multiply-accumulate operator with the result of the multiply-accumulate operation output from the multiply-accumulate operator, each time the feature data of P rows and Q columns is input.

12. The arithmetic processing device according to claim 4, further comprising: a weight storage section for storing weight data of the filter; wherein the arithmetic section has a plurality of arithmetic units corresponding to channels of the output feature surface, where P is a positive integer smaller than M and Q is a positive integer smaller than N, and the arithmetic unit has a plurality of convolution circuits including a convolution multiply-accumulate operator that performs a multiply-accumulate operation on the input weight data and feature data of P rows and Q columns, and a register that stores a new integrated value obtained by integrating a stored integrated value with the result of the multiply-accumulate operation output from the multiply-accumulate operator; wherein the M row and N column window is moved so that reading from the same position on a plurality of channels using the M row and N column window is performed continuously, and feature data of P rows and Q columns at mutually different center positions extracted from the M row and N column window for one channel is input to the plurality of convolution circuits, and each time the M row and N column window moves to the next channel, the filter weight data corresponding to that channel is input to the plurality of convolution circuits.

13. A weight storage unit is provided which stores weight data of the filter, and the calculation unit has a plurality of calculation units corresponding to channels of the output feature surface, where P is a positive integer smaller than M and Q is a positive integer smaller than N, and the calculation unit has a plurality of convolution circuits including a convolution product-sum calculator which performs a product-sum operation on the input weight data and feature data of P rows and Q columns, and a register unit having a plurality of registers which newly store an integrated value obtained by integrating the stored integrated value with the product-sum operation result output from the product-sum calculator, 5. The arithmetic processing device according to claim 4, wherein, each time the M-row, N-column window is moved to a plurality of positions in one channel and then moved to a next channel, feature data of P-row, Q-column positions extracted from the M-row, N-column window and having mutually different center positions are input to the plurality of convolution circuits, and the register that newly holds an integrated value obtained by integrating a product-sum operation result output from the product-sum calculator with a held integrated value is switched, and each time the M-row, N-column window is moved to a next channel, filter weight data corresponding to that channel is input to the plurality of convolution circuits.

14. The processing unit according to claim 4, wherein said feature surface storage unit is configured as a non-volatile memory.

15. The arithmetic processing device according to claim 13, further comprising a first power gating unit that supplies power to each storage area that stores the input-side feature plane and the output-side feature plane of the feature plane storage unit, and cuts off the power supply to storage areas that store other feature planes.

16. The arithmetic processing device according to any one of claims 10 to 13, wherein the weight storage unit is configured as a non-volatile memory.

17. The processing device according to claim 16, characterized in that the weight memory unit stores the weight data of the filter used in the convolution calculation for each layer of the convolutional neural network in a storage area that stores the weight data for each layer, and further comprises a second power gating unit that supplies power to the storage area from which the weight data is read in the convolution calculation and cuts off the power supply to other storage areas.

Images