Neural network operation device and method, computer readable storage medium

Abstract

A neural network operation device and method, and a computer readable storage medium, the neural network operation device comprising: an external storage space, an on-chip storage space, the on-chip storage space comprising an on-chip input data cache, an on-chip output data cache and an on-chip internal data cache, wherein: the external storage space is adapted to cache data to be processed of an i-th cascade group and output data of the i-th cascade group; 1 < i ≤ N, N is the total number of cascade groups, and N is a positive integer; the on-chip input data cache is adapted to cache input data of a first layer in the i-th cascade group, the input data of the first layer being a part of the data to be processed; the on-chip output data cache is adapted to cache operation results corresponding to the input data; and the on-chip internal data cache is adapted to cache operation results of each layer in the i-th cascade group by a computing unit. By using the above scheme, the number of banks in the neural network operation device can be reduced, the hardware complexity can be reduced, and the circuit area can be reduced.

Description

Technical Field

[0001] This invention relates to the field of neural network technology, and in particular to a neural network computing device and method, and a computer-readable storage medium. Background Technology

[0002] Existing neural network processing units (NPUs) operate at speeds far exceeding their ability to access external storage. During operation, the inability of the neural network chip to retrieve data from external storage in a timely manner can cause the computational units on the chip to stall, resulting in low computational efficiency.

[0003] To improve computational efficiency, most current neural network chips employ a layer-by-layer cascading approach, processing data from multiple network layers on-chip memory to reduce the frequency of interaction between on-chip memory and external storage. However, due to memory read / write limitations, computational units still cannot operate in parallel, resulting in relatively low computational efficiency.

[0004] To ensure that each computing unit can perform parallel operations, two banks are typically provided for the output data of each computing unit. If there are n computing units, then 2n banks are needed to store intermediate computing results. Summary of the Invention

[0005] This invention addresses the technical problem of high hardware complexity and large circuit area caused by a large number of banks in neural network computing devices.

[0006] To address the aforementioned technical problems, this invention provides a neural network computing device, comprising: an external storage space and an on-chip storage space. The on-chip storage space includes an on-chip input data cache, an on-chip output data cache, and an on-chip internal data cache. Specifically: the external storage space is adapted to cache the data to be processed in the i-th cascaded group and the output data of the i-th cascaded group; 1 < i ≤ N, where N is the total number of cascaded groups and N is a positive integer; the on-chip input data cache is adapted to cache the input data of the first layer in the i-th cascaded group, where the input data of the first layer is a portion of the data to be processed; the on-chip output data cache is adapted to cache the computation results corresponding to the input data; and the on-chip internal data cache is adapted to cache the computation results of the computing unit for each layer in the i-th cascaded group.

[0007] Optionally, the on-chip internal data cache includes M banks, where M is a positive integer.

[0008] Optionally, there may be multiple computing units, and when at least two computing units perform parallel operations on computing tasks of different layers in the i-th cascade group, the computing tasks of different layers are associated with different banks.

[0009] Optionally, M is associated with the capacity of the on-chip storage space.

[0010] Optionally, the number M of on-chip internal data caches has a different parity than the number of computing units.

[0011] Optionally, the number M of on-chip internal data caches is the sum of the number of computing units and 1.

[0012] Optionally, the on-chip input data buffer is released after the input of the data to be processed is completed.

[0013] Optionally, after the input of the data to be processed is completed, the on-chip input data buffer is adapted to buffer the input data of the first layer in the (i+1)th cascade group.

[0014] Optionally, the on-chip input data buffer includes one bank; and / or, the on-chip output data buffer includes one bank.

[0015] The present invention also provides a neural network operation method, comprising: obtaining input data of the first layer in the i-th cascade group from an on-chip input data buffer; performing operations on the input data in the i-th cascade group using a corresponding computing unit; and storing the operation result corresponding to the input data in an on-chip output data buffer.

[0016] Optionally, the neural network operation method further includes: after completing the input of the data to be processed in the i-th cascade group, releasing the corresponding on-chip internal data cache in the i-th cascade group based on the computing progress of the computing unit.

[0017] The present invention also provides a computer-readable storage medium, which is a non-volatile storage medium or a non-transient storage medium, on which a computer program is stored, wherein the computer program is executed by a processor to perform the steps of any of the neural network operation methods described above.

[0018] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0019] For a cascaded group, an on-chip input data buffer and an on-chip output data buffer are configured. The on-chip input data buffer stores the input data required by the cascaded group, and the on-chip output data buffer stores the computation results obtained by the computing unit. Therefore, the neural network computing device only needs one on-chip input data buffer and one on-chip output data buffer, eliminating the need for two banks for each computing unit. This reduces the number of banks and lowers the hardware complexity and circuit area of ​​the neural network computing device.

[0020] Furthermore, the computational tasks of different layers in the i-th cascade group can be performed in parallel, which can improve the performance of the neural network computing device.

[0021] Furthermore, the number of on-chip internal data caches M has a different parity than the number of computing units, so there is always a computing unit capable of performing computing tasks at any given time, thus improving the execution efficiency of the computing units. Attached Figure Description

[0022] Figure 1 This is a flowchart of a neural network operation method according to an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of an on-chip storage space structure in an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of the structure of a neural network model in an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of the data distribution corresponding to a cascade group 1 in an embodiment of the present invention;

[0026] Figure 5 This is a schematic diagram of the data distribution corresponding to a cascade group 2 in an embodiment of the present invention;

[0027] Figures 6-9 This is the computation flow graph of cascade group 1 in one cycle in this embodiment of the invention;

[0028] Figures 10-28 This is the computation flow graph between cascade group 1 and cascade group 2 in this embodiment of the invention. Detailed Implementation

[0029] As described in the background section above, to ensure that each computing unit can perform parallel operations, two banks are typically provided for the output data of each computing unit. If there are n computing units, then 2n banks are needed to store intermediate computing results.

[0030] In this embodiment of the invention, for a cascaded group, an on-chip input data buffer and an on-chip output data buffer are provided. The on-chip input data buffer stores the input data required by the cascaded group, and the on-chip output data buffer stores the computation results obtained by the computing unit. Therefore, the neural network computing device only needs to provide one on-chip input data buffer and one on-chip output data buffer, eliminating the need for two banks for each computing unit. This reduces the number of banks and lowers the hardware complexity and circuit area of ​​the neural network computing device.

[0031] To make the above-mentioned objectives, features and beneficial effects of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0032] In practical implementation, the banks described below consist of multiple rows and columns of storage units, and only one bank can be accessed during a single read / write operation. Single-port RAM is RAM with only one set of data and address lines; read and write operations cannot be performed simultaneously. Random Access Memory (RAM) is internal memory that can be read and written at any time and is relatively fast; it is typically used as temporary data storage for the operating system or other running programs.

[0033] A convolutional layer (conv) is a common form of computation in neural networks. It performs specific numerical calculations on the input data and obtains the features of the input data as the output of the convolutional layer.

[0034] Pooling is a common computational technique in neural networks. It divides the input image into several rectangular sub-regions, performs calculations on each sub-region, and outputs a specific value as the output of the pooling layer.

[0035] A neural network is an algorithmic mathematical model that mimics the behavioral characteristics of animal neural networks to perform distributed parallel information processing. It processes information by adjusting the relationships between the interconnections between multiple layers of nodes within the system, depending on the complexity of the system.

[0036] Specifically, the specific concepts of the above terms can be referred to in the existing technology, and will not be described in detail here.

[0037] This invention provides a neural network operation method, referring to... Figure 1 The following will provide a detailed explanation through specific steps.

[0038] Step 101: Obtain the input data of the first layer in the i-th cascade group from the on-chip input data buffer.

[0039] Step 102: Using the corresponding computing unit, perform calculations on the input data within the i-th cascade group.

[0040] Step 103: Store the calculation results corresponding to the input data in the external storage space.

[0041] In this embodiment of the invention, the neural network computing device may include external storage space and on-chip storage space. The external storage space may be a memory independently configured relative to the neural network computing device, and the external storage space may communicate with the neural network computing device through a data transmission channel.

[0042] In some embodiments, external storage space may include flash memory chips, erasable programmable read-only memory (EEPROM) chips, etc.

[0043] On-chip storage space can refer to the storage module built into the neural network computing device. In some embodiments, the neural network computing device can be a neural network processing unit (NPU), and the on-chip storage space is the storage module inside the neural network chip.

[0044] In this embodiment of the invention, the on-chip storage space can be divided into three parts, wherein:

[0045] The first part is constructed using a single-port RAM to store data read from external storage. It is capable of being written to via Direct Memory Access (DMA) and read by the computing unit, but reading and writing do not occur simultaneously. In the following embodiments, the first part of the on-chip storage space can be simply referred to as the on-chip input data cache.

[0046] The second part is constructed using a single-port RAM, used to store the calculated results in external storage space. It can be read by DMA and written by the computation unit, but reading and writing will not occur simultaneously. In the following embodiments, the second part of the on-chip storage space can be simply referred to as the on-chip output data cache.

[0047] The third part is constructed using a single-port RAM to store intermediate data generated during the computation process of the computing unit. It can only be read and written by the computing unit, and reading and writing will not occur simultaneously. In the following embodiments, the on-chip storage space of the third part can be simply referred to as the on-chip internal data cache.

[0048] In this embodiment of the invention, the on-chip input data cache can be considered as one bank, the on-chip output data cache as another bank, and the on-chip internal data cache can be divided into N+1 banks, where N is the number of computing units. Therefore, the number of banks included in the on-chip internal data cache has a different parity than the number of computing units.

[0049] In some embodiments, the computation unit may include a convolution computation unit and a pooling computation unit. In this scenario, the computation unit comprises two computation units. Correspondingly, the on-chip internal data cache comprises three banks.

[0050] Reference Figure 2 A schematic diagram of an on-chip storage space structure in an embodiment of the present invention is provided. Figure 2 The on-chip storage space can include an on-chip input data cache, an on-chip output data cache, and an on-chip internal data cache. Specifically: the on-chip input data cache includes one bank, the on-chip output data cache includes one bank, and the on-chip internal data cache includes banks0 to 2. Data from external storage space is input into the on-chip input data cache, and data stored in the on-chip output data cache is output to external storage space.

[0051] Data transmission channels exist between the on-chip input data buffer and the on-chip internal data buffer, and between the on-chip internal data buffer and the on-chip external data buffer, to enable data transmission between the on-chip input data buffer and the on-chip internal data buffer, and between the on-chip internal data buffer and the on-chip external data buffer. Data transmission channels can also exist between the banks of the on-chip internal data buffer to enable data transmission between the banks.

[0052] In practical applications, it is known that neural networks can include multiple layers. In this embodiment of the invention, the maximum number of cascaded layers in each cascade group can be calculated based on the capacity of the on-chip storage space, thereby dividing the neural network into multiple cascade groups. For any cascade group, the input data of the first layer is cached in the on-chip input data cache, and the input data of other layers is cached in the on-chip internal data cache.

[0053] In practice, within a cascaded group, the input data of all layers except the first layer is essentially the output data of the layer preceding it. For example, the input data of layer 2 is essentially the output data of layer 1. Correspondingly, the output data of layer 2 is the input data of layer 3.

[0054] For a cascaded array, the input data for the first layer can be retrieved from external storage using Direct Memory Access (DMA) and cached in the on-chip input data buffer. The input data sequentially passes through each computation unit. After passing through the last computation unit, the input data re-enters the first computation unit, repeating this process until all layers within the cascaded array have been computed. Intermediate computation results from each computation unit can be stored in the on-chip internal data buffer, while the computation result of the final layer is stored in the on-chip output data buffer and written back to external storage using DMA.

[0055] For each computing unit, the flow of intermediate calculation results in the on-chip internal data cache is as follows: intermediate calculation results are stored in the on-chip internal data cache. The first calculation result of the first computing unit is stored in the first bank of the on-chip internal data cache. Subsequently, as intermediate data passes through each computing unit, the resulting intermediate calculation result is stored in the next bank, and the data in the last bank is stored in the first bank after passing through the computing unit.

[0056] For data flow between cascaded groups, once a computing unit reads the input data from the last line of any layer in the previous cascaded group, it can release the storage space occupied by that layer's input data. The released storage space can then be used by the next cascaded group. When the released storage space is large, even if there is still uncomputed data in the previous cascaded group, the data of the next cascaded group can be prefetched and computed, thereby achieving seamless connection between cascaded groups and avoiding pauses in computing units.

[0057] In this embodiment of the invention, a calculation task table can be prepared in advance for the number of calculation units to ensure that each calculation unit can work simultaneously.

[0058] Specifically, we can first label the computing unit corresponding to each computing task, as well as the bank identifier of the on-chip internal data cache used by each computing task. Each computing task can correspond to a layer in the neural network, and one layer can execute one computing task.

[0059] Next, determine the computational tasks that can be processed in parallel, that is, whether there are other computational tasks that can be processed in parallel during the processing of a certain computational task. The conditions for determining whether computational tasks can be processed in parallel may include: at the same time, different computational tasks use different computing units, and computational tasks that perform read operations on the same bank and computational tasks that perform write operations cannot be processed in parallel.

[0060] Referring to Table 1 below, an example of a calculation task table in an embodiment of the present invention is given.

[0061] Computational tasks Computing unit Enter bank Output bank Time 1 Time 2 Time 3 Layer 1 Unit 1 - 0 √ Layer 2 Unit 2 0 1 √ Layer 3 Unit 1 1 2 √ Floor 4 Unit 2 2 0 √ Floor 5 Unit 1 0 1 √ Floor 6 Unit 2 1 - √

[0062] Table 1

[0063] In Table 1, at time 1, the computing tasks corresponding to layer 1 of computing unit 1 and layer 4 of computing unit 2 are listed; at time 2, the computing tasks corresponding to layer 2 of computing unit 2 and layer 5 of computing unit 1 are listed; at time 3, the computing tasks corresponding to layer 3 of computing unit 1 and layer 6 of computing unit 2 are listed.

[0064] The memory access method provided in the above embodiments of the present invention will be described below through specific examples.

[0065] The neural network is configured with 14 layers, numbered layer 1 through layer 14. (Refer to...) Figure 3 A schematic diagram of the structure of a neural network model in an embodiment of the present invention is provided.

[0066] Layers 1, 3, 5, 7, 9, 11, and 13 correspond to convolutional computation, while layers 2, 4, 6, 8, 10, 12, and 14 correspond to pooling computation. The computation units include both convolutional and pooling computation units. Correspondingly, the on-chip internal data cache includes three banks, named bank0 through bank2.

[0067] The above 14 layers are divided into two cascade groups: cascade group 1 includes layers 1 to 8, and cascade group 2 includes layers 9 to 14.

[0068] Reference Figure 4 A schematic diagram of the data distribution corresponding to cascade group 1 is given.

[0069] The input and output data of cascade group 1 are stored in external storage space. The input data of layer 1 is stored in the on-chip input data buffer. The output data of layer 8 is stored in the on-chip output data buffer. The output data of layer 1, layer 4, and layer 7 are stored in bank0 of the on-chip internal data buffer. The output data of layer 2 and layer 5 are stored in bank1. The output data of layer 3 and layer 6 are stored in bank2.

[0070] Reference Figure 5 A schematic diagram of the data distribution corresponding to cascade group 2 is given.

[0071] The input and output data of cascade group 2 are stored in external storage space, the input data of layer 9 is stored in on-chip input data buffer, the output data of layer 14 is stored in on-chip output data buffer, the data of layer 9 and layer 12 are stored in bank0, the data of layer 10 and layer 13 are stored in bank1, and the data of layer 11 is stored in bank2.

[0072] Referring to Table 2, the calculation task table corresponding to cascade group 1 is given.

[0073]

[0074]

[0075] Table 2

[0076] The operation process of cascade group 1 and cascade group 2 will be explained below with reference to Table 2.

[0077] Cascade group 1 consists of 8 layers. Two layers can be processed in parallel at any given time, so one loop takes 4 time steps to complete. Correspondingly, cascade group 2 consists of 6 layers, so one loop takes 3 time steps to complete.

[0078] Reference Figures 6-9 The computation flow graph of cascade group 1 within one cycle is given. One cycle can include 4 time points. The operations performed by each computation unit in different cycles are the same, and the only difference is the data being processed.

[0079] Figure 6 In the process, at time 1, computing unit 1 executes the computation task of layer 1, reads data (i.e., the input data of layer 1) from the on-chip input data buffer, performs calculations, and stores the calculation result (i.e., the output data of layer 1) in bank0. Computing unit 2 retrieves the output data of layer 5 from bank1, performs calculations, and stores the obtained calculation result (i.e., the output data of layer 6) in bank2. Data in the on-chip output data buffer can be output to external storage space via DMA. Data stored in the on-chip output data buffer (output data of layer 8) is output to external storage space via DMA.

[0080] At time 1, the data stored in the on-chip input data buffer is the data to be processed by the neural network. Typically, the amount of data to be processed is large, while the on-chip input data buffer is small. Therefore, the data to be processed can be divided into multiple segments, and one data segment is input into the on-chip input data buffer in each cycle. The length of the data segment does not exceed the maximum capacity of the on-chip input data buffer.

[0081] Figure 7 At time 2, the external storage space inputs the data to be processed (the input data corresponding to layer 1) to the input data buffer via DMA. Computation unit 1 retrieves the output data of layer 4 from bank0, performs calculations, and stores the result (i.e., the output data of layer 5) in bank1. Computation unit 2 retrieves the output data of layer 1 from bank0, performs calculations, and stores the result (i.e., the output data of layer 2) in bank1.

[0082] The last line of data output by layer 1 has been used at time 2, and the storage space occupied by layer 1 can be released at the end of time 2.

[0083] Figure 8At time 3, external storage space inputs data to be processed into the input data buffer via DMA. Computation unit 1 obtains the output data of layer 2 from bank 1, performs calculations, and stores the result (i.e., the output data of layer 3) in bank 2. Computation unit 2 reads the output data of layer 7 from bank 0, performs calculations, and stores the result (i.e., the output data of layer 8) in the on-chip output buffer.

[0084] The last line of data output by layer 2 has been used at time 3, and the storage space occupied by layer 2 can be released at the end of time 3.

[0085] Figure 9 At time 4, data in the on-chip output data buffer can be output to external storage via DMA. Computation unit 1 retrieves the output data of layer 6 from bank 2, performs calculations, and stores the result (i.e., the output data of layer 7) in bank 0. Computation unit 2 reads the output data of layer 3 from bank 2, performs calculations, and stores the result (i.e., the output data of layer 4) in bank 0.

[0086] The last line of data output by layer 3 has been used at time 4, and the storage space occupied by layer 3 can be released at the end of time 4.

[0087] Combination Figures 6-9 It is evident that the execution of computation unit 1 and computation unit 2 in cascade group 1 is not in the order of layers 1 to 8, but rather computation unit 1 and computation unit 2 perform calculations on data from different layers in parallel. Since the execution of computation unit 1 and computation unit 2 is not sequential, at some times, computation unit 1 and computation unit 2 continue to perform calculations on the unfinished tasks from the previous cycle.

[0088] Specifically, such as Figure 6 As shown, at time 1 of the current cycle, the operation of reading data segments from external storage space and transferring them to the on-chip input data buffer is not actually performed. The data corresponding to layer 1 stored in the on-chip input data buffer are: time 2 and time 3 of the previous cycle. The external storage space inputs the data corresponding to layer 1 to the on-chip input data buffer via DMA.

[0089] Furthermore, at time 1 of the current cycle, computing unit 1 reads the output data of layer 5 from bank 1. The output data of layer 5 in bank 1 is obtained by computing unit 1 from the output data of layer 4 in bank 0 at time 2 of the previous cycle. In other words, the output data of layer 6 in the current cycle is calculated based on the output data of layer 5 in the previous cycle.

[0090] Similarly, at time 2 of the current cycle, the data of layer 4 in bank2 is obtained by computation unit 2 from the output data of layer 3 in the previous cycle at time 4. In other words, computation unit 1 calculates the output data of layer 5 in the current cycle based on the output data of layer 4 in the previous cycle.

[0091] At time 3 of the current cycle, computing unit 2 calculates the output data of layer 8 of the current cycle based on the output data of layer 7 of the previous cycle, and stores the output data of layer 8 in the on-chip output data buffer.

[0092] At time 4 of the current cycle, computation unit 1 calculates the output data of layer 6 based on the output data of layer 7 obtained at time 1. The data stored in the on-chip output data buffer is output to external storage space via DMA.

[0093] Once a computing unit has read the input data from the last row of any layer in cascade group 1 from the on-chip input data buffer, it can release the storage space occupied by the input data of that layer to provide for the next cascade group (such as cascade group 2). As long as enough space is released, even if some layers in the previous cascade group have not yet been computed, the data corresponding to the next cascade group can be prefetched and processed, thereby achieving the connection between cascade groups and avoiding interruptions between computing units.

[0094] Reference Figures 10-28 The computation flow graph between cascade group 1 and cascade group 2 is given.

[0095] Figure 10 In the process, at a set time n, computation unit 1 reads the last segment of the data to be processed from the on-chip input data buffer, performs calculations on the last segment, and stores the result in bank0. Computation unit 2 retrieves the output data of layer 5 from bank1, performs calculations, and stores the result (i.e., the output data of layer 6) in bank2. Data in the on-chip output data buffer can be output to external storage via DMA. Data stored in the on-chip output data buffer (output data of layer 8) is also output to external storage via DMA.

[0096] It should be noted that at time n, some layers of cascade group 1 have not yet completed the calculation; only the data that needs to be calculated has been read.

[0097] Figure 11At time n+1, computation unit 1 retrieves the output data of layer 4 from bank0, performs calculations, and stores the result (i.e., the output data of layer 5) in bank1. Computation unit 2 retrieves the output data of layer 1 from bank0, performs calculations, and stores the result (i.e., the output data of layer 2) in bank1. Since all the data required by layer 1 has been read, the external storage space can input the data required by layer 9 into the on-chip input data buffer. That is, starting from time n+1, the data stored in the on-chip input data buffer is the data required by layer 9.

[0098] The output data of layer 1 has been used at time n+1, and the storage space occupied by the output data of layer 1 can be released at the end of time n+1.

[0099] Figure 12 At time n+2, external storage space can input the data required by layer 9 into the on-chip input data buffer. Computation unit 1 reads the output data of layer 2 from bank 1, performs calculations, and stores the result (i.e., the output data of layer 3) in bank 2. Computation unit 2 reads the output data of layer 7 from bank 0, performs calculations, and stores the result (i.e., the output data of layer 8) in the on-chip output data buffer.

[0100] Figure 12 Since the output data of layer 1 in cascade group 1 is used at time n+1 and the computation task corresponding to layer 1 has been completed, the output data of layer 1 in bank0 is released at time n+2.

[0101] The output data of layer 2 has been used at time n+2, and the storage space occupied by the output data of layer 2 can be released at the end of time n+2.

[0102] Figure 13 At time n+3, computation unit 1 reads the output data of layer 6 from bank 2, performs the operation, and stores the result (i.e., the output data of layer 7) in bank 0. Computation unit 2 reads the output data of layer 3 from bank 2, performs the operation, and stores the result (i.e., the output data of layer 4) in bank 0. The on-chip output data cache outputs the output data of layer 8 to external storage space via DMA.

[0103] The output data of layer 3 has been used at time n+3, and the storage space occupied by the output data of layer 3 can be released at the end of time n+3.

[0104] Figure 14At time n+4, computation unit 1 reads the input data corresponding to layer 9 from the on-chip input data buffer, performs the operation, and stores the result (i.e., the output data of layer 9) in bank0. Computation unit 2 reads the output data corresponding to layer 5 from bank1, performs the operation, and stores the result (i.e., the output data of layer 6) in bank2. The on-chip output data buffer outputs the output data of layer 8 to external storage space via DMA.

[0105] Figure 15 At time n+5, external storage space can input the data required by the on-chip input data cache for layer 9. Computation unit 2 can read the output data of layer 9 from bank0 and perform calculations, storing the result (output data of layer 10) in bank1. Computation unit 1 reads the output data of layer 4 from bank1 and performs calculations, storing the result (output data of layer 5) in bank2.

[0106] The output data of layer 4 has been used at time n+5, and the storage space occupied by the output data of layer 4 can be released at the end of time n+5.

[0107] Figure 16 At time n+6, external storage space can input the data required by layer 9 into the on-chip input data buffer. Computation unit 1 reads the output data of layer 10 from bank 1 and performs calculations, storing the result (output data of layer 11) in bank 2. Computation unit 2 reads the output data of layer 7 from bank 0 and performs calculations, storing the result (output data of layer 8) in the on-chip output data buffer.

[0108] Figure 17 At time n+7, computation unit 1 reads the output data of layer 6 from bank2, performs calculations, and stores the result (output data of layer 7) in bank0. Computation unit 2 reads the output data of layer 11 from bank2, performs calculations, and stores the result (output data of layer 12) in bank0. The on-chip output data cache outputs the output data of layer 8 to external storage space via DMA.

[0109] Figure 18 At time n+8, computation unit 1 reads the input data of layer 9 from the on-chip input data buffer, performs the operation, and stores the result (output data of layer 9) in bank0. Computation unit 2 reads the output data of layer 5 from bank1, performs the operation, and stores the result (output data of layer 6) in bank2. The on-chip output data buffer outputs the output data of layer 8 to external storage space via DMA.

[0110] The output data of layer 5 has been used at time n+8, and the storage space occupied by the output data of layer 5 can be released at the end of time n+8.

[0111] Figure 19 At time n+9, external storage space can input the data required by the on-chip input data cache to layer 9. Computation unit 2 reads the output data of layer 9 from bank0 and performs calculations, storing the result (output data of layer 10) in bank1. Computation unit 1 reads the output data of layer 12 from bank0 and performs calculations, storing the result (output data of layer 13) in bank1.

[0112] Figure 20 At time n+10, external storage space can continue to input the data required by layer 9 into the on-chip input data buffer. Computation unit 1 reads the output data of layer 10 from bank 1 and performs calculations, storing the result (output data of layer 11) in bank 2. Computation unit 2 reads the output data of layer 7 from bank 0 and performs calculations, storing the result (output data of layer 8) in the on-chip output data buffer.

[0113] Figure 21 At time n+11, computation unit 2 reads the output data of layer 11 from bank2, performs calculations, and stores the result (output data of layer 12) in bank0. Computation unit 1 reads the output data of layer 6 from bank2, performs calculations, and stores the result (output data of layer 7) in bank0. The on-chip output data cache outputs the output data of layer 8 to external storage space via DMA.

[0114] The output data of layer 6 has been used at time n+11, and the storage space occupied by the output data of layer 6 can be released at the end of time n+11.

[0115] Figure 22 At time n+12, since cascade group 1 has 8 layers and cascade group 2 has 7 layers, cascade group 2 does not perform parallel computation for the final computation of the last two layers (layers 7 and 8) of cascade group 1. Therefore, at time n+12, computation unit 1 retrieves the output data of layer 7 from bank0, performs the computation, and stores the result (i.e., the output data of layer 8) in the on-chip output data buffer. Computation unit 1 experiences a pause.

[0116] Figure 23 In the process, at time n+13, the last layer of cascade group 1 performs its last output, which cannot be parallelized with the calculation of the last layer of cascade group 2. In this scenario, the calculation unit 2 stops.

[0117] At time n+13, computation unit 1 reads the input data of layer 9 from the on-chip input data buffer and performs the operation, storing the result (i.e., the output data of layer 9) in bank0. Furthermore, it outputs the output data of layer 8 stored in the on-chip output data buffer to external storage.

[0118] At time n+13, the operation of cascade group 1 ends.

[0119] At time n+14, after the operation of cascade group 1 is completed, the final layer of calculation for cascade group 2 can begin.

[0120] Figure 24 In this process, computing unit 1 reads the input data of layer 9 from the on-chip input data buffer, performs the operation, and stores the result (i.e., the output data of layer 9) in bank0. Computing unit 2 reads the output data of layer 13 from bank1, performs the operation, and stores the result (i.e., the output data of layer 14) in the on-chip output data buffer.

[0121] Figure 25 At time n+15, computing unit 2 reads the output data of layer 9 from bank0 and performs calculations, storing the result (i.e., the output data of layer 10) in bank1. Computing unit 1 reads the output data of layer 12 from bank0 and performs calculations, storing the result (i.e., the output data of layer 13) in bank1.

[0122] Figure 26 At time n+16, the external storage space inputs the data required by layer 9 into the on-chip input data buffer. Computation unit 1 reads the output data of layer 10 from bank 1, performs calculations, and stores the result (i.e., the output data of layer 11) in bank 2. Computation unit 2 reads the output data of layer 13 from bank 1, performs calculations, and stores the result (i.e., the output data of layer 14) in the on-chip output data buffer.

[0123] Figure 27 At time n+17, computation unit 1 reads the input data of layer 9 from the on-chip input data buffer and performs the operation, storing the result (i.e., the output data of layer 9) in bank0. Computation unit 2 reads the output data of layer 11 from bank2 and performs the operation, storing the result (i.e., the output data of layer 12) in bank0. The output data of layer 14 stored in the on-chip output data buffer is output to external storage space via DMA.

[0124] Figure 28At time n+18, computation unit 2 reads the output data of layer 9 from bank0 and performs calculations, storing the result (i.e., the output data of layer 10) in bank1. Computation unit 1 reads the output data of layer 12 from bank0 and performs calculations, storing the result (i.e., the output data of layer 13) in bank1. The output data of layer 14 stored in the on-chip output data buffer is output to external storage space via DMA.

[0125] The operation of one cycle of cascade group 2 can be completed from time n+16 to time n+18.

[0126] In summary, in this embodiment of the invention, for a cascaded group, an on-chip input data buffer and an on-chip output data buffer are provided. The on-chip input data buffer stores the input data required by the cascaded group, and the on-chip output data buffer stores the computation results obtained by the computing unit. Therefore, the neural network computing device only needs to provide one on-chip input data buffer and one on-chip output data buffer, eliminating the need for two banks for each computing unit. This reduces the number of banks and lowers the hardware complexity and circuit area of ​​the neural network computing device.

[0127] This application also provides a computer-readable storage medium, which is a non-volatile storage medium or a non-transient storage medium, on which a computer program is stored. When the computer program is run by a processor, it executes the neural network operation method provided in any of the above embodiments.

[0128] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include ROM, RAM, disk, or optical disk, etc.

[0129] While the present invention has been disclosed above, it is not limited thereto. Any person skilled in the art can make various modifications and alterations without departing from the spirit and scope of the invention; therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A neural network computing device, characterized in that, include: External storage space and on-chip storage space, wherein the on-chip storage space includes on-chip input data cache, on-chip output data cache and on-chip internal data cache, wherein: The external storage space is suitable for caching the data to be processed in the i-th cascade group and the output data of the i-th cascade group; 1 < i ≤ N, where N is the total number of cascade groups and N is a positive integer; The on-chip input data cache is adapted to cache the input data of the first layer in the i-th cascade group, where the input data of the first layer is a part of the data to be processed. The on-chip output data cache is adapted to cache the operation results corresponding to the input data; the i-th cascade group corresponds to one on-chip input data cache and one on-chip output data cache; both the on-chip input data cache and the on-chip output data cache are constructed using single-port RAM; The on-chip internal data cache is adapted to cache the computation results of each layer in the i-th cascade group by the computing units, including M banks, where M is a positive integer; the number of computing units is multiple, and when at least two computing units perform parallel computation tasks on different layers in the i-th cascade group, the computation tasks of different layers are associated with different banks, and different computation tasks use different computing units; the number M of banks in the on-chip internal data cache is the sum of the number of computing units and 1.

2. The neural network computing device as described in claim 1, characterized in that, The M is associated with the capacity of the on-chip storage space.

3. The neural network computing device as described in claim 1, characterized in that, After the input of the data to be processed is completed, the on-chip input data buffer is released.

4. The neural network computing device as described in claim 3, characterized in that, After the input of the data to be processed is completed, the on-chip input data buffer is adapted to buffer the input data of the first layer in the (i+1)th cascade group.

5. The neural network computing device according to any one of claims 1 to 4, characterized in that, The on-chip input data buffer includes one bank; and / or, the on-chip output data buffer includes one bank.

Images