Device, board, method and readable storage medium for computing neural network

Abstract

The present invention relates to a device, board, method, and readable storage medium for computing a neural network. The computing device of the present invention is included in an integrated circuit device, which includes a universal interconnect interface and other processing devices. The computing device interacts with the other processing devices to jointly complete user-specified computing operations. The integrated circuit device may also include a storage device, which is connected to the computing device and the other processing devices, respectively, for storing data in the computing device and the other processing devices.

Description

Technical Field

[0001] The present invention generally relates to the field of neural networks. More specifically, the present invention relates to an apparatus, a board, a method, and a readable storage medium for computing a neural network. Background Art

[0002] A neural network is a system of multiple neurons connected according to certain rules. It is generally composed of the following four layer structures: input layer, convolution layer, pooling layer, and fully connected layer.

[0003] The input layer extracts information from the input data and converts it into a feature matrix, which contains the features corresponding to this information. The convolutional layer is configured to receive the feature matrix from the input layer and extract features from the input data through convolution operations. In practice, convolutional layers can be constructed with multiple layers. The pooling layer is configured to replace a specific region of the data with a single value, which is usually the maximum or average value of all values ​​in that region. Pooling can reduce model size and increase computational speed without losing too much information. The fully connected layer acts as a classifier within the entire convolutional neural network, equivalent to a feature space transformation. It extracts and integrates all previously useful information, and compares information based on different classifications to determine whether the input data is similar to the target of comparison.

[0004] With technological advancements, the number of neural network layers has increased. For example, in the classic VGG architecture, VGG-A has 11 weight layers, VGG-B has 13 weight layers, VGG-C has 16 weight layers, VGG-D has 16 weight layers, and VGG-E has 19 weight layers. Convolutional layers and fully connected layers generally refer to weight layers. Some neural networks have hundreds of layers. Furthermore, as the number of layers increases, the number of parameters in a neural network increases exponentially. For example, AlexNet has 60 million parameters involved in its calculations.

[0005] Multiple layers and parameters require a large amount of on-chip and off-chip input / output access and sufficient storage space. Therefore, a mechanism to reduce input / output access and storage space usage is urgently needed in the field of artificial intelligence. Summary of the Invention

[0006] In order to at least partially solve the technical problems mentioned in the background technology, the solution of the present invention provides a device, a board, a method and a readable storage medium for calculating a neural network.

[0007] In one aspect, the present invention discloses a method for computing a neural network using a computing device connected to an off-chip memory that stores an on-chip cell map and corresponding weights. The computing device includes a plurality of clusters, each cluster including a shared memory unit and a plurality of processor cores.

[0008] The method includes: loading the on-chip unit graph and the i-th sub-weight in the weight from the off-chip memory into the shared memory unit; broadcasting the i-th sub-graph and the i-th sub-weight in the on-chip unit graph from the shared memory unit to at least one of the multiple processor cores, wherein the i-th sub-weight corresponds to the i-th sub-graph; calculating the i-th intermediate result based on the i-th sub-graph and the i-th sub-weight; loading the i+1-th sub-weight in the weight from the off-chip memory into the shared memory unit; broadcasting the i+1-th sub-graph in the on-chip unit graph from the shared memory unit to at least one of the multiple processor cores, wherein the i+1-th sub-weight corresponds to the i+1-th sub-graph; and broadcasting the i+1-th sub-weight from the shared memory unit to at least one of the multiple processor cores.

[0009] In another aspect, the present invention discloses a computing device connected to an off-chip memory, wherein the off-chip memory carries an on-chip unit map and corresponding weights, and the computing device includes multiple clusters, each cluster including a shared storage unit, a direct memory access module, multiple processor cores, and a broadcast bus.

[0010] The direct memory access module is used to: load the on-chip unit map and the i-th sub-weight in the weight from the off-chip memory into the shared storage unit; and load the i+1-th sub-weight in the weight from the off-chip memory into the shared storage unit.

[0011] The broadcast bus is used to: broadcast the i-th subgraph and the i-th sub-weight in the on-chip unit graph from the shared memory unit to at least one of the multiple processor cores, wherein the i-th sub-weight corresponds to the i-th subgraph; broadcast the i+1-th subgraph in the on-chip unit graph from the shared memory unit to at least one of the multiple processor cores, wherein the i+1-th sub-weight corresponds to the i+1-th subgraph; and broadcast the i+1-th sub-weight from the shared memory unit to at least one of the multiple processor cores.

[0012] At least one of the plurality of processor cores calculates an i-th intermediate result according to the i-th subgraph and the i-th sub-weight.

[0013] In another aspect, the present invention discloses an integrated circuit device including the aforementioned computing device, and a board including the aforementioned integrated circuit device.

[0014] In another aspect, the present invention discloses a computer-readable storage medium having stored thereon a computer program code for calculating a neural network using a computing device. When the computer program code is executed by a processing device, the aforementioned method is executed.

[0015] The present invention is based on the three-level operation hierarchy of system on chip - cluster - processor core, and the three-level memory design of DRAM - SRAM - NRAM / WRAM, to establish a two-layer three-level pipeline, fully utilize hardware resources, and improve the efficiency of neural network calculations. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts.

[0017] Figure 1 is a structural diagram showing a board card according to an embodiment of the present invention;

[0018] Figure 2 is a structural diagram showing an integrated circuit device according to an embodiment of the present invention;

[0019] Figure 3 is a schematic diagram showing the internal structure of a computing device according to an embodiment of the present invention;

[0020] Figure 4 is a schematic diagram showing the internal structure of a processor core according to an embodiment of the present invention;

[0021] Figure 5 is a schematic diagram showing when a processor core wants to write data to a processor core of another cluster;

[0022] Figure 6 is a schematic diagram illustrating a two-layer, three-stage pipeline according to an embodiment of the present invention;

[0023] Figure 7A is a schematic diagram illustrating a strategy for migrating weights according to an embodiment of the present invention;

[0024] Figure 7B is a schematic diagram illustrating another strategy for moving weights according to an embodiment of the present invention;

[0025] Figure 7C is a schematic diagram illustrating another strategy for moving weights according to an embodiment of the present invention;

[0026] Figure 7D is a schematic diagram illustrating another strategy for moving weights according to an embodiment of the present invention;

[0027] Figure 8 is a schematic diagram showing the fusion of two convolutional layers according to an embodiment of the present invention;

[0028] Figure 9 is a schematic diagram showing the formats of NCHW and NHWC;

[0029] Figure 10 is a flow chart illustrating a method for dynamically fusing neural networks according to a fusion strategy according to an embodiment of the present invention;

[0030] Figure 11 is a flow chart illustrating a calculation procedure executed in accordance with an embodiment of the present invention;

[0031] Figure 12 is a flow chart illustrating a weight transfer strategy according to an embodiment of the present invention;

[0032] Figure 13 is a flow chart illustrating another weight migration strategy according to an embodiment of the present invention;

[0033] Figure 14 is a flow chart illustrating another weight migration strategy according to an embodiment of the present invention; and

[0034] Figure 15 FIG. 1 is a flow chart illustrating another weight migration strategy according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of protection of the present invention.

[0036] It should be understood that the terms "first," "second," "third," and "fourth," etc. in the claims, description, and drawings of the present invention are used to distinguish different objects, rather than to describe a specific order. The terms "comprise" and "comprising" used in the description and claims of the present invention indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0037] It should also be understood that the terminology used in this specification is for the purpose of describing specific embodiments only and is not intended to limit the present invention. As used in the specification and claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should further be understood that the term "and / or" as used in the specification and claims refers to any and all possible combinations of one or more of the associated listed items, including and including these combinations.

[0038] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context.

[0039] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0040] A neural network is composed of an input layer, a convolutional layer, an activation function, a pooling layer, and a fully connected layer. This can range from a few layers to hundreds of layers. Each layer executes an operator. For example, a convolutional layer executes a convolution operator. This number of operators is determined by the number of layers. In this disclosure, when referring to a specific layer, this refers to the operator corresponding to that layer.

[0041] When performing neural network calculations, the input information and the output results of each layer of the model are different during each inference calculation. They are regarded as variable data. Variable data are generally represented by feature maps (matrices). In the present invention, the input information of the entire neural network model and the input maps of each layer of the model are collectively referred to as feature maps. Once the feature maps are loaded onto the on-chip memory component, they are referred to as on-chip unit maps in the present invention. The parameters of the training network model are usually not frequently changed after the training is stable, or they can be compiled and generated after the network topology and hardware parameters are determined. They will not change during the calculation process, so they can be regarded as constant data. Constant data includes but is not limited to weights, biases, device hardware instructions, mean and variance of batch norm, etc. In the present invention, weights are used to represent all constant data. When "data" is mentioned in the present invention, it generally refers to a graph structure that allows the operation operations of corresponding operators to be fused together in the neural network model according to the fusion strategy. The variable data and constant data involved in the graph structure are, that is, feature maps plus corresponding weights.

[0042] Figure 1 FIG. 1 is a schematic diagram showing the structure of a board 10 according to an embodiment of the present invention. Figure 1As shown, board 10 includes chip 101, which is a system-on-chip (SoC), or system-on-chip, integrated with one or more combined processing devices. The combined processing device is an artificial intelligence computing unit that supports various deep learning and machine learning algorithms to meet the intelligent processing needs in complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in the field of cloud intelligence. A notable feature of cloud-based intelligent applications is the large amount of input data, which places high demands on the platform's storage and computing capabilities. Board 10 of this embodiment is suitable for cloud-based intelligent applications and has extensive off-chip storage, on-chip storage, and a large amount of computing power.

[0043] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. Calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface formats, such as a PCIe interface.

[0044] Board 10 also includes a memory device 104 for storing data, which includes one or more storage units 105. Memory device 104 is connected to a control device 106 and chip 101 via a bus for data transmission. Control device 106 in board 10 is configured to control the state of chip 101. To this end, in one application scenario, control device 106 may include a microcontroller (MCU).

[0045] Figure 2 FIG. 1 is a block diagram showing the combined processing device in the chip 101 of this embodiment. Figure 2 As shown in , the combined processing device 20 includes a computing device 201 , an interface device 202 , a processing device 203 and a DRAM 204 .

[0046] The computing device 201 is configured to perform user-specified operations and is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0047] Interface device 202 is used to transmit data and control instructions between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to a storage device on-chip of computing device 201. Furthermore, computing device 201 can obtain control instructions from processing device 203 via interface device 202 and write them to a control cache on-chip of computing device 201. Alternatively or optionally, interface device 202 can also read data from the storage device of computing device 201 and transmit it to processing device 203.

[0048] The processing device 203 is a general processing device that performs basic controls including but not limited to data handling, starting and / or stopping the computing device 201. Depending on the implementation, the processing device 203 can be a central processing unit (CPU), a graphics processing unit (GPU), or one or more types of processors among other general and / or special processors. These processors include but are not limited to digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, with respect to the computing device 201 of the present invention, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are integrated and considered together, the two are regarded as forming a heterogeneous multi-core structure.

[0049] DRAM 204 is an off-chip memory used to store data to be processed. It is DDR and is typically 16G or larger in size. It is used to store data of the computing device 201 and / or the processing device 203 .

[0050] Figure 3 Figure 2 shows a schematic diagram of the internal structure of computing device 201. Computing device 201 is used to process input data for computer vision, speech, natural language processing, data mining, and other applications. The computing device 201 in the figure adopts a multi-core layered architecture. As a system-on-chip (SoC), computing device 201 includes multiple clusters, each of which includes multiple processor cores. In other words, computing device 201 is constructed in a hierarchy of SoC-cluster-processor cores.

[0051] At the system-on-chip level, Figure 3 As shown, the computing device 201 includes an external storage controller 301 , a peripheral communication module 302 , an on-chip interconnect module 303 , a synchronization module 304 and multiple clusters 305 .

[0052] There can be multiple external storage controllers 301, and two are shown in the figure as an example. They are used to respond to access requests issued by the processor core and access external storage devices, such as Figure 2 DRAM 204 in the chip, thereby reading data from or writing data to the chip. The peripheral communication module 302 is used to receive control signals from the processing device 203 through the interface device 202 to start the computing device 201 to perform tasks. The on-chip interconnect module 303 connects the external storage controller 301, the peripheral communication module 302 and multiple clusters 305 to transmit data and control signals between each module. The synchronization module 304 is a global synchronization barrier controller (GBC) used to coordinate the work progress of each cluster and ensure information synchronization. Multiple clusters 305 are the computing cores of the computing device 201. Four are shown as an example in the figure. With the development of hardware, the computing device 201 of the present invention can also include 8, 16, 64, or even more clusters 305. Clusters 305 are used to efficiently execute deep learning algorithms.

[0053] At the cluster level, Figure 3 As shown, each cluster 305 includes multiple processor cores (IPU cores) 306 and a memory core (MEM core) 307 .

[0054] The figure shows four processor cores 306 as an example, but the present invention does not limit the number of processor cores 306. Figure 4 Each processor core 306 includes three modules: a control module 41 , a calculation module 42 and a storage module 43 .

[0055] The control module 41 coordinates and controls the operations of the computing module 42 and the storage module 43 to complete deep learning tasks. It includes an instruction fetch unit (IFU) 411 and an instruction decode unit (IDU) 412. The instruction fetch unit 411 retrieves instructions from the processing device 203, while the instruction decode unit 412 decodes the retrieved instructions and sends the decoded results as control information to the computing module 42 and the storage module 43.

[0056] The operation module 42 includes a vector operation unit 421 and a matrix operation unit 422. The vector operation unit 421 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformation. The matrix operation unit 422 is responsible for the core calculations of the deep learning algorithm, namely matrix multiplication and convolution.

[0057] The storage module 43 is used to store or move related data and includes a neuron RAM (NRAM) 431, a weight RAM (WRAM) 432, an input / output direct memory access module (IODMA) 433, and a move direct memory access module (MVDMA) 434. NRAM 431 is used to store feature maps and intermediate results calculated by the processor core 306; WRAM 432 is used to store the weights of the deep learning network; IODMA 433 controls memory access between NRAM 431 / WRAM 432 and DRAM 204 via the broadcast bus 309; and MVDMA 434 controls memory access between NRAM 431 / WRAM 432 and SRAM 308.

[0058] Back to Figure 3 The storage core 307 is primarily used for storage and communication, namely, storing shared data or intermediate results between the processor cores 306, and performing communication between the cluster 305 and the DRAM 204, between the clusters 305, and between the processor cores 306. In other embodiments, the storage core 307 has scalar operation capabilities and is used to perform scalar operations.

[0059] The storage core 307 includes a shared memory unit (SRAM) 308, a broadcast bus 309, a cluster direct memory access module (CDMA) 310, and a global direct memory access module (GDMA) 311. The SRAM 308 acts as a high-performance data transfer station. Data reused between different processor cores 306 within the same cluster 305 does not need to be obtained from the DRAM 204 by each processor core 306. Instead, it is transferred between the processor cores 306 via the SRAM 308. The storage core 307 only needs to quickly distribute the reused data from the SRAM 308 to multiple processor cores 306, thereby improving inter-core communication efficiency and significantly reducing on-chip and off-chip input / output access.

[0060] The broadcast bus 309, CDMA 310, and GDMA 311 are used for communication between the processor cores 306, communication between the clusters 305, and data transmission between the clusters 305 and the DRAM 204, respectively. Each of these will be described below.

[0061] The broadcast bus 309 facilitates high-speed communication between the processor cores 306 within the cluster 305. In this embodiment, the broadcast bus 309 supports inter-core communication methods including unicast, multicast, and broadcast. Unicast refers to point-to-point data transmission (i.e., from one processor core to another), multicast transfers a copy of data from the SRAM 308 to a specific number of processor cores 306, and broadcast transfers a copy of data from the SRAM 308 to all processor cores 306, a special case of multicast.

[0062] The CDMA 310 is used to control memory access to the SRAM 308 between different clusters 305 within the same computing device 201 . Figure 5 This diagram illustrates the operation of CDMA 310 when one processor core attempts to write data to a processor core in another cluster. In this scenario, the same computing device includes multiple clusters. For ease of illustration, only Cluster 0 and Cluster 1 are shown. Cluster 0 and Cluster 1 each include multiple processor cores. Similarly, for ease of illustration, only Core 0 is shown in Cluster 0, and Core 1 is shown in Cluster 1. Core 0 attempts to write data to Core 1.

[0063] First, processor core 0 sends a unicast write request to write data to local SRAM 0. CDMA 0 acts as the master and CDMA 1 acts as the slave. The master pushes the write request to the slave. That is, the master sends the write address AW and write data W to transfer the data to SRAM 1 of cluster 1. The slave then sends a write response B in response. Finally, processor core 1 of cluster 1 sends a unicast read request to read the data from SRAM 1.

[0064] Back to Figure 3GDMA 311 works with external memory controller 301 to control memory access from cluster 305's SRAM 308 to DRAM 204, or to read data from DRAM 204 to SRAM 308. As previously mentioned, communication between DRAM 204 and NRAM 431 or WRAM 432 can be achieved through two channels. The first channel directly connects DRAM 204 and NRAM 431 or WRAM 432 via IODAM 433. The second channel first transfers data between DRAM 204 and SRAM 308 via GDMA 311, and then transfers data between SRAM 308 and NRAM 431 or WRAM 432 via MVDMA 434. While the second channel may appear to require more components and result in a longer data flow, in some embodiments, the bandwidth of the second channel is significantly greater than the first channel. Therefore, communication between DRAM 204 and NRAM 431 or WRAM 432 through the second channel may be more efficient. The embodiment of the present invention can select a data transmission channel according to its own hardware conditions.

[0065] In other embodiments, the functions of GDMA 311 and IODMA 433 can be integrated into the same component. For ease of description, the present invention treats GDMA 311 and IODMA 433 as different components. For those skilled in the art, as long as the functions implemented and the technical effects achieved are similar to those of the present invention, they are within the scope of protection of the present invention. Furthermore, the functions of GDMA 311, IODMA 433, CDMA 310, and MVDMA 434 can also be implemented by the same component. Similarly, as long as the functions implemented and the technical effects achieved are similar to those of the present invention, they are within the scope of protection of the present invention.

[0066] In the aforementioned hardware architecture, the components within computing device 201 are referred to as on-chip, while those outside computing device 201 are referred to as off-chip. Executing the various layers of deep learning requires numerous on-chip and off-chip accesses, particularly reading data from DRAM 204 into computing device 201 and storing the computational results of computing device 201 back in DRAM 204. This frequent access consumes significant hardware resources. To address this issue, one embodiment of the present invention significantly reduces on-chip and off-chip accesses by designing a weight transfer mechanism, while also lowering on-chip weight storage requirements.

[0067] An important reason why the computing device 201 has strong computing power is that the three-level computing hierarchy of system on chip - cluster - processor core, combined with the three-layer memory design of DRAM - SRAM - NRAM / WRAM, enables data to be cached and calculated at the appropriate level, forming a sufficient pipeline.

[0068] When the computing device 201 performs a calculation, the process can be divided into three main phases: load phase (load): loading data; compute phase (compute): transferring data, performing calculations, and transferring intermediate results; and store phase (store): storing results.

[0069] In more detail, this embodiment adopts a two-layer three-stage pipeline, such as Figure 6 As shown, the first-level load phase 601, calculation phase 602, and store-back phase 603 occur at the cluster level. In the first-level load phase 601, GDMA 330 loads data from DRAM 204 into SRAM 308. In the first-level calculation phase 602, cluster 305 performs calculations on the loaded on-chip cell graph and generates calculation results. In the first-level store-back phase 603, GDMA 330 stores the calculation results from SRAM 308 back into DRAM 204.

[0070] Because cluster 305 includes multiple processor cores 306, the first-level computation stage 602 actually partitions the on-chip unit graph into corresponding subgraphs via storage core 307 and broadcasts them to at least one processor core 306 for computation. Therefore, the second-level three-level waterline occurs within processor core 306. More specifically, the second-level load stage 604 involves MVDMA 434 loading the subgraph from SRAM 308 into NRAM 431. The second-level computation stage 605 transfers the subgraph and subweights to the computation module 42 for computation, and then transfers the intermediate results back to NRAM 431. The second-level store-back stage 606 involves MVDMA 434 storing the intermediate results from NRAM 431 back into SRAM 308.

[0071] The first-level pipeline means that the first-level load stage 601, the first-level calculation stage 602, and the first-level store-back stage 603 can be executed in parallel. For example, let's take the case where the same cluster 305 wants to process the j-th on-chip cell graph, the j+1-th on-chip cell graph, and the j+2-th on-chip cell graph. First, the j-th on-chip cell graph is loaded into SRAM 308 in the first-level load stage 601. Then, the j-th on-chip cell graph is calculated in the first-level calculation stage 602, and the first calculation result is transferred back to SRAM 308. While the j-th on-chip cell graph is being calculated, the j+1-th on-chip cell graph is loaded into SRAM 308 in the first-level load stage 607. When the first calculation result is stored back to DRAM 204 in the first-level store-back stage 603, the j+1th on-chip unit map is calculated in the first-level calculation stage 608, and the second calculation result is transferred back to SRAM 308. Simultaneously, the j+2th on-chip unit map is loaded into SRAM 308 in the first-level load stage 610. The first-level pipeline continues in this manner.

[0072] To facilitate the aforementioned pipeline operation, the SRAM 308 of this embodiment includes two storage spaces: a ping memory unit and a pong memory unit. Data flow is categorized into three types based on the ping-pong properties of the SRAM 308: input / output ping-pong (IOparity), input ping-pong (input parity), and no ping-pong (no parity). I / O ping-pong supports parallel loading, computation, and storage. To implement I / O ping-pong, the ping memory unit and the pong memory unit must be completely equal, serving as loading and storage, respectively. Input ping-pong only supports storage and computation in parallel, which increases the transfer time on the SRAM 308. Compared to I / O ping-pong, the ping memory unit and the pong memory unit do not need to be completely equal, but an additional cache equal to the storage space is required. No ping-pong refers to serial loading, storage, and computation, requiring no additional space allocation.

[0073] In order to realize the aforementioned first-level pipeline, the SRAM 308 of this embodiment has ping memory units and pong memory units of the same size to achieve the input / output ping-pong effect. Figure 6For illustration, the storage area involved in the first-level loading stage 601, the first-level calculation stage 602 and the first-level store-back stage 603 of the j-th on-chip unit diagram is limited to the ping storage unit, while the storage area involved in the first-level loading stage 607, the first-level calculation stage 608 and the first-level store-back stage 609 of the j+1-th on-chip unit diagram is limited to the pong storage unit, and the storage area involved in the first-level loading stage 610, the first-level calculation stage 611 and the first-level store-back stage 612 of the j+2-th on-chip unit diagram is again limited to the ping storage unit, and in this way, the ping storage unit and the pong storage unit are used alternately for storage.

[0074] The second-level pipeline refers to the parallel execution of the second-level load stage 604, the second-level calculation stage 605, and the second-level store-back stage 606. For example, let's assume that the same processor core 306 is processing the i-th subgraph, the i+1-th subgraph, and the i+2-th subgraph in the j-th on-chip unit graph. First, the i-th subgraph is broadcast to NRAM 431 in the second-level load stage 604. Then, the i-th subgraph is calculated in the second-level calculation stage 605 to generate the i-th intermediate result, which is then transferred back to NRAM 431. Simultaneously, the i+1-th subgraph is broadcast to NRAM 431 in the second-level load stage 613. The i-th intermediate result is stored back to the SRAM 308 in the second-level store-back stage 606, while the i+1-th subgraph is calculated in the second-level calculation stage 614 to generate the i+1-th intermediate result, and the i+1-th intermediate result is moved back to the NRAM 431, and the i+2-th subgraph is loaded into the NRAM 431 in the second-level load stage 615.

[0075] Considering that the tasks of each cluster 305 are different and the completion times are naturally inconsistent, the synchronization module 304 of this embodiment uses synchronization barrier instructions to synchronize the task completion times to avoid timing errors.

[0076] As for weights, in order to implement the aforementioned two-layer pipeline, there are several strategies for moving weights.

[0077] A strategy such as Figure 7A As shown, SRAM 308 is divided into four spaces: other parameter storage unit 701, weight storage unit 702, ping storage unit 703, and pong storage unit 704. Other parameter storage unit 701 is used to load parameters other than weights, such as bias data, from DRAM 204. Weight storage unit 702 is a single block of storage space used to load weight data from DRAM 204. As previously described, ping storage unit 703 and pong storage unit 704 are used to load different on-chip unit maps from DRAM 204 or corresponding intermediate results stored back from NRAM 321. WRAM 432 is a single block of storage space.

[0078] In the first-level load phase 601, the GDMA 311 loads the j-th on-chip cell map from the DRAM 204 into the ping storage unit 703. It also loads the i-th sub-weight (corresponding to the i-th sub-map of the j-th on-chip cell map) from the DRAM 204 into the weight storage unit 702. In the first-level calculation phase 602, the broadcast bus 309 broadcasts the i-th sub-map and the i-th sub-weight of the on-chip cell map from the SRAM 308 to the NRAM 431 and WRAM 432 of at least one of the processor cores 306 in the second-level load phase 604. The arithmetic unit 42 of the processor core 306 calculates the i-th intermediate result based on the i-th sub-map and the i-th sub-weight in the second-level calculation phase 605. Simultaneously, the broadcast bus 309 broadcasts the i+1-th sub-map of the j-th on-chip cell map from the SRAM 308 to the NRAM 431 in the second-level load phase 613. Since the i-th sub-weight has been loaded into the WRAM 432, the space in the weight storage unit 702 can be released. Therefore, at the same time, the GDMA 311 loads the i+1-th sub-weight (corresponding to the i+1-th subgraph) in the weight from the DRAM 204 into the weight storage unit 702. Since the weight storage unit 702 is a whole block of storage space, the i-th sub-weight in the weight storage unit 702 is overwritten by the i+1-th sub-weight.

[0079] After the i-th intermediate result is generated, the broadcast bus 309 broadcasts the i+1-th sub-weight from the weight storage unit 702 to the WRAM 432. Since the WRAM 432 is also a block of storage space, the i-th sub-weight is also overwritten by the i+1-th sub-weight. It is important to note that the broadcast bus 309 broadcasts the i+1-th sub-weight to the WRAM 432 only after the i-th intermediate result is generated. The reason is that if the i+1-th sub-weight is broadcast to the WRAM 432 when the i-th intermediate result is calculated, the i-th sub-weight may need to be read from the WRAM 432 at any time during the calculation process. However, the i-th sub-weight will have been overwritten, resulting in an incorrect calculation result.

[0080] In the second-layer store-recall phase 606, the MVDMA 434 stores the i-th intermediate result back to the ping storage unit 703. At the same time, the operation unit 42 calculates the i+1-th intermediate result based on the i+1-th sub-graph and the i+1-th sub-weight in the second-layer calculation phase 614, and the i+2-th sub-graph enters the second-layer load phase 615. The broadcast bus 309 broadcasts the i+2-th sub-graph from the SRAM 308 to the NRAM 431. At the same time, the GDMA 311 loads the i+2-th sub-weight (corresponding to the i+2-th sub-graph) in the weight from the DRAM 204 to the weight storage unit 702. The i+1-th sub-weight in the weight storage unit 702 is overwritten by the i+2-th sub-weight.

[0081] After the i+1th intermediate result is generated, the broadcast bus 309 broadcasts the i+2th sub-weight from the weight storage unit 702 to the WRAM 432 . The i+1th sub-weight in the WRAM 432 is also overwritten by the i+2th sub-weight.

[0082] In this strategy, the weights required to calculate the j-th on-chip unit graph are not calculated once with the j-th on-chip unit. Figure 1 Instead of loading all weights into SRAM 308, only the sub-weights of the corresponding subgraph are loaded into SRAM 308 at a time. Only one sub-weight is stored in weight storage unit 702 at a time. To this end, before executing a computation task, computing device 201 first identifies the largest of all sub-weights in the weight, and then determines the size of weight storage space 702 based on the largest sub-weight. For example, if a weight is split into five sub-weights, with sizes of 50KB, 100KB, 120KB, 300KB, and 200KB, respectively, and the largest sub-weight is 300KB, computing device 201 allocates 300KB of space to weight storage space 702. This space is sufficient to store all sub-weights, saving on-chip storage space.

[0083] Another strategy is Figure 7B As shown, the SRAM 308 is also divided into four spaces, but the WRAM 432 is divided into multiple storage spaces, three of which are exemplarily shown in the figure, namely the first storage space 705, the second storage space 706, and the third storage space 707.

[0084] In the first level loading stage 601 , the GDMA 311 loads the j-th on-chip unit map from the DRAM 204 into the ping storage unit 703 , and also loads the i-th sub-weight from the DRAM 204 into the weight storage unit 702 . In the first-level calculation stage 602, the broadcast bus 309 broadcasts the i-th subgraph to the NRAM 431 in the second-level loading stage 604, and also broadcasts the i-th sub-weight to the first storage space 705. The operation unit 42 calculates the i-th intermediate result in the second-level calculation stage 605 based on the i-th subgraph and the i-th sub-weight. At the same time, the broadcast bus 309 broadcasts the i+1-th subgraph in the j-th on-chip unit graph from the SRAM 308 to the NRAM 431 in the second-level loading stage 613. At the same time, the GDMA 311 loads the i+1-th sub-weight from the DRAM 204 into the weight storage unit 702. The i-th sub-weight in the weight storage unit 702 is overwritten by the i+1-th sub-weight.

[0085] After the i+1th sub-weight is loaded into the weight storage unit 702, regardless of whether the i-th intermediate result is calculated, the broadcast bus 309 broadcasts the i+1th sub-weight from the weight storage unit 702 to the second storage space 706. Since the i-th sub-weight is stored in the first storage space 705, it will not be overwritten by the i+1th sub-weight, so as not to affect the correctness of the i-th intermediate result.

[0086] In the second-layer store-back stage 606, the MVDMA 434 stores the i-th intermediate result back to the ping storage unit 703. At the same time, the operation unit 42 calculates the i+1-th intermediate result in the second-layer calculation stage 614 based on the i+1-th sub-graph and the i+1-th sub-weight. The i+2-th sub-graph enters the second-layer loading stage 615, and the broadcast bus 309 broadcasts the i+2-th sub-graph from the SRAM 308 to the NRAM 431. At the same time, the GDMA 311 loads the i+2-th sub-weight from the DRAM 204 into the weight storage unit 702. The i+1-th sub-weight in the weight storage unit 702 is overwritten by the i+2-th sub-weight.

[0087] After the i+2th sub-weight is loaded into the weight storage unit 702, regardless of whether the i+1th intermediate result has been calculated, the broadcast bus 309 broadcasts the i+2th sub-weight from the weight storage unit 702 to the third storage space 707. The i+1th sub-weight is not overwritten, so as not to affect the correctness of the i+1th intermediate result.

[0088] In this strategy, WRAM 432 is divided into multiple storage spaces, so the sub-weight does not need to wait for the previous intermediate result to be completed before being broadcast. That is, the sub-weight can be broadcast to WRAM432 while the previous intermediate result is calculated, shortening the pipeline running time.

[0089] Another strategy is Figure 7C As shown, Figure 7A The difference is that the weight storage unit 702 is divided into multiple storage spaces, and the figure shows three spaces as an example, namely the first storage space 708, the second storage space 709, and the third storage space 710.

[0090] In the first-level loading stage 601, GDMA 311 loads the j-th on-chip unit map from DRAM 204 into the ping storage unit 703, and also loads the i-th sub-weight from DRAM 204 into the first storage space 708, loads the i+1-th sub-weight from DRAM 204 into the second storage space 709, and loads the i+2-th sub-weight from DRAM 204 into the third storage space 710. In the first-level calculation stage 602, the broadcast bus 309 broadcasts the i-th subgraph and the i-th sub-weight in the j-th on-chip unit graph from the SRAM 308 to the NRAM 431 and the WRAM 432 in the second-level loading stage 604. The operation unit 42 calculates the i-th intermediate result in the second-level calculation stage 605 based on the i-th subgraph and the i-th sub-weight. At the same time, the broadcast bus 309 broadcasts the i+1-th subgraph in the on-chip unit graph from the SRAM 308 to the NRAM 431 in the second-level loading stage 613.

[0091] After the i-th intermediate result is generated, the broadcast bus 309 broadcasts the i+1-th sub-weight from the second storage space 709 to the WRAM 432. Since the WRAM 432 is a single block of storage space, the i-th sub-weight is overwritten by the i+1-th sub-weight. In the second-level store-back phase 606, the MVDMA 434 stores the i-th intermediate result back to the ping storage unit 703. At the same time, the operation unit 42 calculates the i+1-th intermediate result based on the i+1-th sub-graph and the i+1-th sub-weight in the second-level calculation phase 614. The i+2-th sub-graph enters the second-level load phase 615, and the broadcast bus 309 broadcasts the i+2-th sub-graph from the SRAM 308 to the NRAM 431. After generating the i+1th intermediate result, the broadcast bus 309 broadcasts the i+2th sub-weight from the third storage space 710 to the WRAM 432 . The i+1th sub-weight in the WRAM 432 is overwritten by the i+2th sub-weight.

[0092] In this strategy, multiple sub-weights can be combined with on-chip units at once. Figure 1When loaded into SRAM 308, since on-chip and off-chip input / output access is a major bottleneck of computing efficiency, multiple storage spaces are divided on the weight storage unit 702 to store different sub-weights respectively. Although it occupies more storage space of SRAM 308, it is still greatly helpful to reduce input / output access.

[0093] Another strategy is Figure 7D As shown, the weight storage unit 702 and the WRAM 432 are both divided into multiple storage spaces.

[0094] In the first-level loading stage 601, GDMA 311 loads the j-th on-chip unit map from DRAM 204 into the ping storage unit 703, and also loads the i-th sub-weight from DRAM 204 into the first storage space 708, loads the i+1-th sub-weight from DRAM 204 into the second storage space 709, and loads the i+2-th sub-weight from DRAM 204 into the third storage space 710. In the first-level calculation stage 602, the broadcast bus 309 broadcasts the i-th subgraph and the i-th sub-weight in the j-th on-chip unit graph from the SRAM 308 to the NRAM 431 and the WRAM 432 in the second-level loading stage 604. The operation unit 42 calculates the i-th intermediate result in the second-level calculation stage 605 based on the i-th subgraph and the i-th sub-weight. At the same time, the broadcast bus 309 broadcasts the i+1-th subgraph in the j-th on-chip unit graph from the SRAM 308 to the NRAM 431 in the second-level loading stage 613.

[0095] After the i+1th sub-weight is loaded into the second storage space 709, regardless of whether the i-th intermediate result is calculated, the broadcast bus 309 broadcasts the i+1th sub-weight from the second storage space 709 to the second storage space 706. Since the i-th sub-weight is stored in the first storage space 705, it will not be overwritten by the i+1th sub-weight, so as not to affect the correctness of the i-th intermediate result. In the second-layer store-recall phase 606, the MVDMA 434 stores the i-th intermediate result back into the ping storage unit 703. Simultaneously, the arithmetic unit 42 calculates the i+1-th intermediate result based on the i+1-th subgraph and the i+1-th subweight in the second-layer calculation phase 614. The i+2-th subgraph then enters the second-layer load phase 615, and the broadcast bus 309 broadcasts the i+2-th subgraph from the SRAM 308 to the NRAM 431. Regardless of whether the i+1-th intermediate result has been calculated, the broadcast bus 309 broadcasts the i+2-th subweight from the third storage space 710 to the third storage space 707. The i+1-th subweight is not overwritten, thereby preventing the accuracy of the i+1-th intermediate result from being affected.

[0096] In this strategy, not only can multiple sub-weights be combined with on-chip units at once, Figure 1 At the same time as being loaded into SRAM 308, the sub-weights do not need to wait for the previous calculation result to be completed before being broadcast. In other words, the sub-weights can be broadcast to WRAM 432 while the previous intermediate result is being calculated, thereby shortening the input / output and pipeline operation time.

[0097] As previously mentioned, executing deep learning calculations at each layer requires a significant amount of off-chip and on-chip access, particularly reading data from DRAM 204 into computing device 201 and then storing the calculation results from computing device 201 back into DRAM 204. This frequent access consumes significant hardware resources. In practice, adjacent layers of a neural network are fused, significantly reducing off-chip and on-chip data transfers.

[0098] Figure 8 Figure 8 shows a schematic diagram of fusing two convolutional layers. The first convolutional layer 810 receives a 7×7 feature map 801 as input. This layer convolves feature map 801 with a 3×3 kernel (not shown) to produce feature map 802 for the first convolutional layer 810. The values ​​of the 5×5 feature submap 804 affect the 3×3 feature submap 805. Assuming a stride of 1, after calculating the 5×5 feature submap 804, the first convolutional layer 810 then calculates the 5×5 feature submap 806. The values ​​of the 5×5 feature submap 806 affect the 3×3 feature submap 807.

[0099] During the calculation of the second convolution layer 811, feature map 802 becomes the input of the second convolution layer 811 and is also convolved with the 3×3 kernel to obtain feature map 803 of the second convolution layer 811. The value of 3×3 feature submap 805 affects 1×1 feature submap 808 in feature map 803. After calculating 3×3 feature submap 805, the second convolution layer 811 then calculates 3×3 feature submap 807, and the value of 3×3 feature submap 807 affects 1×1 feature submap 809 in feature map 803.

[0100] If fusion is not performed, when computing device 201 performs the first convolution layer 810, it reads the 5×5 feature subgraph 804 from DRAM 204, stores the 3×3 feature subgraph 805 back to DRAM 204 after calculation, then reads the 5×5 feature subgraph 806 from DRAM 204, stores the 3×3 feature subgraph 807 back to DRAM 204 after calculation. When performing the second convolution layer 811, it also needs to read the 3×3 feature subgraph 805 from DRAM 204, stores the 1×1 feature subgraph 808 back to DRAM 204 after calculation, then reads the 3×3 feature subgraph 807 from DRAM 204, and stores the 1×1 feature subgraph 809 back to DRAM 204 after calculation. As can be seen from the above description, feature graph 802 is repeatedly read and stored off-chip as intermediate data, which consumes considerable system resources.

[0101] If the first convolutional layer 810 and the second convolutional layer 811 are fused, that is, feature map 802 is stored in NRAM 431, the number of accesses between computing device 201 and DRAM 204 can be reduced, thereby improving the overall execution efficiency of the neural network. Because the fused feature maps (such as feature map 801, feature map 802, and feature map 803) look like an inverted pyramid in the context logic of the neural network model, it is called pyramid fusion.

[0102] The following attempts to illustrate the pyramid fusion related to the present invention in a fusion method, which utilizes the aforementioned Figure 1 、 Figure 2 、 Figure 3 and Figure 4 This fusion is implemented using a hardware structure called a template fuse unit (TFU). The TFU flexibly fuses multiple layers into a single layer using a specific fusion strategy to reduce network input / output overhead. The collection of these fused layers is called a template fuse unit and can be considered a new or customized layer.

[0103] This fusion method simultaneously loads the feature maps required by the template fusion unit from DRAM 204 to the on-chip SRAM 308. Once loaded into SRAM 308, the feature maps are called on-chip unit maps. The on-chip unit maps are then divided into sub-maps. Each sub-map is loaded from SRAM 308 into the NRAM 431 of the processor core 306 assigned to calculate that sub-map. The sub-weights required to calculate that sub-map are also loaded from SRAM 308 into the NRAM 432 at appropriate times. After each sub-map is calculated, a corresponding intermediate result is obtained and stored back in SRAM 308. Once all sub-maps are calculated, the results are stored back in DRAM 204 all at once. In other words, the corresponding results obtained from the on-chip unit maps and weights participating in the computational operations of the operators in the neural network model are transferred between DRAM 204 and SRAM 308, and the corresponding outputs (intermediate results) of the sub-maps are transferred between SRAM 308 and NRAM 431. From the perspective of the computing device 201 , the feature map of the template fusion unit is loaded in units of on-chip unit maps, while the computation is performed in units of sub-maps.

[0104] In more detail, SRAM 308 is one of the important reference indicators of the fusion strategy, and its space size determines whether the template fusion unit is in large-image mode or small-image mode. Small-image mode and large-image mode refer to whether a feature map stored in DRAM 204 can be moved to SRAM 308 for processing at one time. The processing device 203 will compare the storage space required for the feature map with the available space in SRAM 308. If the SRAM 308 space is insufficient and the feature map cannot be placed, it is in large-image mode; if the SRAM 308 is large enough to accommodate the entire feature map, it is in small-image mode. It should be noted that in large-image mode, the on-chip unit map is only a part of the feature map; in small-image mode, if the available space in SRAM 308 is large enough, or the feature map is small enough, SRAM 308 may be able to accommodate multiple feature maps at one time, that is, the on-chip unit map can include multiple feature maps.

[0105] If the large-image mode is used, the feature map must be split before it can be loaded into the computing device 201. The processing device 203 will split the feature map on the DRAM 204 until a sufficiently small on-chip unit map is generated to meet the space requirements of the SRAM 308, allowing the on-chip unit map to be moved to the SRAM 308 for processing in one go. However, when the feature map is split, input-dependent and output-dependent operations may be generated.

[0106] Input-dependent operations occur when the split on-chip unit graphs overlap at least partially. Each subset requires additional copies of some inputs to perform a complete operation, resulting in data redundancy during the split operation. Data redundancy refers to the reuse of the same data within the system. Input-dependent operations also occur when the template fusion unit includes layers such as convolution, pooling, or matrix multiplication.

[0107] Output-dependent operations refer to the need for reduction after each subgraph produces an intermediate result to obtain the calculation result. Reduction refers to splitting the on-chip unit graph into subgraphs based on an understanding of the content of the on-chip unit graph itself and performing separate calculations to reduce the scale of calculations. This minimizes the amount of data while maintaining the original appearance of the on-chip unit graph as much as possible, and then restores or integrates the calculation results based on the subgraphs. During reduction, the calculation results are mutually dependent. When the template fusion unit includes layers such as inner product, convolution, matrix multiplication, sorting, and counting, it will lead to output-dependent operations.

[0108] The data format of a feature map consists of the dimensions N, H, W, and C, where N represents the batch, H the height, W the width, and C the channel. For example, in the case of image data, N represents the number of images in the batch, H the vertical pixel count, W the horizontal pixel count, and C the number of channels (e.g., a black and white image has 1 channel, while an RGB image has 3 channels).

[0109] The order of these dimensions determines the composition of the data. Common compositions include NHWC and NCHW. Figure 9The following figure illustrates the difference between the NCHW and NHWC formats. This figure uses an RGB color image as an example, where R represents red pixels, G represents green pixels, and B represents blue pixels. Sequence 91 is in the NCHW format, with N arranged in the outermost layer. Pixels within each channel are closely spaced, and then arranged in RGB order. The offset in storage for an element with coordinates (n, c, h, w) is ((n×C+c)×H+h)×W+w). Sequence 92 is in the NHWC format, with C arranged in the innermost layer. RGB pixels corresponding to spatial locations across multiple channels are closely spaced. The figure also shows the positions of input pixels 901, 902, and 903 under different arrangements. These three pixels, 901, 902, and 903, together represent the color of a point in the image. The coordinate offset for an element with coordinates (n, c, h, w) is calculated as ((n×H+h)×W+w)×C+c). First, NHWC is closer to the BMP image data storage format than NCHW. BMP files store data pixel by pixel, with each pixel storing the color values ​​of all channels. This eliminates the need for additional dimensionality conversion when reading input images. Consequently, NHWC offers better memory locality, generating one output pixel for every three input pixels. NCHW, on the other hand, must wait until all channel inputs are ready before generating the final output, requiring a larger cache space.

[0110] Figure 10 A flowchart showing the fusion method is shown.

[0111] In step 1001, the starting layer of the template fusion unit is selected according to the starting rule of the fusion strategy. The processing device 203 selects the starting layer of the template fusion unit according to the starting rule of the fusion strategy, that is, selects the layer to start fusion from the layers that have not yet been fused in the neural network.

[0112] In one application scenario, the starting rule may be that the starting layer is the first unfused layer in the neural network, and the processing device 203 searches for the first unfused layer. In another application scenario, considering that convolution and pooling layers consume the most input / output resources, the starting rule is that the starting layer is the first unfused convolution or pooling layer. The processing device 203 first finds all unfused convolution and pooling layers in the neural network model and starts fusing from the first unfused convolution or pooling layer.

[0113] In step 1002, fusion is performed based on the starting layer, and all rules of the fusion strategy are checked one by one to establish a template fusion unit. The processing device 203 performs fusion based on the starting layer, and all rules of the fusion strategy are checked one by one to establish a template fusion unit. Under the premise of meeting all rules, the hardware resources of the computing device 201 are sufficient to support the one-time loading of the on-chip unit map required for the calculation template fusion unit, and then perform neural network calculations according to the template fusion unit. In addition to the aforementioned starting rules, the fusion strategy can also illustratively include the following rules:

[0114] Rule 1: Backward Fusion

[0115] Backward fusion refers to fusion from the starting layer toward the inference direction of the neural network model, that is, fusion in the direction of the first layer → the second layer → the third layer. If there are unfused layers before the starting layer, these unfused layers will not be considered for inclusion in the template fusion unit under this rule.

[0116] Rule 2: Prioritize forward integration

[0117] The so-called forward fusion refers to the fusion in the opposite direction from the starting layer to the neural network reasoning, that is, the fusion in the direction of the third layer → the second layer → the first layer. This rule is usually combined with the starting rule that the starting layer is the frontmost unfused convolution or pooling layer, because there may be unfused layers before the convolution or pooling layer. After selecting the starting layer, the processing device 203 gives priority to forward fusion, trying to incorporate the layers that have not been fused before the starting layer into the template fusion unit. Assuming that the 1st layer to the 2nd layer have been fused, the processing device 203 finds that the frontmost unfused convolution or pooling layer is the 5th layer, so the starting layer is the 5th layer, and the 4th and 3rd layers are fused forward first. If fusion can continue, the 6th and 7th layers are fused backward, and so on.

[0118] Rule 3: Single branch output

[0119] The fusion strategy of this fusion method does not support template fusion units with multi-output networks. The reason is that the shape derivation implemented inside the template fusion unit mainly adopts the form of back-to-forward derivation. A multi-output network means that it is necessary to deduce forward from different outputs separately. The derivation results do not necessarily come down to the same feature map, so it cannot converge.

[0120] Rule 4: Include at least 2 main layers

[0121] When the layer logic is too simple, the performance of the template fusion unit is not as good as that of the unfused layer. Therefore, when the layer logic is used as the fusion strategy, the processing device 203 will evaluate whether the operations of the fused layers are complex enough to make the fusion produce benefits. To produce benefits, it is necessary to include the main layer into the template fusion unit as much as possible. The main layer refers to the layer that consumes a lot of input / output resources, such as matrix multiplication, pooling or convolution. The pooling here includes various types of pooling, such as maximum pooling (maxpool) or average pooling (avgpool), and the convolution also includes various types of convolution, such as ordinary convolution, convolution with mean, channel convolution (depthwise conv), etc. This rule is that the template fusion unit includes at least 2 main layers. When the processing unit 203 determines that this rule is not met, the processing device 203 will adjust the template fusion unit until this rule is met.

[0122] Rule 5: A continuous structure consisting of the main layer, the main layer, and the non-main layer.

[0123] This rule requires that the template fusion unit must include a continuous structure of a primary layer, a primary layer, and a non-primary layer, i.e., a continuous structure of a primary layer, a primary layer, and a non-primary layer adjacent to each other in sequence. This computational complexity makes the fusion effective. If the processing unit 203 determines that this rule is not met, the processing device 203 adjusts the template fusion unit until the rule is met.

[0124] Rule 6: The weights of the convolutional layer are not shared with any other layers of the neural network

[0125] Because the weights of the operators in the neural network model involved in the template fusion unit have a special arrangement, when the fused convolution operator shares weights with other operators, the weight arrangement logic will conflict. This rule is that the weights of the convolution operator in the template fusion unit are not shared with any layer of the neural network. When the processing unit 203 determines that this rule is not met, the processing device 203 will remove the convolution operator from the template fusion unit.

[0126] Rule 7: The weight is not greater than the available space of WRAM

[0127] In small image mode, the on-chip unit map may include multiple feature maps. In this case, the required weights will increase, so it is necessary to carefully evaluate whether the available space in WRAM 432 is sufficient. The rule is that the storage space required for the weights in the on-chip unit map should not exceed the available space in WRAM 432. When the processing device 203 determines that this rule is not met, the processing device 203 will reduce the size of the on-chip unit map. If WRAM 432 is split into multiple storage spaces, the rule is that the storage space required for the sub-weights should not exceed the available space in the weight storage space.

[0128] Rule 8: Redundancy Percentage

[0129] The redundancy percentage is the ratio of the sum of the redundancies generated by the input dependency operation and the output dependency operation to the normal input / output volume of the template fusion unit. Here, the normal input / output volume refers to the amount of data without redundancy in the on-chip unit map before it is split. The processing device 203 calculates the memory size of the on-chip unit map from DRAM 204 to SRAM 308 after the template fusion unit fuses the current layer. TFU , and the normal input / output size (excluding redundancy) ori The percentage of access size TFU Refers to the theoretical access size ori Add the redundancy sum. The formula is as follows:

[0130]

[0131] The processing device 203 takes into account the splitting information and shape derivation of the template fusion unit and sets a percentage threshold of 50%, 75%, 100%, 125% or 150%, preferably 100%. Taking a percentage threshold of 100% as an example, it means that when the total redundancy is greater than twice the normal input / output of the template fusion unit, fusion will no longer be performed. This rule is that the total redundancy generated by the split on-chip unit diagram does not exceed a specific ratio related to the percentage threshold. Once it exceeds the ratio, it means that there is too much redundancy, a large amount of resources will be consumed in calculating redundancy, and performance will decrease. Therefore, when the processing device 203 determines that this rule is not met, the processing device 203 will stop fusion.

[0132] It should be noted that in the small image mode, since at least one complete feature map is loaded from the DRAM 204 to the SRAM 308 at a time, no redundancy is generated. This rule does not apply to the small image mode.

[0133] Rule 9: Input and output dimensions of on-chip unit diagrams

[0134] Assuming that the space size of SRAM 308 is S, the storage space required by the on-chip unit map is IN, and the storage space required for the calculation result of the on-chip unit map is OUT, then this rule states that the space size of SRAM 308 must meet the following conditions:

[0135] If IN and OUT cannot reuse storage space, IN+OUT

[0136] If IN and OUT can reuse storage space, MAX(IN,OUT)

[0137] ​​That is, if IN and OUT cannot reuse storage space, the sum of the storage space of the on-chip unit diagram and the storage space of the calculation results is less than the available space of SRAM 308; if IN and OUT can reuse storage space, the larger of the storage space of the on-chip unit diagram and the storage space of the calculation results is less than the available space of SRAM 308.

[0138] If the SRAM 308 is split into ping memory space and pong memory space, the rule is that the sum of the memory space of the on-chip cell map and the memory space of its calculation results is less than the available space of the ping memory space or the pong memory space.

[0139] Rule 10: The storage space required for a subgraph should not exceed the available space in NRAM

[0140] This rule is that the storage space required for the sub-graph is not greater than the available space of NRAM 431. When the on-chip unit graph on SRAM 308 is to be split into sub-graphs and transferred to NRAM 431, the processing device 203 can perform fine-grained splitting in the N, H, and W dimensions. If the space of NRAM 431 is insufficient, the processing device 203 will split the on-chip unit graph into smaller pieces until this rule is met. Generally speaking, NRAM 431 will have a reasonable amount of available space, so that the on-chip unit graph can be loaded at one time after being split to a reasonable degree. From the perspective of the fusion strategy, the template fusion unit will not be affected by the number of batches. However, the smaller the on-chip unit graph is split (i.e., the more sub-graphs there are), the lower the processing speed will be, so the processing device 203 needs to evaluate the space of NRAM 431.

[0141] In some embodiments, the space of SRAM 308 corresponds to the number of NRAM 431 of the processor cores 306 in the cluster 305. For example, if the cluster 305 includes four processor cores 306, the space of SRAM 308 is four times the space of NRAM 431. In other words, the on-chip unit in the large image mode Figure 1 Generally, it can be allocated to four processor cores 306 for processing. This architecture design has taken into account that the data loaded into SRAM 308 can be allocated to all NRAMs 431 at once. Therefore, this rule does not need to be considered in large-image mode.

[0142] Rule 11: Step Redundancy

[0143] Step redundancy refers to the following: when the template fusion unit has too many fusion layers, and the length and width of the convolution and pooling kernels are greater than the step length, the input data required for each output point overlaps, which is the aforementioned input-dependent operation. This overlapping portion is step redundancy. Step redundancy requires each processor core 306 to read more data, but this part of the multiplexed data will occupy on-chip and off-chip access resources. The more layers the template fusion unit includes, the more serious the step redundancy. The rule is that the sum of the difference between the side length of the kernel of the convolution layer or pooling layer and the step length is not greater than the redundancy threshold.

[0144] The redundancy threshold is defined as follows. Assume that the kernel length and width of the convolution and pooling layers are k x and k y , the step lengths in length and width are s respectively x and s y , then the long-direction step redundancy is k for all convolution and pooling layers in the template fusion unit x -s x Similarly, the step redundancy in the width direction is k for all convolution and pooling layers in the template fusion unit. y -s y The redundancy threshold of this embodiment can be 3, 4, 5, or 6, preferably 4. As long as the step redundancy in either the length direction or the width direction is greater than the redundancy threshold, this rule is not satisfied. The processing device 203 adjusts the template fusion unit, usually by reducing the number of fused layers, until this rule is satisfied.

[0145] The fusion strategy sets an exception to the step-redundancy rule. If the layer to be fused contains multiple branches and the template fusion unit can fuse all of them, the performance of the template fusion unit will be superior. In this case, the processing device 203 will ignore the step-redundancy rule, that is, the step-redundancy will not limit the template fusion unit from fusing multiple branches. In other words, in the fusion strategy of this embodiment, fusing multiple branches takes precedence over the step-redundancy limit. In other words, step-redundancy is only considered in the case of a single branch.

[0146] The above rules are only examples. The present invention does not limit the order in which the rules are executed, nor does it limit these rules to be considered simultaneously. Technical personnel in this field can add or delete rules according to actual conditions in different application scenarios to implement a fusion strategy that conforms to the current application scenario.

[0147] Back to Figure 10In step 1003, neural network calculations are performed based on the established template fusion unit. Based on a three-level computing hierarchy of system-on-chip (SoC), cluster, and processor core, and a three-layer memory design of DRAM, SRAM, and NRAM / WRAM, computing device 201 treats the template fusion unit as a custom layer in the neural network. The on-chip unit map required for computing the template fusion unit is loaded from DRAM 204 to SRAM 308 in one go, allowing data to be cached and calculated at the appropriate level, forming a fully streamlined process. After the calculation is complete, the calculation results are transferred from SRAM 308 to DRAM 204, significantly reducing the input / output overhead of neural network calculations.

[0148] When input data from fields such as computer vision, speech, natural language processing, and data mining is subjected to various deep learning and machine learning algorithms, the present invention, based on a template fusion unit, can reduce the input / output overhead of neural network calculations. Another embodiment of the present invention is a method for performing neural network calculations using a template fusion unit.

[0149] In more detail, this embodiment also adopts Figure 6 The two-layer, three-stage pipeline shown. In this embodiment, the on-chip unit map is the input map of the template fusion unit, and the weights refer to the weights required to calculate the input map. As previously mentioned, in large-map mode, the on-chip unit map is only a portion of a feature map; in small-map mode, the on-chip unit map includes at least one feature map. This embodiment also supports multiple weight transfer strategies as shown in Figure 7.

[0150] A strategy such as Figure 7AAs shown, the weight storage unit 702 and WRAM 432 are both complete storage spaces. In the first-level loading phase 601, GDMA 311 loads the template fusion unit input (on-chip unit map) from DRAM 204 into the ping storage unit 703. It also loads the i-th sub-weight in the weight (corresponding to the i-th sub-map of the on-chip unit map) from DRAM 204 into the weight storage unit 702. In the first-level calculation stage 602, the broadcast bus 309 broadcasts the i-th subgraph and the i-th subweight in the on-chip unit map from the SRAM 308 to the NRAM 431 and the WRAM 432 of at least one of the multiple processor cores 306 in the second-level loading stage 604. The operation unit 42 calculates the i-th intermediate result in the second-level calculation stage 605 based on the i-th subgraph and the i-th subweight. At the same time, the broadcast bus 309 broadcasts the i+1-th subgraph in the on-chip unit map from the SRAM 308 to the NRAM 431 in the second-level loading stage 613. Since the i-th sub-weight has been loaded into the WRAM 432, the space in the weight storage unit 702 can be released. Therefore, at the same time, the GDMA 311 loads the i+1-th sub-weight (corresponding to the i+1-th subgraph) in the weight from the DRAM 204 into the weight storage unit 702. Since the weight storage unit 702 is a whole block of storage space, the i-th sub-weight in the weight storage unit 702 is overwritten by the i+1-th sub-weight.

[0151] After generating the i-th intermediate result, the broadcast bus 309 broadcasts the i+1-th sub-weight from the weight storage unit 702 to the WRAM 432. Since the WRAM 432 is also a whole block of storage space, the i-th sub-weight is also covered by the i+1-th sub-weight.

[0152] In the second-layer store-recall phase 606, the MVDMA 434 stores the i-th intermediate result back to the ping storage unit 703. At the same time, the operation unit 42 calculates the i+1-th intermediate result based on the i+1-th sub-graph and the i+1-th sub-weight in the second-layer calculation phase 614, and the i+2-th sub-graph enters the second-layer load phase 615. The broadcast bus 309 broadcasts the i+2-th sub-graph from the SRAM 308 to the NRAM 431. At the same time, the GDMA 311 loads the i+2-th sub-weight (corresponding to the i+2-th sub-graph) in the weight from the DRAM 204 to the weight storage unit 702. The i+1-th sub-weight in the weight storage unit 702 is overwritten by the i+2-th sub-weight.

[0153] After the i+1th intermediate result is generated, the broadcast bus 309 broadcasts the i+2th sub-weight from the weight storage unit 702 to the WRAM 432 . The i+1th sub-weight in the WRAM 432 is also overwritten by the i+2th sub-weight.

[0154] In this strategy, the weights corresponding to the on-chip unit graph are not all matched to the on-chip unit graph at once. Figure 1 Instead of loading the sub-weights of the subgraphs into SRAM 308 at the same time, only the sub-weights of the corresponding subgraphs are loaded into SRAM 308 at a time. Only one sub-weight is stored in weight storage unit 702 at a time. Therefore, computing device 201 first identifies the largest of the multiple sub-weights and then determines the size of weight storage space 702 based on the largest sub-weight. This configuration still frequently accesses DRAM 204 during the calculation of the template fusion unit, which reduces the advantages of fusion. However, if the space in SRAM 308 is limited, this strategy can be regarded as a compromise.

[0155] Another strategy is Figure 7B As shown, WRAM 432 is divided into multiple memory spaces. In the first layer loading stage 601, GDMA 311 loads the input map (on-chip unit map) of the template fusion unit from DRAM 204 into ping storage unit 703, and also loads the i-th sub-weight from DRAM 204 into weight storage unit 702. In the first-layer calculation stage 602, the broadcast bus 309 broadcasts the i-th subgraph to the NRAM 431 in the second-layer loading stage 604, and also broadcasts the i-th sub-weight to the first storage space 705. The operation unit 42 calculates the i-th intermediate result in the second-layer calculation stage 605 based on the i-th subgraph and the i-th sub-weight. At the same time, the broadcast bus 309 broadcasts the i+1-th subgraph from the SRAM 308 to the NRAM 431 in the second-layer loading stage 613. At the same time, the GDMA 311 loads the i+1-th sub-weight from the DRAM 204 into the weight storage unit 702. The i-th sub-weight in the weight storage unit 702 is overwritten by the i+1-th sub-weight.

[0156] After the i+1th sub-weight is loaded into the weight storage unit 702, regardless of whether the i-th intermediate result is calculated, the broadcast bus 309 broadcasts the i+1th sub-weight from the weight storage unit 702 to the second storage space 706. Since the i-th sub-weight is stored in the first storage space 705, it will not be overwritten by the i+1th sub-weight, so as not to affect the correctness of the i-th intermediate result.

[0157] In the second-layer store-back stage 606, the MVDMA 434 stores the i-th intermediate result back to the ping storage unit 703. At the same time, the operation unit 42 calculates the i+1-th intermediate result in the second-layer calculation stage 614 based on the i+1-th sub-graph and the i+1-th sub-weight. The i+2-th sub-graph enters the second-layer loading stage 615, and the broadcast bus 309 broadcasts the i+2-th sub-graph from the SRAM 308 to the NRAM 431. At the same time, the GDMA 311 loads the i+2-th sub-weight from the DRAM 204 into the weight storage unit 702. The i+1-th sub-weight in the weight storage unit 702 is overwritten by the i+2-th sub-weight.

[0158] After the i+2th sub-weight is loaded into the weight storage unit 702, regardless of whether the i+1th intermediate result has been calculated, the broadcast bus 309 broadcasts the i+2th sub-weight from the weight storage unit 702 to the third storage space 707. The i+1th sub-weight is not overwritten, so as not to affect the correctness of the i+1th intermediate result.

[0159] In this strategy, WRAM 432 is divided into multiple storage spaces, so the sub-weight does not need to wait for the previous intermediate result to be completed before being broadcast. That is, the sub-weight can be broadcast to WRAM432 while the previous intermediate result is calculated, shortening the pipeline running time.

[0160] Another strategy is Figure 7C As shown, Figure 7A The difference is that the weight storage unit 702 is divided into multiple storage spaces. In the first-level loading stage 601, GDMA 311 loads the template fusion unit input map (on-chip unit map) from DRAM 204 into the ping storage unit 703. It also loads the i-th sub-weight from DRAM 204 into the first storage space 708, the i+1-th sub-weight from DRAM 204 into the second storage space 709, and the i+2-th sub-weight from DRAM 204 into the third storage space 710. In the first-layer calculation stage 602, the broadcast bus 309 broadcasts the i-th subgraph and the i-th subweight from the SRAM 308 to the NRAM 431 and the WRAM 432 in the second-layer loading stage 604. The operation unit 42 calculates the i-th intermediate result in the second-layer calculation stage 605 based on the i-th subgraph and the i-th subweight. At the same time, the broadcast bus 309 broadcasts the i+1-th subgraph from the SRAM 308 to the NRAM 431 in the second-layer loading stage 613.

[0161] After the i-th intermediate result is generated, the broadcast bus 309 broadcasts the i+1-th sub-weight from the second storage space 709 to the WRAM 432. Since the WRAM 432 is a single block of storage space, the i-th sub-weight is overwritten by the i+1-th sub-weight. In the second-level store-back phase 606, the MVDMA 434 stores the i-th intermediate result back to the ping storage unit 703. At the same time, the operation unit 42 calculates the i+1-th intermediate result based on the i+1-th sub-graph and the i+1-th sub-weight in the second-level calculation phase 614. The i+2-th sub-graph enters the second-level load phase 615, and the broadcast bus 309 broadcasts the i+2-th sub-graph from the SRAM 308 to the NRAM 431. After generating the i+1th intermediate result, the broadcast bus 309 broadcasts the i+2th sub-weight from the third storage space 710 to the WRAM 432 . The i+1th sub-weight in the WRAM 432 is overwritten by the i+2th sub-weight.

[0162] In this strategy, multiple sub-weights can be combined with on-chip units at once. Figure 1 The data are loaded into the SRAM 308 at the same time, so the advantages of fusion can be fully realized, which is of great help to the computing efficiency.

[0163] Another strategy is Figure 7D As shown, the weight storage unit 702 and the WRAM 432 are both divided into multiple storage spaces. In the first layer loading stage 601, the GDMA 311 loads the input map (on-chip unit map) required by the template fusion unit from the DRAM 204 into the ping storage unit 703. It also loads the i-th sub-weight from the DRAM 204 into the first storage space 708, the i+1-th sub-weight from the DRAM 204 into the second storage space 709, and the i+2-th sub-weight from the DRAM 204 into the third storage space 710. In the first-layer calculation stage 602, the broadcast bus 309 broadcasts the i-th subgraph and the i-th subweight from the SRAM 308 to the NRAM 431 and the WRAM 432 in the second-layer loading stage 604. The operation unit 42 calculates the i-th intermediate result in the second-layer calculation stage 605 based on the i-th subgraph and the i-th subweight. At the same time, the broadcast bus 309 broadcasts the i+1-th subgraph in the on-chip unit map from the SRAM 308 to the NRAM 431 in the second-layer loading stage 613.

[0164] Regardless of whether the i-th intermediate result has been calculated, the broadcast bus 309 broadcasts the i+1-th sub-weight from the weight storage unit 702 to the second storage space 706. Since the i-th sub-weight is stored in the first storage space 705, it will not be overwritten by the i+1-th sub-weight, thereby not affecting the correctness of the i-th intermediate result. In the second-level store-back phase 606, the MVDMA 434 stores the i-th intermediate result back to the ping storage unit 703. At the same time, the operation unit 42 calculates the i+1-th intermediate result based on the i+1-th sub-graph and the i+1-th sub-weight in the second-level calculation phase 614. The i+2-th sub-graph enters the second-level load phase 615, and the broadcast bus 309 broadcasts the i+2-th sub-graph from the SRAM 308 to the NRAM 431. Regardless of whether the i+1th intermediate result is calculated, the broadcast bus 309 broadcasts the i+2th sub-weight from the weight storage unit 702 to the third storage space 707. The i+1th sub-weight is not overwritten, which will not affect the correctness of the i+1th intermediate result.

[0165] In this strategy, not only can multiple sub-weights be combined with on-chip units at once, Figure 1 By being loaded into SRAM 308 at the same time, the sub-weights do not need to wait for the previous calculation result to be completed before being broadcast. In other words, the sub-weights can be broadcast to WRAM 432 while the previous intermediate result is being calculated, thus shortening the input / output and pipeline operation time.

[0166] Figure 11 A flowchart showing another embodiment of executing a computing program using the computing device 201 is shown.

[0167] In step 1101, the on-chip cell graph and at least the first sub-weight (corresponding to the first sub-graph of the on-chip cell graph) in the weights are loaded from DRAM 204 to SRAM 308. In step 1102, it is determined whether all sub-graphs of the on-chip cell graph have been calculated. If not, step 1103 is executed to select an uncalculated sub-graph and move the required data to NRAM 411 and WRAM 432. In step 1104, the calculation task is executed. In step 1105, the intermediate results are moved from NRAM 411 to SRAM 308, and the process returns to step 1102.

[0168] If it is determined in step 1102 that all subgraphs have been calculated, step 1106 is executed to reduce all intermediate results to generate a calculation result. In step 1107, the calculation result is transferred from SRAM 308 to DRAM 204.

[0169] Figure 12A flowchart of a two-layer, three-stage pipeline according to another embodiment is shown. In step 1201, the jth on-chip unit map and the i-th sub-weight are loaded. In step 1202, the jth on-chip unit map is synchronously calculated, the jth calculation result is generated, and the j+1th on-chip unit map is loaded. In step 1203, the jth calculation result is synchronously stored, the j+1th on-chip unit map is calculated, the j+1th calculation result is generated, and the j+2th on-chip unit map is loaded.

[0170] When the computing device 201 has Figure 7A When the storage space is configured, step 1202 further includes the following steps.

[0171] In step 1204 , the i-th subgraph and the i-th subweight in the j-th on-chip unit graph are broadcast from the SRAM 308 to the NRAM 431 and the WRAM 432 of at least one of the processor cores 306 .

[0172] In step 1205, the i-th intermediate result is calculated based on the i-th subgraph and the i-th subweight. At the same time, the i+1-th subgraph in the on-chip unit graph is broadcast from the ping storage unit 703 to the NRAM 431. Since the i-th subweight has been loaded into the WRAM 432, the space in the weight storage unit 702 can be released. Therefore, at the same time, the i+1-th subweight (corresponding to the i+1-th subgraph) in the weight is also loaded from the DRAM 204 into the weight storage unit 702. The i-th subweight in the weight storage unit 702 is overwritten by the i+1-th subweight.

[0173] In step 1206, after obtaining the i-th intermediate result, the i+1-th sub-weight is broadcast from the weight storage unit 702 to the WRAM 432. Since the WRAM 432 is also a whole block of storage space, the i-th sub-weight is also overwritten by the i+1-th sub-weight.

[0174] In step 1207, the i-th intermediate result is stored back in the ping storage unit 703. At the same time, the i+1-th intermediate result is calculated based on the i+1-th subgraph and the i+1-th sub-weight, the i+2-th subgraph is broadcast from the SRAM 308 to the NRAM 431, and the i+2-th sub-weight in the weight (corresponding to the i+2-th subgraph) is loaded from the DRAM 204 to the weight storage unit 702. The i+1-th sub-weight in the weight storage unit 702 is overwritten by the i+2-th sub-weight.

[0175] In step 1208, after obtaining the i+1th intermediate result, the i+2th sub-weight is broadcast from the weight storage unit 702 to the WRAM 432, and the i+1th sub-weight in the WRAM 432 is also overwritten by the i+2th sub-weight.

[0176] When the computing device 201 has Figure 7B When the storage space is configured, step 1202 further includes the following steps: Figure 13 Steps shown.

[0177] In step 1301 , the i-th subgraph is broadcast to the NRAM 431 , and the i-th subweight is also broadcast to the first storage space 705 .

[0178] In step 1302, the i-th intermediate result is calculated based on the i-th subgraph and the i-th sub-weight. At the same time, the i+1-th subgraph in the on-chip unit graph is broadcast from SRAM 308 to NRAM 431, and the i+1-th sub-weight is loaded from DRAM 204 to the weight storage unit 702. The i-th sub-weight in the weight storage unit 702 is overwritten by the i+1-th sub-weight.

[0179] In step 1303, regardless of whether the i-th intermediate result has been calculated, the i+1-th sub-weight is broadcast from the weight storage unit 702 to the second storage space 706. Since the i-th sub-weight is stored in the first storage space 705, it will not be overwritten by the i+1-th sub-weight, so as not to affect the correctness of the first calculation result.

[0180] In step 1304, the i-th intermediate result is stored back in the ping storage unit 703. At the same time, the i+1-th intermediate result is calculated based on the i+1-th subgraph and the i+1-th sub-weight, and the i+2-th subgraph is broadcast from the SRAM 308 to the NRAM 431, and the i+2-th sub-weight is loaded from the DRAM 204 to the weight storage unit 702. The i+1-th sub-weight in the weight storage unit 702 is overwritten by the i+2-th sub-weight.

[0181] In step 1305, regardless of whether the i+1th intermediate result is calculated, the i+2th sub-weight is broadcast from the weight storage unit 702 to the third storage space 707. The i+1th sub-weight is not overwritten, so as not to affect the correctness of the second calculation result.

[0182] Figure 14 A flowchart of a two-layer three-stage pipeline according to another embodiment is shown, wherein the computing device 201 has the following Figure 7CThe storage space configuration is as follows. In step 1401, the jth on-chip unit map and the jth weight are loaded. In step 1402, the jth on-chip unit map is synchronously calculated and the jth calculation result is generated, and the j+1th on-chip unit map is loaded. In step 1403, the jth calculation result is synchronously stored back, the j+1th on-chip unit map is calculated and the j+1th calculation result is generated, and the j+2th on-chip unit map is loaded. Step 1402 further includes the following steps.

[0183] In step 1404 , the i-th subgraph and the i-th subweight in the j-th on-chip unit graph are broadcast from the SRAM 308 to the NRAM 431 and the WRAM 432 .

[0184] In step 1405 , the i-th intermediate result is calculated based on the i-th subgraph and the i-th subweight. At the same time, the i+1-th subgraph is broadcast from the SRAM 308 to the NRAM 431 .

[0185] In step 1406, after obtaining the i-th intermediate result, the i+1-th sub-weight is broadcast from the second storage space 709 to the WRAM 432. Since the WRAM 432 is a whole block of storage space, the i-th sub-weight is overwritten by the i+1-th sub-weight.

[0186] In step 1407, the i-th intermediate result is stored back in the ping storage unit 703. At the same time, the i+1-th intermediate result is calculated based on the i+1-th subgraph and the i+1-th subweight, and the i+2-th subgraph is broadcast from the SRAM 308 to the NRAM 431.

[0187] In step 1408 , after obtaining the i+1th intermediate result, the i+2th sub-weight is broadcast from the third storage space 710 to the WRAM 432 , and the i+1th sub-weight in the WRAM 432 is overwritten by the i+2th sub-weight.

[0188] When the computing device 201 has Figure 7D When the storage space is configured, step 1402 further includes the following steps: Figure 15 Steps shown.

[0189] In step 1501 , the i-th subgraph and the i-th subweight in the on-chip unit graph are broadcast from the SRAM 308 to the NRAM 431 and the WRAM 432 .

[0190] In step 1502 , the i-th intermediate result is calculated based on the i-th subgraph and the i-th subweight. At the same time, the i+1-th subgraph in the on-chip unit graph is broadcast from the SRAM 308 to the NRAM 431 .

[0191] In step 1503 , regardless of whether the calculation of the i-th intermediate result is completed, the i+1-th sub-weight is broadcast from the weight storage unit 702 to the second storage space 706 .

[0192] In step 1504, the i-th intermediate result is stored back in the ping storage unit 703, and the i+1-th intermediate result is calculated based on the i+1-th subgraph and the i+1-th subweight. Then, the i+2-th subgraph is broadcast from the SRAM 308 to the NRAM 431.

[0193] In step 1505 , regardless of whether the calculation of the (i+1)th intermediate result is completed, the (i+2)th sub-weight is broadcast from the weight storage unit 702 to the third storage space 707 .

[0194] exist Figure 15 In the process, since the weight storage unit 702 and the WRAM 432 are both divided into multiple storage spaces, this embodiment can broadcast the sub-weights stored in the weight storage unit 702 to the WRAM 432 at one time. The sub-weights stored in the WRAM 432 each have their own storage space and will not overwrite each other, so it will not affect the calculation of the intermediate results.

[0195] Another embodiment of the present invention is a computer-readable storage medium having stored thereon a computer program code for computing a neural network. When the computer program code is executed by a processor, the method disclosed in the aforementioned embodiments is executed.

[0196] The present invention is based on a three-level operation hierarchy of system-on-chip (SoC), cluster, and processor core, and a three-layer memory design of DRAM, SRAM, and NRAM / WRAM. It establishes a two-layer, three-stage pipeline and supports multiple ways of loading / broadcasting weights. This allows technicians in this field to choose to reduce input / output overhead or lower hardware space requirements to run neural network models according to actual needs, fully utilize hardware resources, and improve neural network computing efficiency.

[0197] Depending on the application scenario, the electronic device or device of the present invention may include a server, a cloud server, a server cluster, a data processing device, a robot, a computer, a printer, a scanner, a tablet computer, a smart terminal, a PC device, an Internet of Things terminal, a mobile terminal, a mobile phone, a driving recorder, a navigator, a sensor, a camera, a camera, a video camera, a projector, a watch, a headset, a mobile storage, a wearable device, a visual terminal, an automatic driving terminal, a vehicle, a household appliance, and / or a medical device. The vehicle includes an airplane, a ship and / or a vehicle; the household appliance includes a television, an air conditioner, a microwave oven, a refrigerator, an electric rice cooker, a humidifier, a washing machine, an electric light, a gas stove, and a range hood; the medical device includes an MRI, an ultrasound machine and / or an electrocardiograph. The electronic device or device of the present invention may also be applied to the Internet, the Internet of Things, data centers, energy, transportation, public administration, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, medical care and other fields. Furthermore, the electronic device or device of the present invention may also be used in application scenarios related to artificial intelligence, big data and / or cloud computing, such as the cloud, edge, and terminal. In one or more embodiments, electronic devices or apparatuses with high computing power according to the solution of the present invention can be applied to cloud devices (such as cloud servers), while electronic devices or apparatuses with low power consumption can be applied to terminal devices and / or edge devices (such as smartphones or cameras). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or edge device are compatible with each other, so that according to the hardware information of the terminal device and / or edge device, appropriate hardware resources can be matched from the hardware resources of the cloud device to simulate the hardware resources of the terminal device and / or edge device, so as to complete the unified management, scheduling and collaborative work of end-to-end or cloud-edge-to-end.

[0198] It should be noted that, for the purpose of simplicity, the present invention describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will understand that the scheme of the present invention is not limited by the order of the described actions. Therefore, based on the disclosure or teachings of the present invention, those skilled in the art will understand that some of the steps therein can be performed in other orders or simultaneously. Further, those skilled in the art will understand that the embodiments described in the present invention can be regarded as optional embodiments, that is, the actions or modules involved therein are not necessarily necessary for the implementation of one or more schemes of the present invention. In addition, depending on the different schemes, the present invention also has different emphases on the description of some embodiments. In view of this, those skilled in the art will understand that the parts that are not described in detail in a certain embodiment of the present invention may also refer to the relevant descriptions of other embodiments.

[0199] In terms of specific implementation, based on the disclosure and teachings of the present invention, those skilled in the art can understand that several embodiments disclosed in the present invention can also be implemented in other ways not disclosed herein. For example, with respect to the various units in the electronic device or device embodiments described above, this article splits them based on the consideration of logical functions, and there may be other ways of splitting them in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions in the units or components can be selectively disabled. With respect to the connection relationship between different units or components, the connection discussed above in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0200] In the present invention, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network elements. In addition, according to actual needs, some or all of the units may be selected to achieve the purpose of the solutions described in the embodiments of the present invention. In addition, in some scenarios, multiple units in the embodiments of the present invention may be integrated into a single unit or each unit may exist physically separately.

[0201] In some implementation scenarios, the above-mentioned integrated unit can be implemented in the form of a software program module. If implemented in the form of a software program module and sold or used as an independent product, the integrated unit can be stored in a computer-readable memory. Based on this, when the solution of the present invention is embodied in the form of a software product (such as a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to enable a computer device (such as a personal computer, a server or a network device, etc.) to perform some or all of the steps of the method described in the embodiment of the present invention. The aforementioned memory may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0202] In some other implementation scenarios, the above-mentioned integrated units can also be implemented in the form of hardware, that is, as specific hardware circuits, which may include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit may include but is not limited to physical devices, and the physical devices may include but are not limited to devices such as transistors or memristors. In view of this, the various devices described herein (such as computing devices or other processing devices) can be implemented by appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage unit or storage device can be any appropriate storage medium (including magnetic storage media or magneto-optical storage media, etc.), which can be, for example, resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high bandwidth memory (HBM), hybrid memory cube (HMC), ROM and RAM, etc.

[0203] The embodiments of the present invention are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A method for computing a neural network using a computing device, the computing device being connected to an off-chip memory, the off-chip memory storing an on-chip cell map and corresponding weights, the computing device comprising a plurality of clusters, each cluster comprising a shared memory unit and a plurality of processor cores, the method comprising: Loading the on-chip unit map and the i-th sub-weight in the weight from the off-chip memory to the shared storage unit; wherein i is a positive integer; Broadcasting the i-th subgraph and the i-th subweight in the on-chip unit graph from the shared storage unit to at least one of the plurality of processor cores, wherein the i-th subweight corresponds to the i-th subgraph; Calculate an i-th intermediate result according to the i-th subgraph and the i-th subweight; Loading the (i+1)th sub-weight in the weight from the off-chip memory to the shared storage unit; Broadcasting the (i+1)th subgraph in the on-chip unit graph from the shared storage unit to at least one of the plurality of processor cores, wherein the (i+1)th subweight corresponds to the (i+1)th subgraph; and The (i+1)th sub-weight is broadcasted from the shared storage unit to at least one of the plurality of processor cores. 2 . The method according to claim 1 , wherein the step of loading the (i+1)th sub-weight is performed synchronously while calculating the (i)th intermediate result.

3. The method according to claim 2, wherein the weight comprises a plurality of sub-weights, the method further comprising: Identifying the one with the largest storage space among the plurality of sub-weights; as well as The size of the weight storage space in the shared storage unit is determined according to the largest storage space among the multiple sub-weights. 4 . The method according to claim 3 , wherein the (i+1)th sub-weight covers the (i)th sub-weight in the weight storage space. 5 . The method according to claim 1 , wherein the step of loading the on-chip unit map and the i-th sub-weight is performed synchronously with the step of loading the (i+1)-th sub-weight.

6. The method according to claim 5, wherein the weight storage space in the shared storage unit includes a first storage space and a second storage space, and the method further comprises: Storing the i-th sub-weight in the first storage space; as well as The (i+1)th sub-weight is stored in the second storage space.

7. The method according to any one of claims 1 to 6, wherein the step of broadcasting the (i+1)th sub-weight is performed synchronously with the step of calculating.

8. The method according to claim 7, wherein each processor core includes a weight storage unit, the weight storage unit including a first storage space and a second storage space, the method further comprising: Storing the i-th sub-weight in the first storage space; as well as The (i+1)th sub-weight is stored in the second storage space.

9. The method according to any one of claims 1 to 6, wherein the step of broadcasting the (i+1)th sub-weight is performed after the step of calculating.

10. The method according to claim 9, wherein each processor core includes a weight storage unit, the method further comprising: Storing the i-th sub-weight in the weight storage unit; as well as The i+1th sub-weight is stored in the weight storage unit to cover the i-th sub-weight.

11. A computing device connected to an off-chip memory, wherein the off-chip memory carries an on-chip cell map and corresponding weights, the computing device comprising a plurality of clusters, each cluster comprising: shared storage unit; Direct Memory Access Module, used to: Loading the on-chip unit map and the i-th sub-weight in the weight from the off-chip memory into the shared storage unit; as well as Loading the (i+1)th sub-weight in the weight from the off-chip memory to the shared storage unit; Multiple processor cores; as well as Broadcast bus, used to: Broadcasting the i-th subgraph and the i-th subweight in the on-chip unit graph from the shared storage unit to at least one of the plurality of processor cores, wherein the i-th subweight corresponds to the i-th subgraph; Broadcasting the i+1th subgraph in the on-chip unit graph from the shared storage unit to at least one of the plurality of processor cores, wherein the i+1th subweight corresponds to the i+1th subgraph; broadcasting the (i+1)th sub-weight from the shared storage unit to at least one of the plurality of processor cores; At least one of the plurality of processor cores calculates an i-th intermediate result according to the i-th subgraph and the i-th sub-weight. 12 . The computing device according to claim 11 , wherein when at least one of the plurality of processor cores calculates the i-th intermediate result, the direct memory access module synchronously loads the (i+1)-th sub-weight.

13. The computing device according to claim 12, wherein the weight comprises a plurality of sub-weights, and the computing device determines the size of the weight storage space in the shared storage unit based on the sub-weight with the largest storage space. The computing device according to claim 13 , wherein the (i+1)th sub-weight covers the (i)th sub-weight in the weight storage space. 15 . The computing device according to claim 12 , wherein the direct memory access module loads the (i+1)th sub-weight simultaneously when loading the on-chip unit map and the i-th sub-weight.

16. The computing device according to claim 15, wherein the weight storage space in the shared storage unit comprises: A first storage space for storing the i-th sub-weight; as well as The second storage space is used to store the (i+1)th sub-weight.

17. The computing device according to any one of claims 11 to 16, wherein when at least one of the plurality of processor cores calculates the i-th intermediate result, the broadcast bus synchronously broadcasts the (i+1)-th sub-weight.

18. The computing device according to claim 17, wherein each processor core comprises a weight storage unit, the weight storage unit comprising: A first storage space for storing the i-th sub-weight; as well as The second storage space is used to store the (i+1)th sub-weight.

19. The computing device according to any one of claims 11 to 16, wherein after at least one of the plurality of processor cores calculates the i-th intermediate result, the broadcast bus broadcasts the (i+1)-th sub-weight.

20. The computing device according to claim 19, wherein each processor core comprises a weight storage unit for storing the i-th sub-weight and the (i+1)-th sub-weight, wherein the (i+1)-th sub-weight covers the i-th sub-weight.

21. An integrated circuit device comprising a computing device according to any one of claims 11 to 20.

22. A board comprising the integrated circuit device according to claim 21.

23. A computer-readable storage medium having stored thereon a computer program code for calculating a neural network using a computing device, wherein when the computer program code is executed by a processing device, the method according to any one of claims 1 to 10 is executed.

Images