Apparatus, board, method and readable storage medium for computing neural network

Abstract

The present disclosure relates to apparatuses, boards, methods, and readable storage media for computing neural networks, where the computing apparatuses of the present disclosure are included in an integrated circuit apparatus that includes a general purpose interconnect interface and other processing apparatuses. The computing apparatuses interact with the other processing apparatuses to collectively complete user-specified computing operations. The integrated circuit apparatus can also include storage apparatuses connected to the computing apparatuses and the other processing apparatuses, respectively, for data storage of the computing apparatuses and the other processing apparatuses.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of neural networks. More particularly, the present disclosure relates to an apparatus, a board card, a method and a readable storage medium for computing neural networks. BACKGROUND

[0002] A neural network is a system of multiple neurons connected in a certain rule, which is generally composed of four kinds of layer structures: an input layer, a convolution layer, a pooling layer, and a fully connected layer.

[0003] The input layer is to intercept part of the information from the input data and convert it into a feature matrix, which contains the features corresponding to the part of the information. The convolution layer is configured to receive the feature matrix from the input layer and extract features from the input data through convolution operation. In practical application, multiple convolution layers can be built. The pooling layer is configured to replace a region of data with a value, which is usually the maximum value or the average value of all values in the region. Through pooling, the model size can be reduced and the calculation speed can be improved without losing too much information. The fully connected layer acts as a classifier in the entire convolutional neural network, which is equivalent to a feature space transformation. It extracts and integrates all useful information from the previous layers, compares the information based on different classifications, and judges whether the input data is similar to the target of comparison.

[0004] With the development of technology, the number of layers of neural networks is increasing. For example, in the classic VGG architecture, VGG-A has 11 weight layers, VGG-B has 13 weight layers, VGG-C has 16 weight layers, VGG-D has 16 weight layers, and VGG-E has 19 weight layers. Among them, the convolution layer and the fully connected layer are referred to as weight layers. Some neural networks have hundreds of layers. Moreover, with the increase in the number of layers, the number of parameters of the neural network also increases exponentially, for example, AlexNet has 60 million parameters participating in the calculation.

[0005] Multiple layers and multiple parameters require a large number of on-chip and off-chip input / output accesses, which will consume many resources and delay the operation time. Therefore, a mechanism to reduce input / output access is urgently needed in the field of artificial intelligence. SUMMARY

[0006] To at least partially solve the technical problems mentioned in the background, the present disclosure provides an apparatus, a board card, a method and a readable storage medium for computing neural networks.

[0007] In one aspect, the disclosure discloses a computing device connected to off-chip memory, comprising a plurality of clusters, each cluster comprising: a shared memory unit; a first direct memory access module for loading a first on-chip graph from the off-chip memory into the shared memory unit; and a plurality of processor cores for respectively computing the first on-chip graph and generating a first computation result; wherein the first direct memory access module stores the first computation result from the shared memory unit back to the off-chip memory.

[0008] In another aspect, the disclosure discloses an integrated circuit device comprising the aforementioned computing device, and a board card comprising the aforementioned integrated circuit device.

[0009] In another aspect, the disclosure discloses a method for computing a neural network using a computing device comprising a plurality of clusters, each cluster comprising a shared memory unit, the method comprising: a first layer loading phase of loading a first on-chip graph from off-chip memory into the shared memory unit; a first layer computing phase of respectively computing the first on-chip graph and generating a first computation result; and a first layer storing phase of storing the first computation result from the shared memory unit back to the off-chip memory.

[0010] In another aspect, the disclosure discloses a computer readable storage medium having stored thereon computer program code for computing a neural network using a computing device, which when executed by a processing device, performs the aforementioned method.

[0011] The disclosure is based on a three-level operation hierarchy of system-on-chip-cluster-processor core, and a three-layer memory design of DRAM-SRAM-NRAM / WRAM, establishing a two-layer three-stage pipeline, fully utilizing hardware resources, and improving the efficiency of neural network computation. BRIEF DESCRIPTION OF DRAWINGS

[0012] The above and other objects, features and advantages of the example embodiments of the disclosure will be readily understood through reading the detailed description of the embodiments of the disclosure below in conjunction with the accompanying drawings. In the drawings, several embodiments of the disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals are used to indicate like or corresponding parts throughout the several views:

[0013] Figure 1 is a structural diagram of a board card showing an embodiment of the disclosure;

[0014] Figure 2 is a structural diagram of an integrated circuit device showing an embodiment of the disclosure;

[0015] Figure 3 is a schematic diagram of the internal structure of a computing device showing an embodiment of the disclosure;

[0016] Figure 4 is a schematic diagram illustrating the internal structure of a processor core according to an embodiment of the present disclosure;

[0017] Figure 5 is a schematic diagram illustrating when one processor core wants to write data to another processor core of a cluster;

[0018] Figure 6 is a schematic diagram illustrating an AlexNet model;

[0019] Figure 7 is a schematic diagram illustrating an exemplary neural network model;

[0020] Figure 8 is a schematic diagram illustrating two convolution layers fused together according to an embodiment of the present disclosure;

[0021] Figure 9 is a schematic diagram illustrating the formats of NCHW and NHWC;

[0022] Figure 10 is a flowchart illustrating a neural network calculation performed by a template fusion unit according to an embodiment of the present disclosure;

[0023] Figure 11 is a flowchart illustrating a dynamic fusion of a neural network according to a fusion strategy according to an embodiment of the present disclosure;

[0024] Figure 12 is a flowchart illustrating a neural network calculation performed by a template fusion unit according to an embodiment of the present disclosure;

[0025] Figure 13 is a schematic diagram illustrating a division of a sub-template fusion unit according to an embodiment of the present disclosure;

[0026] Figure 14 is a schematic diagram illustrating a two-layer three-stage pipeline according to an embodiment of the present disclosure;

[0027] Figure 15 is a flowchart illustrating a calculation procedure performed based on a template fusion unit according to an embodiment of the present disclosure; and

[0028] Figure 16 is a flowchart illustrating a two-layer three-stage pipeline according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of, rather than all of, the embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present disclosure.

[0030] It should be understood that the terms "first", "second", "third", and "fourth" and the like in the description and in the claims of the present disclosure are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. The terms "including", "containing", and "comprising" and the like are used herein to indicate the inclusion as set out above, but do not preclude the addition or presence of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0031] It should also be understood that the terms used in the present disclosure are merely for the purpose of describing particular embodiments and are not intended to limit the present disclosure. As used in the specification and claims of this disclosure, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or", as used in the specification and in claims, indicate one and / or any combinations of the associated listed items.

[0032] As used in the specification and claims, the term "if can be construed to mean "when" or "upon" or "in response to determining" or "in response to detecting" depending on the context.

[0033] Specific embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings.

[0034] A neural network is composed of input layer, convolution layer, activation function, pooling layer, fully connected layer, less than several layers, more than hundreds of layers, each layer executes an operator, for example, convolution layer executes convolution operator, how many layers need to execute how many operators. In the present disclosure, when referring to a specific layer, it means the corresponding operator of the layer.

[0035] In neural network computation, the input information and the output results of each layer of the model are different in each inference computation. They are considered variable data, which are generally represented by feature maps (matrices). In this disclosure, the input information of the entire neural network model and the input maps of each layer of the model are collectively referred to as feature maps. Once the feature maps are loaded onto the on-chip memory component, they are referred to as on-chip cell maps in this disclosure. The parameters of the trained network model are usually not frequently changed after training is stable, or they can be compiled and generated after the network topology and hardware parameters are determined and do not change during the computation process. Therefore, they can be regarded as constant data. Constant data includes, but is not limited to, weights, biases, device hardware instructions, the mean and variance of batch normalization, etc. In this disclosure, weights are used to represent all constant data. When "data" is mentioned in this disclosure, it generally refers to the graph structure that allows the operation of corresponding operators in the neural network model to be fused together according to the fusion strategy. The variable data and constant data involved in this graph structure are the feature maps plus the corresponding weights.

[0036] Figure 1 A schematic diagram of the structure of a board 10 according to an embodiment of this disclosure is shown. Figure 1 As shown, board 10 includes chip 101, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 10 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and substantial computing power.

[0037] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.

[0038] The board card 10 further comprises a storage device 104 for storing data, which comprises one or more storage units 105. The storage device 104 is connected and transmits data with the control device 106 and the chip 101 through a bus. The control device 106 in the board card 10 is configured to regulate the state of the chip 101. To this end, in one application scenario, the control device 106 can comprise a micro controller unit (MCU).

[0039] Figure 2 is a structural diagram of the combined processing apparatus in the chip 101 of this embodiment. As shown in Figure 2 The combined processing apparatus 20 comprises a computing apparatus 201, an interface apparatus 202, a processing apparatus 203 and a DRAM 204.

[0040] The computing apparatus 201 is configured to perform user-specified operations, mainly implemented as a single-core or multi-core intelligent processor to perform deep learning or machine learning calculations, which can interact with the processing apparatus 203 through the interface apparatus 202 to jointly complete the user-specified operations.

[0041] The interface apparatus 202 is used to transmit data and control instructions between the computing apparatus 201 and the processing apparatus 203. For example, the computing apparatus 201 can obtain input data from the processing apparatus 203 via the interface apparatus 202 and write it into the storage device on the computing apparatus 201. Further, the computing apparatus 201 can obtain control instructions from the processing apparatus 203 via the interface apparatus 202 and write them into the control cache on the computing apparatus 201. Alternatively or optionally, the interface apparatus 202 can also read data in the storage device of the computing apparatus 201 and transmit it to the processing apparatus 203.

[0042] The processing device 203, as a general processing device, performs basic control including but not limited to data carrying, starting and / or stopping of the computing device 201, etc. Depending on the implementation, the processing device 203 can be one or more types of processors of central processing unit (CPU), graphics processing unit (GPU), or other general purpose and / or special purpose processors, including but not limited to digital signal processor (DSP), application specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc., and the number thereof can be determined according to actual needs. As mentioned before, only in terms of the computing device 201 of the present disclosure, it can be considered as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are considered together, both are considered to form a heterogeneous multi-core structure.

[0043] The DRAM 204 is used to store data to be processed, is a DDR memory, and usually has a size of 16G or more, and is used to save data of the computing device 201 and / or the processing device 203.

[0044] Figure 3 The internal structure of the computing device 201 is shown. The computing device 201 is used to process input data of computer vision, voice, natural language, data mining, etc. The computing device 201 in the figure adopts a multi-core hierarchical structure design. The computing device 201 is a system on a chip, which includes a plurality of clusters, and each cluster includes a plurality of processor cores. In other words, the computing device 201 is composed of a system on a chip-cluster-processor core hierarchy.

[0045] From the perspective of the system on a chip hierarchy, as shown in Figure 3 The computing device 201 includes an external storage controller 301, a peripheral communication module 302, an on-chip interconnection module 303, a synchronization module 304, and a plurality of clusters 305.

[0046] The external storage controller 301 can be multiple, and two are exemplarily shown in the figure. The external storage controller 301 is used to access an external storage device in response to an access request issued by a processor core, for example Figure 2DRAM 204, thereby reading data from off-chip or writing data to. The peripheral communication module 302 is used to receive control signals from the processing device 203 through the interface device 202, to start the computing device 201 to perform tasks. The on-chip interconnection module 303 connects the external storage controller 301, the peripheral communication module 302 and the plurality of clusters 305, to transmit data and control signals between the modules. The synchronization module 304 is a global barrier controller (GBC) used to coordinate the work progress of each cluster, to ensure the synchronization of information. The plurality of clusters 305 is the computing core of the computing device 201, which is exemplarily shown as four in the figure, and the computing device 201 of the present disclosure can also include 8, 16, 64 or even more clusters 305 as the hardware develops. The cluster 305 is used to efficiently execute deep learning algorithms.

[0047] As shown in the cluster level, Figure 3 each cluster 305 includes a plurality of processor cores (IPU core) 306 and a memory core (MEM core) 307.

[0048] The processor core 306 is exemplarily shown as four in the figure, and the number of the processor core 306 is not limited by the present disclosure. The internal architecture is shown as Figure 4 Each processor core 306 includes three modules: a control module 41, an operation module 42 and a storage module 43.

[0049] The control module 41 is used to coordinate and control the operation of the operation module 42 and the storage module 43 to complete the task of deep learning, which includes an instruction fetch unit (IFU) 411 and an instruction decode unit (IDU) 412. The instruction fetch unit 411 is used to obtain instructions from the processing device 203, and the instruction decode unit 412 decodes the obtained instructions and sends the decoding results as control information to the operation module 42 and the storage module 43.

[0050] The operation module 42 includes a vector operation unit 421 and a matrix operation unit 422. The vector operation unit 421 is used to perform vector operations, which can support vector multiplication, addition, nonlinear transformation and other complex operations; the matrix operation unit 422 is responsible for the core calculation of deep learning algorithms, i.e. matrix multiplication and convolution.

[0051] The storage module 43 is configured to store or transfer relevant data, including a neuron RAM (NRAM) 431, a weight RAM (WRAM) 432, an input / output direct memory access (IODMA) 433, and a move direct memory access (MVDMA) 434. The NRAM 431 is configured to store feature maps and intermediate results calculated by the processor cores 306. The WRAM 432 is configured to store weights of a deep learning network. The IODMA 433 is configured to control access of the NRAM 431 and the WRAM 432 to the DRAM 204 via the broadcast bus 309. The MVDMA 434 is configured to control access of the NRAM 431 and the WRAM 432 to the SRAM 308.

[0052] Returning to Figure 3 , the storage core 307 is mainly configured to store and communicate, i.e., store shared data or intermediate results among the processor cores 306, perform communication between the execution cluster 305 and the DRAM 204, perform communication among the execution clusters 305, perform communication among the processor cores 306, and the like. In other embodiments, the storage core 307 has the capability of scalar operation, and is configured to perform scalar operation.

[0053] The storage core 307 includes an SRAM 308, a broadcast bus 309, a cluster direct memory access (CDMA) 310, and a global direct memory access (GDMA) 311. The SRAM 308 is configured to serve as a high-performance data transfer station. Data reused among different processor cores 306 within the same execution cluster 305 does not need to be obtained from the DRAM 204 by each of the processor cores 306, but is transferred among the processor cores 306 via the SRAM 308. The storage core 307 only needs to quickly distribute the reused data from the SRAM 308 to the multiple processor cores 306, so as to improve the efficiency of inter-core communication, and greatly reduce on-chip and off-chip input / output access.

[0054] The broadcast bus 309, the CDMA 310, and the GDMA 311 are respectively configured to perform communication among the processor cores 306, communication among the execution clusters 305, and data transmission between the execution clusters 305 and the DRAM 204. The following will be described respectively.

[0055] The broadcast bus 309 is used to accomplish high-speed communication between the processor cores 306 within the cluster 305. The broadcast bus 309 in this embodiment supports inter-core communication modes including unicast, multicast and broadcast. Unicast refers to point-to-point (i.e. single processor core to single processor core) data transmission, multicast is a communication mode in which a piece of data is transmitted from the SRAM 308 to specific processor cores 306, and broadcast is a communication mode in which a piece of data is transmitted from the SRAM 308 to all processor cores 306, which is a special case of multicast.

[0056] The CDMA 310 is used to control access to the SRAM 308 between different clusters 305 within the same computing device 201. Figure 5 A schematic diagram is shown when a processor core wants to write data to a processor core of another cluster to illustrate the working principle of the CDMA 310. In this application scenario, the same computing device includes multiple clusters, for the convenience of illustration, only cluster 0 and cluster 1 are shown in the figure, and cluster 0 and cluster 1 respectively include multiple processor cores. For the convenience of illustration, cluster 0 in the figure only shows processor core 0, and cluster 1 only shows processor core 1. Processor core 0 wants to write data to processor core 1.

[0057] First, processor core 0 sends a unicast write request to write data into the local SRAM 0, CDMA 0 as the master and CDMA 1 as the slave, the master pushes the write request to the slave, i.e. the master sends the write address AW and the write data W, and the data is transmitted to the SRAM 1 of cluster 1, then the slave sends the write response B as a response, and finally the processor core 1 of cluster 1 sends a unicast read request to read the data from the SRAM 1.

[0058] Back to Figure 3, the GDMA 311 cooperates with the external memory controller 301 to control the access of the SRAM 308 of the cluster 305 to the DRAM 204 or to read data from the DRAM 204 to the SRAM 308. As mentioned above, the communication between the DRAM 204 and the NRAM 431 or the WRAM 432 can be achieved through two channels. The first channel is to directly contact the DRAM 204 and the NRAM 431 or the WRAM 432 through the IODMA 433; the second channel is to first transfer data between the DRAM 204 and the SRAM 308 through the GDMA 311, and then transfer data between the SRAM 308 and the NRAM 431 or the WRAM 432 through the MVDMA 434. Although the second channel seems to need more elements to participate and the data flow is longer, in fact, in some embodiments, the bandwidth of the second channel is much larger than that of the first channel, so that the communication between the DRAM 204 and the NRAM 431 or the WRAM 432 through the second channel can be more efficient. The embodiments of the present disclosure can select the data transmission channel according to the hardware conditions.

[0059] In other embodiments, the functions of the GDMA 311 and the functions of the IODMA 433 can be integrated in the same component. For the convenience of description, the GDMA 311 and the IODMA 433 are regarded as different components in the present disclosure, and for those skilled in the art, as long as the functions achieved and the technical effects achieved are similar to the present disclosure, they belong to the protection scope of the present disclosure. Further, the functions of the GDMA 311, the functions of the IODMA 433, the functions of the CDMA 310, the functions of the MVDMA 434 can also be realized by the same component, similarly, as long as the functions achieved and the technical effects achieved are similar to the present disclosure, they all belong to the protection scope of the present disclosure.

[0060] The structure of the neural network related to the present disclosure is divided into two categories: long-chain structure and block structure. The long-chain structure refers to the neural network model composed of single-chain concatenated layers, each layer has only one input and one output, and the whole belongs to single branch, for example, the VGG16 model or the AlexNet model shown in Figure 6 The block structure refers to the subnetwork in the neural network having only one input and one output, but there are multiple branches in the subnetwork, that is, part of the layers of the subnetwork have multiple inputs or outputs, for example, the resblock structure of resnet50, the block structure of inception_v3, etc. Figure 7A schematic diagram showing an exemplary neural network model comprising subnetwork 701 and subnetwork 702. Subnetwork 701 has only one input and one output, which comprises first layer to sixth layer, the first layer has 2 outputs and the sixth layer has 2 inputs, thus subnetwork 701 comprises 2 branches, one branch is first layer→second layer→third layer→sixth layer, and the other branch is first layer→fourth layer→fifth layer→sixth layer, subnetwork 701 constitutes a block structure. Similarly, subnetwork 702 also constitutes a block structure.

[0061] In performing the computation of each layer of deep learning, a large amount of off-chip on-chip access is required, especially reading the input data from DRAM 204 into the computing device 201, and storing the computing result of the computing device 201 to DRAM 204. Such frequent access will consume a lot of hardware resources. In order to solve this problem, the present disclosure greatly reduces the off-chip on-chip data transmission by fusing adjacent layers of neural networks.

[0062] Figure 8 A schematic diagram showing the fusion of two convolutional layers is shown. The input of the first layer convolutional layer 810 is a 7x7 feature map 801, and this layer obtains the feature map 802 of the first layer convolutional layer 810 by convolving the feature map 801 with a 3x3 kernel (not shown). Among them, the value of the 5x5 feature subgraph 804 will affect the 3x3 feature subgraph 805. Assuming the stride is 1, after calculating the 5x5 feature subgraph 804, the first layer convolutional layer 810 will then calculate the 5x5 feature subgraph 806, and the value of the 5x5 feature subgraph 806 will affect the 3x3 feature subgraph 807.

[0063] In the calculation of the second layer convolutional layer 811, the feature map 802 becomes the input of the second layer convolutional layer 811, and is also convolved with a 3x3 kernel to obtain the feature map 803 of the second layer convolutional layer 811. Among them, the value of the 3x3 feature subgraph 805 will affect the 1x1 feature subgraph 808 in the feature map 803. After calculating the 3x3 feature subgraph 805, the second layer convolutional layer 811 will then calculate the 3x3 feature subgraph 807, and the value of the 3x3 feature subgraph 807 will affect the 1x1 feature subgraph 809 in the feature map 803.

[0064] If not fused, the computing device 201 reads the 5x5 feature map 804 from the DRAM 204 when performing the first layer convolution 810, stores the 3x3 feature map 805 back to the DRAM 204 after computation, then reads the 5x5 feature map 806 from the DRAM 204, stores the 3x3 feature map 807 to the DRAM 204 after computation. When performing the second layer convolution 811, the 3x3 feature map 805 is read from the DRAM 204, the 1x1 feature map 808 is stored to the DRAM 204 after computation, then the 3x3 feature map 807 is read from the DRAM 204, the 1x1 feature map 809 is stored to the DRAM 204 after computation. As can be seen from the above description, the feature map 802 is repeatedly read and stored on the off-chip on-chip as intermediate data, which occupies a lot of system resources.

[0065] If the first layer convolution 810 and the second layer convolution 811 are fused, that is, the feature map 802 is stored in the NRAM 431 (the weights of the first layer convolution 810 and the second layer convolution 811 can also be stored in the WRAM 432), the number of accesses between the computing device 201 and the DRAM 204 can be reduced, and the execution efficiency of the overall neural network can be improved. Since the feature maps participating in the fusion (such as the feature map 801, the feature map 802, and the feature map 803) look like an inverted pyramid in the context logic of the neural network model, it is called pyramid fusion.

[0066] Pyramid fusion is usually based on a specific convolution layer and a pooling layer in a neural network to perform fusion backward, that is, the starting layer of fusion is a convolution layer or a pooling layer, and a plurality of layers are fused backward according to the hardware conditions thereof, which can include a plurality of convolution layers and pooling layers. However, with the development of deep learning and neural networks, the order of layers becomes complex, for example, an activation layer is arranged in front of a convolution layer, and the activation layer should also be considered how to be fused with the convolution layer behind it. Therefore, in addition to simply fusing the convolution layer and the pooling layer as the core, the present disclosure provides various fusion methods, which do not necessarily take the convolution layer and the pooling layer as the core, but adopt a specific strategy to flexibly select each layer of the neural network for fusion, even if the layer is user-defined, as long as it meets the fusion strategy, it can be fused to optimize the overall performance.

[0067] Another embodiment of the present disclosure is a new fusion method, which uses the foregoing Figure 1 , Figure 2 , Figure 3 and Figure 4The hardware structure of the template fusion unit is shown in FIG. 3. The template fusion unit is implemented by a hardware structure, and the fusion is called template fusion unit (TFU). The template fusion unit mainly flexibly fuses multiple layers into one layer by a certain fusion strategy to reduce the input / output overhead of the network, which includes the aforementioned pyramid fusion and other fusion methods. The set of layers that are fused is called the template fusion unit, which can be regarded as a new layer or a custom layer.

[0068] In this embodiment, the feature maps, weight values and the like required by the template fusion unit are loaded from the DRAM 204 to the on-chip SRAM 308 at one time. After the feature maps are loaded into the SRAM 308, they are called on-chip unit maps. The on-chip unit maps are cut into sub-maps, and each time a sub-map is loaded from the SRAM 308 to the NRAM 431 of the processor core 306 assigned to calculate the sub-map, and the weight values required for calculating the sub-map are also loaded from the SRAM 308 to the WRAM 432. After each sub-map is calculated, the corresponding intermediate result is obtained, and the intermediate result is stored back to the SRAM 308. After all the sub-maps are calculated, the calculation results are stored back to the DRAM 204 at one time. That is, the corresponding results obtained by the on-chip unit maps and the weight values participating in the operator operation in the neural network model are transferred between the DRAM 204 and the SRAM 308, and the output (intermediate result) corresponding to the sub-map is transferred between the SRAM 308 and the NRAM 431. From the perspective of the computing device 201, the data loading of the template fusion unit is in units of on-chip unit maps, and the calculation is in units of sub-maps.

[0069] In more detail, the SRAM 308 is one of the important reference indicators of the fusion strategy, and its space size determines whether the template fusion unit is in a large map mode or a small map mode. The small map mode and the large map mode refer to whether a feature map stored in the DRAM 204 can be moved to the SRAM 308 for processing at one time. The processing device 203 compares the storage space required by the feature map with the available space of the SRAM 308. If the SRAM 308 is not enough, the feature map cannot be placed, and it is in the large map mode. If the SRAM 308 is enough to accommodate the entire feature map, it is in the small map mode. It should be particularly noted that in the large map mode, the on-chip unit map is only a part of the feature map. In the small map mode, if the available space of the SRAM 308 is large enough, or the feature map is small enough, the SRAM 308 can accommodate multiple feature maps at one time, that is, the on-chip unit map can include multiple feature maps.

[0070] If so, the feature map must be split to be loaded into the computing device 201. The processing device 203 splits the feature map on the DRAM 204 until a small enough on-chip unit graph is generated to meet the space requirement of the SRAM 308, so that the on-chip unit graph can be moved to the SRAM 308 for processing at one time. When the feature map is split, input-dependent operations and output-dependent operations may be generated.

[0071] Input-dependent operations refer to the fact that each on-chip unit graph after splitting at least partially overlaps, and each subset needs some additional copies of input to perform complete operations, resulting in data redundancy in the splitting operation. Data redundancy refers to the reuse of the same data in the system. Input-dependent operations are caused when the template fusion unit includes convolution, pooling, or matrix multiplication.

[0072] Output-dependent operations refer to the fact that after each subgraph outputs an intermediate result, a reduction (reduce) operation is needed to obtain the calculation result. Reduction refers to the fact that based on the understanding of the content of the on-chip unit graph, the on-chip unit graph is split into subgraphs for calculation to reduce the calculation scale, so as to maximize the data reduction while maintaining the original appearance of the on-chip unit graph as much as possible. The calculation results are dependent on each other when the reduction is performed. Output-dependent operations are caused when the template fusion unit includes inner product, convolution, matrix multiplication, sorting, counting, and the like.

[0073] The data format of the feature map that can be processed by this embodiment includes N, H, W, and C dimensions, where N represents batch, H represents height, W represents width, and C represents channel. Taking image data as an example, N indicates the number of images in the batch, H indicates the number of pixels in the vertical direction of the image, W indicates the number of pixels in the horizontal direction, and C indicates the number of channels (for example, the number of channels C of a black-and-white image is 1, and the number of channels C of an RGB color image is 3).

[0074] The order of these dimensions determines the composition of the data, and common composition methods include NHWC and NCHW, Figure 9Figure 1 shows the difference between NCHW and NHWC format. This figure is an example of RGB color image. R represents red pixel, G represents green pixel, and B represents blue pixel. Sequence 91 is NCHW format. N is arranged in the outer layer, and pixels in each channel are arranged next to each other in the order of RGB. The offset of the element with coordinates (n, c, h, w) in the storage is ((n x C + c) x H + h) x W + w. Sequence 92 is NHWC format. C is arranged in the innermost layer, and RGB pixels corresponding to a spatial position are arranged next to each other. The figure also shows the positions of input pixel 901, input pixel 902, and input pixel 903 in different arrangement modes. The three input pixels 901, 902, and 903 together represent the color of a point in the image. The conversion method of the corresponding coordinate offset of the element with coordinates (n, c, h, w) is ((n x H + h) x W + w) x C + c. NHWC is closer to the BMP picture data storage format than NCHW. In the BMP format file, data is stored according to individual pixel points, and each pixel point stores color values of all channels. This makes it unnecessary to perform additional dimension conversion when reading the input picture. Therefore, the memory locality of NHWC is better, and each three input pixels can obtain an output pixel. NCHW must wait for all channel inputs to be ready before obtaining the final output result, which requires a large cache space.

[0075] In this embodiment, the data fusion neural network can be divided into a plurality of template fusion units according to layers of the data fusion neural network, Figure 10 A corresponding flowchart is shown in Figure 10.

[0076] In step 1001, the processing device 203 determines whether the required storage space of the feature map is greater than the available space of the SRAM 308. If yes, it means that the feature map cannot be loaded into the SRAM 308 at one time, and therefore step 1002 is performed to split the feature map. In this embodiment, the processing device 203 preferentially selects splitting in the N dimension, because no input or output dependent operation is generated. If splitting in the N dimension cannot meet the requirement, splitting in the H or W dimension is considered, which may generate input or output dependent operation. This embodiment also supports splitting in the C dimension, in particular, splitting in the Cout direction. In this way, one convolution is split into multiple convolutions through data optimization, so that the WRAM 432 can accommodate the weights, for example, the weights are split to four processor cores 306. Therefore, as long as splitting in a certain dimension is processable by the computing device 201, it is within the scope of the disclosure.

[0077] Further, the processing device 203 can sequentially split the feature map or weight in N, H, W dimensions with a certain granularity, which can be a fixed or variable ratio, or represented by a function. In one application scenario, the processing device 203 splits the feature map or weight from large to small. Taking the feature map as an example, first, the feature map with dimension NHWC is split into a feature map with dimension N1HWC and a feature map with dimension N2HWC in the N dimension, where the certain granularity is a fixed ratio, and N1 and N2 are each half of N. If it is still not small enough, the processing device 203 continues to split the feature map with dimension N1HWC into a feature map with dimension N1H1WC and a feature map with dimension N1H2WC in the H dimension, where H1 and H2 are each half of H. If it is still not small enough, the processing device 203 continues to split the feature map with dimension N1H1WC into a feature map with dimension N1H1W1C and a feature map with dimension N1H1W2C in the W dimension, where W1 and W2 are each half of W. The processing device 203 can continue to split in the N, W, H dimensions with smaller granularity, such as quarter, eighth, or sixteenth, until the feature map is small enough to be loaded into the on-chip unit graph of the SRAM 308.

[0078] It can be understood that the processing device 203 can continue to split in one dimension until it cannot be split, and then select another dimension to continue splitting. For example, if splitting in the H dimension continues, and the smallest unit still cannot be loaded into the SRAM 308, then splitting in the W dimension is selected until the smallest unit is obtained.

[0079] It should be noted that since the splitting is from large to small, when the split feature map meets the condition, the size of the required storage space is usually close to the available space of the SRAM 308. In other words, in the large graph mode, the DRAM 204 can only transfer one split feature map to the SRAM 308 at a time, but in the small graph mode, the space of the SRAM 308 can load multiple feature maps from the DRAM 204 at a time.

[0080] In another application scenario, the processing device 203 splits from small to large, and the certain granularity can also be a fixed or variable ratio, or represented by a function. For example, first, the feature map is split in the N dimension with a certain granularity of the smallest unit, i.e., 1xHxWxC. If the SRAM 308 can load, the processing device 203 continues to enlarge the split of the feature map, for example, to 2xHxWxC. If it can still be loaded, it continues to enlarge until n x H x W x C cannot be loaded, and the size of the on-chip unit graph is (n-1) x H x W x C.

[0081] If the required storage space of 1 x H x W x C already exceeds the available space of SRAM 308, the processing device 203 continues to split from another dimension, for example, from the H dimension, and then the processing device 203 judges 1 x 1 x W x C. If it is small enough, it increases along the H dimension until it finds that the required storage space of 1 x (h-1) x W x C is just close to but not greater than the available space of SRAM 308. If it still exceeds the available space of SRAM 308, the processing device 203 continues to split from another dimension, for example, from the W dimension. In this way, the optimal input data that can be loaded into SRAM 308 at one time is found. In this case, the optimal means that the required storage space of the on-chip unit graph is just close to but not greater than the available space of SRAM 308.

[0082] After the processing device 203 splits the feature map, it returns to step 1001 and judges whether the required storage space of the split feature map is still greater than the available space of SRAM 308. If so, it executes step 1002 again to continue to split down.

[0083] If the processing device 203 judges that the required storage space of the split feature map is not greater than the available space of SRAM 308, it means that SRAM 308 can load the split feature map at one time, and then it executes step 1003 to set the split feature map as the on-chip unit graph.

[0084] Finally, it executes step 1004 to determine the template fusion unit according to the size of the on-chip unit graph. This step will be described in detail later.

[0085] In other application scenarios, when the processing device 203 repeatedly executes multiple times between step 1001 and step 1002, it means that the required storage space of the split feature map is getting closer and closer to the available space of SRAM 308. For example, assuming that the required storage space of the feature map is 100k and the available space of SRAM 308 is 40k, in step 1001, the processing device 203 judges that the required storage space of the feature map is greater than the available space of SRAM 308, so it executes step 1002 to split it into half along the N dimension. At this time, the required storage space of the split feature map is 50k. Then it returns to step 1001, and the required storage space of the split feature map is still greater than the available space of SRAM 308, so it continues to execute step 1002 to split it into half again along the N dimension. At this time, the required storage space of the split feature map is 25k. Then it returns to step 1001, and the required storage space of the split feature map is less than the available space of SRAM 308, so it executes step 1003 to set the split feature map (with a size of 25k) as the on-chip unit graph.

[0086] The available space of the SRAM 308 is 40k, while the required storage space of the on-chip unit graph is 25k, leaving 15k of space idle. The reason for this is that the step 1002 splits in half each time, so that the last split is too large in granularity. This embodiment can gradually reduce the specific granularity of the split with the number of splits, so that the required storage space of the split on-chip unit graph is as close as possible to the available space of the SRAM 308. For example, the specific granularity can be set to one-half at the beginning, one-third next, and one-fourth at the end. Similarly, taking the feature map required storage space of 100k and the available space of the SRAM 308 of 40k as an example, in step 1001, the processing device 203 determines that the feature map required storage space is greater than the available space of the SRAM 308, so step 1002 is performed, the specific granularity is set to one-half, and the split feature map is 50k. Then, step 1001 is returned to, the split feature map required storage space is still greater than the available space of the SRAM 308, and step 1002 is continued to be performed. At this time, the specific granularity is adjusted to one-third, and the split feature map is 37.5k. Then, step 1001 is returned to, the split feature map required storage space is less than the available space of the SRAM 308, so step 1003 is performed, and the processing device 203 sets the split feature map (size 37.5k) as the on-chip unit graph. 37.5k is closer to 40k than 25k, and the latter way makes more efficient use of the available space of the SRAM 308. This embodiment does not limit the size of the specific granularity, which can be set according to the application scenario.

[0087] After determining the size of the on-chip unit graph, step 1004 is performed, which dynamically fuses the neural network according to the fusion strategy. Figure 11 This embodiment shows a method for dynamically fusing a neural network according to a fusion strategy.

[0088] In step 1101, the starting layer of the template fusion unit is selected according to the starting rule of the fusion strategy. The processing device 203 selects the starting layer of the template fusion unit according to the starting rule of the fusion strategy, that is, selects the layer to start fusion from among the layers that have not yet been fused in the neural network.

[0089] In one application scenario, the starting rule can be that the starting layer is the first un-fused layer in the neural network, and the processing device 203 searches for the first un-fused layer. Taking the AlexNet neural network model of Figure 6 For example, taking the AlexNet neural network model of 23 layers, assuming that the first layer to the fifth layer have been fused, when the starting rule is that the starting layer is the first un-fused layer in the neural network, the processing device 203 selects the ReLU activation layer of the sixth layer as the starting layer and fuses backward (i.e., in the direction of the seventh layer). It should be noted that under this starting rule, the starting layer does not necessarily have to be a convolutional layer or a pooling layer.

[0090] In another application scenario, considering that convolution and pooling layers consume the most input / output resources, the starting rule is that the starting layer is the first unmerged convolution or pooling layer. The processing device 203 first finds all the convolution and pooling layers of the unmerged layers in the neural network model, and then starts merging from the first unmerged convolution or pooling layer. For example, in the AlexNet neural network model of Figure 6 , it is assumed that the first layer to the ninth layer have been merged. The processing device 203 finds all the convolution and pooling layers of the unmerged layers in the neural network model, that is, the eleventh layer, the thirteenth layer, and the fifteenth layer. Then, starting from the first unmerged convolution or pooling layer, the starting layer is the eleventh layer.

[0091] In step 1102, merging is performed based on the starting layer, and all rules of the merging strategy are checked one by one to establish a template merging unit. The processing device 203 performs merging based on the starting layer, and all rules of the merging strategy are checked one by one to establish a template merging unit. Under the premise of meeting all the rules, the hardware resources of the computing device 201 are sufficient to support the loading of the data required by the computing template merging unit at one time, and then the neural network computing is performed according to the template merging unit. In addition to the aforementioned starting rule, the merging strategy exemplarily includes the following rules:

[0092] Rule one: backward merging

[0093] Backward merging refers to merging from the starting layer to the inference direction of the neural network model, for example, in the AlexNet neural network model of Figure 6 , it is merged in the direction of the first layer→the second layer→the third layer. If there are unmerged layers before the starting layer, these unmerged layers will not be considered in the template merging unit under this rule.

[0094] Rule two: preferentially forward merging

[0095] Forward merging refers to merging from the starting layer to the opposite direction of the inference of the neural network, for example, in the AlexNet neural network model of FIG. 6, it is merged in the direction of the third layer→the second layer→the first layer. This rule is usually combined with the aforementioned starting rule that the starting layer is the first unmerged convolution or pooling layer, because there may be unmerged layers before the convolution or pooling layer. After selecting the starting layer, the processing device 203 preferentially merges forward to try to include the layers before the starting layer that have not been merged into the template merging unit. For example, in the AlexNet neural network model of Figure 6 , it is assumed that the first layer to the second layer have been merged. The processing device 203 finds that the first unmerged convolution or pooling layer is the fifth layer, so the starting layer is the fifth layer. The fourth layer and the third layer are preferentially merged forward. If the merging can continue, then the sixth layer, the seventh layer, and the like are merged backward.

[0096] Rule Three: Prefer block structure

[0097] When the neural network model has block structure, this rule requires the processing device 203 to prefer to add or delete template fusion units in block structure rather than in layer, if the operation logic fusion of an entire block is unsuccessful, then consider fusion from the layers on each branch. For example, the neural network model of Figure 7 , the processing device 203 will prefer to consider the subnetwork 701 or the subnetwork 702 as a unit for fusion.

[0098] When the neural network is a long chain structure, since there is no block structure, directly add or delete template fusion units in layer. This rule is not applicable to neural network models with long chain structure.

[0099] Rule Four: Single branch output

[0100] The fusion strategy of this embodiment does not support template fusion units as multi-output networks, because the shape derivation implemented inside the template fusion unit mainly adopts the form of backward-to-forward derivation, and multi-output network means that different outputs need to be derived forward respectively, and the results of derivation do not necessarily converge to the same feature map, so as to fail to converge.

[0101] In other words, the output of the template fusion unit needs to be single branch output, that is, the last layer of the template fusion unit can only have one output. Figure 7 Two fusion ways of the subnetwork 701 are marked, the first one is to fuse the first layer to the fifth layer into a template fusion unit 703, and the second one is to fuse the first layer to the sixth layer into a template fusion unit 704. Since the outputs of the third layer and the fifth layer are the outputs of the template fusion unit 703, the template fusion unit 703 belongs to multi-output network, that is, multi-branch output. The output of the sixth layer is the output of the template fusion unit 704, which only produces one output data, so the template fusion unit 704 belongs to single-output network, that is, single-branch output. The processing unit 203 will judge whether the output of the template fusion unit is single-branch output, if this rule is not satisfied, the processing device 203 adds or deletes layers inside the template fusion unit until this rule is satisfied.

[0102] Rule Five: Include at least 2 main layers

[0103] When the layer logic is too simple, the performance of the template fusion unit is worse than that of the unfused layers. Therefore, when using layer logic as the fusion strategy, the processing device 203 evaluates whether the operations of each fused layer are complex enough to generate benefits from the fusion. To generate benefits, the main layers should be included in the template fusion unit as much as possible. The main layers refer to layers that consume a lot of input / output resources, such as matrix multiplication, pooling, or convolution. Pooling here includes various types of pooling, such as max pooling or average pooling, and convolution also includes various types of convolution, such as ordinary convolution, convolution with average, and depthwise convolution. This rule requires that the template fusion unit includes at least two main layers. When the processing unit 203 determines that this rule is not met, the processing device 203 will adjust the template fusion unit until the rule is met.

[0104] Rule 6: Includes a continuous structure with the main layer, the main layer, and non-main layers adjacent in sequence.

[0105] This rule requires that template fusion units must include a continuous structure of main layers, main layers, and non-main layers; that is, a continuous structure where main layers, main layers, and non-main layers are sequentially adjacent. Such computation is complex enough to make fusion efficient. See also... Figure 6 The template fusion unit consists of layers 4, 5, and 6, where layer 4 is a max pooling layer, layer 5 is a convolutional layer, and layer 6 is a ReLU activation layer. This conforms to a continuous structure where the main layers and non-main layers are sequentially adjacent. Therefore, the template fusion unit including layers 4, 5, and 6 satisfies this rule. When the processing unit 203 determines that this rule is not satisfied, the processing device 203 adjusts the template fusion unit until the rule is satisfied.

[0106] Rule 7: Includes adjacent, continuous structures of scalar computation layers and vector computation layers.

[0107] This rule refers to a template fusion unit comprising a continuous structure of scalar computation layers and vector computation layers, i.e., a continuous structure where scalar computation layers and vector computation layers are sequentially adjacent. The scalar computation layer refers to an addition layer, subtraction layer, or multiplication layer, and the vector computation layer refers to an activation layer, batch normalization layer, or scaling layer. When processing unit 203 determines that this rule is not satisfied, processing device 203 will adjust the template fusion unit until the rule is satisfied.

[0108] Rule 8: The weights of a convolutional layer are not equal to the output of any other layer.

[0109] This rule states that the weights of the convolutional layers in the template fusion unit are not the outputs of any layer in the neural network, regardless of whether that layer is included in the template fusion unit. When the processing unit 203 determines that this rule is not met, the processing device 203 will remove this convolutional layer from the template fusion unit.

[0110] Rule 9: The weights of the convolution layer are not shared with any layer of the neural network

[0111] Since the weights of the operators involved in the template fusion unit have a special placement form, when the convolution operator to be fused shares weights with other operators, the placement logic of the weights will conflict. This rule is that the weights of the convolution operator in the template fusion unit are not shared with any layer of the neural network. When the processing unit 203 judges that this rule is not satisfied, the processing device 203 removes the convolution operator from the template fusion unit.

[0112] Rule 10: The weights are not greater than the available space of the WRAM

[0113] The large graph mode has less restriction on the WRAM 432 because the on-chip unit graph loaded into the SRAM 308 is only a part of the feature map, and when calculating the template fusion unit, the WRAM 432 only needs to store all the weights of the feature map. However, since the small graph mode may load multiple feature maps into the SRAM 308, the number of weights required will increase, and it is necessary to carefully evaluate whether the available space of the WRAM 432 is sufficient. This rule is that the required storage space of the weights in the on-chip unit graph is not greater than the available space of the WRAM 432. When the processing device 203 judges that this rule is not satisfied, the processing device 203 reduces the size of the on-chip unit graph.

[0114] If the weights are split based on the output channel parameter Cout of the C dimension, since the weights will be evenly distributed to multiple processor cores 306, the rule is adjusted to:

[0115]

[0116] where W j is the required storage space of the weights involved in the on-chip unit graph j, n is the number of processor cores in the cluster, and W is the available space of the WRAM 432.

[0117] Rule 11: Redundancy percentage

[0118] The redundancy percentage is the ratio of the total redundancy generated by the input-dependent operation and the output-dependent operation to the normal input / output amount of the template fusion unit. Here, the normal input / output amount refers to the amount of data without redundancy before the on-chip unit graph is split. The processing device 203 calculates the percentage of the memory access amount size TFU of the on-chip unit graph from the DRAM 204 to the SRAM 308 after the template fusion unit fuses the current layer, to the normal input / output amount (without redundancy) size ori , where the memory access amount size TFU refers to the theoretical memory access amount size oriThe sum of redundancies is added. The formula is as follows:

[0119]

[0120] The processing device 203 takes into account the split information and shape derivation calculation of the template fusion unit, and sets the percentage threshold to 50%, 75%, 100%, 125%, or 150%, preferably 100%. Taking the percentage threshold of 100% as an example, it means that when the sum of redundancies is greater than twice the normal input / output amount of the template fusion unit, fusion is no longer performed. This rule is that the sum of redundancies generated by the split tile unit graph does not exceed a certain proportion related to the percentage threshold. Once it is exceeded, it means that the redundant part is too much, and a large amount of resources will be consumed in calculating the redundancy, and the performance will be reduced. Therefore, when the processing device 203 determines that this rule is not met, the processing device 203 will stop fusion.

[0121] It should be noted that in the small graph mode, since at least one whole complete feature map is loaded at a time from the DRAM 204 to the SRAM 308, no redundancy is generated. This rule does not apply to the small graph mode.

[0122] Rule Twelve: Tile Unit Graph Input / Output Size

[0123] Assuming that the spatial size of the SRAM 308 is S, the storage space required by the tile unit graph is IN, and the storage space required by the calculation result of the tile unit graph is OUT, then this rule is that the spatial size of the SRAM 308 needs to meet the following conditions:

[0124] If IN and OUT cannot share storage space, IN+OUT<S

[0125] If IN and OUT can share storage space, MAX(IN, OUT)<S

[0126] That is, if IN and OUT cannot share storage space, the sum of the storage space of the tile unit graph and the storage space of the calculation result is less than the available space of the SRAM 308; if IN and OUT can share storage space, the larger one of the storage space of the tile unit graph and the storage space of the calculation result is less than the available space of the SRAM 308.

[0127] Rule Thirteen: W i +IN1+IN2≤S

[0128] In the small graph mode, this rule is that the spatial size of the SRAM 308 needs to meet the following conditions:

[0129] W i +IN1+IN2≤S That is, the storage space required by the weight of the subgraph i W iThe sum of the required storage space of the on-chip unit graph IN1, the required storage space of the cache space IN2 is not greater than the available space of the SRAM 308. When the processing device 203 judges that this rule is not satisfied, the processing device 203 reduces the number of on-chip unit graphs until the rule is satisfied.

[0130] Rule Fourteen: SubINi+W i +IN2≤S

[0131] In the small graph mode, this rule requires that the space size of the SRAM 308 satisfies the following condition:

[0132] SubINi+W i +IN2≤S

[0133] That is, the required storage space of the subgraph i SubINi, the required storage space of the weight of the subgraph i W i , the required storage space of the cache space IN2 is not greater than the available space of the SRAM 308. When the processing device 203 judges that this rule is not satisfied, the processing device 203 reduces the number of on-chip unit graphs until the rule is satisfied.

[0134] Rule Fifteen: SubOUTi+W i+1 +IN2≤S

[0135] In the small graph mode, this rule requires that the space size of the SRAM 308 satisfies the following condition:

[0136] SubOUTi+W i+1 +IN2≤S

[0137] That is, the required storage space of the intermediate result of the subgraph i SubOUTi, the required storage space of the weight of the next subgraph W i+1 , the required storage space of the cache space IN2 is not greater than the available space of the SRAM 308. When the processing device 203 judges that this rule is not satisfied, the processing device 203 reduces the number of on-chip unit graphs until the rule is satisfied.

[0138] Rule Sixteen: W i +W i+1 ≤W

[0139] The weights participating in the convolution operation in the template fusion unit are independently transported and reside on the WRAM 432. In the small graph mode, if the subgraph includes multiple feature maps, considering the pipeline between the subgraphs, the WRAM 432 stores at most the weights of two adjacent subgraphs at the same time. Assuming that the required storage space of each subgraph i is W i , and the total space of the WRAM 432 is W, this rule requires that the space size of the WRAM 432 satisfies the following condition:

[0140] W i +W i+1 ≤W

[0141] i.e. the storage space W required by the weight of subgraph i i the storage space W required by the weight of next subgraph i+1 the sum is not greater than the available space of WRAM 432. When the processing device 203 judges that this rule is not satisfied, the processing device 203 reduces the number of on-chip unit graphs until the rule is satisfied.

[0142] Rule Seventeen: The storage space required by subgraph is not greater than the available space of NRAM

[0143] This rule is that the storage space required by subgraph is not greater than the available space of NRAM 431. When the on-chip unit graph on SRAM 308 is to be split into subgraphs and carried to NRAM 431, the processing device 203 can split in fine granularity in N, H, and W dimensions. If the space of NRAM 431 is insufficient, the processing device 203 splits the on-chip unit graph more finely until the rule is satisfied. Generally, NRAM 431 has reasonable available space so that the on-chip unit graph is split to a reasonable degree and can be loaded at one time, and from the perspective of fusion strategy, the template fusion unit is not affected by the number of batches. However, the smaller the on-chip unit graph is split (i.e. the more subgraphs), the processing speed will decrease, so the processing device 203 needs to evaluate the space of NRAM 431.

[0144] In some embodiments, the space of SRAM 308 corresponds to the number of NRAM 431 of processor cores 306 in cluster 305, for example, if cluster 305 includes 4 processor cores 306, the space of SRAM 308 is 4 times the space of NRAM 431. In other words, the on-chip unit graph in large graph mode Figure 1 Generally, can be allocated to 4 processor cores 306 for processing, and this architecture design has considered that the data loaded in SRAM 308 can be allocated to all NRAM 431 at one time. Therefore, this rule does not need to be considered in the large graph mode.

[0145] Rule Eighteen: The number of feature maps is not greater than the feature map threshold

[0146] In the tile mode, the on-chip unit graph can include multiple feature maps, the more feature maps, the more sub-graph transmission between SRAM 308 and NRAM 431, the efficiency will be reduced, therefore, the more feature maps in the on-chip unit graph is not necessarily better, the processing device 203 will calculate the appropriate fusion layer number according to the number of feature maps in the on-chip unit graph to maximize its efficiency. The rule is that the number of feature maps in the on-chip unit graph is not greater than the feature map threshold, when the processing device 203 judges that this rule is not met, the processing device 203 reduces the number of feature maps in the on-chip data until the rule is met.

[0147] Rule nineteen: step redundancy

[0148] Step redundancy refers to: when the template fusion unit fusion layer number is too many, plus the length and width of the kernel of convolution and pooling is greater than the step, the input data required by each output point has an overlapping part, that is, the input dependent operation mentioned above, and the overlapping part is the step redundancy. Step redundancy makes each processor core 306 need to read some data, but this part of the reused data will occupy the access resources on-chip and off-chip, the more the number of layers included in the template fusion unit, the more serious the step redundancy. The rule is that the difference between the edge length of the kernel of the convolution layer or the pooling layer and the step is not greater than the redundancy threshold.

[0149] In this embodiment, the redundancy threshold is defined as follows. Assuming that the length and width of the kernel of the convolution and pooling layer are k x and k y , the steps in the length and width directions are s x and s y , the step redundancy in the length direction is the sum of k x -s x of all convolution and pooling layers in the template fusion unit; similarly, the step redundancy in the width direction is the sum of k y -s y of all convolution and pooling layers in the template fusion unit. The redundancy threshold of this embodiment can be 3, 4, 5 or 6, preferably 4. As long as the step redundancy in either the length direction or the width direction is greater than the redundancy threshold, this rule is not met. The processing device 203 adjusts the template fusion unit, usually reduces the number of fused layers, until this rule is met.

[0150] The fusion strategy sets an exception rule for step redundancy. If there are multiple branches in the layers to be fused and the template fusion unit can fuse the entire multiple branches, the performance of the template fusion unit will be more excellent, in this case, the processing device 203 will ignore the rule of step redundancy, that is, step redundancy will not limit the template fusion unit to fuse multiple branches, that is, in the fusion strategy of this embodiment, fusing multiple branches is preferred to the limitation of step redundancy. That is, step redundancy is only considered in the case of single branch.

[0151] The above rules are only examples, and the disclosure does not limit the order in which the rules are executed, nor does it limit the rules that need to be considered at the same time. A person skilled in the art can add or delete rules according to actual conditions in different application scenarios to achieve a fusion strategy that meets the current application scenario.

[0152] Returning to Figure 11 In step 1103, the neural network calculation is performed according to the established template fusion unit. Based on the three-level operation hierarchy of the system-on-a-chip-cluster-processor core, the three-layer memory design of DRAM-SRAM-NRAM / WRAM, and the template fusion unit being regarded as a custom layer in the neural network, the computing device 201 loads the data required for the calculation of the template fusion unit from the DRAM 204 to the SRAM 308 at one time, so that the data can be cached and calculated in the appropriate level to form sufficient pipelining, and the calculation result is transmitted from the SRAM 308 to the DRAM 204 after the calculation is completed, greatly reducing the input / output overhead in neural network calculation.

[0153] When input data in the fields of computer vision, speech, natural language processing, data mining, etc. are subjected to various deep learning and machine learning algorithms, the disclosure based on the template fusion unit can reduce the input / output overhead in neural network calculation. Another embodiment of the disclosure is a method for performing neural network calculation using a template fusion unit. Figure 12 The flow thereof is shown.

[0154] In step 1201, the template fusion unit is determined according to the fusion strategy. The processing device 203 selects the starting layer of the template fusion unit according to the starting rule of the fusion strategy, and performs fusion based on the starting layer, and checks all the rules of the fusion strategy one by one to establish the template fusion unit. The previous embodiment has illustrated various rules of the fusion strategy in detail, and will not be repeated here.

[0155] In this step, the template fusion unit is presented in the form of source code, and the source code needs to be converted into object code (machine code) by a compiler. The following steps are the process of converting the source code of the template fusion unit into object code by the compiler.

[0156] In step 1202, the shape of the template fusion unit is derived. For the data to be processed by the template fusion unit, this embodiment uses a reverse derivation method, in which the compiler derives from the output how many sizes of input are needed, so that the template fusion unit can be derived. Figure 8For example, the feature map 803 is deduced reversely to the feature map 802, and the feature map 802 is deduced reversely to the feature map 801. In this step, the compiler not only deduces the required input data according to the template fusion unit, but also further deduces the redundancy.

[0157] Next, step 1203 of deducing addresses is performed. According to the shape of the template fusion unit, the compiler deduces the on-chip storage space addresses of the entire control flow graph, and implements the access of general addresses, so as to achieve the purpose of reducing computing resources and shortening computing time. The control flow graph is an abstract data structure used in the compiler, which represents all paths that a program can execute, and reflects the possible flow direction of all nodes in the process in the form of a flowchart. The control flow graph is composed of nodes and the relationship between nodes. The node is also called a basic block (BB), which is a sequence of statements that is executed in the program to the maximum extent. Each basic block has only one entrance and exit, and is executed from the entrance to the exit. The basic block has the characteristic that if the first instruction is executed, all instructions in the basic block will be executed in sequence.

[0158] Each basic block contains at least one instruction, and the instructions in the basic block can use a pointer to point to a specific on-chip storage space. The pointer is a variable used to save the address of a specific address space. Through the pointer, the processor core 306 can load data into the specific address space pointed to by the pointer, or take out data from the specific address pointed to by the pointer.

[0159] According to the division of the template fusion unit, the compiler initially divides the basic blocks, and after iterative operation, the basic blocks and their relationships are confirmed, thereby completing the target code for implementing the template fusion unit.

[0160] Moreover, the compiler also analyzes the reuse data of the two template fusion units in the neural network in sequence, judges how much data in the previous template fusion unit can be left on the chip for use by the next template fusion unit, and plans the storage addresses of the data according to the judgment result.

[0161] In this step, the compiler completes the deduction of the addresses in the control flow graph.

[0162] In step 1204, the on-chip storage space is allocated. The processing device 203 allocates the physical space of the SRAM 308, the NRAM 431 and the WRAM 432 based on the deduction of the template fusion unit address. In this step, the compiler completes the pointing of the pointers in the control flow graph.

[0163] Finally, step 1205 is performed to generate executable instructions. In this step, a linker links the object code generated by the compiler and libraries to form an executable file. In more detail, the object code is a program module including machine code and information available to the linker, and the linker resolves undefined symbolic references and replaces placeholders in the object code with addresses of the symbols to generate executable instructions. The executable instructions can be directly executed by the computing device 201 to complete the computation of the neural network.

[0164] The present disclosure dynamically determines a template fusion unit to fuse multiple layers in a neural network to form a new custom layer and loads data required by the template fusion unit at one time to reduce input / output overhead by setting a fusion strategy.

[0165] When the template fusion unit is determined according to the rules of the fusion strategy, it is not necessarily necessary to expand the fusion starting from a convolutional layer or a pooling layer. The foregoing embodiment mentions that in one application scenario, the starting rule can be that the starting layer is the first un-fused layer in the neural network, which can be a layer other than a convolutional layer or a pooling layer. Such a starting rule makes the establishment of the template fusion unit more flexible, enabling appropriate selection of the starting layer to start the fusion based on the order of the layers in different neural networks, without being limited by the position and number of convolutional layers or pooling layers in the neural network model, thereby adapting to various network models and making the fusion more comprehensive to improve the overall efficiency.

[0166] For example, taking the neural network model of Figure 6 as an example, assuming that the first layer to the fifth layer have been fused, when the next template fusion unit is established, if the starting rule adopts the starting layer as the first un-fused convolutional or pooling layer, the next convolutional or pooling layer is the eighth layer, in other words, the sixth layer and the seventh layer can not be fused, which affects the overall efficiency.

[0167] Another embodiment of the present disclosure is a scheme for fusing a neural network, in which the starting layer is a layer other than a convolutional layer and a pooling layer, i.e., a non-convolutional layer and a non-pooling layer. This embodiment is also implemented based on the framework of Figure 1 to Figure 4 . This embodiment also performs the flowchart shown in Figure 11 .

[0168] In step 1101, a starting layer is selected according to a fusion strategy. The processing device 203 selects a starting layer according to a fusion strategy, for example, the starting rule of the fusion strategy is that the starting layer is the first un-fused layer in the neural network, which is a layer other than a convolutional layer or a pooling layer.

[0169] It is noted that the step does not use a starting rule that the starting layer is the first unmerged convolutional or pooling layer. If the starting layer is selected according to the starting rule, the starting layer must be a convolutional or pooling layer, and the advantage of the embodiment that the position and number of convolutional layers or pooling layers in the neural network model are not limited does not exist.

[0170] In one application scenario, the starting layer can be an element-wise layer, which is a layer that operates on each element of a vector. The input data and the output data of such an operation have the same shape. Element-wise layers include the following types:

[0171] 1. Basic operations: vector addition, vector subtraction, vector multiplication, etc.

[0172] 2. Advanced operations: absolute value, square root, division, exponentiation, modulus, power, etc.

[0173] 3. Trigonometric function operations

[0174] 4. Rounding operations: rounding up, rounding, rounding down, keeping only the integer, etc.

[0175] 5. Activation functions: sigmoid, tanh, ReLU, etc.

[0176] In another application scenario, the starting layer can be an add padding layer. Adding padding is to not discard the original image information and to keep the size of the input data consistent with the original image. The add padding layer adds blank elements around the input data. Figure 1

[0177] In another application scenario, the starting layer can be a custom layer. With the development of deep learning and the complexity of neural networks, known or standard operators are not enough, and more and more custom operation rules are applied to neural networks. This embodiment can select a custom layer as the starting layer.

[0178] In another application scenario, the starting rule of the fusion strategy of this embodiment causes the processing device 203 to further determine whether the neural network includes a block structure. If not, it indicates that the neural network is a long chain structure, and the processing device 203 can select the first unmerged layer in the neural network according to the aforementioned starting rule; if yes, this embodiment refers to the aforementioned rule three, and preferentially fuses in the unit of the block structure. Therefore, the processing device 203 then determines whether the first layer in the block structure is a layer other than a convolutional layer and a pooling layer. If yes, the processing device 203 takes the first layer as the starting layer.

[0179] ​The input of the first layer and the output of the last layer of the template fusion unit are the interaction data of the template fusion unit with the DRAM 204, and the calculation of each layer does not need to access the DRAM 204 again. In this embodiment, the processing device 203 further divides the template fusion unit into a plurality of sub-template fusion units according to the sizes of the NRAM 431 and the WRAM 432.

[0180] Figure 13 A schematic diagram showing the division of the sub-template fusion unit is shown. The T1 layer to the T11 layer are a section of a specific deep learning network, and the processing device 203 further divides the template fusion unit 1301 into a first sub-template fusion unit 1311 and a second sub-template fusion unit 1312 according to the sizes of the NRAM 431 and the WRAM 432. In other embodiments, the processing device 203 can divide the template fusion unit into a number of sub-template fusion units.

[0181] Before starting the calculation of the template fusion unit 1301, the GDMA 311 once moves the data required by the template fusion unit 1301 from the DRAM 204 to the SRAM 308, including the on-chip unit graph corresponding to the template fusion unit and the corresponding weights of the on-chip unit graph, the sub-graph corresponding to each sub-template fusion unit and the corresponding weights of the sub-graph, which is a part of the on-chip unit graph and a part of the corresponding weights of the on-chip unit graph. Then the MVDMA 434 moves the sub-graph required for executing the first sub-template fusion unit 1311 to the NRAM 431 and loads the corresponding weights of the sub-graph into the WRAM 432, and the operation module 42 starts to execute the task of the first sub-template fusion unit 1311, that is, to calculate the T1 layer to the T6 layer, without accessing the SRAM 308 again. When the first sub-template fusion unit 1311 is calculated, the first intermediate result is output, and the MVDMA 434 moves the first intermediate result from the NRAM 431 to the SRAM 308.

[0182] Then, the MVDMA 434 moves the sub-graphs required for the execution of the second sub-template fusion unit 1312 from the SRAM 308 to the NRAM 431, loads the corresponding weights of the sub-graphs into the WRAM 432, and the operation module 42 performs the task of the second sub-template fusion unit 1312, i.e., calculates the T7 layer to the T11 layer, without accessing the SRAM 308 again. When the second sub-template fusion unit 1312 is calculated, the second intermediate result is output, and the MVDMA 434 moves the second intermediate result from the NRAM 431 to the SRAM 308. One processor core 306 reduces the first intermediate result and the second intermediate result to generate a calculation result, and finally the GDMA 311 moves the calculation result from the SRAM 308 to the DRAM 204 at one time, thereby completing the task of the template fusion unit 1301, i.e., completing the task of the T1 layer to the T11 layer, and only accessing the DRAM 204 at the beginning and the end of the template fusion unit 1301, thereby greatly reducing the number of input / output times.

[0183] One important reason why the computing device 201 has strong computing power is that the three-level operation hierarchy of the system-on-chip-cluster-processor core, combined with the three-layer memory design of DRAM-SRAM-NRAM / WRAM, enables data to be cached and calculated at the appropriate level, forming sufficient pipelining.

[0184] When the computing device 201 performs calculation, it can be mainly divided into the following three stages. The load stage (load): loading data; the compute stage (compute): moving data, calculating, and moving intermediate results; and the store stage (store): storing results.

[0185] More specifically, this embodiment adopts a two-layer three-level pipeline, as shown in FIG. 14. Figure 14 As shown in FIG. 14, the load stage 1401, the compute stage 1402, and the store stage 1403 of the first layer occur in the cluster level. The first layer load stage 1401 is to execute the template fusion unit, the GDMA 330 loads data from the DRAM 204 to the SRAM 308, the first layer compute stage 1402 is to calculate the loaded on-chip graph by the cluster 305, and the first layer store stage 1403 is to store the calculation result from the SRAM 308 to the DRAM 204 by the GDMA 330.

[0186] Since the cluster 305 includes multiple processor cores 306, the first-tier computation stage 1402 actually partitions the on-chip unit graph into corresponding sub-graphs, which are broadcasted to the processor cores 306 for computation via the storage core 307, and thus the three-stage pipeline of the second tier occurs in the processor cores 306. In more detail, the second-tier load stage 1404 is to perform the sub-template fusion unit to load the sub-graphs from the SRAM 308 into the NRAM 431 and load the required weights into the WRAM 432 by the MVDMA 434, the second-tier computation stage 1405 is to carry the sub-graphs and weights to the operation module 42 for computation and carry the intermediate results back to the NRAM 431, and the second-tier store stage 1406 is to store the intermediate results from the NRAM 431 back to the SRAM 308 by the MVDMA 434.

[0187] The pipeline of the first tier means that the first-tier load stage 1401, the first-tier computation stage 1402, and the first-tier store stage 1403 can be simultaneously parallel. Taking the same cluster 305 to process the first on-chip unit graph, the second on-chip unit graph, and the third on-chip unit graph as an example, first, the first on-chip unit graph is loaded into the SRAM 308 in the first-tier load stage 1401. Then, the first on-chip unit graph is computed in the first-tier computation stage 1402 and the first computation result is carried back to the SRAM 308, and at the same time, the second on-chip unit graph is loaded into the SRAM 308 in the first-tier load stage 1407. When the first computation result is stored back to the DRAM 204 in the first-tier store stage 1403, the second on-chip unit graph is computed in the first-tier computation stage 1408 and the second computation result is carried back to the SRAM 308, and at the same time, the third on-chip unit graph is loaded into the SRAM 308 in the first-tier load stage 1410.

[0188] In order to cooperate with the operation of the aforementioned pipeline, the SRAM 308 of this embodiment includes two storage spaces: ping storage space and pong storage space. The flow of the template fusion unit is divided into three types according to the ping-pong property of the SRAM 308: input / output ping-pong (IO parity), input ping (input parity), and no ping (no parity). The input / output ping-pong can support the parallel of loading, computing, and storing, and in order to achieve the input / output ping-pong, the ping storage space and the pong storage space need to be completely equal, and are respectively used for loading and storing. The input ping only supports the parallel of storing and computing, and will additionally increase the carrying time on the SRAM 308, and compared with the input / output ping-pong, the ping storage space and the pong storage space do not need to be completely equal, but need to additionally allocate a cache equal to the size of the storage space for storing. The no ping means that the loading / storing and computing are serial, and the space does not need to be additionally allocated.

[0189] To achieve the first pipeline as mentioned above, the SRAM 308 of this embodiment has the same size of ping storage space and pong storage space to achieve the effect of input / output ping-pong. Again, taking the first sub-graph as an example, the first layer loading stage 1401, the first layer calculation stage 1402 and the first layer store-back stage 1403 are executed on the ping storage space, while the first layer loading stage 1407, the first layer calculation stage 1408 and the first layer store-back stage 1409 of the second sub-graph are executed on the pong storage space, and the first layer loading stage 1410, the first layer calculation stage 1411 and the first layer store-back stage 1412 of the third sub-graph are executed on the ping storage space, and so on. Figure 14 For illustration, the first layer loading stage 1401, the first layer calculation stage 1402 and the first layer store-back stage 1403 of the first sub-graph are executed on the ping storage space, while the first layer loading stage 1407, the first layer calculation stage 1408 and the first layer store-back stage 1409 of the second sub-graph are executed on the pong storage space, and the first layer loading stage 1410, the first layer calculation stage 1411 and the first layer store-back stage 1412 of the third sub-graph are executed on the ping storage space, and so on.

[0190] The pipeline of the second layer means that the second layer loading stage 1404, the second layer calculation stage 1405 and the second layer store-back stage 1406 can be simultaneously parallel. Taking the same processor core 306 as an example which is intended to process the first sub-graph, the second sub-graph and the third sub-graph. First, the first sub-graph is loaded into the NRAM 431 in the second layer loading stage 1404, and the required weight is loaded into the WRAM 432. Then the first sub-graph is calculated and reduced in the second layer calculation stage 1405, and the reduced intermediate result is carried back to the NRAM 431, at the same time, the second sub-graph is loaded into the NRAM 431 in the second layer loading stage 1413, and the required weight is loaded into the WRAM 432. Finally, the first intermediate result is stored back to the SRAM 308 in the second layer store-back stage 1406, at the same time, the second sub-graph is calculated and reduced in the second layer calculation stage 1414, and the reduced intermediate result is carried back to the NRAM 431, and the third sub-graph is loaded into the NRAM 431 in the second layer loading stage 1415, and the required weight is loaded into the WRAM 432.

[0191] Considering that the time to complete the tasks of each cluster 305 is naturally inconsistent, the synchronization module 304 of this embodiment uses the synchronization barrier instruction to synchronize the time to complete the tasks to avoid timing errors.

[0192] In one application scenario, the embodiment does not turn on weight replacement, that is, when a sub-graph is calculated in the foregoing pipelining process, the weight of the next sub-graph is broadcast synchronously, so that the WRAM 432 stores the weights of two adjacent sub-graphs at the same time. Since the spaces of multiple sub-graphs in the WRAM 432 affect each other, the spaces occupied by the weights of the two adjacent sub-graphs in the WRAM 432 are greater than the total of the weights of the two adjacent sub-graphs. For the SRAM 308, in order to facilitate direct broadcasting instead of multiple accesses to the DRAM 204 in the case of multiple batches, the processing device 203 needs to allocate multiple spaces for storing weights to the SRAM 308, and these weights are always resident in the SRAM 308. If the template fusion unit includes multiple convolution layers, the space of the SRAM 308 or the WRAM 432 can not be large enough to load all the weights, so that multiple layers cannot be fused.

[0193] When the processing device 203 determines that the template fusion unit includes multiple convolution layers, the embodiment switches to the weight replacement mode. The weight replacement refers to that when a sub-graph is calculated, the processing device 203 loads the weight of the next sub-graph from the DRAM 204 to the SRAM 308. When the next sub-graph is calculated, the broadcast bus 309 broadcasts the weight to the WRAM 432. Compared with not turning on the weight replacement, although the weight replacement increases the number of off-chip on-chip accesses, the SRAM 308 only needs to be configured with the storage space of the maximum weight, and at any time, the SRAM 308 only stores the weight of one sub-graph, and this space can be reused. The rules of the fusion strategy can include the switching of the weight replacement. When the total weight of the template fusion unit is small, the weight replacement is not used to strive for faster calculation speed; when the total weight is large, the weight replacement is used to strive for more fused layers.

[0194] The embodiment establishes two three-level pipelines based on the three-level operation hierarchy of the system-on-chip-cluster-processor core and the three-layer memory design of the DRAM-SRAM-NRAM / WRAM, fully utilizes the hardware resources, and improves the neural network calculation efficiency.

[0195] Figure 15Fig. 16 shows a flow chart of a two-layer three-stage pipeline according to another embodiment. In step 1601, a first on-chip graph is loaded. In step 1602, the first on-chip graph is computed synchronously and a first intermediate result is generated, and a second on-chip graph is loaded. In step 1603, the first intermediate result is stored back synchronously, the second on-chip graph is computed and a second intermediate result is generated, and a third on-chip graph is loaded. Step 1602 further comprises the following steps. In step 1604, a first sub-graph is loaded, wherein the first sub-graph is at least a portion of the first on-chip graph. In step 1605, the first sub-graph is computed synchronously and a first intermediate result is generated, and a second sub-graph is loaded, wherein the second sub-graph is at least a portion of the first on-chip graph. In step 1606, the first intermediate result is stored back synchronously, the second sub-graph is computed, and a third sub-graph is loaded, wherein the third sub-graph is also at least a portion of the first on-chip graph.

[0196] If in step 1502, it is determined that all sub-template fusion units have finished execution, then step 1506 is executed to reduce all intermediate results to generate a computation result. In step 1507, the computation result is moved from SRAM 308 to DRAM 204. At this point, the task of the template fusion unit is completed.

[0197] Figure 16 Fig. 16 shows a flow chart of a two-layer three-stage pipeline according to another embodiment. In step 1601, a first on-chip graph is loaded. In step 1602, the first on-chip graph is computed synchronously and a first intermediate result is generated, and a second on-chip graph is loaded. In step 1603, the first intermediate result is stored back synchronously, the second on-chip graph is computed and a second intermediate result is generated, and a third on-chip graph is loaded. Step 1602 further comprises the following steps. In step 1604, a first sub-graph is loaded, wherein the first sub-graph is at least a portion of the first on-chip graph. In step 1605, the first sub-graph is computed synchronously and a first intermediate result is generated, and a second sub-graph is loaded, wherein the second sub-graph is at least a portion of the first on-chip graph. In step 1606, the first intermediate result is stored back synchronously, the second sub-graph is computed, and a third sub-graph is loaded, wherein the third sub-graph is also at least a portion of the first on-chip graph.

[0198] Another embodiment of the present disclosure is a computer readable storage medium having stored thereon computer program codes for dynamically fusing a neural network according to a fusion strategy, which, when executed by a processor, performs the steps of Figure 10 、 Figure 11 、 Figure 12 、 Figure 15 ,Figure 16 The method described.

[0199] This disclosure dynamically determines the template fusion unit by setting a fusion strategy, fusions multiple layers in the neural network to form a new custom layer, and loads the data required to compute the template fusion unit at once to reduce input / output overhead.

[0200] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0201] It should be noted that, for the purpose of clarity, the disclosure describes some methods and embodiments thereof as a series of acts and / or combinations thereof, but the skilled person would understand that the scope of the disclosure is not limited to the order of the acts described. Accordingly, some of the acts can be performed in other orders or even concurrently with one another. Furthermore, not all of the acts described need to be performed in order to implement some embodiments of the disclosure. In addition, the description of several embodiments of the disclosure have different focuses. In view of this, the skilled person can understand that some parts not described in detail in some embodiments of the disclosure can also be seen from the relevant description of other embodiments.

[0202] In terms of specific implementation, based on the disclosure and teaching of the disclosure, the skilled person can understand that the several embodiments disclosed by the disclosure can also be implemented in other manners not disclosed herein. For example, as to each unit in the foregoing electronic device or apparatus embodiments, the division thereof is made on the basis of logical functions, and the actual implementation can be different. For example, the units or components can be combined or integrated into another system, or some features or functions of the units or components can be selectively disabled. As to the connection relationship between different units or components, the connection discussed above can be direct or indirect coupling between units or components. In some scenarios, the foregoing direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0203] In the disclosure, the units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units. The foregoing components or units can be located in the same place or distributed on multiple network units. In addition, according to actual needs, some or all of the units can be selected to achieve the purpose of the scheme described in the embodiments of the disclosure. In addition, in some scenarios, multiple units in the embodiments of the disclosure can be integrated into one unit or each unit physically exists separately.

[0204] In some implementation scenarios, the above-mentioned integrated units can be implemented in the form of software program modules. If implemented in the form of software program modules and sold or used as independent products, the integrated units can be stored in a computer-readable memory. Based on this, when the schemes of the present disclosure are embodied in the form of software products (for example, computer-readable storage media), the software products can be stored in the memory, which can include a number of instructions to make a computer device (for example, a personal computer, a server or a network device, etc.) execute part or all of the steps of the method described in the embodiments of the present disclosure. The aforementioned memory can include, but is not limited to, a U disk, a flash disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0205] In some other implementation scenarios, the above-mentioned integrated units can also be implemented in the form of hardware, that is, specific hardware circuits, which can include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit can include, but is not limited to, physical devices, and the physical devices can include, but are not limited to, transistors or memristors, etc. In view of this, various devices described herein (for example, computing devices or other processing devices) can be implemented by appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs and ASICs, etc. Further, the aforementioned storage units or storage devices can be any appropriate storage medium (including magnetic storage media or magneto-optical storage media, etc.), which can be, for example, resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high bandwidth memory (HBM), hybrid memory cube (HMC), ROM and RAM, etc.

[0206] The above has carried out the detailed introduction to the embodiment of the disclosure, the principle and implementation mode of the disclosure are described in the text by applying specific examples, the above embodiment is only used to help understanding the method of the disclosure and its core idea; at the same time, for the general technical personnel in the art, according to the idea of the disclosure, there will be changes in specific implementation mode and application range, and the above is described, the content of the specification should not be understood as the limitation of the disclosure.

Claims

1. A computing device connected to off-chip memory, the computing device comprising a plurality of clusters, each cluster comprising: a shared memory unit; a first direct memory access module to load a first on-chip graph from the off-chip memory into the shared memory unit; and a plurality of processor cores to respectively compute on the first on-chip graph and generate a first computation result; wherein the first direct memory access module stores the first computation result from the shared memory unit back to the off-chip memory; a template fusion unit to fuse a plurality of layers of a neural network into one layer; a feature map of the template fusion unit loaded into the shared memory unit is referred to as an on-chip graph; a processing device to select a starting layer according to a fusion strategy and establish a template fusion unit; wherein the fusion strategy is that a sum of redundancies resulting from splitting into the on-chip graph does not exceed a percentage threshold, and when the processing device determines that the fusion strategy is not satisfied, the processing device stops fusion; the sum of redundancies is a difference between a memory access amount and a theoretical memory access amount. 2.The computing device of claim 1, wherein when the plurality of processor cores compute on the first on-chip graph to generate the first computation result, the first direct memory access module loads a second on-chip graph from the off-chip memory into the shared memory unit. 3.The computing device of claim 2, wherein when the first computation result is stored back to the off-chip memory, the plurality of processor cores respectively compute on the second on-chip graph and generate a second computation result, and the first direct memory access module loads a third on-chip graph from the off-chip memory into the shared memory unit. 4.The computing device of claim 3, wherein each cluster further comprises a storage core to split the first on-chip graph into a plurality of first sub-graphs and broadcast to the plurality of processor cores for computation. 5.The computing device of claim 4, wherein the processor core comprises: a neuron storage unit; a second direct memory access module to load one of the plurality of first sub-graphs from the shared memory unit into the neuron storage unit; a computation module to compute on the one of the plurality of first sub-graphs to generate a first intermediate result; wherein the second direct memory access module stores the first intermediate result back to the shared memory unit. 6.The computing device of claim 5, wherein one of the plurality of processor cores reduces each first intermediate result into the first computation result. 7.The computing device of claim 6, wherein when the computation module computes on the first sub-graph, the storage core splits the second on-chip graph into a plurality of second sub-graphs and broadcasts to the plurality of processor cores. ​ 8. The computing device of claim 7, wherein when the second DMA module stores the first intermediate result back into the shared memory unit, the computation module performs computation according to the second sub-graph to generate a second intermediate result, the storage core partitions the third on-chip unit graph into a plurality of third sub-graphs, and broadcasts to the plurality of processor cores.

9. The computing device of claim 3, wherein the shared memory unit comprises: a ping memory space to store the first on-chip unit graph, the first computation result, and a third on-chip unit graph; and a pong memory space to store the second on-chip unit graph and the second computation result.

10. The computing device of claim 9, wherein the ping memory space and the pong memory space have the same size.

11. The computing device of claim 5, wherein the second DMA module is a transfer DMA module.

12. The computing device of claim 1, wherein the first DMA module is a global DMA module.

13. An integrated circuit device comprising the computing device of any one of claims 1-12.

14. A board card comprising the integrated circuit device of claim 13.

15. A method for computing a neural network using a computing device, the computing device comprising a plurality of clusters, each cluster comprising a shared memory unit, the method comprising: a first layer loading stage of loading a first on-chip unit graph from off-chip memory into the shared memory unit; a first layer computation stage of computing the first on-chip unit graph respectively and generating a first computation result; and a first layer storing back stage of storing back the first computation result from the shared memory unit into the off-chip memory; a template fusion unit fusing a plurality of layers of the neural network into one layer; a feature map of the template fusion unit being loaded into the shared memory unit is referred to as an on-chip unit graph; a processing device selecting a starting layer according to a fusion strategy and establishing a template fusion unit; wherein the fusion strategy is that a sum of redundancies generated by splitting into the on-chip unit graph does not exceed a percentage threshold, when the processing device determines that the fusion strategy is not satisfied, the processing device stops fusion; the sum of redundancies is a difference between a memory access amount and a theoretical memory access amount.

16. The method of claim 15, wherein when the first on-chip unit graph performs the first layer computation stage, a second on-chip unit graph performs the first layer loading stage and is loaded from the off-chip memory into the shared memory unit.

17. The method of claim 16, wherein when the first computation result performs the first layer storing back stage, the second on-chip unit graph performs the first layer computation stage and generates a second computation result, and a third on-chip unit graph performs the first layer loading stage and is loaded from the off-chip memory into the shared memory unit.

18. The method of claim 17, wherein each cluster further comprises a plurality of processor cores, each processor core comprising a neuron memory unit, the method further comprising: ​ ​ a second layer load stage of splitting the first on-chip graph into a plurality of first sub-graphs, loading one of the plurality of first sub-graphs from the shared memory unit into the neuron memory unit; a second layer compute stage of performing computations based on the one of the plurality of first sub-graphs to generate a first intermediate result; and a second layer store-back stage of storing the first intermediate result back into the shared memory unit.

19. The method of claim 18, wherein while the first sub-graph is performing the second layer compute stage, the second on-chip graph is performing the second layer load stage, splitting the second on-chip graph into a plurality of second sub-graphs, loading one of the plurality of second sub-graphs from the shared memory unit into the neuron memory unit.

20. The method of claim 19, wherein while the first intermediate result is performing the second layer store-back stage, the second sub-graph is performing the second layer compute stage to generate a second intermediate result, the third on-chip graph is performing the second layer load stage, splitting the third on-chip graph into a plurality of third sub-graphs, loading one of the plurality of third sub-graphs from the shared memory unit into the neuron memory unit.

21. A computer readable storage medium having stored thereon computer program code for computing a neural network using a computing device, which when executed by a processing device, performs the method of any one of claims 15 to 20.

Images