Computing devices, boards, methods, and readable storage media for computing neural networks

Abstract

The present disclosure relates to a computing apparatus, a board card, a method and a readable storage medium for computing a neural network according to a template fusion unit, wherein the computing apparatus of the present disclosure is included in an integrated circuit apparatus, which includes a general interconnection interface and other processing apparatuses. The computing apparatus interacts with the other processing apparatuses to jointly complete a user-specified computing operation. The integrated circuit apparatus can further include a storage apparatus connected with the computing apparatus and the other processing apparatuses respectively for data storage of the computing apparatus and the other processing apparatuses.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to the field of neural networks. More particularly, the present disclosure relates to a computing device, a board card, a method and a readable storage medium for computing a neural network according to a template fusion unit. BACKGROUND

[0002] A neural network is a system of multiple neurons connected according to certain rules, which is generally composed of four kinds of layer structures: an input layer, a convolution layer, a pooling layer, and a fully connected layer.

[0003] The input layer is to intercept part of the information from the input data and present it in the form of a feature matrix, which contains the features corresponding to the part of information. The convolution layer is configured to receive the feature matrix from the input layer and extract features from the input data through convolution operation. In practical application, multiple convolution layers can be built. The pooling layer is configured to replace a region of data with a value, which is usually the maximum value or the average value of all values in the region. Through pooling, the model size can be reduced and the calculation speed can be improved without losing too much information. The fully connected layer acts as a classifier in the entire convolutional neural network, which is equivalent to a feature space transformation. It extracts and integrates all useful information from the previous layers, compares the information based on different classifications, and judges whether the input data is similar to the target of comparison.

[0004] With the development of technology, the number of layers of neural networks is increasing. For example, in the classic VGG architecture, VGG-A has 11 weight layers, VGG-B has 13 weight layers, VGG-C has 16 weight layers, VGG-D has 16 weight layers, and VGG-E has 19 weight layers. Among them, the convolution layer and the fully connected layer are referred to as weight layers. Some neural networks have hundreds of layers. Moreover, with the increase in the number of layers, the number of parameters of the neural network also increases exponentially, for example, AlexNet has 60 million parameters participating in the calculation.

[0005] Multiple layers and multiple parameters require a large number of on-chip and off-chip input / output accesses, which will consume many resources and delay the operation time. Therefore, a mechanism to reduce input / output access is urgently needed in the field of artificial intelligence. SUMMARY

[0006] To at least partially solve the technical problems mentioned in the background, the present disclosure provides a computing device, a board card, a method and a readable storage medium for computing a neural network according to a template fusion unit.

[0007] In one aspect, the disclosure discloses a computing device for computing a neural network according to a template fusion unit that fuses multiple layers of the neural network, the computing device comprising a plurality of clusters, each cluster comprising a shared memory unit and a plurality of processor cores. The shared memory unit is configured to load an on-chip unit graph from an off-chip memory. Each processor core comprises a neuron memory unit configured to load a subgraph from the shared memory unit, the subgraph being a part of the on-chip unit graph, and an operation module configured to compute the subgraph and generate an intermediate result. The intermediate results are reduced across the plurality of processor cores to generate a computation result corresponding to the on-chip unit graph, and the shared memory unit is configured to store the computation result back to the off-chip memory.

[0008] In another aspect, the disclosure discloses an integrated circuit device for computing a neural network according to a template fusion unit, comprising an off-chip memory, a processing device, and a computing device. The off-chip memory is configured to store feature maps of the neural network; the processing device is configured to fuse multiple layers of the neural network according to a fusion strategy to generate the template fusion unit, and to split the feature maps into on-chip unit graphs. The computing device comprises a plurality of clusters, each cluster comprising a shared memory unit and a plurality of processor cores. The shared memory unit is configured to load an on-chip unit graph from an off-chip memory. Each processor core comprises a neuron memory unit configured to load a subgraph from the shared memory unit, the subgraph being a part of the on-chip unit graph, and an operation module configured to compute the subgraph and generate an intermediate result. The intermediate results are reduced across the plurality of processor cores to generate a computation result corresponding to the on-chip unit graph, and the shared memory unit is configured to store the computation result back to the off-chip memory.

[0009] In another aspect, the disclosure discloses a board card comprising the integrated circuit device according to the foregoing.

[0010] In another aspect, the disclosure discloses a method for computing a neural network according to a template fusion unit that fuses multiple layers of the neural network, the method comprising: loading the on-chip unit graph; loading a subgraph, the subgraph being a part of the on-chip unit graph; computing the subgraph and generating an intermediate result; reducing the intermediate results to generate a computation result corresponding to the on-chip unit graph; and storing the computation result back.

[0011] In another aspect, the disclosure discloses a computer readable storage medium having stored thereon computer program code for computing a neural network according to a template fusion unit, the computer program code, when executed by a processing device, performs the method according to the foregoing.

[0012] The present disclosure relates to a scheme of fusing a template unit to compute a neural network, neural network computation based on a template fusion unit to reduce input / output overhead. BRIEF DESCRIPTION OF DRAWINGS

[0013] The above and other objects, features and advantages of the example embodiments of the present disclosure will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0014] Figure 1 is a structural diagram of a board card showing an embodiment of the present disclosure;

[0015] Figure 2 is a structural diagram of an integrated circuit device showing an embodiment of the present disclosure;

[0016] Figure 3 is an internal structural diagram of a computing device showing an embodiment of the present disclosure;

[0017] Figure 4 is an internal structural diagram of a processor core showing an embodiment of the present disclosure;

[0018] Figure 5 is a diagram showing when one processor core wants to write data to another processor core of a cluster;

[0019] Figure 6 is a diagram showing an AlexNet model;

[0020] Figure 7 is a diagram showing an exemplary neural network model;

[0021] Figure 8 is a diagram showing two convolution layers fused together in an embodiment of the present disclosure;

[0022] Figure 9 is a diagram showing NCHW and NHWC formats;

[0023] Figure 10 is a flowchart showing a neural network computation using a template fusion unit in an embodiment of the present disclosure;

[0024] Figure 11 is a flowchart showing dynamic fusion of a neural network according to a fusion policy in an embodiment of the present disclosure;

[0025] Figure 12 is a flowchart showing a neural network computation using a template fusion unit in an embodiment of the present disclosure;

[0026] Figure 13is a schematic diagram showing a neural network model with a block structure;

[0027] Figure 14 is a flowchart showing a neural network calculated based on executable instructions according to embodiments of the present disclosure;

[0028] Figure 15 is a schematic diagram showing a ring all-reduce framework;

[0029] Figure 16 is a schematic diagram showing a plurality of clusters in a logical loop;

[0030] Figure 17A is a schematic diagram showing a first iteration of a ring all-reduce;

[0031] Figure 17B is a schematic diagram showing a second iteration of a ring all-reduce;

[0032] Figure 17C is a schematic diagram showing a third iteration of a ring all-reduce;

[0033] Figure 18A is a schematic diagram showing a ring all-reduce with each cluster having a processor core performing a full reduction calculation; and

[0034] Figure 18B is a schematic diagram showing a ring all-reduce after performing a full calculation. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of, rather than all of, the embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present disclosure.

[0036] It should be understood that the terms “first”, “second”, “third”, and “fourth” and the like in the claims, specification, and drawings of the present disclosure are used to distinguish different objects, and are not used to describe a particular order. The terms “include” and “contain” used in the specification and claims of the present disclosure indicate the presence of the described features, whole, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, whole, steps, operations, elements, components, and / or sets thereof.

[0037] It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used in this disclosure and the claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or," as used herein, refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0038] As used in this specification and claims, the terms "if' and "when" can, depending upon the context, be interpreted to mean either "when... or "as soon as... or "upon determining," or "in response to determining" or "in response to detecting."

[0039] A detailed description of specific embodiments of the disclosure follows.

[0040] A neural network is composed of input layer, convolution layer, activation function, pooling layer, fully connected layer, less than several layers, more than hundreds of layers, each layer executes an operator, for example, the convolution layer executes the convolution operator, how many layers need to execute how many operators. In the present disclosure, when referring to a specific layer, it means the corresponding operator of the layer.

[0041] In the process of neural network calculation, the input information and the output results of each layer of the model are different in each inference calculation, they are regarded as variable data, and the variable data is generally represented by feature map (matrix). In the present disclosure, the input information and the input map of each layer of the model of the entire neural network model are collectively referred to as feature map, and once the feature map is loaded onto the on-chip memory component, it is referred to as on-chip unit map in the present disclosure. The parameters of the trained network model are generally not frequently changed after the training is stable, or the network topology structure and hardware parameters are determined and can be compiled to generate, and they will not be changed in the calculation process, so they can be regarded as constant data, including but not limited to weight, bias, device hardware instruction, mean and variance of batch normalization (batchnorm), etc. In the present disclosure, all constant data are uniformly represented by weight. When referring to "data" in the present disclosure, it generally refers to the graph structure that allows the operation of the corresponding operator in the neural network model to be fused together according to the fusion strategy. The variable data and constant data involved in this graph structure are also the feature map plus the corresponding weight.

[0042] Figure 1 A structural schematic diagram of a board card 10 according to an embodiment of the present disclosure is shown. As shown in FIG. 1, the board card 10 includes a plurality of processing units 100, a plurality of memory units 200, a plurality of input / output units 300, a plurality of communication units 400, and a plurality of power supply units 500. Figure 1As shown, the board card 10 includes a chip 101, which is a system on chip (SoC) integrated with one or more combined processing devices, which is an artificial intelligence operation unit to support various deep learning and machine learning algorithms to meet the intelligent processing needs in complex scenarios in the fields of computer vision, speech, natural language processing, data mining, etc. In particular, deep learning technology is widely used in cloud intelligent fields. A significant feature of cloud intelligent applications is the large amount of input data, which has high requirements for the storage capacity and computing capacity of the platform. The board card 10 of this embodiment is suitable for cloud intelligent applications and has a large off-chip storage, on-chip storage and a large amount of computing capacity.

[0043] The chip 101 is connected with an external device 103 through an external interface device 102. The external device 103 is, for example, a server, a computer, a camera, a display, a mouse, a keyboard, a network card or a wifi interface, etc. The data to be processed can be transmitted from the external device 103 to the chip 101 through the external interface device 102. The computing result of the chip 101 can be transmitted back to the external device 103 through the external interface device 102. According to different application scenarios, the external interface device 102 can have different interface forms, such as a PCIe interface, etc.

[0044] The board card 10 further includes a storage device 104 for storing data, which includes one or more storage units 105. The storage device 104 is connected and transmits data with the control device 106 and the chip 101 through a bus. The control device 106 in the board card 10 is configured to regulate the state of the chip 101. For this purpose, in one application scenario, the control device 106 can include a micro controller unit (MCU).

[0045] Figure 2 is a structural diagram of the combined processing device in the chip 101 of this embodiment. As Figure 2 shown in the figure, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203 and a DRAM 204.

[0046] The computing device 201 is configured to perform user-specified operations, mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations, which can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0047] The interface device 202 is used to transmit data and control instructions between the computing device 201 and the processing device 203. For example, the computing device 201 can obtain input data from the processing device 203 via the interface device 202 and write the data into the storage device on the computing device 201. Further, the computing device 201 can obtain control instructions from the processing device 203 via the interface device 202 and write the control instructions into the control buffer on the computing device 201. Alternatively or additionally, the interface device 202 can also read data from the storage device of the computing device 201 and transmit the data to the processing device 203.

[0048] The processing device 203 is a general-purpose processing device, which performs basic controls including but not limited to data transfer, starting and / or stopping of the computing device 201, etc. Depending on the implementation, the processing device 203 can be one or more types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), or other general-purpose and / or special-purpose processors, including but not limited to a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc., and the number thereof can be determined according to actual needs. As mentioned above, only in terms of the computing device 201 of the present disclosure, it can be considered as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0049] The DRAM 204 is used to store data to be processed, is a DDR memory, and usually has a size of 16G or more, and is used to save data of the computing device 201 and / or the processing device 203.

[0050] Figure 3 The internal structure of the computing device 201 is shown. The computing device 201 is used to process input data such as computer vision, speech, natural language, data mining, etc. The computing device 201 in the figure adopts a multi-core hierarchical structure design. The computing device 201 is a system-on-a-chip, which includes multiple clusters, and each cluster includes multiple processor cores. In other words, the computing device 201 is composed of a system-on-a-chip-cluster-processor core hierarchy.

[0051] In terms of the system-on-a-chip hierarchy, as shown in the figure, Figure 3As shown, the computing device 201 includes an external storage controller 301, a peripheral communication module 302, an on-chip interconnect module 303, a synchronization module 304, and multiple clusters 305.

[0052] There can be multiple external storage controllers 301; two are shown as an example in the figure. These controllers are used to respond to access requests from the processor core to access external storage devices, such as… Figure 2 The DRAM 204 in the chip allows data to be read from or written to external devices. The peripheral communication module 302 receives control signals from the processing device 203 via the interface device 202, initiating the computing device 201 to execute tasks. The on-chip interconnect module 303 connects the external storage controller 301, the peripheral communication module 302, and multiple clusters 305 to transmit data and control signals between modules. The synchronization module 304 is a global barrier controller (GBC) used to coordinate the working progress of each cluster and ensure information synchronization. The multiple clusters 305 are the computing core of the computing device 201. Four are shown exemplary in the figure; however, with hardware development, the computing device 201 disclosed herein may also include 8, 16, 64, or even more clusters 305. The clusters 305 are used to efficiently execute deep learning algorithms.

[0053] From the perspective of cluster hierarchy, such as Figure 3 As shown, each cluster 305 includes multiple processor cores (IPU cores) 306 and one memory core (MEM core) 307.

[0054] Four processor cores 306 are shown in the figure as an example; this disclosure does not limit the number of processor cores 306. Its internal architecture is as follows: Figure 4 As shown. Each processor core 306 includes three main modules: a control module 41, an arithmetic module 42, and a storage module 43.

[0055] The control module 41 coordinates and controls the operation of the computation module 42 and the storage module 43 to complete the deep learning task. It includes an instruction fetch unit (IFU) 411 and an instruction decode unit (IDU) 412. The instruction fetch unit 411 fetches instructions from the processing device 203, and the instruction decode unit 412 decodes the fetched instructions and sends the decoding result as control information to the computation module 42 and the storage module 43.

[0056] The operation module 42 includes a vector operation unit 421 and a matrix operation unit 422. The vector operation unit 421 is configured to perform vector operations, and can support complex operations such as vector multiplication, addition, nonlinear transformation, etc. The matrix operation unit 422 is responsible for the core calculation of the deep learning algorithm, i.e. matrix multiplication and convolution.

[0057] The storage module 43 is configured to store or transfer related data, including a neuron storage unit (NRAM) 431, a weight storage unit (WRAM) 432, an input / output direct memory access module (IODMA) 433, and a move direct memory access module (MVDMA) 434. The NRAM 431 is configured to store feature maps and intermediate results calculated by the processor core 306. The WRAM 432 is configured to store weights of the deep learning network. The IODMA 433 controls access of the NRAM 431 / WRAM 432 to the DRAM 204 through the broadcast bus 309. The MVDMA 434 is configured to control access of the NRAM 431 / WRAM 432 to the SRAM 308.

[0058] Returning to Figure 3 , the storage core 307 is mainly configured to store and communicate, i.e. store shared data or intermediate results among the processor cores 306, and perform communication between the execution cluster 305 and the DRAM 204, communication between the execution clusters 305, communication between the processor cores 306, etc. In other embodiments, the storage core 307 has the ability of scalar operation, and is configured to perform scalar operations.

[0059] The storage core 307 includes a shared storage unit (SRAM) 308, a broadcast bus 309, a cluster direct memory access module (CDMA) 310, and a global direct memory access module (GDMA) 311. The SRAM 308 serves as a high-performance data transfer station. Data reused among different processor cores 306 in the same execution cluster 305 does not need to be obtained from the DRAM 204 by each processor core 306, but is transferred among the processor cores 306 through the SRAM 308. The storage core 307 only needs to quickly distribute the reused data from the SRAM 308 to the multiple processor cores 306, so as to improve the efficiency of inter-core communication, and greatly reduce input / output access on-chip and off-chip.

[0060] The broadcast bus 309, the CDMA 310 and the GDMA 311 are respectively used to perform communication between the processor cores 306, communication between the clusters 305 and data transmission between the clusters 305 and the DRAM 204. The following will be described respectively.

[0061] The broadcast bus 309 is used to complete high-speed communication between the processor cores 306 in the cluster 305. The broadcast bus 309 of the embodiment supports inter-core communication modes including unicast, multicast and broadcast. Unicast refers to point-to-point (i.e. single processor core to single processor core) data transmission, multicast is a communication mode of transmitting a piece of data from the SRAM 308 to a specific few processor cores 306, and broadcast is a communication mode of transmitting a piece of data from the SRAM 308 to all processor cores 306, which is a special case of multicast.

[0062] The CDMA 310 is used to control access to the SRAM 308 between different clusters 305 in the same computing device 201. Figure 5 A schematic diagram is shown when a processor core wants to write data to a processor core of another cluster, to illustrate the working principle of the CDMA 310. In this application scenario, the same computing device includes multiple clusters, for the convenience of description, only cluster 0 and cluster 1 are shown in the figure, and cluster 0 and cluster 1 respectively include multiple processor cores. Similarly, for the convenience of description, only processor core 0 of cluster 0 and only processor core 1 of cluster 1 are shown in the figure. The processor core 0 wants to write data to the processor core 1.

[0063] Firstly, the processor core 0 sends a unicast write request to write data into the local SRAM 0, the CDMA 0 acts as a master and the CDMA 1 acts as a slave, the master pushes the write request to the slave, that is, the master sends a write address AW and write data W, and transmits the data to the SRAM 1 of the cluster 1, then the slave sends a write response B as a response, and finally the processor core 1 of the cluster 1 sends a unicast read request to read the data from the SRAM 1.

[0064] Back to Figure 3The GDMA 311, in conjunction with the external memory controller 301, controls memory access from the SRAM 308 to the DRAM 204 of the cluster 305, or reads data from the DRAM 204 into the SRAM 308. As previously described, communication between the DRAM 204 and the NRAM 431 or WRAM 432 can be achieved through two channels. The first channel is a direct connection between the DRAM 204 and the NRAM 431 or WRAM 432 via the IODAM 433; the second channel involves first transmitting data between the DRAM 204 and SRAM 308 via the GDMA 311, and then transmitting data between the SRAM 308 and the NRAM 431 or WRAM 432 via the MVDMA 434. Although the second channel appears to require more components and has a longer data flow, in some embodiments, the bandwidth of the second channel is actually much greater than that of the first channel. Therefore, communication between the DRAM 204 and the NRAM 431 or WRAM 432 may be more efficient via the second channel. The embodiments disclosed herein allow for the selection of data transmission channels based on the hardware capabilities available.

[0065] In other embodiments, the functions of GDMA 311 and IODMA 433 can be integrated into the same component. For ease of description, this disclosure treats GDMA 311 and IODMA 433 as different components. For those skilled in the art, any component whose implemented functions and achieved technical effects are similar to those disclosed herein falls within the scope of protection of this disclosure. Furthermore, the functions of GDMA 311, IODMA 433, CDMA 310, and MVDMA 434 can also be implemented by the same component. Similarly, any component whose implemented functions and achieved technical effects are similar to those disclosed herein falls within the scope of protection of this disclosure.

[0066] The neural network structures related to this disclosure are divided into two categories: long-chain structures and block structures. Long-chain structures refer to neural network models composed of layers connected in a single chain, with each layer having only one input and one output, and the overall structure is single-branched, such as the VGG16 model or... Figure 6 The AlexNet model shown is an example. A block structure refers to a subnetwork in a neural network that has only one input and one output, but contains multiple branches. That is, some layers of the subnetwork have multiple inputs or outputs, such as the resblock structure of resnet50 and the block structure of inception_v3. Figure 7A schematic diagram showing an exemplary neural network model comprising subnetwork 701 and subnetwork 702. Subnetwork 701 has only one input and one output, which comprises first layer to sixth layer, the first layer has 2 outputs and the sixth layer has 2 inputs, thus subnetwork 701 comprises 2 branches, one branch is first layer→second layer→third layer→sixth layer, and the other branch is first layer→fourth layer→fifth layer→sixth layer, subnetwork 701 constitutes a block structure. Similarly, subnetwork 702 also constitutes a block structure.

[0067] In performing the computation of each layer of deep learning, a large amount of off-chip on-chip access is required, especially reading the input data from DRAM 204 into computing device 201, and storing the computation result of computing device 201 to DRAM 204. Such frequent access will consume a lot of hardware resources. In order to solve this problem, the present disclosure greatly reduces the off-chip on-chip data transmission by fusing adjacent layers of neural network.

[0068] Figure 8 A schematic diagram showing the fusion of two convolution layers is shown. The input of first layer convolution layer 810 is 7x7 feature map 801, and this layer convolves feature map 801 with a 3x3 kernel (not shown) to obtain the feature map 802 of first layer convolution layer 810. Among them, the values of 5x5 feature subgraph 804 will affect 3x3 feature subgraph 805. Assuming the stride is 1, after calculating 5x5 feature subgraph 804, first layer convolution layer 810 will continue to calculate 5x5 feature subgraph 806, and the values of 5x5 feature subgraph 806 will affect 3x3 feature subgraph 807.

[0069] In the calculation of second layer convolution layer 811, feature map 802 becomes the input of second layer convolution layer 811, and is also convolved with a 3x3 kernel to obtain feature map 803 of second layer convolution layer 811. Among them, the values of 3x3 feature subgraph 805 will affect 1x1 feature subgraph 808 in feature map 803. After calculating 3x3 feature subgraph 805, second layer convolution layer 811 will continue to calculate 3x3 feature subgraph 807, and the values of 3x3 feature subgraph 807 will affect 1x1 feature subgraph 809 in feature map 803.

[0070] If not fused, the computing device 201 reads the 5x5 feature map 804 from the DRAM 204 when performing the first layer convolution 810, stores the 3x3 feature map 805 back to the DRAM 204 after computation, then reads the 5x5 feature map 806 from the DRAM 204, stores the 3x3 feature map 807 to the DRAM 204 after computation. When performing the second layer convolution 811, the 3x3 feature map 805 is read from the DRAM 204, the 1x1 feature map 808 is stored to the DRAM 204 after computation, then the 3x3 feature map 807 is read from the DRAM 204, the 1x1 feature map 809 is stored to the DRAM 204 after computation. As can be seen from the above description, the feature map 802 is repeatedly read and stored on the off-chip on-chip as intermediate data, which occupies a lot of system resources.

[0071] If the first layer convolution 810 and the second layer convolution 811 are fused, that is, the feature map 802 is stored in the NRAM 431 (the weights of the first layer convolution 810 and the second layer convolution 811 can also be stored in the WRAM 432), the number of accesses between the computing device 201 and the DRAM 204 can be reduced, and the execution efficiency of the overall neural network can be improved. Since the feature maps participating in the fusion (such as the feature map 801, the feature map 802, and the feature map 803) look like an inverted pyramid in the context logic of the neural network model, it is called pyramid fusion.

[0072] Pyramid fusion is usually based on a specific convolution layer and a pooling layer in a neural network to perform fusion backward, that is, the starting layer of fusion is a convolution layer or a pooling layer, and a plurality of layers are fused backward according to the hardware conditions thereof, which can include a plurality of convolution layers and pooling layers. However, with the development of deep learning and neural networks, the order of layers becomes complex, for example, an activation layer is arranged in front of a convolution layer, and the activation layer should also be considered how to be fused with the convolution layer behind it. Therefore, in addition to simply fusing the convolution layer and the pooling layer as the core, the present disclosure provides various fusion methods, which do not necessarily take the convolution layer and the pooling layer as the core, but adopt a specific strategy to flexibly select each layer of the neural network for fusion, even if the layer is user-defined, as long as it meets the fusion strategy, it can be fused to optimize the overall performance.

[0073] Another embodiment of the present disclosure is a new fusion method, which uses the foregoing Figure 1 , Figure 2 , Figure 3 and Figure 4The hardware structure of the template fusion unit is shown in FIG. 2. The template fusion unit is implemented by a hardware structure, and the fusion is called template fusion unit (TFU). The template fusion unit mainly flexibly fuses multiple layers into one layer by a certain fusion strategy to reduce the input / output overhead of the network, which includes the aforementioned pyramid fusion and other fusion methods. The set of layers that are fused is called the template fusion unit, which can be regarded as a new layer or a custom layer.

[0074] In this embodiment, the feature maps, weights, etc. required by the template fusion unit are loaded from the DRAM 204 to the on-chip SRAM 308 at one time. After the feature maps are loaded into the SRAM 308, they are called on-chip unit maps. The on-chip unit maps are cut into sub-maps, and each time a sub-map is loaded from the SRAM 308 to the NRAM 431 of the processor core 306 assigned to calculate the sub-map, and the weights required for calculating the sub-map are also loaded from the SRAM 308 to the WRAM 432. After each sub-map is calculated, the corresponding intermediate result is obtained, which is stored back to the SRAM 308. After all the sub-maps are calculated, the calculation results are stored back to the DRAM 204 at one time. That is, the corresponding results obtained by the on-chip unit maps and the weights participating in the operator operation in the neural network model are transferred between the DRAM 204 and the SRAM 308, and the output (intermediate result) of the sub-map is transferred between the SRAM 308 and the NRAM 431. From the perspective of the computing device 201, the data loading of the template fusion unit is in units of on-chip unit maps, and the calculation is in units of sub-maps.

[0075] In more detail, the SRAM 308 is one of the important reference indicators of the fusion strategy, and its space size determines whether the template fusion unit is in a large map mode or a small map mode. The small map mode and the large map mode refer to whether a feature map stored in the DRAM 204 can be moved to the SRAM 308 for processing at one time. The processing device 203 compares the storage space required by the feature map with the available space of the SRAM 308. If the SRAM 308 is not enough, the feature map cannot be placed, and it is in the large map mode. If the SRAM 308 is sufficient to accommodate the entire feature map, it is in the small map mode. It should be particularly noted that in the large map mode, the on-chip unit map is only a part of the feature map; in the small map mode, if the available space of the SRAM 308 is large enough, or the feature map is small enough, the SRAM 308 can accommodate multiple feature maps at one time, that is, the on-chip unit map can include multiple feature maps.

[0076] If so, the feature map must be split to be loaded into the computing device 201. The processing device 203 will split the feature map on the DRAM 204 until a small enough on-chip unit graph is generated to meet the space requirement of the SRAM 308, so that the on-chip unit graph can be moved to the SRAM 308 for processing at one time. When the feature map is split, input-dependent operations and output-dependent operations may be generated.

[0077] Input-dependent operations refer to the fact that each on-chip unit graph after splitting at least partially overlaps, and each subset needs some additional copies of input to perform complete operations, resulting in data redundancy in the splitting operation. Data redundancy refers to the reuse of the same data in the system. Input-dependent operations are caused when the template fusion unit includes convolution, pooling, or matrix multiplication.

[0078] Output-dependent operations refer to the fact that after each subgraph outputs an intermediate result, it needs to be reduced to obtain the calculation result. Reduction refers to the fact that based on the understanding of the content of the on-chip unit graph itself, after splitting into subgraphs, the calculation is performed separately to reduce the calculation scale, so as to maximize the data amount under the premise of maintaining the original appearance of the on-chip unit graph as much as possible. The calculation result is dependent on each other when reduction is performed. Output-dependent operations are caused when the template fusion unit includes inner product, convolution, matrix multiplication, sorting, counting, etc.

[0079] The data format of the feature map that can be processed by this embodiment includes N, H, W, and C dimensions, where N represents batch, H represents height, W represents width, and C represents channel. Taking image data as an example, N indicates the number of images in this batch, H indicates the number of pixels in the vertical direction of the image, W indicates the number of pixels in the horizontal direction, and C indicates the number of channels (for example, the number of channels C of a black and white image is 1, and the number of channels C of an RGB color image is 3).

[0080] The order of these dimensions determines the composition of the data, and the common composition methods are NHWC and NCHW, Figure 9Figure 1 shows the difference between NCHW and NHWC format. This figure is an example of RGB color image. R represents red pixel, G represents green pixel, and B represents blue pixel. Sequence 91 is NCHW format. N is arranged in the outer layer, and pixels in each channel are arranged next to each other in the order of RGB. The offset of the element with coordinates (n, c, h, w) in the storage is ((n x C + c) x H + h) x W + w. Sequence 92 is NHWC format. C is arranged in the innermost layer, and RGB pixels corresponding to a spatial position in multiple channels are arranged next to each other. The figure also shows the positions of input pixel 901, input pixel 902, and input pixel 903 under different arrangement methods. The three input pixels 901, 902, and 903 together represent the color of a point in the image. The conversion method of the corresponding coordinate offset of the element with coordinates (n, c, h, w) is ((n x H + h) x W + w) x C + c. NHWC is closer to the BMP picture data storage format than NCHW. In the BMP format file, data is stored according to individual pixel points, and each pixel point stores color values of all channels. This makes it unnecessary to perform additional dimension conversion when reading the input picture. Therefore, the memory locality of NHWC is better, and each three input pixels can obtain an output pixel. NCHW must wait for all channel inputs to be ready before obtaining the final output result, which requires a large cache space.

[0081] In this embodiment, the data fusion neural network can be divided into a plurality of template fusion units according to layers of the data fusion neural network, Figure 10 A corresponding flowchart is shown in Figure 10.

[0082] In step 1001, the processing device 203 determines whether the required storage space of the feature map is greater than the available space of the SRAM 308. If yes, it means that the feature map cannot be loaded into the SRAM 308 at one time, and therefore step 1002 is performed to split the feature map. In this embodiment, the processing device 203 preferentially selects splitting in the N dimension, because no input or output dependent operation is generated. If splitting in the N dimension cannot meet the requirements, splitting in the H or W dimension is considered, which may generate input or output dependent operation. This embodiment also supports splitting in the C dimension, in particular, splitting in the Cout direction. In this way, one convolution is split into multiple convolutions through data optimization, so that the WRAM 432 can accommodate the weights, for example, the weights are split to four processor cores 306. Therefore, as long as splitting in a certain dimension is processable by the computing device 201, it is within the scope of the disclosure.

[0083] Further, the processing device 203 can sequentially split the feature map or weight in N, H, W dimensions with a certain granularity, which can be a fixed or variable ratio, or represented by a function. In one application scenario, the processing device 203 splits the feature map or weight from large to small. Taking the feature map as an example, first, the feature map with dimension NHWC is split into a feature map with dimension N1HWC and a feature map with dimension N2HWC in the N dimension, where the certain granularity is a fixed ratio, and N1 and N2 are each half of N. If it is still not small enough, the processing device 203 continues to split the feature map with dimension N1HWC into a feature map with dimension N1H1WC and a feature map with dimension N1H2WC in the H dimension, where H1 and H2 are each half of H. If it is still not small enough, the processing device 203 continues to split the feature map with dimension N1H1WC into a feature map with dimension N1H1W1C and a feature map with dimension N1H1W2C in the W dimension, where W1 and W2 are each half of W. The processing device 203 can continue to split in the N, W, H dimensions with smaller granularity, such as quarter, eighth, or sixteenth, until the feature map is small enough to be loaded into the on-chip unit graph of the SRAM 308.

[0084] It can be understood that the processing device 203 can continue to split in one dimension until it cannot be split, and then select another dimension to continue splitting. For example, if splitting in the H dimension continues, and the smallest unit still cannot be loaded into the SRAM 308, then splitting in the W dimension is selected until the smallest unit is obtained.

[0085] It should be noted that since the splitting is from large to small, when the split feature map meets the condition, the size of the required storage space is usually close to the available space of the SRAM 308. In other words, in the large graph mode, the DRAM 204 can only transfer one split feature map to the SRAM 308 at a time, but in the small graph mode, the space of the SRAM 308 can load multiple feature maps from the DRAM 204 at a time.

[0086] In another application scenario, the processing device 203 splits from small to large, and the certain granularity can also be a fixed or variable ratio, or represented by a function. For example, first, the feature map is split in the N dimension with a certain granularity of the smallest unit, i.e., 1xHxWxC. If the SRAM 308 can load, the processing device 203 continues to enlarge the split of the feature map, for example, to 2xHxWxC. If it can still be loaded, it continues to enlarge until n x H x W x C cannot be loaded, and the size of the on-chip unit graph is (n-1) x H x W x C.

[0087] If the required storage space of 1 x H x W x C already exceeds the available space of SRAM 308, the processing device 203 continues to split from another dimension, for example, from the H dimension, and then the processing device 203 judges 1 x 1 x W x C. If it is small enough, it increases along the H dimension until it finds 1 x (h-1) x W x C, which is just close to but not greater than the available space of SRAM 308. If it still exceeds the available space of SRAM 308, the processing device 203 continues to split from another dimension, for example, from the W dimension. In this way, the optimal input data that can be loaded into SRAM 308 at one time is found. Here, the optimal means that the required storage space of the on-chip unit graph is just close to but not greater than the available space of SRAM 308.

[0088] After the processing device 203 splits the feature map, it returns to step 1001 and judges whether the required storage space of the split feature map is still greater than the available space of SRAM 308. If so, it executes step 1002 again to continue to split down.

[0089] If the processing device 203 judges that the required storage space of the split feature map is not greater than the available space of SRAM 308, it means that SRAM 308 can load the split feature map at one time, and then it executes step 1003 to set the split feature map as the on-chip unit graph.

[0090] Finally, it executes step 1004 to determine the template fusion unit according to the size of the on-chip unit graph. This step will be described in detail later.

[0091] In other application scenarios, when the processing device 203 repeatedly executes steps 1001 and 1002 multiple times, it means that the required storage space of the split feature map is getting closer and closer to the available space of SRAM 308. For example, assuming that the required storage space of the feature map is 100k and the available space of SRAM 308 is 40k, in step 1001, the processing device 203 judges that the required storage space of the feature map is greater than the available space of SRAM 308, so it executes step 1002 to split it into half along the N dimension. At this time, the required storage space of the split feature map is 50k. Then it returns to step 1001, and the required storage space of the split feature map is still greater than the available space of SRAM 308, so it continues to execute step 1002 to split it into half again along the N dimension. At this time, the required storage space of the split feature map is 25k. Then it returns to step 1001, and the required storage space of the split feature map is less than the available space of SRAM 308, so it executes step 1003 to set the split feature map (with a size of 25k) as the on-chip unit graph.

[0092] The available space of the SRAM 308 is 40k, while the required storage space of the on-chip unit graph is 25k, leaving 15k of space idle. The reason for this is that the step 1002 splits in half each time, so that the last split is too large in granularity. This embodiment can gradually reduce the specific granularity of the split with the number of splits, so that the required storage space of the split on-chip unit graph is as close as possible to the available space of the SRAM 308. For example, the specific granularity can be set to one-half at the beginning, one-third next, and one-fourth at the end. Similarly, taking the feature map required storage space of 100k and the available space of the SRAM 308 of 40k as an example, in step 1001, the processing device 203 determines that the required storage space of the feature map is greater than the available space of the SRAM 308, so step 1002 is performed, the specific granularity is set to one-half, and the split feature map is 50k. Then, step 1001 is returned to, the required storage space of the split feature map is still greater than the available space of the SRAM 308, and step 1002 is continued to be performed. At this time, the specific granularity is adjusted to one-third, and the split feature map is 37.5k. Then, step 1001 is returned to, the required storage space of the split feature map is less than the available space of the SRAM 308, so step 1003 is performed, and the processing device 203 sets the split feature map (size 37.5k) as the on-chip unit graph. 37.5k is closer to 40k than 25k, and the latter way makes more efficient use of the available space of the SRAM 308. This embodiment does not limit the size of the specific granularity, which can be set according to the application scenario.

[0093] After determining the size of the on-chip unit graph, step 1004 is performed, which dynamically fuses the neural network according to the fusion strategy. Figure 11 This embodiment shows a method for dynamically fusing a neural network according to a fusion strategy.

[0094] In step 1101, the starting layer of the template fusion unit is selected according to the starting rule of the fusion strategy. The processing device 203 selects the starting layer of the template fusion unit according to the starting rule of the fusion strategy, that is, selects the layer to start fusion from among the layers that have not yet been fused in the neural network.

[0095] In one application scenario, the starting rule can be that the starting layer is the first un-fused layer in the neural network, and the processing device 203 searches for the first un-fused layer. Taking the AlexNet neural network model of Figure 6 For example, the AlexNet neural network model has 23 layers, and suppose that the first layer to the fifth layer have been fused, then when the starting rule is that the starting layer is the first un-fused layer in the neural network, the processing device 203 selects the ReLU activation layer of the sixth layer as the starting layer and fuses backward (i.e., in the direction of the seventh layer). Note that under this starting rule, the starting layer does not have to be a convolutional layer or a pooling layer.

[0096] In another application scenario, considering that convolution and pooling layers consume the most input / output resources, the starting rule is that the starting layer is the first unmerged convolution or pooling layer, and the processing device 203 first finds all the convolution and pooling layers of the unmerged layers in the neural network model, and then merges from the first unmerged convolution or pooling layer backward. For example, in the AlexNet neural network model of Figure 6 , assuming that the first layer to the ninth layer have been merged, the processing device 203 finds all the convolution and pooling layers of the unmerged layers in the neural network model, that is, the eleventh layer, the thirteenth layer, and the fifteenth layer, and then merges from the first unmerged convolution or pooling layer, that is, the starting layer is the eleventh layer.

[0097] In step 1102, the starting layer is taken as the reference for merging, and all the rules of the merging strategy are checked one by one to establish the template merging unit. The processing device 203 takes the starting layer as the reference for merging, and checks all the rules of the merging strategy one by one to establish the template merging unit. On the premise that all the rules are met, the hardware resources of the computing device 201 are sufficient to support the loading of the data required by the computing template merging unit at one time, and then the neural network computing is performed according to the template merging unit. In addition to the aforementioned starting rule, the merging strategy exemplarily includes the following rules:

[0098] Rule one: backward merging

[0099] The so-called backward merging refers to merging from the starting layer to the inference direction of the neural network model, for example, in the AlexNet neural network model of Figure 6 , it is merged in the direction of the first layer→the second layer→the third layer. If there are unmerged layers before the starting layer, these unmerged layers will not be considered into the template merging unit under this rule.

[0100] Rule two: preferentially forward merging

[0101] The so-called forward merging refers to merging from the starting layer to the opposite direction of the inference of the neural network, for example, in the AlexNet neural network model of Figure 6 , it is merged in the direction of the third layer→the second layer→the first layer. This rule is usually matched with the aforementioned starting rule that the starting layer is the first unmerged convolution or pooling layer, because there may be unmerged layers before the convolution or pooling layer. After the starting layer is selected, the processing device 203 preferentially merges forward to try to include the layers before the starting layer that have not been merged into the template merging unit. For example, in the AlexNet neural network model of Figure 6 , assuming that the first layer to the second layer have been merged, the processing device 203 finds that the first unmerged convolution or pooling layer is the fifth layer, so the starting layer is the fifth layer, and preferentially merges the fourth layer and the third layer forward, and if it can continue to merge, then it merges the sixth layer, the seventh layer, and so on backward.

[0102] Rule Three: Prefer block structure

[0103] When the neural network model has block structure, this rule requires the processing device 203 to prefer to add or delete template fusion units in block structure rather than in layer, if the operation logic fusion of a whole block is unsuccessful, then consider fusion from the layers on each branch. For example, the neural network model of Figure 7 , the processing device 203 will prefer to consider the subnetwork 701 or the subnetwork 702 as a unit for fusion.

[0104] When the neural network is a long chain structure, since there is no block structure, directly add or delete template fusion units in layer. This rule is not applicable to neural network models with long chain structure.

[0105] Rule Four: Single branch output

[0106] The fusion strategy of this embodiment does not support the template fusion unit as a multi-output network, because the shape derivation implemented inside the template fusion unit mainly adopts the form of backward-to-forward derivation, and the multi-output network means that different outputs need to be derived forward respectively, and the results of the derivation do not necessarily converge to the same feature map, so as to fail to converge.

[0107] In other words, the output of the template fusion unit needs to be single branch output, that is, the last layer of the template fusion unit can only have one output. Figure 7 Two fusion ways of the subnetwork 701 are marked, the first one is to fuse the first layer to the fifth layer into a template fusion unit 703, and the second one is to fuse the first layer to the sixth layer into a template fusion unit 704. Since the outputs of the third layer and the fifth layer are the outputs of the template fusion unit 703, the template fusion unit 703 belongs to a multi-output network, that is, a multi-branch output. While the output of the sixth layer is the output of the template fusion unit 704, only one output data is generated, so the template fusion unit 704 belongs to a single-output network, that is, a single-branch output. The processing unit 203 will judge whether the output of the template fusion unit is a single-branch output, if this rule is not met, the processing device 203 adds or deletes layers inside the template fusion unit until this rule is met.

[0108] Rule Five: Include at least 2 main layers

[0109] When the layer logic is too simple, the performance of the template fusion unit is not better than that of the unfused layers. Therefore, when the layer logic is used as the fusion strategy, the processing device 203 evaluates whether the operations of the fused layers are complex enough to make the fusion beneficial. To be beneficial, the main layers, which refer to layers such as matrix multiplication, pooling, or convolution that consume a large amount of input / output resources, should be included in the template fusion unit as much as possible. Here, the pooling includes various types of pooling such as max pooling or average pooling, and the convolution includes various types of convolution such as normal convolution, convolution with mean, depthwise convolution, etc. This rule requires the template fusion unit to include at least two main layers. When the processing unit 203 determines that this rule is not satisfied, the processing device 203 adjusts the template fusion unit until this rule is satisfied.

[0110] Rule Six: Include a continuous structure of main layer, main layer, and non-main layer in sequence

[0111] This rule requires the template fusion unit to include a continuous structure of main layer, main layer, and non-main layer, i.e., a continuous structure of main layer, main layer, and non-main layer in sequence. Such operations are complex enough to make the fusion beneficial. Referring to FIG. 4, the 4th layer, the 5th layer, and the 6th layer in the 4th layer-5th layer-6th layer satisfy the continuous structure of main layer, main layer, and non-main layer in sequence, and thus the template fusion unit including the 4th layer, the 5th layer, and the 6th layer satisfies this rule. When the processing unit 203 determines that this rule is not satisfied, the processing device 203 adjusts the template fusion unit until this rule is satisfied. Figure 6

[0112] Rule Seven: Include a continuous structure of scalar calculation layer and vector calculation layer

[0113] This rule requires the template fusion unit to include a continuous structure of scalar calculation layer and vector calculation layer, i.e., a continuous structure of scalar calculation layer and vector calculation layer in sequence. The scalar calculation layer refers to an addition layer, a subtraction layer, or a multiplication layer, and the vector calculation layer refers to an activation layer, a batch normalization layer, or a scaling layer. When the processing unit 203 determines that this rule is not satisfied, the processing device 203 adjusts the template fusion unit until this rule is satisfied.

[0114] Rule Eight: The weights of the convolution layer are not the output of a certain layer

[0115] This rule requires the weights of the convolution layer in the template fusion unit to be not the output of any layer of the neural network, regardless of whether the layer is included in the template fusion unit. When the processing unit 203 determines that this rule is not satisfied, the processing device 203 removes the convolution layer from the template fusion unit.

[0116] ​Rule Nine: The weights of the convolution layer are not shared with any layer of the neural network

[0117] Since the weights of the operators involved in the template fusion unit have a special placement form, when the convolution operator to be fused shares weights with other operators, the placement logic of the weights will conflict. This rule is that the weights of the convolution operator in the template fusion unit are not shared with any layer of the neural network. When the processing unit 203 judges that this rule is not satisfied, the processing device 203 removes the convolution operator from the template fusion unit.

[0118] Rule Ten: The weights are not greater than the available space of the WRAM

[0119] The large graph mode has less restriction on the WRAM 432, because the on-chip unit graph loaded into the SRAM 308 is only a part of the feature map, and when calculating the template fusion unit, the WRAM 432 only needs to store all the weights of the feature map. However, since the small graph mode may load multiple feature maps into the SRAM 308, the number of weights required will increase, and it is necessary to carefully evaluate whether the available space of the WRAM 432 is sufficient. This rule is that the required storage space of the weights in the on-chip unit graph is not greater than the available space of the WRAM 432. When the processing device 203 judges that this rule is not satisfied, the processing device 203 reduces the size of the on-chip unit graph.

[0120] If the weights are split based on the output channel parameter Cout of the C dimension, since the weights will be evenly distributed to multiple processor cores 306, the rule is adjusted to:

[0121]

[0122] where W j is the required storage space of the weights involved in the on-chip unit graph j, n is the number of processor cores in the cluster, and W is the available space of the WRAM 432.

[0123] Rule Eleven: Redundancy Percentage

[0124] The redundancy percentage is the ratio of the total redundancy generated by the input-dependent operation and the output-dependent operation to the normal input / output amount of the template fusion unit. Here, the normal input / output amount refers to the amount of data without redundancy before the on-chip unit graph is split. The processing device 203 calculates the percentage of the memory access amount size TFU of the on-chip unit graph from the DRAM 204 to the SRAM 308 after the template fusion unit fuses the current layer, to the normal input / output amount (without redundancy) size ori , where the memory access amount size TFU refers to the theoretical memory access amount size oriThe sum of redundancies is added. The formula is as follows:

[0125] The processing device 203 takes into account the split information and shape derivation calculation of the template fusion unit, and sets the percentage threshold to 50%, 75%, 100%, 125%, or 150%, preferably 100%. Taking the percentage threshold of 100% as an example, it means that when the sum of redundancies is greater than twice the normal input / output amount of the template fusion unit, fusion is no longer performed. This rule is that the sum of redundancies generated by the split tile unit graph does not exceed a certain proportion related to the percentage threshold. Once it is exceeded, it means that the redundant part is too much, and a large amount of resources will be consumed in calculating the redundancy, and the performance will be reduced. Therefore, when the processing device 203 determines that this rule is not met, the processing device 203 will stop fusion.

[0126] It should be noted that in the small graph mode, since at least one whole complete feature map is loaded at a time from the DRAM 204 to the SRAM 308, no redundancy is generated. This rule does not apply to the small graph mode.

[0127] Rule Twelve: Tile Unit Graph Input / Output Size

[0128] Assuming that the spatial size of the SRAM 308 is S, the storage space required by the tile unit graph is IN, and the storage space required by the calculation result of the tile unit graph is OUT, then this rule is that the spatial size of the SRAM 308 needs to meet the following conditions:

[0129] If IN and OUT cannot share storage space, IN+OUT<S

[0130] If IN and OUT can share storage space, MAX(IN, OUT)<S

[0131] That is, if IN and OUT cannot share storage space, the sum of the storage space of the tile unit graph and the storage space of the calculation result is less than the available space of the SRAM 308; if IN and OUT can share storage space, the larger one of the storage space of the tile unit graph and the storage space of the calculation result is less than the available space of the SRAM 308.

[0132] Rule Thirteen: W i +IN1+IN2≤S

[0133] In the small graph mode, this rule is that the spatial size of the SRAM 308 needs to meet the following conditions:

[0134] W i +IN1+IN2≤S

[0135] That is, the storage space required by the weight of the subgraph i W iThe sum of the required storage space of the on-chip unit graph IN1, the required storage space of the cache space IN2 is not greater than the available space of the SRAM 308. When the processing device 203 judges that this rule is not satisfied, the processing device 203 reduces the number of on-chip unit graphs until the rule is satisfied.

[0136] Rule Fourteen: SubINi+W i +IN2≤S

[0137] In the small graph mode, this rule requires that the space size of the SRAM 308 satisfies the following condition:

[0138] SubINi+W i +IN2≤S

[0139] That is, the required storage space of the subgraph i SubINi, the required storage space of the weight of the subgraph i W i , the required storage space of the cache space IN2 is not greater than the available space of the SRAM 308. When the processing device 203 judges that this rule is not satisfied, the processing device 203 reduces the number of on-chip unit graphs until the rule is satisfied.

[0140] Rule Fifteen: SubOUTi+W i+1 +IN2≤S

[0141] In the small graph mode, this rule requires that the space size of the SRAM 308 satisfies the following condition:

[0142] SubOUTi+W i+1 +IN2≤S

[0143] That is, the required storage space of the intermediate result of the subgraph i SubOUTi, the required storage space of the weight of the next subgraph W i+1 , the required storage space of the cache space IN2 is not greater than the available space of the SRAM 308. When the processing device 203 judges that this rule is not satisfied, the processing device 203 reduces the number of on-chip unit graphs until the rule is satisfied.

[0144] Rule Sixteen: W i +W i+1 ≤W

[0145] The weights participating in the convolution operation in the template fusion unit are independently transported and reside on the WRAM 432. In the small graph mode, if the subgraph includes multiple feature maps, considering the pipeline between the subgraphs, the WRAM 432 stores at most the weights of two adjacent subgraphs at the same time. Assuming that the required storage space of each subgraph i is W i , and the total space of the WRAM 432 is W, this rule requires that the space size of the WRAM 432 satisfies the following condition:

[0146] Wi +W i+1 ≤W

[0147] i.e. the storage space W required by the weight of sub-graph i i , the storage space W required by the weight of next sub-graph i+1 The sum is not greater than the available space of WRAM 432. When the processing device 203 judges that this rule is not satisfied, the processing device 203 reduces the number of on-chip unit graphs until the rule is satisfied.

[0148] Rule Seventeen: The storage space required by sub-graph is not greater than the available space of NRAM

[0149] This rule is that the storage space required by sub-graph is not greater than the available space of NRAM 431. When the on-chip unit graph on SRAM 308 is to be split into sub-graphs and carried to NRAM 431, the processing device 203 can split the on-chip unit graph in the N, H, and W dimensions in a fine-grained manner. If the space of NRAM 431 is insufficient, the processing device 203 splits the on-chip unit graph more finely until the rule is satisfied. Generally, NRAM 431 has reasonable available space, so that the on-chip unit graph is split to a reasonable degree and can be loaded at one time, and from the perspective of the fusion strategy, the template fusion unit is not affected by the number of batches. However, the smaller the on-chip unit graph is split (i.e. the more sub-graphs), the lower the processing speed will be, so the processing device 203 needs to evaluate the space of NRAM 431.

[0150] In some embodiments, the space of SRAM 308 corresponds to the number of NRAM 431 of processor cores 306 within the cluster 305, for example, the cluster 305 includes 4 processor cores 306, and the space of SRAM 308 is 4 times the space of NRAM 431. In other words, the on-chip unit graph in the large graph mode Figure 1 Generally, can be allocated to 4 processor cores 306 for processing, and this architecture design has considered that the data loaded into SRAM 308 can be allocated to all NRAM 431 at one time. Therefore, this rule does not need to be considered in the large graph mode.

[0151] Rule Eighteen: The number of feature maps is not greater than the feature map threshold

[0152] In the tile mode, the on-chip unit graph can include multiple feature maps, the more feature maps, the more sub-graph transmission between the SRAM 308 and the NRAM 431, the efficiency will be reduced, therefore, the more feature maps in the on-chip unit graph is not necessarily better, the processing device 203 will calculate the appropriate fusion layer number according to the number of feature maps in the on-chip unit graph to maximize the benefit. The rule is that the number of feature maps in the on-chip unit graph is not greater than the feature map threshold, when the processing device 203 judges that the rule is not met, the processing device 203 reduces the number of feature maps in the on-chip data until the rule is met.

[0153] Rule nineteen: step redundancy

[0154] Step redundancy refers to: when the template fusion unit fusion layer number is too much, plus the length and width of the kernel of convolution and pooling is greater than the step, the input data required by each output point has an overlapping part, that is, the input dependent operation mentioned above, and the overlapping part is the step redundancy. Step redundancy makes each processor core 306 need to read some data, but this part of the reused data will occupy the access resources on-chip and off-chip, the more the number of layers included in the template fusion unit, the more serious the step redundancy. The rule is that the difference between the edge length of the kernel of the convolution layer or the pooling layer and the step is not greater than the redundancy threshold.

[0155] In this embodiment, the redundancy threshold is defined as follows. Assuming that the length and width of the kernel of the convolution and pooling layer are k x and k y , the steps in the length and width directions are s x and s y , the step redundancy in the length direction is the sum of k x -s x of all convolution and pooling layers in the template fusion unit; similarly, the step redundancy in the width direction is the sum of k y -s y of all convolution and pooling layers in the template fusion unit. The redundancy threshold of this embodiment can be 3, 4, 5 or 6, preferably 4. As long as the step redundancy in either the length direction or the width direction is greater than the redundancy threshold, the rule is not met. The processing device 203 adjusts the template fusion unit, usually reduces the number of fused layers, until the rule is met.

[0156] The fusion strategy sets an exception rule for step redundancy. If there are multiple branches in the layers to be fused and the template fusion unit can fuse the entire multiple branches, the performance of the template fusion unit will be more excellent, in this case, the processing device 203 will ignore the rule of step redundancy, that is, step redundancy will not limit the template fusion unit to fuse multiple branches, that is, in the fusion strategy of this embodiment, fusing multiple branches is preferred to the limitation of step redundancy. That is, step redundancy is only considered in the case of single branch.

[0157] The above rules are only examples, and the disclosure does not limit the order in which the rules are executed, nor does it limit the rules that need to be considered at the same time. A person skilled in the art can add or delete rules according to actual conditions in different application scenarios to achieve a fusion strategy that meets the current application scenario.

[0158] Back to Figure 11 In step 1103, the neural network calculation is performed according to the established template fusion unit. Based on the three-level operation hierarchy of the system-on-a-chip-cluster-processor core, the three-layer memory design of DRAM-SRAM-NRAM / WRAM, and the template fusion unit being regarded as a custom layer in the neural network, the computing device 201 loads the data required for the calculation of the template fusion unit from the DRAM 204 to the SRAM 308 at one time, so that the data can be cached and calculated in the appropriate level to form sufficient pipelining, and the calculation result is transmitted from the SRAM 308 to the DRAM 204 after the calculation is completed, greatly reducing the input / output overhead in neural network calculation.

[0159] When the input data in the fields of computer vision, speech, natural language processing, data mining, etc. are subjected to various deep learning and machine learning algorithms, the disclosure based on the template fusion unit can reduce the input / output overhead in neural network calculation. Another embodiment of the disclosure is a method for performing neural network calculation using a template fusion unit. Figure 12 The flow thereof is shown.

[0160] In step 1201, the template fusion unit is determined according to the fusion strategy. The processing device 203 selects the starting layer of the template fusion unit according to the starting rule of the fusion strategy, and performs fusion based on the starting layer, and checks all the rules of the fusion strategy one by one to establish the template fusion unit. The previous embodiment has illustrated various rules of the fusion strategy in detail, and will not be repeated here.

[0161] In this step, the template fusion unit is presented in the form of source code, and the source code needs to be converted into object code (machine code) by a compiler. The following steps are the process of converting the source code of the template fusion unit into object code by the compiler.

[0162] In step 1202, the shape of the template fusion unit is derived. For the data to be processed by the template fusion unit, this embodiment adopts a reverse derivation method, and the compiler derives how many sizes of input are needed from the output forward to Figure 8For example, the feature map 803 is deduced reversely to the feature map 802, and the feature map 802 is deduced reversely to the feature map 801. In this step, the compiler not only deduces the required input data according to the template fusion unit, but also further deduces the redundancy.

[0163] Next, step 1203 of deducing addresses is performed. According to the shape of the template fusion unit, the compiler deduces the on-chip storage space addresses of the entire control flow graph, and implements the access of general addresses, so as to achieve the purpose of reducing computing resources and shortening computing time. The control flow graph is an abstract data structure used in the compiler, which represents all paths that a program can execute, and reflects the possible flow direction of all nodes in the process in the form of a flowchart. The control flow graph is composed of nodes and the relationship between nodes. The node is also called a basic block (BB), which is a sequence of statements that is executed in the program to the maximum extent. Each basic block has only one entrance and exit, and is executed from the entrance to the exit. The basic block has the characteristic that if the first instruction is executed, all instructions in the basic block will be executed in sequence.

[0164] Each basic block contains at least one instruction, and the instructions in the basic block can use a pointer to point to a specific on-chip storage space. The pointer is a variable used to save the address of a specific address space. Through the pointer, the processor core 306 can load data into the specific address space pointed to by the pointer, or take out data from the specific address pointed to by the pointer.

[0165] According to the division of the template fusion unit, the compiler initially divides the basic blocks, and after iterative operation, the basic blocks and their relationships are confirmed, thereby completing the target code for implementing the template fusion unit.

[0166] Moreover, the compiler also analyzes the reuse data of the two template fusion units in the neural network in sequence, judges how much data in the previous template fusion unit can be left on the chip for use by the next template fusion unit, and plans the storage addresses of the data according to the judgment result.

[0167] In this step, the compiler completes the deduction of the addresses in the control flow graph.

[0168] In step 1204, the on-chip storage space is allocated. The processing device 203 allocates the physical space of the SRAM 308, the NRAM 431 and the WRAM 432 based on the deduction of the template fusion unit address. In this step, the compiler completes the pointing of the pointers in the control flow graph.

[0169] Finally, step 1205 is performed to generate executable instructions. In this step, a linker links the object code generated by the compiler with libraries to form an executable file. In more detail, the object code is a program module including machine code and information available to the linker, and the linker resolves undefined symbolic references and replaces placeholders in the object code with addresses of the symbols to generate executable instructions. The executable instructions can be directly executed by the computing device 201 to complete the computation of the neural network.

[0170] The present disclosure dynamically determines a template fusion unit to fuse multiple layers in a neural network to form a new custom layer and loads data required by the template fusion unit at one time to reduce input / output overhead by setting a fusion strategy.

[0171] When the template fusion unit is determined according to the rules of the fusion strategy, it is not necessarily necessary to expand the fusion starting from a convolutional layer or a pooling layer. The foregoing embodiment mentions that in one application scenario, the starting rule can be that the starting layer is the first un-fused layer in the neural network, which can be a layer other than a convolutional layer or a pooling layer. Such a starting rule makes the establishment of the template fusion unit more flexible, enabling appropriate selection of the starting layer to start the fusion based on the order of the layers in different neural networks, without being limited by the position and number of convolutional layers or pooling layers in the neural network model, thereby adapting to various network models and making the fusion more comprehensive to improve the overall efficiency.

[0172] For example, taking the neural network model of Figure 6 as an example, assuming that the first layer to the fifth layer have been fused, when the next template fusion unit is established, if the starting rule adopts the starting layer as the first un-fused convolutional or pooling layer, the next convolutional or pooling layer is the eighth layer, in other words, the sixth layer and the seventh layer can not be fused, which affects the overall efficiency.

[0173] Another embodiment of the present disclosure is a scheme for fusing a neural network, in which the starting layer is a layer other than a convolutional layer and a pooling layer, i.e., a non-convolutional layer and a non-pooling layer. This embodiment is also implemented based on the framework of Figures 1 to 4 . This embodiment also performs the flowchart shown in Figure 11 .

[0174] In step 1101, a starting layer is selected according to a fusion strategy. The processing device 203 selects a starting layer according to a fusion strategy, for example, the starting rule of the fusion strategy is that the starting layer is the first un-fused layer in the neural network, which is a layer other than a convolutional layer or a pooling layer.

[0175] It is noted that the step does not use the starting rule that the starting layer is the first unmerged convolutional or pooling layer. If the starting layer is selected according to the starting rule, the starting layer must be a convolutional or pooling layer, and the advantage of the embodiment that the position and number of convolutional layers or pooling layers in the neural network model are not limited does not exist.

[0176] In one application scenario, the starting layer can be an element-wise layer, which is a layer that operates on each element of a vector. The input data and the output data of such an operation have the same shape. Element-wise layers include the following types:

[0177] 1. Basic operations: vector addition, vector subtraction, vector multiplication, etc.

[0178] 2. Advanced operations: absolute value, square root, division, exponentiation, modulus, power, etc.

[0179] 3. Trigonometric function operations

[0180] 4. Rounding operations: rounding up, rounding, rounding down, keeping only the integer, etc.

[0181] 5. Activation functions: sigmoid, tanh, ReLU, etc.

[0182] In another application scenario, the starting layer can be an add padding layer. Adding padding is to not discard the original image information and to keep the size of the input data the same as the original image. The padding layer adds blank elements around the input data. Figure 1

[0183] In another application scenario, the starting layer can be a custom layer. With the development of deep learning and the complexity of neural networks, known or standard operators are not enough, and more and more custom operation rules are applied to neural networks. This embodiment can select a custom layer as the starting layer.

[0184] In another application scenario, the starting rule of the fusion strategy of this embodiment causes the processing device 203 to further determine whether the neural network includes a block structure. If not, it indicates that the neural network is a long chain structure, and the processing device 203 can select the first unmerged layer in the neural network according to the aforementioned starting rule; if yes, this embodiment refers to the aforementioned rule three, and preferentially fuses in the unit of the block structure. Therefore, the processing device 203 then determines whether the first layer in the block structure is a layer other than a convolutional layer and a pooling layer. If yes, the processing device 203 takes the first layer as the starting layer.

[0185] ​When the processing device 203 determines that the frontmost layer is one of the convolutional layer and the pooling layer, the processing device 203 can directly select the convolutional layer or the pooling layer as the starting layer, or select a layer closest to the frontmost layer and other than the convolutional layer and the pooling layer as the starting layer. Figure 13 A neural network model with a block structure is shown, and the example neural network model includes a subnetwork 1301 and a subnetwork 1302. The subnetwork 1301 includes a first layer to a sixth layer, and the subnetwork 1302 includes an eighth layer to an eleventh layer. The subnetwork 1301 and the subnetwork 1302 are connected at a seventh layer. Assuming that the subnetwork 1301 has been fused, when the subnetwork 1302 is fused, according to the foregoing rules, the processing device 203 determines whether a frontmost layer (i.e., the eighth layer) of the subnetwork 1302 is a layer other than the convolutional layer and the pooling layer. If so, the eighth layer is directly selected as the starting layer for fusion. If the eighth layer is the convolutional layer or the pooling layer, the processing device 203 can also select the eighth layer as the starting layer, or select a layer closest to the frontmost layer and other than the convolutional layer and the pooling layer as the starting layer. The layer closest to the eighth layer is the seventh layer, the seventh layer has not been fused, and it is assumed that the seventh layer is neither the convolutional layer nor the pooling layer. Therefore, the processing device 203 selects the seventh layer as the starting layer. If the seventh layer is also the convolutional layer or the pooling layer, this embodiment can select the seventh layer or the eighth layer as the starting layer.

[0186] This embodiment can preferentially fuse the entire block structure to improve the fusion efficiency. However, in a specific application scenario, the processing device 203 cannot select a layer closest to the frontmost layer and other than the convolutional layer and the pooling layer as the starting layer. For example, in the neural network model of FIG. 7, assuming that the subnetwork 701 has been fused, when the subnetwork 702 is fused, if the seventh layer is the convolutional layer or the pooling layer, and in the case where the subnetwork 701 has been fused, the processing device 203 cannot select a layer closest to the frontmost layer and other than the convolutional layer and the pooling layer as the starting layer. In this case, the processing device 203 selects a layer closest to the frontmost layer and other than the convolutional layer and the pooling layer (i.e., the eighth layer) as the starting layer. However, in this way, the entire block structure cannot be included in the template fusion unit. Since the fusion effect of the eighth layer as the starting layer is not ideal, the processing device 203 can also directly select the seventh layer as the starting layer. Figure 7

[0187] After the starting layer is selected, step 1102 is performed, and a template fusion unit is established based on the starting layer. The processing device 203 can establish the template fusion unit according to the rules (rules 1 to 19) exemplified in the foregoing embodiments. These rules are only examples, and this embodiment does not limit the order in which the rules are executed, nor does it limit that these rules need to be considered at the same time. A person skilled in the art can add or delete rules according to actual conditions in different application scenarios to achieve a fusion strategy that meets the current application scenario.

[0188] ​Steps 1101 and 1102 correspond to step 1201, which determines the template fusion unit based on the fusion strategy. Then, the compiler derives the shape of the template fusion unit (step 1202), derives the address (step 1203), allocates on-chip memory space (step 1204), and finally the linker generates executable instructions (step 1205).

[0189] In step 1103, neural network calculations are performed based on the established template fusion unit. The computing device 201 executes the aforementioned executable instructions to perform neural network calculations based on the template fusion unit.

[0190] The starting layer in this embodiment can be any layer other than convolution and pooling. This starting rule makes the establishment of template fusion units more flexible. It can appropriately select the starting layer to start fusion for different neural networks, without being limited by the position and number of convolution or pooling layers in the neural network model. This allows it to adapt to various network models, making the fusion more comprehensive and improving the overall efficiency.

[0191] After generating executable instructions, the computing device 201 can infer the neural network based on the executable instructions, using template fusion units as units. Another embodiment disclosed herein is a scheme for computing a neural network based on executable instructions, which also has... Figures 1 to 4 The architecture is used to compute the graph of template fusion units, and its implementation is as follows: Figure 14 The process is shown below.

[0192] In step 1401, the feature maps of the neural network are stored. As described in the previous embodiments, the processing device 203 fuses multiple layers of the neural network according to the fusion strategy to generate template fusion units, and appropriately splits the feature maps into on-chip unit maps based on each rule.

[0193] More specifically, when the processing device 203 is in Figure 12 In step 1201, the template fusion unit is determined according to the fusion strategy. When it is determined that the feature map is larger than the available space of SRAM 308 (i.e., large image mode), the feature map needs to be split so that it can be loaded into SRAM 308 multiple times. The splitting method can be a specific granularity split in at least one of the N, H, and W dimensions. In this embodiment, the specific granularity can be, but is not limited to, half. When the processing device 203 determines that the feature map is not larger than the available space of SRAM 308 (i.e., small image mode), the on-chip cell map may include one or more feature maps, depending on how many feature maps can be loaded into the available space of SRAM 308. The technical details of converting the feature map into an on-chip cell map have been described in the foregoing embodiments for large image mode and small image mode, and will not be repeated here.

[0194] The feature maps to be used for neural network calculations are all stored in DRAM 204.

[0195] In step 1402, the on-chip unit graph is loaded. Since the executable instructions are based on template fusion unit computing neural network, when the computing device 201 executes the executable instructions, it is according to the template fusion unit to compute the neural network, instead of computing layer by layer according to each layer of the neural network. The executable instructions carry information on how to split the feature map into the on-chip unit graph, that is, carry address information of the on-chip unit graph, and the SRAM 308 loads the on-chip unit graph from the appropriate address of the DRAM 204 according to the address information through the GMDA 311.

[0196] In step 1403, the sub-graphs are loaded. The NRAM 432 loads the sub-graphs through the MVDMA 434. Taking an example of 1 cluster 305 including 4 processor cores 306, the on-chip unit graph will be split into 4 sub-graphs, and one processor core 306 in the cluster 305 splits the on-chip unit graph into 4 sub-graphs in a certain granularity in at least one of the N, H, and W dimensions, and sends the sub-graphs to the NRAM 432 of each processor core 306 through the MVDMA 434. In this embodiment, the certain granularity can be but is not limited to half.

[0197] In step 1404, the sub-graphs are computed and corresponding intermediate results are generated. The operation module 42 of each processor core 306 takes out the sub-graphs from the NRAM 431 for computation, and stores the intermediate results back into the NRAM 431 after generating the intermediate results. It should be noted that since each processor core 306 is allocated a sub-graph belonging to a different part of the on-chip unit graph, each intermediate result also reflects a part of the computation result.

[0198] In step 1405, the intermediate results are reduced to generate the computation result of the corresponding on-chip unit graph. Reduction refers to combining the intermediate results into the computation result, that is, the aforementioned output-dependent operation. The broadcast bus 309 transmits the intermediate result of each processor core 306 to the next processor core 306, and the processor core 306 computes the intermediate result of the previous processor core 306 with the corresponding intermediate result stored to generate the computation result. There are various ways to implement reduction, and the following takes an example of ring allreduce to illustrate how to implement the reduction in this embodiment.

[0199] Figure 15 A ring allreduce framework is shown. The ring allreduce framework 1500 exemplarily shows 4 clusters in a computing device 201: a first cluster 1501, a second cluster 1502, a third cluster 1503, and a fourth cluster 1504, each including 4 processor cores. The ring allreduce framework 1500 organizes these clusters into a logical loop. Each cluster is connected to only the previous cluster and the next cluster, and receives and transmits data in the same direction. For example, the first cluster 1501 receives data from the fourth cluster 1504, and transmits data to the second cluster 1502.Figure 15 Each cluster receives data from the previous cluster in a clockwise direction, and sends data to the next cluster in a counterclockwise direction, as indicated by the arrows. The data transmission is performed through the CDMA 310 under the control and coordination of the synchronization module 304, and the framework can make full use of the input / output bandwidth of each cluster.

[0200] The multiple clusters in the logical loop are shown in FIG. 15A. Figure 16 Before the circular all-reduce is performed, the processor cores in each cluster have completed the subgraph computation of the core, that is, the intermediate results are generated and stored in the NRAM of each processor core. Taking the first cluster 1501 as an example, the four processor cores generate intermediate results a0, b0, c0, and d0, respectively.

[0201] Next, the reduction procedure is performed, and the clusters will perform N-1 times (N is 4 here) of reduction iterations. In each iteration, the clusters will send all the intermediate results to the next cluster and receive all the intermediate results from the previous cluster for computation, and the intermediate results sent and received by each cluster are different in each iteration.

[0202] For the convenience of illustration, it is assumed here that the output dependent operation only needs to add the intermediate results to generate the computation result. Figure 17A It is shown that in the first iteration, the intermediate result a0 of the first cluster 1501 is transmitted to the second cluster 1502 to be added to the intermediate result a1, the intermediate result b1 of the second cluster 1502 is transmitted to the third cluster 1503 to be added to the intermediate result b2, the intermediate result c2 of the third cluster 1503 is transmitted to the fourth cluster 1504 to be added to the intermediate result c3, and the intermediate result d3 of the fourth cluster 1504 is transmitted to the first cluster 1501 to be added to the intermediate result d0.

[0203] Figure 17B It is shown that in the second iteration, the intermediate result a0+a1 of the second cluster 1502 is transmitted to the third cluster 1503 to be added to the intermediate result a2, the intermediate result b1+b2 of the third cluster 1503 is transmitted to the fourth cluster 1504 to be added to the intermediate result b3, the intermediate result c2+c3 of the fourth cluster 1504 is transmitted to the first cluster 1501 to be added to the intermediate result c0, and the intermediate result d0+d3 of the first cluster 1501 is transmitted to the second cluster 1502 to be added to the intermediate result d1.

[0204] Figure 17CAt the third iteration, the intermediate result a0+a1+a2 of the third cluster 1503 is transferred to the fourth cluster 1504 to be added to the intermediate result a3, the intermediate result b1+b2+b3 of the fourth cluster 1504 is transferred to the first cluster 1501 to be added to the intermediate result b0, the intermediate result c0+c2+c3 of the first cluster 1501 is transferred to the second cluster 1502 to be added to the intermediate result c1, and the intermediate result d0+d1+d3 of the second cluster 1502 is transferred to the third cluster 1503 to be added to the intermediate result d2.

[0205] After the aforementioned addition calculation is performed, the state shown in FIG. 15B can be obtained, in which each cluster has a processor core that performs a complete reduction calculation, i.e., adds all the corresponding intermediate results together, for example, the second processor core of the first cluster 1501 carries the calculation result b0+b1+b2+b3, the third processor core of the second cluster 1502 carries the calculation result c0+c1+c2+c3, the fourth processor core of the third cluster 1503 carries the calculation result d0+d1+d2+d3, and the first processor core of the fourth cluster 1504 carries the intermediate result a0+a1+a2+a3. Figure 18A

[0206] In order to implement the full reduction, the clusters must exchange these calculation results so that all clusters have the same final value, which is called allgather. The process of the allgather program is similar to that of the reduction program, i.e., N-1 iterations are performed, but the values received by the clusters are not added but overwritten, and finally the result shown in FIG. 15C is obtained, in which all processor cores carry the complete calculation result, which is stored in the SRAM 308. Figure 18B

[0207] The above-described ring full reduction operation is only used to illustrate one implementation of the reduction of this embodiment, and the disclosure does not limit the manner of reduction.

[0208] Finally, step 1406 is performed to store the calculation result back. The SRAM 308 stores the calculation result back to the DRAM 204 through the GDMA 311. The calculation result is the result of the on-chip unit graph calculation of the cluster. At this point, the computing device 201 completes the on-chip unit graph calculation.

[0209] Another embodiment of the disclosure is a computer readable storage medium having stored thereon computer program code for dynamically fusing neural networks according to a fusion policy, which, when executed by a processor, performs the method as shown in Figure 10 、 Figure 11 、 Figure 12 、 Figure 14

[0210] ​​​The present disclosure dynamically determines a template fusion unit and fuses multiple layers in a neural network to form a new customized layer by setting a fusion strategy, and loads data required for calculating the template fusion unit at one time to reduce input / output overhead.

[0211] According to different application scenarios, the electronic device or apparatus of the present disclosure can include a server, a cloud server, a server cluster, a data processing apparatus, a robot, a computer, a printer, a scanner, a tablet computer, a smart terminal, a PC device, an Internet of Things terminal, a mobile terminal, a mobile phone, a vehicle event data recorder, a navigation device, a sensor, a camera, a camera, a video camera, a projector, a watch, a headset, a mobile storage, a wearable device, a visual terminal, an autonomous driving terminal, a vehicle, a household appliance, and / or a medical device. The vehicle includes an airplane, a ship, and / or a vehicle; the household appliance includes a television, an air conditioner, a microwave oven, a refrigerator, an electric rice cooker, a humidifier, a washing machine, an electric lamp, a gas stove, an exhaust hood; the medical device includes a nuclear magnetic resonance instrument, a B-ultrasound instrument, and / or an electrocardiograph. The electronic device or apparatus of the present disclosure can also be applied to the fields of Internet, Internet of Things, data center, energy, transportation, public management, manufacturing, education, power grid, telecommunications, finance, retail, construction site, medical treatment, etc. Further, the electronic device or apparatus of the present disclosure can also be used in cloud, edge, terminal, etc. application scenarios related to artificial intelligence, big data, and / or cloud computing. In one or more embodiments, the electronic device or apparatus with high computing power according to the present disclosure can be applied to a cloud device (such as a cloud server), and the electronic device or apparatus with small power consumption can be applied to a terminal device and / or an edge device (such as a smart phone or a camera). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or the edge device are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud device according to the hardware information of the terminal device and / or the edge device to simulate the hardware resources of the terminal device and / or the edge device, so as to complete unified management, scheduling and collaborative work of end-cloud integration or cloud-edge integration.

[0212] It should be noted that, for the purpose of clarity, the disclosure describes some methods and embodiments thereof as a series of acts and / or combinations thereof, but those skilled in the art will understand that the present disclosure is not limited to the order of the acts described. Those skilled in the art will understand and appreciate that some steps of the methods can be decided to be executed in other orders or at the same time with other steps. Further, those skilled in the art will understand and appreciate that some of the embodiments described in the disclosure can be considered optional, i.e., the acts or modules involved therein are not necessarily essential for the implementation of one or more of the aspects of the present disclosure. In addition, the disclosure describes some embodiments with different focuses according to different aspects. In view of this, those skilled in the art will understand that the parts not described in detail in some embodiments of the disclosure can also be seen from the relevant description of other embodiments.

[0213] In specific implementation aspects, based on the disclosure and teachings of the present disclosure, those skilled in the art will understand that some of the embodiments disclosed in the present disclosure can also be implemented in other ways not disclosed herein. For example, as for each unit in the electronic device or apparatus embodiments described above, the units are split based on the logical functions considered herein, and there can be other splitting manners in actual implementation. For another example, a plurality of units or components can be combined or integrated into another system, or some features or functions of the units or components can be selectively disabled. As for the connection relationship between different units or components, the connections discussed above in conjunction with the drawings can be direct or indirect coupling between the units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, where the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0214] In the present disclosure, the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units. The aforementioned components or units can be located in the same place or distributed on a plurality of network units. In addition, according to actual needs, some or all of the units can be selected to achieve the purpose of the aspects described in the embodiments of the present disclosure. In addition, in some scenarios, a plurality of units in the embodiments of the present disclosure can be integrated into one unit or each unit physically exists separately.

[0215] In some implementation scenarios, the integrated units described above can be implemented in the form of software program modules. If implemented in the form of software program modules and sold or used as independent products, the integrated units can be stored in a computer readable memory. Based on this, when the schemes of the disclosure are embodied in the form of software products (for example, computer readable storage media), the software products can be stored in the memory, which can include a plurality of instructions to make a computer device (for example, a personal computer, a server or a network device, etc.) execute part or all of the steps of the method described in the embodiments of the disclosure. The aforementioned memory can include, but is not limited to, a variety of media capable of storing program codes, such as a U disk, a flash disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, etc.

[0216] In some implementation scenarios, the integrated units described above can be implemented in the form of software program modules. If implemented in the form of software program modules and sold or used as independent products, the integrated units can be stored in a computer readable memory. Based on this, when the schemes of the disclosure are embodied in the form of software products (for example, computer readable storage media), the software products can be stored in the memory, which can include a plurality of instructions to make a computer device (for example, a personal computer, a server or a network device, etc.) execute part or all of the steps of the method described in the embodiments of the disclosure. The aforementioned memory can include, but is not limited to, a variety of media capable of storing program codes, such as a U disk, a flash disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, etc.

[0217] The embodiments of the disclosure are described in detail above, and the principles and implementation manners of the disclosure are described by applying specific examples. The above description of the embodiments is only used to help understand the method of the disclosure and its core idea; meanwhile, for those skilled in the art, according to the idea of the disclosure, the specific implementation manner and application range can be changed, and the above description of the specification should not be understood as a limitation of the disclosure.

Claims

1. A computing apparatus for computing a neural network based on a template fusion unit, wherein the template fusion unit fuses multiple layers of the neural network, and the computing apparatus comprises multiple clusters, each cluster comprising: Shared memory units are used to load on-chip cell diagrams from off-chip memory; as well as Multiple processor cores, each processor core including: A neuron storage unit for loading a subgraph from the shared storage unit, the subgraph being a portion of the on-chip unit graph; and The computation module is used to calculate the subgraph and generate intermediate results; The intermediate results are reduced among the multiple processor cores to generate calculation results corresponding to the on-chip cell graph, and the shared memory unit stores the calculation results back to the off-chip memory. Specifically, when the storage space required by the feature map is greater than the available space of the shared storage unit, the feature map is split into the on-chip cell map; the fusion strategy is that the total redundancy generated by splitting into the on-chip cell map does not exceed a percentage threshold, and fusion is stopped when the fusion strategy is not satisfied; the total redundancy is the difference between the memory access volume and the theoretical memory access volume.

2. The computing device according to claim 1, wherein the feature map includes N, H, W, and C dimensions, and the on-chip cell map is the feature map split at a specific granularity in at least one of the N, H, and W dimensions.

3. The computing device according to claim 1, wherein the off-chip memory stores a plurality of feature maps, and when the storage space required by the plurality of feature maps is not greater than the available space of the shared storage unit, the on-chip unit map includes the plurality of feature maps.

4. The computing device according to claim 1, wherein the on-chip unit graph includes N, H, W, and C dimensions, and the subgraph is a specific-granularity split of the on-chip unit graph in at least one of the N, H, and W dimensions.

5. The computing device of claim 1, wherein each cluster further includes a broadcast bus, one of the plurality of processor cores splits the on-chip cell graph according to the number of the plurality of processor cores, the broadcast bus transmits intermediate results of each processor core to the next processor core, and the processor core performs calculations on the intermediate results of the previous processor core and the corresponding stored intermediate results to generate the calculation result.

6. An integrated circuit device for computing a neural network based on template fusion units, comprising: Off-chip memory is used to store the feature maps of the neural network; A processing device for fusing multiple layers of the neural network according to a fusion strategy to generate the template fusion unit and splitting the feature map into on-chip unit maps; as well as Computing device, comprising multiple clusters, each cluster comprising: A shared storage unit is used to load the on-chip cell diagram; Multiple processor cores, each processor core including: A neuron storage unit for loading a subgraph from the shared storage unit, the subgraph being a portion of the on-chip unit graph; The computation module is used to calculate the subgraph and generate intermediate results; The intermediate results are reduced among the multiple processor cores to generate calculation results corresponding to the on-chip cell graph, and the shared memory unit stores the calculation results back to the off-chip memory. When the processing device determines that the storage space required by the feature map is greater than the available space of the shared storage unit, it splits the feature map into the on-chip unit map. The fusion strategy is that the total redundancy generated by splitting into the on-chip cell graph does not exceed a percentage threshold. When the processing device determines that the fusion strategy is not satisfied, the processing device stops fusion. The total redundancy is the difference between the memory access volume and the theoretical memory access volume.

7. The integrated circuit device of claim 6, wherein the feature map includes N, H, W, and C dimensions, and the processing device splits the feature map at a specific granularity in at least one of the N, H, and W dimensions.

8. The integrated circuit device according to claim 6, wherein when the processing device determines that the storage space required by the feature map is not greater than the available space of the shared memory cell, the on-chip cell map includes a plurality of feature maps.

9. The integrated circuit device of claim 6, wherein the on-chip cell diagram includes N, H, W, and C dimensions, and one of the plurality of processor cores splits the on-chip cell diagram into sub-diagrams at a specific granularity in at least one of the N, H, and W dimensions.

10. The integrated circuit device of claim 6, wherein each cluster further includes a broadcast bus, one of the plurality of processor cores splits the on-chip cell diagram according to the number of the plurality of processor cores, the broadcast bus transmits intermediate results of each processor core to the next processor core, and the processor core performs calculations on the intermediate results of the previous processor core and the corresponding stored intermediate results to generate the calculation result.

11. A board comprising an integrated circuit device according to any one of claims 6 to 10.

12. A method for computing a neural network based on a template fusion unit, wherein the template fusion unit fuses multiple layers of the neural network, the method comprising: Load the on-chip cell diagram into the shared memory unit; Load a sub-graph, which is a part of the on-chip cell graph; Calculate the subgraph and generate intermediate results; The intermediate results are reduced to produce the calculation results corresponding to the on-chip unit diagram; as well as The calculation results are stored off-chip. The method further includes: when the storage space required by the feature map is greater than the available space of the shared storage unit, splitting the feature map into the on-chip unit map; The fusion strategy ensures that the total redundancy generated by splitting the chip into the on-chip cell graph does not exceed a percentage threshold. When the fusion strategy is not met, fusion is stopped. The total redundancy is the difference between the memory access volume and the theoretical memory access volume.

13. The method of claim 12, further comprising: Store the feature map of the neural network.

14. A computer-readable storage medium having stored thereon computer program code for computing a neural network based on template fusion units, wherein when the computer program code is run by a processing device, it performs the method of any one of claims 12 to 13.

Images