Compilation and scheduling methods of computational graphs and related products

By splitting the computational graph into execution blocks and using the runtime scheduling method of the context storage circuit, the execution efficiency problem of dynamic neural networks on the neural network accelerator is solved, and high-performance parallel processing of dynamic neural networks is realized, and suitable for static Neural network scenario.

CN114035916BActive Publication Date: 2024-11-26CAMBRICON SINGGO (NANJING) TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111291728.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-01
Publication Date
2024-11-26
Estimated Expiration
2041-11-01

AI Technical Summary

Technical Problem

Existing neural network accelerators mainly optimize static neural networks during compilation, making it difficult to efficiently execute dynamic neural networks, especially due to the dynamic tensor shape and control flow, the scheduling optimization effect is poor, and the register resource management is complex.

Method used

Provide a runtime scheduling method and compilation method for computing graphs, splitting the computation graph into multiple execution blocks, using context storage circuits to build and schedule execution blocks at runtime, and implement parallel flow processing, suitable for dynamic and static neural networks .

Benefits of technology

The performance of dynamic neural networks has been significantly improved, the performance of dynamic neural networks has been improved by 1.7 times, and the performance of static neural networks has been maintained by more than 96%, which is suitable for hybrid scenarios of dynamic and static neural networks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114035916B_ABST
    Figure CN114035916B_ABST
Patent Text Reader

Abstract

The present disclosure discloses a method for compiling and scheduling a computational graph and related products. An accelerator for executing the scheduling of a computational graph can be implemented as a computing device, included in a combined processing device, and the combined processing device can also include an interface device and other processing devices. The computing device interacts with other processing devices to jointly complete the computing operations specified by the user. The combined processing device can also include a storage device, which is respectively connected to the computing device and other processing devices, and is used to store data of the computing device and other processing devices. The solution disclosed in the present disclosure can facilitate pipeline processing by decoupling dynamic information from scheduling, thereby improving the processing efficiency of the machine.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure generally relates to the field of data processing. More specifically, the present disclosure relates to a method for compiling a computational graph, a runtime scheduling method, a compiler, an accelerator, a chip, and a board. Background Art

[0002] Deep Neural Network (DNN) has demonstrated its powerful capabilities in a wide range of applications, including but not limited to image processing, natural language processing, and gaming. At the same time, the continued development of DNN technology has also brought new opportunities for architectural innovation in specific fields. Many architecture and system studies of machine learning accelerators are dedicated to accelerating DNN training and reasoning to obtain better computing power and higher power efficiency.

[0003] Recently, dynamic neural network technology has attracted more and more attention from researchers due to its powerful expressiveness for complex network architectures with dynamic control flow and variable data size. As dynamic neural networks become increasingly important in natural language processing and semantic segmentation, currently widely used frameworks also begin to support dynamic neural network technology.

[0004] However, existing optimization work based on neural network accelerators usually focuses on the optimization of static neural networks, and uses static scheduling and other methods to optimize static computation graphs during compilation. Currently, there is still a lack of a systematic and complete solution for efficiently executing dynamic neural networks on neural network accelerators. Summary of the invention

[0005] In order to at least partially solve one or more technical problems mentioned in the background technology, the present disclosure provides solutions from multiple aspects. On the one hand, an improved accelerator and runtime scheduling method for computational graphs are provided, which can help to achieve effective pipeline processing. On the other hand, a programming interface or compiler and a corresponding compilation method are provided, which can facilitate programmers to perform stage optimization and provide high-order semantics for scheduling optimization.

[0006] In a first aspect, the present disclosure discloses a runtime scheduling method for a computation graph, wherein the computation graph is a compiled computation graph, and the compiled computation graph includes a plurality of execution blocks, wherein the execution blocks represent sub-operations of each node in the computation graph, and the method includes:

[0007] When the compiled computation graph is run, in response to the preparation of the execution block, the information of the execution block to be executed in the compiled computation graph is stored to construct a runtime computation graph; and according to the dependency relationship, the stored execution blocks are scheduled based on the runtime computation graph to implement pipeline operation.

[0008] In a second aspect, the present disclosure discloses a method for compiling a computation graph, comprising:

[0009] Each node in the computation graph is split into a plurality of execution blocks to generate a compiled computation graph, each execution block represents a sub-operation of a corresponding node, and the execution blocks are used to construct a runtime computation graph and perform scheduling based on the execution blocks when running the compiled computation graph.

[0010] In a third aspect, the present disclosure discloses an accelerator, comprising:

[0011] A parallel operation circuit, used for executing operations in parallel;

[0012] A context storage circuit is used to store information of an execution block that needs to be executed in the compiled computation graph to construct a runtime computation graph in response to a prepared execution of the execution block when running the compiled computation graph, wherein the compiled computation graph includes a plurality of execution blocks, and the execution blocks represent sub-operations of each node in the computation graph; and

[0013] The scheduling circuit is used to schedule the stored execution blocks based on the runtime calculation graph according to the dependency relationship to control the parallel operation circuit to realize pipeline operation.

[0014] In a fourth aspect, the present disclosure provides a compiler, comprising a processing circuit configured to execute the compilation method of the computation graph described in the second aspect of the present disclosure.

[0015] In a fifth aspect, the present disclosure provides a chip, comprising the accelerator of the third aspect and / or the compiler of the fourth aspect.

[0016] In a sixth aspect, the present disclosure provides a board comprising the chip of the fifth aspect.

[0017] According to the scheduling scheme provided above, parallel pipeline processing can be implemented for dynamic neural networks, thereby obtaining significant performance improvements. In addition, the scheduling scheme provided can also be applied to static neural networks, which only introduces negligible overhead. Therefore, the scheduling scheme of the disclosed embodiment can also be applied to mixed scenarios where dynamic neural networks and static neural networks exist at the same time, thereby obtaining overall performance improvements. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] By reading the detailed description below with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0019] Figure 1A structural diagram of a board according to an embodiment of the present disclosure is shown;

[0020] Figure 2 A structural diagram showing a combined processing device according to an embodiment of the present disclosure;

[0021] Figure 3 A schematic diagram showing the internal structure of a multi-core computing device according to an embodiment of the present disclosure;

[0022] Figure 4 shows an example of pipelining between nodes;

[0023] Figure 5 A schematic overview diagram showing a dynamic water flow system according to an embodiment of the present disclosure;

[0024] Figure 6 A schematic simplified structural block diagram of an accelerator according to an embodiment of the present disclosure is shown;

[0025] Figure 7 An exemplary flow chart showing a method for compiling a computational graph according to an embodiment of the present disclosure;

[0026] Figure 8 An example of computing graph splitting according to an embodiment of the present disclosure is shown;

[0027] Fig. 9 An exemplary rewritten program according to an embodiment of the present disclosure is shown;

[0028] Fig.10 An exemplary programming interface according to an embodiment of the present disclosure is shown;

[0029] Fig.11 An exemplary flow chart showing a runtime scheduling method of a computation graph according to an embodiment of the present disclosure; and

[0030] Fig.12 An exemplary execution process of a runtime scheduling scheme according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0032] It should be understood that the terms "first", "second", "third", and "fourth" etc. that may appear in the claims, specifications, and drawings of the present disclosure are used to distinguish different objects rather than to describe a specific order. The terms "include" and "comprise" used in the specifications and claims of the present disclosure indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their collections.

[0033] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.

[0034] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context.

[0035] Existing research mainly focuses on optimizing static neural networks with fixed input / output shapes and static computational graphs. For example, Auto TVM builds a statistical cost model and designs an exploration model to search for the best configuration to run the network on hardware. DNNVM designs a cycle-accurate simulator to find the best execution strategy for fusion nodes. These optimizations require knowledge of a predefined fixed network architecture and are therefore difficult to apply to dynamic neural networks.

[0036] By analyzing the reasoning process of dynamic neural networks, the inventors noticed that dynamic tensor shapes and control flows hinder scheduling optimization from obtaining better computational parallelism and hardware utilization. Specifically, for static neural networks, the shapes of inputs and outputs are known in advance, and the network structure is also fixed. Therefore, compilers for DNN accelerators can optimize hardware utilization and computational throughput based on dependency analysis of static computational graphs and using software pipelining technology. However, for dynamic neural networks, the computational load of control flows and tensors can only be determined at runtime. Therefore, it is extremely risky to make the best schedule in advance during static analysis. In addition, for dynamic neural networks, more context needs to be recorded, which undoubtedly increases the burden on register resources.

[0037] In view of one or more of the technical problems mentioned above, the present disclosure provides solutions in multiple aspects. On the one hand, an improved accelerator is provided, especially an improved DNN accelerator, which can help to achieve effective pipeline processing. On the other hand, a programming interface or compiler is provided, which can facilitate programmers to perform stage optimization and provide high-order semantics for scheduling optimization. On the other hand, a scheduling scheme is provided, which effectively performs dynamic pipeline processing based on the improved accelerator.

[0038] The specific implementation of the present disclosure is described in detail below with reference to the accompanying drawings.

[0039] Neural Networks and Computational Graphs

[0040] Deep neural network (DNN) models usually use symbolic representation to represent the structure of the network computational graph. For example, TensorFlow uses a directed graph consisting of a set of nodes and edges to describe the computational process. This directed graph is called a computational graph.

[0041] Regarding the terms "node" and "operator" mentioned in this disclosure, it should be noted that the term "operator" is from the computing level of the computer (or from the software level or algorithm level); and the term "node" is a more figurative term (from the graphical level or a more intuitive level). In terms of the content referred to, the terms "operator" and "node" actually refer to the same thing. That is, in this disclosure, it can be considered that the terms "operator" and "node" have the same meaning and can be used interchangeably, but are described from different aspects.

[0042] Static neural networks have a fixed network structure and fixed tensor shapes. The definition phase of the computational graph is called static declaration. Static neural networks enable DNN models to be deployed simply and efficiently. The compiler can optimize the network through complex optimization methods at compile time. Batch processing techniques can be used to improve the efficiency of multi-core processors (such as GPUs). Due to these advantages, static declaration is the main programming paradigm for DNN compilers.

[0043] However, with the continuous development of natural language processing and semantic segmentation, dynamic neural networks are being applied in more and more DNNs. Compared with static neural networks, dynamic neural networks have unfixed computational graphs, which include variable size, variable structure or control flow. Dynamic neural network technology supports variable network structure through dynamic declaration at runtime, thus enabling applications that require complex neural network structures.

[0044] Specifically, dynamic neural networks are often applied to the following scenarios: 1) Sequence language models. The input of these models is sentences that are usually of variable length. 2) Tree-structured recurrent neural networks (RNNs). For language models with sentiment analysis, the input is a tree structure, and they vary for different sentences. 3) Neural architecture search (NAS). NAS aims to find the optimal model for a specific task by repeatedly testing the performance of different network architectures. During the execution of the task, the network structure continues to evolve.

[0045] In some cases, a dynamic neural network can be simplified to a static neural network. For example, for a sequential language model with variable sentence length, all sentences can be aligned to the longest sentence by adding redundant padding. However, this will result in a lot of redundant and unnecessary computation.

[0046] Exemplary Hardware Environment

[0047] DNN accelerators are domain-specific processors designed to improve the computational efficiency and energy efficiency of DNN applications. The architectural characteristics of DNN accelerators are very different from traditional CPUs or GPUs, which greatly affects the program model and compiler optimization.

[0048] Figure 1 FIG. 1 is a schematic diagram showing the structure of a board 10 according to an embodiment of the present disclosure. Figure 1 As shown, the board 10 includes a chip 101, which is a system-on-chip (SoC), or system on chip, integrated with one or more combined processing devices, which are artificial intelligence computing units that support various deep learning and machine learning algorithms to meet the intelligent processing needs in complex scenarios in the fields of computer vision, speech, natural language processing, data mining, etc. In particular, deep learning technology is widely used in the field of cloud intelligence. A notable feature of cloud intelligence applications is the large amount of input data, which places high demands on the storage capacity and computing power of the platform. The board 10 of this embodiment is suitable for cloud intelligence applications and has huge off-chip storage, on-chip storage, and powerful computing power.

[0049] The chip 101 is connected to an external device 103 via an external interface device 102. The external device 103 is, for example, a server, a computer, a camera, a display, a mouse, a keyboard, a network card, or a wifi interface. The data to be processed can be transmitted from the external device 103 to the chip 101 via the external interface device 102. The calculation result of the chip 101 can be transmitted back to the external device 103 via the external interface device 102. According to different application scenarios, the external interface device 102 can have different interface forms, such as a PCIe interface.

[0050] The board 10 also includes a storage device 104 for storing data, which includes one or more storage units 105. The storage device 104 is connected to the control device 106 and the chip 101 through a bus and transmits data. The control device 106 in the board 10 is configured to control the state of the chip 101. To this end, in an application scenario, the control device 106 may include a microcontroller (MCU).

[0051] Figure 2 1 is a block diagram showing the combined processing device in the chip 101 of this embodiment. Figure 2 As shown in , the combined processing device 20 includes a computing device 201 , an interface device 202 , a processing device 203 and a storage device 204 .

[0052] The computing device 201 is configured to execute user-specified operations, and is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0053] The interface device 202 is used to transmit data and control instructions between the computing device 201 and the processing device 203. For example, the computing device 201 can obtain input data from the processing device 203 via the interface device 202 and write it into the storage device on the computing device 201 chip. Further, the computing device 201 can obtain control instructions from the processing device 203 via the interface device 202 and write them into the control cache on the computing device 201 chip. Alternatively or optionally, the interface device 202 can also read data in the storage device of the computing device 201 and transmit it to the processing device 203.

[0054] The processing device 203, as a general processing device, performs basic controls including but not limited to data handling, starting and / or stopping the computing device 201. Depending on the implementation, the processing device 203 can be a central processing unit (CPU), a graphics processing unit (GPU) or one or more types of processors in other general and / or special processors, these processors include but are not limited to digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, only with respect to the computing device 201 disclosed in the present invention, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are integrated and considered together, the two are regarded as forming a heterogeneous multi-core structure.

[0055] The storage device 204 is used to store data to be processed, which may be DRAM or DDR memory, and is usually 16G or larger in size, and is used to store data of the computing device 201 and / or the processing device 203 .

[0056] Figure 3 The schematic diagram of the internal structure of the computing device 201 is shown as a multi-core. The multi-core computing device 300 adopts a hierarchical structure design. The multi-core computing device 300 is a system on chip, which includes at least one computing cluster, and each computing cluster includes multiple processor cores. In other words, the multi-core computing device 300 is composed of a hierarchy of system on chip-computing cluster-processor core.

[0057] At the system-on-chip level, Figure 3 As shown, the multi-core computing device 300 includes an external storage controller 31 , a peripheral communication module 32 , an on-chip interconnect module 33 , a synchronization module 34 , and a plurality of computing clusters 35 .

[0058] There may be multiple external storage controllers 31, two of which are shown in the figure as an example, and are used to respond to access requests issued by the processing core and access external storage devices, such as Figure 2DRAM 204 in the chip, thereby reading data from outside the chip or writing data. The peripheral communication module 32 is used to receive control signals from the processing device 203 through the interface device 202, and start the computing device 201 to perform tasks. The on-chip interconnection module 33 connects the external storage controller 31, the peripheral communication module 32 and multiple computing clusters 35 to transmit data and control signals between each module. The synchronization module 34 is a global synchronization barrier controller (GBC) used to coordinate the work progress of each computing cluster and ensure information synchronization. Multiple computing clusters 35 are the computing cores of the multi-core computing device 300. Four are shown in the figure as an example. With the development of hardware, the multi-core computing device 300 disclosed in the present invention can also include 8, 16, 64, or even more computing clusters 35. The computing cluster 35 is used to efficiently execute deep learning algorithms. A grid structure that can communicate in a ring can be formed between multiple computing clusters 35, that is, a grid-like interconnection circuit can be provided between multiple computing clusters.

[0059] From the perspective of computing clusters, Figure 3 As shown in the upper right corner, each computing cluster 35 includes a processing unit 302 and a memory core (MEM core) 304. The processing unit 302 performs various computing tasks. In some implementations, the processing unit can be a multi-core architecture, for example, including multiple processing cores (IPU cores) 311-1 to 311-n to complete tasks such as large-scale vector computing. The present disclosure does not limit the number of processing cores 311.

[0060] The internal structure of the processing core 311 is as follows Figure 3 As shown below, each processing core 311 may have a plurality of computing modules 324-1 to 324-m for executing computing tasks, and a local storage module 323 required for executing computing tasks.

[0061] The computing module 324 is a basic tensor computing unit on the chip, including but not limited to a vector operation unit, a tensor operation unit that performs matrix multiplication, an operation unit capable of directly performing convolution operations, or a convolution computing unit that integrates img2col (image to row and column) and gemm (general matrix multiplication).

[0062] The local storage module 323 can be used as a cache level (e.g., the first-level cache L1cache) within the computing cluster 35, which may include a neuron RAM (NRAM) and a weight RAM (WRAM). NRAM is used to store input neurons, output neurons, and intermediate results after calculation; WRAM is used to store the convolution kernel of the deep learning network, i.e., the weight. It should be noted that the processing core may also include various communication modules to exchange data with an external storage unit. For example, the local storage module 323 communicates with the shared storage module 315 in the storage core 304 through the communication module 321. The communication module 321 may be, for example, a move direct memory access module (MVDMA). The local storage module 323 may also exchange data with an off-chip memory, such as DRAM 308, through a communication module 322. The communication module 322 may be, for example, an input / output direct memory access module (IODMA). The IODMA 322 controls the memory access between the NRAM / WRAM in the local memory module 323 and the DRAM 308 ; the MVDMA 321 controls the memory access between the NRAM / WRAM in the local memory module 323 and the shared memory module 315 .

[0063] continue Figure 3 In the upper right view, the storage core 304 is mainly used for storage and communication, that is, to store shared data or intermediate results between the processing cores 311, and to perform communication between the computing cluster 35 and the DRAM 308, communication between the computing clusters 35, and communication between the processing cores 311. In other embodiments, the storage core 304 has the ability of scalar operation and is used to perform scalar operation to realize the operation task in data communication.

[0064] The storage core 304 includes a relatively large shared storage module (SRAM) 315, a broadcast bus 314, a cluster direct memory access module (CDMA) 318, a global direct memory access module (GDMA) 316, and a communication computing module 317. The SRAM 315 plays the role of a high-performance data transfer station. The data reused between different processing cores 311 in the same computing cluster 35 does not need to be obtained from the DRAM 308 by each processing core 311, but is transferred between the processing cores 311 through the SRAM 315. The storage core 304 only needs to quickly distribute the reused data from the SRAM 315 to multiple processing cores 311, so as to improve the efficiency of inter-core communication and greatly reduce on-chip and off-chip input / output access.

[0065] The broadcast bus 314, CDMA 318 and GDMA 316 are respectively used to perform communication between the processing cores 311, communication between the computing clusters 35 and data transmission between the computing clusters 35 and the DRAM 308. They will be described below.

[0066] The broadcast bus 314 is used to complete high-speed communication between the processing cores 311 in the computing cluster 35. The broadcast bus 314 of this embodiment supports inter-core communication methods including unicast, multicast and broadcast. Unicast refers to point-to-point (e.g., single processing core to single processing core) data transmission, multicast is a communication method for transmitting a copy of data from the SRAM 315 to a specific number of processing cores 311, and broadcast is a communication method for transmitting a copy of data from the SRAM 315 to all processing cores 311, which is a special case of multicast.

[0067] In each computing cluster 35, each processing core 311 can initiate a broadcast to simultaneously broadcast data to the local storage modules 323 (e.g., NRAM or WRAM) of each core. Broadcasting to NRAM and WRAM belongs to two data channels and can be performed concurrently, but at a certain time point, each processing core can only initiate one broadcast, that is, broadcasting to WRAM and NRAM cannot be initiated simultaneously on the same core.

[0068] CDMA 318 is used to control the memory access of SRAM 315 between different computing clusters 35 in the same computing device 301. GDMA 316 cooperates with external memory controller 31 to control the memory access of SRAM 315 of computing cluster 35 to DRAM 308, or reads data from DRAM 308 to SRAM 315. As can be seen from the above, the communication between DRAM 308 and NRAM / WRAM in local storage module 323 can be realized through two channels. The first channel is to directly connect DRAM 308 and local storage module 323 through IODMA 322; the second channel is to first transmit data between DRAM 308 and SRAM 315 through GDMA 316, and then transmit data between SRAM 315 and local storage module 323 through MVDMA 321. Although it seems that the second channel requires more components to participate and the data flow is longer, in fact, in some embodiments, the bandwidth of the second channel is much greater than the first channel, so the communication between DRAM 308 and local storage module 323 through the second channel may be more efficient. The embodiments of the present disclosure can select the data transmission channel according to their own hardware conditions.

[0069] In some embodiments, the storage core 304 can be used as a cache level (e.g., second-level cache L2 cache) within the computing cluster 35 to achieve the effect of widening the communication bandwidth. Furthermore, the storage core 304 can also complete the communication with other computing clusters 35. The storage core 304 can, for example, realize communication functions such as broadcast, scatter, gather, reduce, and all-reduce between computing clusters 35. Among them, broadcast refers to distributing the same data to all computing clusters; scatter refers to distributing different data to different computing clusters; gather refers to gathering data from multiple computing clusters together; reduce refers to performing operations on data in multiple computing clusters according to a specified mapping function to obtain the final result and sending it to a certain computing cluster; and the difference between all-reduce and reduce is that the final result of the latter is only sent to one computing cluster, while the full-reduce needs to be sent to all computing clusters.

[0070] The computing module 317 during communication can be used to complete computing tasks in the communication such as the above-mentioned protocol and the full protocol during the communication process without the need for the processing unit 302, thereby improving the communication efficiency and achieving the effect of "storage and computing in one". Depending on different hardware implementations, the computing module 317 during communication and the shared storage module 315 can be integrated into the same or different components. The embodiments of the present disclosure are not limited in this regard. As long as the functions implemented and the technical effects achieved are similar to those of the present disclosure, they all belong to the protection scope of the present disclosure.

[0071] The DNN accelerator can be implemented as Figure 2 The computing device 201, from Figure 3 It can be seen from the internal structure of the computing device 201 shown that DNN accelerators usually have complex storage architectures that need to be explicitly managed by software. In addition, the accelerator uses VLIW (very long instruction word) to design its instruction set architecture (ISA) for vector or matrix operations. This ISA needs to be scheduled by software at runtime to obtain better module parallelism. Based on these characteristics, soft pipelining is one of the most effective optimization methods on DNN accelerators.

[0072] To date, existing tools and systems are based on CPUs or GPUs without considering the architectural characteristics of DNN accelerators. Due to the completely different programming model, these existing solutions are not suitable for DNN accelerators. Therefore, in DNN accelerators, it is urgent to develop efficient architectures and system optimization solutions for dynamic neural networks.

[0073] Exemplary dynamic water flow system

[0074] The dynamic neural network tool supports neural networks and dynamic declarations that vary according to input. Due to these dynamic characteristics, it is extremely challenging to implement efficient task scheduling and computational pipelining in dynamic neural networks.

[0075] Figure 4 An example of pipelines between nodes is shown to illustrate the challenges faced in dynamic neural networks. In this example, the pipelines within the nodes are omitted. (a) shows the soft pipelining of a static neural network with a fixed structure, and (b) shows the soft pipelining of a dynamic neural network with control flow.

[0076] As shown in (a), static neural networks usually use soft pipelining, which can perform calculations and data transfers simultaneously to obtain better hardware utilization, thereby improving computational parallelism. This soft pipelining is achieved by statically scheduling the execution order at compile time. When the structure of the neural network model is fixed, this soft pipelining is simple and efficient.

[0077] However, for dynamic neural networks with control flow, the scheduling order cannot be determined in advance, so it is difficult to achieve optimal static scheduling. As shown in (b), in this computational graph, the execution of node B or C depends on the result of the conditional judgment. Therefore, it is not clear until runtime whether B or C will be executed. In this example, simple soft pipelining requires the insertion of bubbles, which causes the pipeline to stall, hindering the improvement of computational parallelism.

[0078] In addition, dynamic neural networks are also more complex in terms of register management. Usually for static neural networks, context information such as tensor shape, tensor address, and loop count can be calculated in advance as intermediate values. But for dynamic neural networks, these are variable values, and all context information needs to be saved in registers. This requirement makes register allocation more challenging, especially for processors with fewer registers or without hardware memory management units.

[0079] In view of this, in the disclosed embodiments, a dynamic pipeline solution is provided, which can decouple the dynamic structure and scheduling from each other by adjusting them in the compilation stage and the runtime stage respectively, thereby eliminating the inefficiency caused by the dynamic characteristics in the dynamic neural network.

[0080] Figure 5 A schematic overview of a dynamic water flow system according to an embodiment of the present disclosure is shown.

[0081] like Figure 5 As shown, the computational graph of the dynamic neural network in this example is shown as 510, which shows four nodes A, B, C and D, and also includes a control flow (Switch), which determines whether to execute B or C according to conditions at runtime.

[0082] During compilation, the compiler converts the computation graph described in a high-level language into a series of low-level machine language instructions for machine execution. As shown in 520 in the figure, each node in the computation graph 510 is compiled into machine instructions. Since the dynamic characteristics cannot be determined at this time, that is, it is impossible to determine which of node B and node C will execute, all nodes will be compiled. The figure shows that nodes A, B, C, and D are compiled in serial order.

[0083] In the disclosed embodiments, in order to support pipeline processing at runtime, each node is split into several execution blocks during compilation, and these execution blocks represent the sub-operations of the node in the computation graph. Furthermore, these execution blocks are split according to whether the hardware modules involved in the node can be executed in parallel. For example, in the example in the figure, node A is split into three execution blocks: L (load), C (calculate), and S (store back), and nodes B, C, and D are similarly split. It can be understood that the load operation involves loading data from an external storage circuit to a local; the calculation operation involves the calculation circuit; and the store back operation involves storing the calculation result back to the external storage circuit. From the previous Figure 3 The storage architecture of the deep learning accelerator described above shows that the parallel execution of load and restore operations can be supported by different memory access interfaces or by allocating different levels of storage circuits. Therefore, when the processed data are not dependent on each other, the three operations can be processed in parallel.

[0084] Next, when running the compiled computation graph, when dynamic information such as graph structure and tensor shape is determined, the execution blocks split at compile time can be used to construct the runtime computation graph. Since the dynamic characteristics of the dynamic neural network can be determined at runtime, such as the shape, address, control flow, etc. of the tensor, the constructed runtime computation graph is determined. Specifically, as shown in 530, a context storage circuit 532 is introduced in the accelerator that runs the computation graph, which can be implemented using a buffer, so it is also called a context buffer (Context Buffer), abbreviated as CB. The context storage circuit is used to store information of the execution blocks that need to be executed in the compiled computation graph at runtime.

[0085] For example, in the example in the figure, assuming that the control flow conditions are judged according to the dynamic information at runtime and it is determined that the branch of node B will be executed, then the relevant information of each execution block in nodes A, B, and D is stored in the context storage circuit. In this way, the dynamic pipeline system of the disclosed embodiment converts the dynamic graph into a deterministic graph stored in the context storage circuit.

[0086] Then, the scheduler 531 can perform scheduling based on the deterministic graph constructed by nodes A, B, and D, thereby realizing pipeline processing. As shown in 533 in the figure, the scheduler can schedule to realize three pipelines in parallel. For example, at the same time, D can be loaded, B can be calculated, and A can be stored back. Therefore, by scheduling these execution blocks determined to be executed in the context storage circuit 532, the scheduler can realize efficient pipeline without being affected by dynamic information.

[0087] From the above description of the dynamic pipeline system, it can be seen that the disclosed embodiment provides an improved accelerator on the one hand, which introduces a context storage circuit to maintain the structural information of the runtime calculation graph; on the other hand, it provides a compiler or programming interface, which splits the calculation graph into multiple execution blocks during compilation, so that the runtime calculation graph can be constructed in units of execution blocks during runtime; on the other hand, it also provides a scheduler, which schedules each operation based on the deterministic calculation graph stored in the context storage circuit during runtime to achieve parallel pipeline operations. The following will be described in detail for each aspect.

[0088] Exemplary Accelerators

[0089] Figure 6 A schematic simplified structural block diagram of an accelerator according to an embodiment of the present disclosure is shown.

[0090] As shown in the figure, the accelerator 600 includes a parallel operation circuit 610, a context storage circuit 620 and a scheduling circuit 630. It can be understood that Figure 6In the accelerator 600, only the components related to this embodiment are shown. Therefore, it is obvious to a person skilled in the art that the accelerator 600 may also include components related to the present embodiment. Figure 6 The components shown in the figure are different from common components, such as control circuits.

[0091] The parallel operation circuit 610 is used to perform operations in parallel. The parallel operation circuit may include, for example, a plurality of computing units capable of performing operations in parallel, including but not limited to a vector operation unit, a tensor operation unit for performing matrix multiplication, an operation unit capable of directly performing convolution operations, or a convolution operation unit integrating img2col (image to row and column) and gemm (general matrix multiplication). The parallel operation circuit may be, for example, Figure 3 The calculation module 324 is shown.

[0092] The context storage circuit 620 is used to store information of the execution blocks determined to be executed in the compiled computation graph at runtime. At runtime, in response to the preparation of the execution blocks, it is possible to determine which execution blocks in the compiled computation graph are to be executed according to the determined dynamic information, thereby storing the information of the corresponding execution blocks in the context storage circuit 620.

[0093] The information of the execution block may include context information and a program counter. The context information refers to all execution information of the operator, which may include but is not limited to the data source address, data destination address, tensor data shape, and calculation configuration. For the convolution operator, the calculation configuration, for example, involves convolution parameters, such as the size parameters of the convolution kernel, the convolution step size, and the convolution padding.

[0094] The program counter (PC) is an identifier for each machine instruction that constitutes the compiled computation graph, which points to a machine language instruction in the memory and the address containing the instruction. The context storage circuit maintains the information of the execution block based on the program counter, thereby facilitating the subsequent scheduling of the execution block by the scheduler 630.

[0095] In order to improve the implementation efficiency, the context storage circuit 620 can be an on-chip storage circuit of the accelerator 600, especially a global cache circuit, so that compared with the implementation using registers, it can provide a larger storage space and a lower access delay. The context storage circuit 620 can reduce the complexity of maintaining the context. During the optimization and execution of dynamic neural networks, a large amount of context information is variable at runtime. If registers are used, due to limited register resources, frequent register spill and refill operations are required to maintain such a large amount of context information. Register allocation is an NP-complete problem, and inappropriate register allocation may block the pipeline. Therefore, compared with limited register resources, the context storage circuit 620 provides a larger and more sufficient storage space for storing these contexts.

[0096] The context storage circuit 620 may be an additionally configured buffer, or may be a storage area divided from an existing storage circuit on the accelerator 600, for example Figure 3 The shared storage module 315, or the NRAM or WRAM in the local storage module 323, etc., the disclosed embodiments are not limited in this regard.

[0097] The scheduling circuit 630 is used to schedule these execution blocks to be executed on the parallel operation circuit 610 according to the dependencies and based on the deterministic runtime calculation graph constructed by the execution blocks stored in the context storage circuit 620 to achieve pipeline processing.

[0098] Specifically, dependencies may include hardware module dependencies and / or data dependencies. Hardware module dependencies indicate whether multiple hardware modules that perform an operation can be processed in parallel. For example, the hardware modules involved in the aforementioned L (load), C (calculate), and save back (S) can be processed in parallel. Data dependencies refer to whether there are read-write conflicts between data. For example, if the second execution block needs to use the calculation result of the first execution block, the calculation result of the first execution block needs to be written to the specified location first and then read by the second execution block to ensure correct operation.

[0099] Therefore, in some embodiments, the scheduling circuit 630 determines the execution blocks that can be executed in parallel within the same node and / or between different nodes of the runtime computation graph based on the hardware module dependencies and / or data dependencies of executing each execution block, so that pipeline processing can be implemented based on these parallelisms.

[0100] As mentioned above, the context storage circuit 620 maintains the information of the execution block based on the program counter. Therefore, when scheduling, the scheduling circuit 630 can call the corresponding execution block according to the program counter stored in the context storage circuit 620. When executing, the called execution block loads the corresponding context information from the context storage circuit 620 to perform the corresponding operation.

[0101] Therefore, the accelerator including the context storage circuit 620 provided in this embodiment can schedule pipeline processing based on the constructed deterministic computation graph at runtime, thereby improving hardware utilization and parallelism, thereby improving computing efficiency.

[0102] Example compiler / programming interface

[0103] The scheduling of the pipeline needs to distinguish which execution blocks or codes can be executed in parallel on different hardware modules, which is very time-consuming for dynamic neural networks at runtime. In order to overcome this problem, the dynamic pipeline system of the disclosed embodiment splits the nodes in the computation graph into several execution blocks during compilation, thereby explicitly indicating the mapping between hardware modules and code segments to facilitate scheduling at runtime.

[0104] Figure 7 An exemplary flowchart of a computational graph compilation method according to an embodiment of the present disclosure is shown.

[0105] As shown in the figure, in step 710, each node in the computation graph is split into several execution blocks to generate a compiled computation graph, each execution block represents a sub-operation of the corresponding node. These split execution blocks can be used to build a runtime computation graph and schedule it in units of execution blocks when running the compiled computation graph.

[0106] In some embodiments, the splitting step 710 may include the following sub-step 711: splitting each node into several execution blocks according to whether the hardware modules involved can be executed in parallel.

[0107] Optionally or additionally, the splitting step 710 may also include a sub-step 712:.

[0108] Figure 8 An example of computational graph splitting according to an embodiment of the present disclosure is shown.

[0109] As shown in the figure, according to the hardware modules required to execute the code, it can be correspondingly divided into an L (load) execution block 810, a C (calculation) execution block 820 and an S (save back) execution block 830.

[0110] Furthermore, adjacent codes executed in the same hardware module can be divided into the same execution block. In the example in the figure, three adjacent vector loading instructions LoadV for loading bias, weight and input respectively are executed by the same hardware module, and all load data from the off-chip storage circuit to the on-chip storage circuit, so they are all divided into the L execution block 810; three adjacent instructions for executing convolution Conv, vector addition AddV and activation Active respectively are all executed by the computing module, so they are all divided into the C execution block 820. The S execution block 830 contains only one storage back instruction, which is used to store the operation result back to the off-chip storage circuit.

[0111] By splitting the execution blocks in this way, the runtime scheduler only needs to schedule different execution blocks, which greatly simplifies the scheduling process and reduces the runtime scheduling overhead.

[0112] On the other hand, in order to maintain the context information of the execution block, the original program needs to be appropriately rewritten to save the context information during runtime.

[0113] Fig. 9 An exemplary rewritten program according to an embodiment of the present disclosure is shown, and the program shows the context information of saving the L execution block.

[0114] As shown in the figure, when the L execution block is ready, all the variables involved are stored in the context storage circuit CB. These variables include, for example, the source address, the destination address, and the size of each data. The source address is, for example, the address on the off-chip storage circuit DRAM, such as &in_ddr, &bias_ddr, &weight_ddr in the program; the destination address is the address of the on-chip storage circuit, such as &in, &bias, &weight in the program. The data size is, for example, in_size, bias_size, weight_size in the program. In addition, the program counter PC of the L execution block is also saved in CB.

[0115] As can be seen from the above exemplary rewriting program, the relevant information of the execution block and / or its storage format that needs to be stored in the context storage circuit at runtime can also be customized through the programming interface. As a result, the context storage circuit can also be implemented more flexibly. No matter what form is customized, such as which parameters are specifically stored, the parameter format, etc., it is only necessary to keep consistency in storing and reading the context information.

[0116] In addition, the dynamic pipeline system of the disclosed embodiment also provides a programming interface to facilitate the definition of each execution block, including defining the block name, execution action, etc. of the execution block.

[0117] Fig.10 An exemplary programming interface according to an embodiment of the present disclosure is shown, and this example shows the definition of an L execution block.

[0118] As shown in the figure, a keyword is added when defining the execution block: block name, which is used to indicate the hardware module involved in executing the execution block. In this example, the block name of the L execution block is IO, indicating that the L execution block involves an input and output hardware module.

[0119] In addition, the execution action or behavior of the execution block is also defined as first loading corresponding context information from the context storage circuit CB, and then executing the operation corresponding to the execution block based on the context information.

[0120] For example, in the example in the figure, context information is first read from CB, including destination address dst, source address src, data size size, etc. Then, the action of L execution block is executed, that is, data of size indicated by size is read from address src of off-chip storage circuit DRAM and stored in address dst of on-chip storage circuit.

[0121] As can be seen from the above, the compiler provided by the disclosed embodiment decomposes the operation program into two parts: one part is to maintain the context information and PC, which is convenient for subsequent flexible execution block calls; the other part is to define different execution blocks and clearly indicate the block names of the execution blocks, thereby indicating the hardware module dependencies between the execution blocks. Based on these pre-settings, the runtime scheduler can use the execution block as the basic unit to implement dynamic pipelines, and only requires a small runtime overhead.

[0122] Example Scheduling Process

[0123] Fig.11 An exemplary flow chart of a runtime scheduling method for a computation graph according to an embodiment of the present disclosure is shown. From the above description, it can be seen that based on the splitting of the computation graph at compile time and the additional context storage circuit provided, the accelerator can implement pipeline processing at runtime even for dynamic neural networks.

[0124] As shown in the figure, in step 1110, when running the compiled computation graph, in response to the preparation of the execution blocks in the computation graph, the information of the execution blocks that need to be executed in the computation graph is stored to construct the runtime computation graph.

[0125] When dynamic information such as graph structure and tensor shape in a dynamic neural network is determined, it can be determined which nodes in the compiled computational graph need to be executed and which do not. At this time, the information of the execution blocks of the nodes that are determined to be executed can be stored in a specially provided context storage circuit. Thus, the stored execution blocks can constitute a deterministic runtime computational graph. The stored execution block information may include context information and a program counter. The specific details are described above and will not be repeated here.

[0126] Next, in step 1120, according to the dependency relationship, the stored execution blocks are scheduled based on the deterministic runtime computation graph to implement pipeline operation.

[0127] Specifically, the execution blocks that can be executed in parallel within the same node and / or between different nodes of the runtime computation graph can be determined based on the hardware module dependencies and / or data dependencies of the execution of each execution block, thereby achieving pipeline processing through parallel execution on different hardware modules.

[0128] During scheduling, the corresponding execution block can be called according to the stored program counter. When the called execution block is executed, according to its definition (see the description of the compiler section above), it first loads the corresponding context information from the context storage circuit, and then performs the corresponding operation according to the context information.

[0129] Fig.12 An exemplary execution process of a runtime scheduling scheme according to an embodiment of the present disclosure is shown.

[0130] Figure (a) shows that when compiling a dynamic computation graph containing control flow, all nodes are split into several execution blocks and the corresponding compiled computation graph is generated. The execution blocks of all relevant nodes are defined at compile time according to the programming interface. The execution blocks of nodes A, B, C, and D in Figure (a) are all pre-defined.

[0131] Figures (b)-(d) show the dynamic scheduling process at runtime. At runtime, based on the determined dynamic information, only the execution block information of the nodes that need to be executed is saved in the context storage circuit CB. For example, when the condition of the control flow of the computation graph in the figure is judged to be true, the execution blocks of nodes A, B, and D will be stored in CB. Thus, a deterministic runtime computation graph can be constructed in CB.

[0132] Specifically, as shown in (b), in response to node A preparing to execute, the information of each execution block of node A to be executed is stored in the context storage circuit CB. In this example, node A is not only split into L, C and S execution blocks according to the executed hardware module, but also splits the data according to the maximum single processing capacity of the hardware, so that there can be corresponding multiple L, C and S execution blocks. For example, when the data is split into a1, a2, and a3, the corresponding execution blocks are La1, Ca1, Sa1, La2, Ca2, Sa2, La3, Ca3 and Sa3. At this time, the scheduler can schedule the execution blocks stored in CB according to the dependency relationship.

[0133] As shown in (c), in response to the node B preparing to execute, the information of each execution block of the node B to be executed is stored in the CB. In this example, the node B is also split into multiple L, C and S execution blocks according to the executed hardware module and the maximum single processing capacity of the hardware. For example, it is split into execution blocks Lb1, Cb1, Sb1, Lb2, Cb2, Sb2, Lb3, etc. At this time, the scheduler can schedule the execution blocks stored in the CB according to the dependency relationship.

[0134] The process in Figure (d) is similar to the previous one and will not be described in detail here.

[0135] In specific scheduling, scheduling within the same node and / or between different nodes may be involved. Fig.12 Take (b) in Figure 1 for example, which involves scheduling within node A. In the figure, Lai represents the load execution block of node A for the i-th data block, and Cai represents the calculation execution block of node A for the i-th data block. Due to the hardware module dependency, La2 should be executed after La1 because they both involve the same hardware module; and due to the data dependency, Ca1 should also be executed after La1. However, La2 and Ca1 can be executed in parallel. Therefore, in (b), La2 and Ca1 can be scheduled for execution at the same time.

[0136] Fig.12 Figure (c) shows the scheduling between different nodes. When node B is ready, its execution block is also stored in CB. At the same time, there are still unprocessed execution blocks of node A in CB, namely San-1, Can and San. In this scenario, the scheduler can call Lb1, San-1 and Can for parallel execution.

[0137] It can be seen that, through the dynamic water flow system provided by the embodiment of the present disclosure, there is no need to add redundant bubbles to interrupt the water flow, so efficient pipeline processing can be achieved.

[0138] The dynamic pipeline scheme of the disclosed embodiment is described above in conjunction with various drawings. Experimental results show that the dynamic pipeline scheme of the disclosed embodiment can achieve significant performance improvement for dynamic neural networks. For example, it can achieve a 1.7-fold speed improvement on dynamic neural networks. In addition, experimental results also show that the dynamic pipeline scheme of the disclosed embodiment can also be applied to static neural networks, and the overhead introduced by the context storage circuit is almost negligible. For example, it can maintain more than 96% of the performance on a static neural network. Therefore, the dynamic pipeline scheme of the disclosed embodiment is not only applicable to dynamic neural networks, but also to static neural networks, and can also be applied to mixed scenarios where both dynamic neural networks and static neural networks exist. The description above using dynamic neural networks as an example is also applicable to static neural networks or mixed scenarios.

[0139] Some embodiments of the present disclosure also provide chips and boards including the above-mentioned accelerator and / or compiler, which may include the corresponding features described above, which will not be repeated here.

[0140] According to different application scenarios, the electronic equipment or device disclosed herein may include servers, cloud servers, server computing clusters, data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, driving recorders, navigators, sensors, cameras, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, automatic driving terminals, transportation, household appliances, and / or medical equipment. The transportation includes airplanes, ships and / or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lights, gas stoves, and range hoods; the medical equipment includes magnetic resonance imaging, ultrasound machines and / or electrocardiographs. The electronic equipment or device disclosed herein may also be applied to the Internet, IoT, data centers, energy, transportation, public administration, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and medical fields. Further, the electronic equipment or device disclosed herein may also be used in cloud, edge, and terminal applications related to artificial intelligence, big data, and / or cloud computing. In one or more embodiments, electronic devices or devices with high computing power according to the disclosed solution can be applied to cloud devices (such as cloud servers), while electronic devices or devices with low power consumption can be applied to terminal devices and / or edge devices (such as smart phones or cameras). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or edge device are compatible with each other, so that according to the hardware information of the terminal device and / or edge device, appropriate hardware resources can be matched from the hardware resources of the cloud device to simulate the hardware resources of the terminal device and / or edge device, so as to complete the unified management, scheduling and collaborative work of end-to-end or cloud-edge-to-end.

[0141] It should be noted that, for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will appreciate that the scheme of the present disclosure is not limited by the order of the actions described. Therefore, based on the disclosure or teaching of the present disclosure, those skilled in the art will appreciate that some of the steps therein may be performed in other orders or simultaneously. Further, those skilled in the art will appreciate that the embodiments described in the present disclosure may be regarded as optional embodiments, i.e., the actions or modules involved therein may not necessarily be necessary for the implementation of one or some of the schemes of the present disclosure. In addition, depending on the different schemes, the present disclosure also has different focuses on the description of some embodiments. In view of this, those skilled in the art may appreciate the parts that are not described in detail in a certain embodiment of the present disclosure, and may also refer to the relevant descriptions of other embodiments.

[0142] In terms of specific implementation, based on the disclosure and teaching of this disclosure, those skilled in the art can understand that several embodiments disclosed in this disclosure can also be implemented by other methods not disclosed herein. For example, with respect to the various units in the electronic device or device embodiments described above, this article splits them on the basis of considering the logical functions, and there may be other splitting methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions in the units or components can be selectively disabled. In terms of the connection relationship between different units or components, the connection discussed in the above text in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0143] In the present disclosure, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed on multiple network units. In addition, according to actual needs, some or all of the units may be selected to achieve the purpose of the scheme described in the embodiments of the present disclosure. In addition, in some scenarios, multiple units in the embodiments of the present disclosure may be integrated into one unit or each unit may exist physically separately.

[0144] In some other implementation scenarios, the above-mentioned integrated unit can also be implemented in the form of hardware, that is, a specific hardware circuit, which may include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit may include but is not limited to physical devices, and the physical devices may include but are not limited to devices such as transistors or memristors. In view of this, the various devices described herein (such as computing devices or other processing devices) can be implemented by appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs, etc. Further, the aforementioned storage unit or storage device may be any appropriate storage medium (including magnetic storage media or magneto-optical storage media, etc.), which may be, for example, a variable resistive memory (Resistive Random Access Memory, RRAM), a dynamic random access memory (Dynamic Random Access Memory, DRAM), a static random access memory (Static Random Access Memory, SRAM), an enhanced dynamic random access memory (Enhanced Dynamic Random Access Memory, EDRAM), a high bandwidth memory (High Bandwidth Memory, HBM), a hybrid memory cube (Hybrid Memory Cube, HMC), ROM and RAM, etc.

[0145] The embodiments of the present disclosure are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method of the present disclosure and its core idea. At the same time, for those skilled in the art, according to the ideas of the present disclosure, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present disclosure.

Claims

1. A runtime scheduling method for a computation graph, wherein the computation graph is a compiled computation graph, and the compiled computation graph includes a plurality of execution blocks, wherein the execution blocks represent sub-operations of each node in the computation graph, and the method comprises: When running the compiled computation graph, in response to the preparation of execution blocks, storing information of execution blocks that need to be executed in the compiled computation graph to construct a runtime computation graph; and According to the dependency relationship, the stored execution blocks are scheduled based on the runtime computation graph to implement pipeline operation.

2. The method according to claim 1, wherein the scheduling comprises: According to the hardware module dependency and / or data dependency of executing the execution blocks, the execution blocks that can be executed in parallel within the same node and / or between different nodes of the runtime computation graph are determined.

3. The method according to claim 1, wherein the storing comprises: The context information and the program counter of the execution block to be executed are stored.

4. The method according to claim 3, wherein the context information includes at least one of the following: a data source address, a data destination address, a tensor data shape, and a computing configuration.

5. The method of claim 3, wherein the scheduling comprises: According to the stored program counter, the corresponding execution block is called.

6. The method according to claim 5, further comprising: When the called execution block is executed, the stored context information is loaded to perform corresponding operations.

7. The method according to any one of claims 1 to 6, wherein each node in the compiled computation graph is split into a plurality of execution blocks as follows during compilation: Each node is split according to whether the hardware modules involved can be executed in parallel.

8. The method according to claim 7, wherein each node in the compiled computation graph is further split into several execution blocks as follows during compilation: Group adjacent code executed in the same hardware module into the same execution block.

9. The method according to any one of claims 1-6, wherein each of the execution blocks has a block name to indicate the hardware modules involved in executing the current execution block, so as to indicate the hardware module dependencies during scheduling.

10. The method according to any one of claims 1 to 6, wherein: The compiled computation graph is a dynamic computation graph, and the runtime computation graph is a deterministic computation graph.

11. A method for compiling a computational graph, comprising: Each node in the computation graph is split into a plurality of execution blocks to generate a compiled computation graph, each execution block represents a sub-operation of a corresponding node, and the execution blocks are used to construct a runtime computation graph and perform scheduling based on the execution blocks when running the compiled computation graph.

12. The method according to claim 11, further comprising: Each node is split into several execution blocks according to whether the hardware modules involved can be executed in parallel.

13. The method according to claim 12, further comprising: Group adjacent code executed in the same hardware module into the same execution block.

14. The method according to any one of claims 11 to 13, further comprising: A block name is defined for each execution block to indicate the hardware module involved in executing the current execution block.

15. The method according to any one of claims 11 to 13, further comprising: Customize the information about the execution block that needs to be stored at runtime and / or its storage format. The method according to claim 15 , wherein the relevant information of the execution block comprises context information and a program counter of the execution block.

17. The method according to claim 16, wherein the context information comprises at least one of the following: Data source address, data destination address, tensor data shape, and calculation configuration.

18. The method according to any one of claims 16-17, further comprising: The behavior of the execution block is defined as first loading the stored context information, and then executing the operation corresponding to the execution block based on the context information.

19. An accelerator, comprising: A parallel operation circuit, used for executing operations in parallel; A context storage circuit is used to store information of an execution block that needs to be executed in the compiled computation graph to construct a runtime computation graph in response to a prepared execution of the execution block when running the compiled computation graph, wherein the compiled computation graph includes a plurality of execution blocks, and the execution blocks represent sub-operations of each node in the computation graph; as well as The scheduling circuit is used to schedule the stored execution blocks based on the runtime calculation graph according to the dependency relationship to control the parallel operation circuit to realize pipeline operation. 20 . The accelerator according to claim 19 , wherein the information of the execution block comprises context information and a program counter. 21 . The accelerator according to claim 20 , wherein the context information comprises at least one of the following: a data source address, a data destination address, a tensor data shape, and a computing configuration.

22. The accelerator according to claim 19, wherein the scheduling circuit is further configured to: According to the hardware module dependency and / or data dependency of executing the execution blocks, the execution blocks that can be executed in parallel within the same node and / or between different nodes of the runtime computation graph are determined.

23. The accelerator according to claim 20, wherein the scheduling circuit is further configured to: According to the stored program counter, the corresponding execution block is called. 24 . The accelerator according to claim 23 , wherein the called execution block loads the corresponding context information from the context storage circuit when executing to perform the corresponding operation.

25. The accelerator according to any one of claims 19 to 24, wherein each node in the compiled computation graph is split into a plurality of execution blocks during compilation in the following manner: Each node is split according to whether the hardware modules involved can be executed in parallel.

26. The accelerator according to claim 25, wherein each node in the compiled computation graph is further split into a plurality of execution blocks during compilation as follows: Group adjacent code executed in the same hardware module into the same execution block.

27. The accelerator according to any one of claims 19-24, wherein each of the execution blocks has a block name to indicate the hardware modules involved in executing the current execution block, so as to indicate the hardware module dependency to the scheduling circuit.

28. The accelerator according to any one of claims 19 to 24, wherein the context storage circuit is an on-chip storage circuit of the accelerator.

29. An accelerator according to any one of claims 19-24, wherein the compiled computation graph is a dynamic computation graph, and the runtime computation graph is a deterministic computation graph.

30. A compiler, comprising: A processing circuit configured to execute the compiling method according to any one of claims 11-18.

31. A chip comprising the accelerator according to any one of claims 19-29 and / or the compiler according to claim 30.

32. A board comprising the chip according to claim 31.