Computational graph compilation methods and related products

Abstract

The present disclosure discloses a method for compiling a computational graph and related products. The method includes: obtaining a computational graph to be compiled, expressed by a second intermediate representation, performing forward deduction of the shape, and obtaining complete shape information based on the forward deduction and tensor data splitting information; using the complete shape information, determining whether the tensor data splitting information needs to be adjusted; determining tensor data splitting information that meets the requirements based on the judgment result; deducing memory access modes for operators in the computational graph based on the tensor data splitting information that meets the requirements; determining address domain-related parameters of instructions involved in a loop in the code logic of the computational graph based on the deduced memory access mode; pipeline-arranging the instructions in the loop; and compiling the computational graph based on the address domain-related parameters of the instructions involved in the loop and the result of the instruction pipeline arrangement to obtain a binary file recognizable by an intelligent processor.

Description

Technical Field

[0001] The present disclosure generally relates to the field of intelligent computing, and more particularly to the field of neural networks. More specifically, the present disclosure relates to a method for compiling a computational graph, a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] In intelligent computing systems, deep learning frameworks serve as the primary programming interface for building corresponding machine learning / deep learning networks for machine learning algorithm models in various machine learning applications. Deep learning frameworks include a variety of programming frameworks, such as Google's TensorFlow, and deep learning frameworks like Caffe, Caffe2, MXNet, and PyTorch. These frameworks typically use computational graphs (or intermediate representations (IRs)) to describe the computational process of machine learning algorithms. Tensors represent all data in the computational graph, and operators represent various operations.

[0003] Various graph-level optimizations can be performed on the IR graph. These optimizations include, but are not limited to, constant folding, operator fusion, and graph substitution. The optimized IR graph can be compiled into an abstract language representation, or various instructions. These instructions can be executed by the underlying machine learning hardware.

[0004] During the compilation process from computational graphs to underlying machine instruction code, several fixed levels of abstraction are typically introduced, gradually descenting the language's intermediate representation into an intermediate representation supported by the target compilation framework. Designing and implementing a universal compilation technology approach across these multiple levels of abstraction is a key challenge facing optimizing compilers. Summary of the Invention

[0005] To address at least one or more of the technical issues mentioned above, this disclosure proposes a computational graph compilation solution in multiple aspects. This solution introduces a TFUIR layer during compilation. This layer optimizes the computational graph from the perspective of operator fusion, effectively taking into account information from both the algorithm software and the execution hardware, thereby improving compiler optimization effectiveness. In other aspects, this disclosure also provides compilation solutions for transitioning from a higher level to the TFUIR layer, as well as for transitioning from the TFUIR layer to a lower level.

[0006] In a first aspect, the present disclosure discloses a method for compiling a computation graph, wherein the method comprises:

[0007] Obtaining a computational graph to be compiled, expressed by a second intermediate representation, performing forward derivation of the shape, and obtaining complete shape information based on the forward derivation and tensor data splitting information; wherein the second intermediate representation is obtained by conversion based on the first intermediate representation, and the first intermediate representation is obtained by unified conversion based on a task model expressed in at least one front-end language; and the computational graph is a task executed on an intelligent processor;

[0008] Using the complete shape information, determining whether it is necessary to adjust the tensor data splitting information;

[0009] Determine the tensor data splitting information that meets the requirements according to the judgment result;

[0010] Derivation of memory access modes for operators in the computation graph based on the tensor data splitting information that meets the requirements;

[0011] Determining address domain-related parameters of instructions involved in a loop in the code logic of the computation graph based on the derived memory access pattern;

[0012] Pipelining the instructions in the loop based on the instruction stream supported by the intelligent processor;

[0013] The computation graph is compiled according to address domain related parameters of the instructions involved in the loop and the result of the instruction pipeline arrangement in the loop to obtain a binary file recognizable by the intelligent processor.

[0014] In a second aspect, the present disclosure provides a processing device for performing compilation on a computation graph, comprising:

[0015] a processor configured to execute program instructions; and

[0016] A memory configured to store the program instructions, which, when loaded and executed by the processor, causes the processor to execute the computation graph compilation method according to the first aspect of the present disclosure.

[0017] In a third aspect, the present disclosure provides a computer-readable storage medium having program instructions stored therein. When the program instructions are loaded and executed by a processor, the processor executes the computation graph compilation method according to the first aspect of the present disclosure.

[0018] In a fourth aspect, the present disclosure provides a computer program product, comprising a computer program or instructions, which, when executed by a processor, implements the method for compiling the computational graph according to the first aspect of the present disclosure.

[0019] In the compilation scheme of the computational graph provided above, the disclosed embodiment specifically introduces the concept of the second intermediate expression TFUIR level and the inter-level conversion scheme associated with this level, targeting operator fusion, an important optimization method in neural networks, thereby providing an efficient optimization compilation scheme. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts.

[0021] Figure 1 A structural diagram of a board according to an embodiment of the present disclosure is shown;

[0022] Figure 2 A structural diagram showing a combined processing device according to an embodiment of the present disclosure;

[0023] Figure 3a A schematic diagram showing the internal structure of a single processor core of a single-core computing device according to an embodiment of the present disclosure;

[0024] Figure 3b A simplified schematic diagram showing the internal structure of a multi-core computing device according to an embodiment of the present disclosure;

[0025] Figure 4 Shows the schematic structure of the software stack;

[0026] Figure 5 The hierarchical structure of the optimization compiler of the embodiment of the present disclosure is exemplarily shown;

[0027] Figure 6 An exemplary flow chart illustrating a second conversion process from IRO to TFUIR implemented by a processing device according to an embodiment of the present disclosure;

[0028] Figure 7 An exemplary flow chart illustrating a third conversion process from TFUIR to IR1 implemented by a processing device according to an embodiment of the present disclosure;

[0029] Figure 8 A flowchart illustrating an exemplary method for generating instructions at compile time, implemented by a computing device, according to some embodiments of the present disclosure;

[0030] Figure 9 A flowchart of an exemplary method for generating instructions at compile time, performed by a computing device, according to other embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0031] The following will clearly and completely describe the technical solutions in the embodiments of this disclosure in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this disclosure, not all of them. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this disclosure.

[0032] It should be understood that the terms "first," "second," "third," and "fourth," etc., which may appear in the claims, specification, and drawings of this disclosure, are used to distinguish different objects rather than to describe a specific order. The terms "include" and "comprising" used in the specification and claims of this disclosure indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0033] It should also be understood that the terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the disclosure. As used in this disclosure and the claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" as used in this disclosure and the claims refers to any and all possible combinations of one or more of the associated listed items, including and including these combinations.

[0034] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context.

[0035] The specific embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0036] Exemplary Hardware Architecture

[0037] Figure 1 FIG. 1 is a schematic diagram showing the structure of a board 10 according to an embodiment of the present disclosure. Figure 1As shown, board 10 includes chip 101, which is a system-on-chip (SoC), or system-on-chip, integrated with one or more combined processing devices. The combined processing device is an artificial intelligence computing unit that supports various deep learning and machine learning algorithms to meet the intelligent processing needs in complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in the field of cloud intelligence. A notable feature of cloud-based intelligent applications is the large amount of input data, which places high demands on the platform's storage and computing capabilities. Board 10 of this embodiment is suitable for cloud-based intelligent applications and has extensive off-chip storage, on-chip storage, and powerful computing capabilities.

[0038] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. Calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface formats, such as a PCIe interface.

[0039] Board 10 also includes a memory device 104 for storing data, which includes one or more storage units 105. Memory device 104 is connected to a control device 106 and chip 101 via a bus for data transmission. Control device 106 in board 10 is configured to control the state of chip 101. To this end, in one application scenario, control device 106 may include a microcontroller (MCU).

[0040] Figure 2 FIG. 1 is a block diagram showing the combined processing device in the chip 101 of this embodiment. Figure 2 As shown in , the combined processing device 20 includes a computing device 201 , an interface device 202 , a processing device 203 and a storage device 204 .

[0041] The computing device 201 is configured to perform user-specified operations and is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0042] Interface device 202 is used to transmit data and control instructions between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to a storage device on-chip of computing device 201. Furthermore, computing device 201 can obtain control instructions from processing device 203 via interface device 202 and write them to a control cache on-chip of computing device 201. Alternatively or optionally, interface device 202 can also read data from the storage device of computing device 201 and transmit it to processing device 203.

[0043] The processing device 203 is a general processing device that performs basic controls including but not limited to data handling, starting and / or stopping the computing device 201. Depending on the implementation, the processing device 203 can be a central processing unit (CPU), a graphics processing unit (GPU) or one or more types of processors in other general and / or special processors, which include but are not limited to digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, with respect to the computing device 201 disclosed herein, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are integrated and considered together, the two are regarded as forming a heterogeneous multi-core structure.

[0044] The storage device 204 is used to store data to be processed, which may be DRAM or DDR memory, and is typically 16G or larger in size, for storing data of the computing device 201 and / or the processing device 203 .

[0045] When computing device 201 runs a neural network, it typically first compiles the neural network using processing device 203 to produce an executable file. This executable file contains device information, indicating which device in the heterogeneous computer system the executable file should be executed on. After the executable files are assembled and linked, the neural network executable program is generated and stored in storage device 204.

[0046] The processing device 203 can read the executable program from its storage location and obtain multiple tasks of the program according to the executable program. These tasks are distributed to the computing device 201 for execution via the interface device 202 to finally obtain the calculation results.

[0047] Figure 3a Shown Figure 2 Schematic diagram of the internal structure of the processing core when computing device 201 in FIG. is a single-core device. Computing device 301 is used to process input data for computer vision, speech, natural language processing, data mining, and other purposes. Computing device 301 includes three modules: a control module 31 (also known as a controller), an operation module 32 (also known as an operator), and a storage module 33 (also known as a memory).

[0048] The control module 31 coordinates and controls the operations of the computing module 32 and the storage module 33 to complete deep learning tasks. It includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 retrieves instructions from the processing device 203, while the instruction decode unit 312 decodes the retrieved instructions and sends the decoded results as control information to the computing module 32 and the storage module 33.

[0049] The operation module 32 includes a vector operation unit 321 and a matrix operation unit 322. The vector operation unit 321 is used to perform vector operations, supporting complex operations such as vector multiplication, addition, and nonlinear transformations. The matrix operation unit 322 is responsible for the core calculations of the deep learning algorithm, namely: matrix multiplication and convolution.

[0050] Storage module 33 is used to store or transfer relevant data and includes neuron RAM (NRAM) 331, weight RAM (WRAM) 332, and direct memory access (DMA) 333. NRAM 331 is used to store input neurons, output neurons, and intermediate results after calculation; WRAM 332 is used to store the convolution kernels of the deep learning network, i.e., the weights; DMA 333 is connected to DRAM 204 via bus 34 and is responsible for data transfer between computing device 301 and DRAM 204. It should be noted that the NRAM and WRAM here can be two storage areas formed by logically dividing the same memory, or they can be two independent memories, and this is not specifically limited here.

[0051] Figure 3bThe figure shows a simplified schematic diagram of the internal structure of a multi-core computing device 201. A multi-core computing device can be abstracted using a hierarchical hardware model. As shown in the figure, the multi-core computing device can be abstracted into three levels: chip level (Chip) 360, processor cluster level (Cluster) 370, and processor core level (Core) 380. The disclosed embodiments mainly involve data transmission of storage units and computing units. Therefore, the figures and description briefly illustrate and introduce the relevant computing structures, omitting other parts.

[0052] At the chip level, each chip contains local DDR storage, each processor chip serves as a computing and control unit, and each processor chip contains multiple multiprocessors as computing units.

[0053] At the processor cluster level, each multiprocessor includes multiple accelerator cores as control and computing units, and a shared memory SRAM as a storage unit.

[0054] At the processor core level, each accelerator core contains local storage and a local processing unit array. NFU refers to the Neuron Function Unit, which is used for convolution calculations. The structure of a single processor core can be similar to Figure 3a The structural diagram of the single-core computing device shown is not described in detail here.

[0055] In this multi-core computing device, the storage model includes the global memory of the board, SRAM (shared memory) on the Cluster, NRAM, WRAM and registers on the Core, etc. In order to obtain better performance, the data movement between the storage levels below the Card and the balance between memory access / computation can be explicitly controlled. SRAM is contained in the memory processing unit MPU (Memory Process Unit Core, abbreviated as MPU, or Mem Core). Core refers to the intelligent processing core (Intelligent Process Unit Core, abbreviated as IPU Core or Core) in the multi-core computing device. 1 IPU Core contains NRAM, WRAM, NFU, etc. Cluster refers to a processor cluster or a computing cluster. Usually a multi-core computing device contains several Clusters, and a Cluster contains 1 Mem Core + N IPU Cores.

[0056] Exemplary Software Stack

[0057] Figure 4 FIG4 shows a schematic structure of a software stack 400. As shown in the figure, the software stack 400 includes a machine learning application 410, a deep learning framework 420, an IR graph 430, machine execution instructions 440, and machine learning hardware 450.

[0058] The machine learning application 410 provides corresponding machine learning algorithm models for different application scenarios. These algorithm models can be written in various high-level programming languages.

[0059] The deep learning framework 420 may include a variety of programming frameworks, such as Google's tensor flow graph TensorFlow, deep learning frameworks Caffe, Caffe2, MXNet, PyTorch, etc. For the machine learning algorithm models of various machine learning applications 410, the deep learning framework 420 is used as the main programming interface to parse these algorithm models to build corresponding machine learning / deep learning networks. During the compilation period, the deep learning framework 420 can provide trained algorithm models. The deep learning framework 420 can convert the constructed network into a graph-level intermediate representation (IR), referred to as an IR graph or computational graph 430. The nodes in the IR graph represent operations or operators, which can be applied to a certain input and can generate outputs passed to other nodes. The edges in the IR graph represent the flow of data.

[0060] During compilation, various graph-level optimizations can be performed on the IR graph 430. These optimizations include, but are not limited to, constant folding, operator fusion, and graph substitution. The optimized IR graph can be compiled into an assembly-like file. This file is converted into machine language instructions, which are then packaged into a relocatable target program format.

[0061] For machine-executable instructions 440, at runtime, the relocatable target program is just-in-time compiled into machine-executable instructions 440 by the runtime compilation library. These instructions can be executed by the underlying machine learning hardware 450. The machine learning hardware 450 can be, for example, a chip in the exemplary hardware architecture described above.

[0062] The compiler is responsible for converting hardware-independent IR graphs 430, written in a high-level language, into machine-executable instructions 440 that can be efficiently executed on specific underlying hardware. During the conversion process, the compiler provides various optimizations, enabling the optimized program to run more efficiently on the hardware during the execution phase.

[0063] Because compilers must bridge the significant gap between the upper-level programming model and the underlying hardware, using a single layer of compiler abstraction is difficult to fully improve program performance. Consequently, an increasing number of compilers are adopting a multi-level abstraction approach. Designing and implementing a common compilation technology approach within this multi-level abstraction hierarchy is a major challenge facing optimizing compilers.

[0064] Example compilation architecture

[0065] In an embodiment of the present disclosure, an optimizing compiler is provided, which divides the intermediate representation IR into three levels: IR0, TFUIR and IR1. At different IR levels, optimization processing with different focuses is performed, so that hardware-independent programs can be converted into machine instructions that can be efficiently executed on specific hardware.

[0066] Figure 5 The hierarchical structure of the optimizing compiler 500 according to the embodiment of the present disclosure is exemplified.

[0067] As shown in the figure, the compiler 500 can receive trained algorithm models 510 provided by various deep learning frameworks. The compiler 500 converts these algorithm models 510 programmed based on the deep learning framework into a unified computational graph representation, namely the first layer IR: IR0 520 through the first conversion (501).

[0068] The first conversion is mainly used to set the operator type based on different frameworks, and to convert the current numerous deep learning frameworks that use task models written in various languages ​​into a computational graph represented by IR0, so that subsequent model optimization and code generation can be implemented based on this unified computational graph. Since the first conversion focuses on processing the conversion of various front-end languages ​​to a unified form of computational graph, IR0 only reflects the information of the algorithm model itself, which is not related to the specific hardware to be executed. IR0 can be an intermediate representation defined based on the MLIR (multi-level IR) compiler framework.

[0069] Here is an example of IR0, which has the following text form:

[0070] module{

[0071] func.foo(%arg0,%arg1,%arg2){

[0072] %0=mm.conv2d(%arg0,%arg1);

[0073] %1=mm.scale(%0,%arg2);

[0074] %2=mm.relu(%1);

[0075] return %2;}

[0076] }

[0077] In the above example, a function foo is described, which takes three parameters, arg0, arg1, and arg2, and sequentially performs a convolution operation (conv2d), a scaling operation (scale), and an activation operation (relu), and finally returns the result. As can be seen from this example, IR0 only reflects the operational logic at the software algorithm level and does not contain any hardware-related information.

[0078] Next, the compiler 500 converts IR0 520 into a second-level IR: TFUIR 530 through a second conversion (502). The second conversion primarily maps the operators expressed in IR0 to operators expressed in TFUIR from the perspective of operator fusion, based on hardware resource information such as hardware fusion instructions and hardware-supported data copy behaviors. For example, after the second conversion, the conv operator in IR0 may correspond to the conv operator, fusion_conv operator, or fold_conv operator expressed in TFUIR, depending on the scale parameter. The TFUIR operator 530 is an abstract representation based on the operator fusion solution. Based on the expression of TFU IR 530, optimization operations such as pipeline drainage and graph optimization are carried out for the TFU-IR operator during the operator fusion process.

[0079] Operator fusion is a typical computational graph optimization method that can calculate multiple operators together in a single kernel without saving the intermediate results back to the global memory, thereby effectively reducing input / output access in network model operations, that is, reducing IO memory access.

[0080] In the second conversion, automatic fusion technology is used to automatically search the computational graph expressed by IR0 based on data-driven and performance evaluation models, and finally obtain the optimal operator fusion method. Then, IR conversion and optimization processing are performed based on the fused operators.

[0081] Specifically, the automatic fusion technology traverses various fusion feasibility scenarios and data splitting feasibility scenarios in the computation graph represented by IR0 to obtain a list of operators in the computation graph with fusion potential and a scheduling strategy. Within the scheduling strategy, information influencing operator fusion performance, such as the list of operators with fusion potential, data splitting method, pipelining method, weight residency, and weight preloading, is uniformly defined as control data. This transforms the search for the optimal operator fusion strategy into a search for optimal control data. In this way, operators with connected inputs / outputs in the computation graph are allowed to fuse. Furthermore, during the automatic fusion process, a performance evaluation function for the fused operator can be generated based on the control data of the fused operator, tailored to specific hardware information. Based on this performance evaluation function, the fusion benefit of the fused operator can be calculated, and the optimal fusion strategy can be selected based on the fusion benefit. This approach automatically finds the optimal operator fusion strategy for various networks.

[0082] As can be seen from the second conversion process, the converted TFUIR not only reflects the logical operation information of the algorithm model, but also reflects the underlying hardware information due to the need to consider specific hardware information during the automatic fusion process. The granularity of this hardware information depends on the automatic fusion process. In other words, the TFU IR incorporates some hardware information based on operator fusion. Through operator fusion, the computational graph represented by IR0 is converted to that represented by TFUIR. During the execution of tasks, the AI ​​processor does not save intermediate results back to global memory, effectively reducing input / output accesses in network model operations, that is, reducing I / O memory accesses, thereby reducing I / O bottlenecks.

[0083] Here is an example of a TFUIR, which has the following text form:

[0084] module{

[0085] func.foo(%arg0,%arg1,%arg2){

[0086] %0=tfu.fusion_conv(%arg0,%arg1,%arg2);

[0087] return %0;}

[0088] }

[0089] It can be seen that this TFUIR example continues the previous IR0 example, fusing the three operators of convolution operation (conv2d), scaling operation (scale) and activation operation (relu) into a fused convolution operator (fusion_conv).

[0090] Next, compiler 500 converts TFUIR 530 into a third-level IR: IR1 540, through a third transformation (503). The third transformation primarily optimizes the IR from a hardware execution perspective, while IR1 540 is a completely hardware-specific abstract representation. IR1 contains rich hardware information, which the underlying hardware can use to execute tasks corresponding to the neural network model.

[0091] In the third conversion, the IR is optimized based on information about the underlying execution hardware. This involves adjusting the splitting scheme in the previously determined automatic fusion solution. Code generation is then performed to obtain an abstract representation that can be provided to the runtime compilation library for further compilation and ultimately the generation of machine-readable binary instructions. The TFU IR expression is converted to an IR1 expression, clarifying which hardware information appears in the TFU IR expression and which in the IR1 expression, similar to stacking building blocks. Because the software stack itself is designed to fully utilize hardware resources, locating software code issues can be more easily done, ultimately leading to more efficient problem resolution.

[0092] Thus, the disclosed embodiments provide a compiler comprising three IR levels, specifically introducing the intermediate TFUIR level for operator fusion, an important computational graph optimization method. The TFUIR level, based on the operator fusion scheme, performs more refined computational graph optimization processing at the granularity of fused operators. More specifically, by converting the first level IR0 into the intermediate TFUIR level, hardware characteristics can be displayed at the first granularity (e.g., the granularity of operator fusion), and graph-level optimization can be performed based on this granularity. This graph-level optimization is an optimization with hardware information, which is different from the graph-level optimization at the IR0 stage. In addition, the TFUIR expression does not contain all hardware information, which is more programmer-friendly and more efficient when locating code problems. Subsequently, after the code generation (Codegen) stage, the TFUIR level is converted to the third level IR1, which can display hardware characteristics at a finer granularity. Thus, by introducing hardware information layer by layer, a high-level, hardware-independent computational graph can be ultimately converted into a low-level, fully hardware-dependent computational graph and provided to the runtime compilation library to generate machine-recognized binary instructions. In summary, the compilation scheme of the disclosed embodiment provides an effective performance optimization solution for accelerating the processing of neural network models on hardware execution platforms.

[0093] The following describes the conversion process between the various IR levels. The first conversion from the algorithm model provided by the deep learning framework to IR0 focuses on the differences between various front-end languages. Various known conversion methods can be used, which are not limited or described in detail in the disclosed embodiments.

[0094] Conversion from IR0 to TFUIR

[0095] As mentioned above, in the second conversion from IR0 to TFUIR, the computational graph is converted and expressed mainly from the perspective of operator fusion.

[0096] Figure 6FIG2 is an exemplary flow chart illustrating a second conversion process from IR0 to TFUIR implemented by a processing device according to an embodiment of the present disclosure. The processing device may be, for example, Figure 2 processing device 203.

[0097] As shown in the figure, in step 610, automatic fusion technology is first used to automatically search for the optimal operator fusion method from the computation graph represented by IR0. For more information on automatic fusion technology, please refer to Chinese invention patent application CN202211372965.4, entitled "Automatic Fusion Method for Operators in Computational Graphs and Related Products." Since this is not the focus of the present embodiment, this article will not elaborate on this aspect and will only describe the parts related to the present embodiment.

[0098] Automatic fusion technology traverses various fusion feasibility scenarios and data splitting feasibility scenarios in the computation graph, considers various scheduling strategies (for example, pipelining, whether weight retention is enabled, whether weight preloading is enabled, etc.), and selects the optimal operator fusion method through performance evaluation functions. Therefore, after automatic fusion processing, the optimal operator fusion method is provided.

[0099] The operator fusion method can include the following information: the operator sequence that can be fused and has the best performance; descriptive files or structured information about the fusion level; and data splitting information.

[0100] An operator sequence can be a sequence of operators extracted directly from the computational graph represented by IR0 based on the results of the automatic fusion process. For example, the convolution operator (conv2d), scaling operator (scale), and activation operator (relu) shown in the previous example. This operator sequence only provides a list of operators that can be fused, without reflecting further fusion information, such as the fusion level.

[0101] The fusion level can be represented by a descriptive file or structured information of the fusion level. Since operator fusion aims to reduce IO memory access, the fusion level can be determined based on the storage level of the hardware. Figure 3bAs can be seen from the described hardware architecture, the storage hierarchy can include three layers: the NRAM, WRAM, and other memories on the IPUs of each processor core in the processor core level 380 are abstracted as the L1 storage level, the SRAM memory on the MPUs in the processor cluster level 370 are abstracted as the L2 storage level, and the global memory of the chip level 360 is the third storage level. The fusion-optimized computational graph can also be divided into graphs and subgraphs accordingly. When the fusion-optimized network model is executed in these hardware environments, the input and output of the graph are the global memory of the chip level 360, such as DDR, and the intermediate results reside in the L2 storage level. One graph corresponds to one fusion operator. A graph can include several subgraphs (fusion sub-operators). The input and output of each subgraph are the L2 storage level, and the intermediate results reside in the L1 storage level. A subgraph can include several operators. The input and output of each operator are the L1 storage level.

[0102] In some embodiments, the fusion level information of the fusion operator can be represented by a structured text description of the graph and subgraph. For example, an example of a structured text description of the fusion level is given below.

[0103] Graph1{

[0104] subgraph1{

[0105] Conv1

[0106] Add1

[0107] Relu1

[0108] }

[0109] subgraph2{

[0110] conv2

[0111] scale2

[0112] bn1

[0113] }

[0114] Graph2{ ...

[0116] }

[0117] The above example shows two graphs: graph1 and graph2, whose inputs and outputs are all chip-level global memory (such as DDR). That is, graph1 and graph2 both fetch data from DDR, perform internal operations, and then store the final results back to DDR. The above example further shows that graph1 includes two subgraphs: subgraph1 and subgraph2, whose inputs and outputs are all L2 storage levels. That is, subgraph1 and subgraph2 both fetch data from the shared cache SRAM on the MPU, perform internal operations, and then store the final results back to SRAM. Each subgraph includes several operators, and the inputs and outputs of these operators are L1 storage levels, that is, NRAM, WRAM, etc. on the IPU.

[0118] In other embodiments, fusion-level information can be described directly at the IRO level using operator calls. In these embodiments, each subgraph can be considered an operator call, for example, using a callop. Furthermore, the operators contained in the corresponding callop can be defined.

[0119] Data splitting information is used to represent the data splitting scheme in the operator fusion scheme. Specifically, the data splitting scheme includes parallel and serial data splitting between different levels of storage. Each splitting scheme involves the data splitting dimension and split block size, etc.

[0120] For example, in Figure 3b In the environment shown in the figure where the chip contains two storage layers, the data splitting scheme of the fusion operator includes a parallel splitting method of multiple L2 storage layers of the fusion operator, a serial splitting method of a single L2 storage layer of the fusion operator, a parallel splitting method of multiple L1 storage layers of each fusion sub-operator, and a serial splitting method of a single L1 storage layer of each fusion sub-operator.

[0121] The parallel splitting method of multiple L2 storage levels of the fusion operator involves the hardware information of the computing device running the computation graph, such as Figure 3b In the hardware environment shown, the splitting between clusters is involved at the processor cluster level.

[0122] The serial splitting method of a single L2 storage level of the fusion operator involves the splitting of the input data, and the parameters mainly include the dimensions of the splitting. For example, when the HW (height and width) dimension of the input feature map (input data) does not exceed the first threshold, such as when the entire feature map can be stored on a single L2 storage level, it can be considered as a "small image mode" at this time, and its splitting dimension is N (batch) dimension. For another example, when the HW dimension of the input feature map exceeds the second threshold, it can be considered as a "large image mode" at this time, and its splitting dimension can include three dimensions: N, H, and W. It can be understood that the parameter setting of the serial splitting method is only exemplary, and those skilled in the art can also construct more splitting methods, such as "medium image mode", "super large image mode", etc., and the disclosed embodiment has no limitation in this respect. In addition, the parameters of the serial splitting method can also include the specific splitting block size of each splitting dimension.

[0123] The parallel splitting of the multiple L1 storage levels of each fusion operator is mainly related to the dimensions of the input feature map. These dimensions may include, for example, the N (batch) dimension, H (height) dimension, W (width) dimension, and C (channel) dimension of the input feature map. The specific split value of each dimension can be determined, which depends on, for example, the number of computing components in the hardware implementation.

[0124] Similarly, the serial splitting method of a single L1 storage level of each fusion operator also involves the splitting of the input feature map, and the parameters mainly include the split dimension and the split block size.

[0125] In summary, data splitting information can include the following aspects:

[0126] Whether to split the data serially or in parallel can be indicated by the value of the parameter by-factor-or-by-part, for example: 1 for serial, 0 for parallel, and vice versa. The factor in serial splitting indicates the size of each part, which usually depends on the processing power of the computing hardware. The part in parallel splitting indicates the number of parts to be split, which usually depends on the number of processor cores in the parallel processing hardware.

[0127] b. Split dimension: This parameter indicates the dimension on which the data is split. It can be any dimension such as N, H, W, C, etc.

[0128] c. Split coefficient. This parameter represents the value of the factor or part above, and has corresponding meanings depending on whether the split is serial or parallel.

[0129] d. Bind-axis: This parameter is used to indicate the mapping or binding relationship between the parallel splitting scheme and the hardware. It can be understood that the parallel splitting scheme can be split in parallel at different levels, such as the L2 level or the L1 level. Accordingly, these parallel splitting schemes require the support of parallel processing hardware at different levels. The binding axis is used to indicate the corresponding parallel processing hardware dimension. For example, assuming that the parallel split part = 4, the splitting dimension is N dimension, and the binding axis is dim-y, it means that the N dimension is split into 4 parts, each corresponding to a dim-y.

[0130] The dim-x, dim-y, and dim-z of the binding axis can be defined as hierarchical dimensions corresponding to different hardware (operators) according to different hardware architectures. In one example, tasks are issued in units of jobs. Usually, jobs represent independent tasks that have no data dependencies on each other, such as processing different data blocks. Each job has its own instruction file. Each job can have different hardware units on different execution hardware. For example, on execution hardware A, a job is issued in units of processor clusters (see Figure 3b On the same execution hardware, a job can also have different job types, and the number of hardware units contained in different job types can be different. For example, job types can include U1 (1 processor cluster), U2 (2 processor clusters), U4 (4 processor clusters), etc. At this time, dim-x can represent the task type of the currently issued job. More specifically, dim-x can indicate the number of executable processing cores in the current job. For example, Figure 3b For example, if a processor cluster contains four cores, then a U1 job corresponds to a dim-x value of 4, a U2 job corresponds to a dim-x value of 8, and so on. dim-y represents the number of jobs. dim-z can represent higher dimensions, such as the number of kernels to start.

[0131] Optionally or additionally, the operator fusion method may also include some configuration information, such as whether to enable pipelining, whether to enable weight retention, and so on.

[0132] The pipelining method for IO / compute parallelism of a fused operator or fused sub-operators primarily involves whether pipelining is enabled, and if so, the number of pipelining stages used. For example, the configuration information can be set to disable pipelining, so that IO access and computation are not performed in parallel. Another example is that the configuration information can be set to enable pipelining, and for some modes, the three-stage LCS pipelining is enabled, while for other modes, the five-stage LLCS pipelining is enabled, where L stands for load, C stands for compute, and S stands for store.

[0133] Typically in neural network calculations, the weights of each network layer are fixed, so the resident data is usually the weights. In this regard, data residency is sometimes also called weight residency. Through data residency, the time for repeated data loading can be effectively reduced, the number of data reuses can be increased, and the overall calculation can be accelerated. The configuration information can be set to non-resident or resident. When selecting residency, it can further include which model data (such as which operator weights) resides in the L1 storage layer and which model data resides in the L2 storage layer.

[0134] continue Figure 6 Then, in step 620, based on the optimal operator fusion method determined in the previous step, the computation graph at the IR0 level is converted to a computation graph at the TFUIR level. During the conversion process, various basic processing is performed on the operators in the computation graph represented by IR0 using the optimal operator fusion method determined in the previous step. Optionally or additionally, some optimization processing may also be performed.

[0135] Basic processing may include, but is not limited to, replacement of instructions, addition of memory scopes, addition of memory operations, etc.

[0136] The replacement at the instruction level includes replacing multiple operator instructions separated in the computation graph at the IR0 level with a single operator instruction based on information about the operator fusion method, such as the descriptive file of the previous fusion level.

[0137] Specifically, in some implementations, based on the fusion operator instructions supported by the hardware that will execute the computation graph, the corresponding multiple separate operator instructions are replaced with a single fusion operator instruction. For example: if the hardware supports convolution fusion instructions (for example, the convolution operator and the subsequent addition operator and relu operator are fused into one convolution instruction, which is called a convolution fusion instruction), and the determined operator fusion method indicates that the three operators are fused together, then the three operator instructions in the computation graph at the IR0 level can be replaced with a single convolution fusion instruction. In some implementations, certain models of hardware can support multiple fusion operator instructions, such as depth convolution fusion instructions, fused deconvolution instructions, folded convolution post-fusion instructions, instructions for merging several vector operations, and so on.

[0138] In other implementations, if there is no fusion operator instruction corresponding to the fused operator, the multiple operator instructions to be fused can be replaced with an operator call instruction, callop, and the operators contained in the corresponding callop can be defined in the file. In some implementations, based on the structured text of the fusion level, a subgraph is replaced with a callop, and the operators contained in the callop are defined afterwards.

[0139] Adding memory scopes involves adding attributes to operators based on their own definitions to describe the memory scopes where the operator's input and output data are stored. At this stage, the granularity of the scope can be the memory scope divided according to the data content to be stored. For example, based on Figure 3b Hardware architecture, assuming that the convolution operator itself requires that the input neurons be stored on NRAM, the input convolution kernel be stored on WRAM, and the output neurons be stored on NRAM, then add these memory scopes to the convolution operator according to the requirements of the convolution operator.

[0140] Furthermore, information such as data type and data shape can also be described. The data type refers to whether the data is integer or floating point. These data types may include, but are not limited to, int8, int16, int32, fp16, fp32, tf32, and so on. The data shape at this level refers to the complete shape of the data, which has not yet been split. For example, the complete shape of the input neuron of a convolution operator is 1×24×24×64, the data type is uint8, and it is stored in NRAM. In some implementations, memref can be used to describe the above information, thereby indicating the layout of the tensor data in the memory space.

[0141] The addition of memory operations includes: adding memory operations to move data between different storage levels based on the information of the fusion level. As mentioned earlier, the input and output of the graph are global memory at the chip level, such as DDR. One graph corresponds to one fusion operator or kernel. The input and output of each subgraph are the L2 storage level, and the intermediate results reside on the L1 storage level. Therefore, between different graphs, it is necessary to add memory operations that interact with global memory (such as DDR); and between different subgraphs, it is necessary to add memory operations that interact with the L2 storage level (for example, shared storage SRAM). For example, in a subgraph Figure 1 A copy instruction from the L1 storage level to the L2 storage level is inserted after the output of the L1 storage level to store the output data back to the L2 storage level; and the following sub-level Figure 2 Insert a copy instruction from the L2 storage level to the L1 storage level before the input of to obtain the input data from the L2 storage level. Figure 1 A copy instruction from the L2 storage level to the chip-level global memory is inserted after the output of Figure 2 A copy instruction from the chip-level global memory to the L2 storage hierarchy is inserted before the input of .

[0142] Furthermore, these added memory operations can further restrict the memory source and / or destination of their operations based on the memory scope previously added to the operator. For example, for a fused convolution operator (assuming it corresponds to a subgraph), before its input, a memory copy instruction from shared storage SRAM to NRAM can be inserted for the input neuron data; and for the convolution kernel, a memory copy instruction from shared storage SRAM to WRAM can be inserted.

[0143] Optionally or additionally, during the conversion of the IRO-level computation graph to the TFUIR-level computation graph, some optimizations may be performed. These optimizations may include, but are not limited to: Co dimension splitting, alignment optimization, setting weight residency, reusing weights between operators, reusing activation tables, marking shape inference entries, etc.

[0144] After the above conversion process, in step 630, the converted TFUIR-level computation graph is output for subsequent processing.

[0145] As can be seen from the conversion process above, the computation graph at the IR0 level is completely hardware-independent and simply describes network-level computations from a software perspective. However, the computation graph at the TFUIR level introduces operator fusion and adds hardware-related information from the perspective of operator fusion, such as hardware-supported fusion instructions and memory operations between hardware storage levels corresponding to fused operators. Furthermore, during this conversion process, optimizations can be performed on a per-fused operator basis, such as vector merging and memory operation merging, allowing compilation optimization of the computation graph at the granularity of fused operators.

[0146] Conversion from TFUIR to IR1

[0147] After converting from IR0 to TFUIR, a third conversion is required to convert TFUIR to IR1. This conversion process requires shape deduction, memory allocation, and back-end code generation. The generated code text is also IR1, so IR1 is a completely hardware-specific abstract representation.

[0148] In this article, shape refers to the dimensional information of data, including the number of dimensions and the size of each dimension. In intelligent computing, the shape of data is reflected in the data's memory access address. Therefore, in this context, the memory access pattern refers to the memory access address used to access each data block after the data is split or sliced ​​along a certain dimension according to the dimensionality splitting scheme.

[0149] In this article, shape deduction, memory access mode deduction or memory access address deduction means that when a certain dimension of the data is split or divided, it is necessary to deduce the memory access address of each time / each data block of the associated operator in the computation graph for the input data and / or output data after the split according to the split data blocks. The associated operator here refers to the operator in the computation graph that needs to directly or indirectly access the data, for example, it is necessary to directly or indirectly use the data to perform calculations, or it is necessary to directly or indirectly write the calculation results to the data. In addition, in the following description, unless otherwise specified, the data involved are tensor data.

[0150] Figure 7 FIG1 is an exemplary flow chart illustrating a third conversion process from TFUIR to IR1 implemented by a processing device according to an embodiment of the present disclosure. The processing device may be, for example, Figure 2 processing device 203.

[0151] As shown in the figure, in step 710, the memory access mode of the operators in the TFUIR graph is derived based on the data split information. The data split information here refers to the data split information determined by the automatic fusion technology during the second conversion process from IR0 to TFUIR. Please refer to the previous description and will not be repeated here.

[0152] In some implementations, data splitting information typically refers to the splitting information of the output data. Therefore, when performing memory access mode derivation, the operator of the output data (i.e., the end operator in the computation graph) is used as the entry operator, i.e., the starting point of the derivation, to perform memory access mode derivation. When the end operator is selected as the entry operator, the memory access mode derivation direction is reversed, that is, the memory access mode of the input data is deduced from the memory access mode of the output data in the opposite direction of the data flow in the computation graph.

[0153] In this memory access pattern derivation, the entry operator data is split into one or more data blocks according to the dimension splitting scheme of the entry operator data. Next, based on the memory access pattern of this entry operator data, the memory access pattern of the data associated with the entry operator data in the computation graph is derived. When the entry operator data is split according to the dimension splitting scheme, its corresponding memory access pattern can also be determined. The memory access pattern includes the memory access address of each data block into which the data set is split.

[0154] In some embodiments, memory access mode derivation may include: taking the end operator as the entry operator, starting from the entry operator, traversing each layer of operators in the computational graph from back to front, and performing reverse shape deduction to obtain the memory access addresses of one or more input data blocks and output data blocks that each layer of operators needs to split to meet the dimension splitting scheme of the entry operator.

[0155] Through studying deep learning algorithms, the inventors discovered that although the network structure of a neural network is complex, its basic components are relatively stable, mainly including operators such as convolution (conv), pooling (pool), padding (pad), and vector operations. These basic operators take up most of the time for the overall calculation. Further analysis of these basic operators revealed that they can be decomposed into some basic operations. In this article, these basic operations are referred to as atomic operations. Note that since the processing of data in intelligent computing involves division by dimension, the atomic operations here are also determined in units of dimension. In the disclosed embodiments, by analyzing the memory access mode of the basic operators, memory access mode derivation formulas for several types of atomic operations can be abstracted. Thus, the memory access mode derivation of various operator data can be performed based on these memory access mode derivation formulas.

[0156] Specifically, during the traversal of the above-mentioned computational graph, for each layer of operators in the computational graph, reverse shape deduction may include: decomposing the operator's operations into several atomic operations by dimension; and based on the input and output memory access address relationship of the atomic operation, that is, the memory access mode deduction formula, reversely deriving the memory access address of the input data block required for the operator to calculate an output data block.

[0157] In some embodiments, the atomic operations may include one or more of the following: a forward sliding window operation, such as used in operators such as convolution (conv) and pooling (pool), usually in the width (W) and height (H) dimensions; a forward padding operation, such as used in padding operations of operators such as convolution and pooling, or in an independent padding (pad) operator (pad size is a negative number); a backward sliding window operation, such as used in operators such as deconv and unpooling (unpool), usually also in the width (W) and height (H) dimensions; a backward padding operation, such as used in deconv, or in an independent padding operator (pad size is a negative number); a vector operation, used for various vector operations in operators, such as addition, subtraction, multiplication, activation calculations, etc.; a broadcast operation, used for broadcast copy operations, similar to numpy's broadcast operations on a certain dimension, such as in cycle add; and an aggregation operation (reduce), used to aggregate data, such as the accumulation operation on the Ci dimension in the convolution (conv) operator, reduce calculations, etc.

[0158] Based on the abstracted atomic operations described above, we can derive a memory access pattern derivation formula based on their specific operation methods. This formula, specifically, the relationship between the input and output memory access addresses of the atomic operations, can be used to derive the memory access pattern of the current operator's data. This allows us to decompose the current operator's operation into several atomic operations by dimension when traversing the computation graph. This can then be deduced using the atomic operation memory access pattern derivation formula to obtain the memory access pattern of the current operator's data.

[0159] Next, in step 720, memory allocation is performed based on the derived memory access pattern. The derived memory access pattern indicates the size of each split block of input and output data of each operator, that is, the memory size required for each split block. Based on this information, memory can be allocated for the split block. In a specific implementation, memory allocation is also performed on the corresponding memory scope based on the memory scope added to the operator during the conversion to TFUIR. The memory allocation information may include, for example, the memory scope, the starting position (first address) in the memory scope, and the allocated space size (split block size).

[0160] Finally, in step 730, according to the result of memory allocation, a code generation process is performed based on TFUIR, and compilation and optimization are performed to obtain a binary file that can be recognized by the intelligent processor.

[0161] The above describes an exemplary implementation flow for the third conversion process. As can be seen from the conversion process above, the computation graph at the IR1 level is completely hardware-dependent, describing the execution of the computation graph at the hardware instruction level, including the specific execution implementation of operators and the instruction pipeline. Furthermore, operator implementation is based on the computation graph at the TFUIR level, i.e., at the granularity of fused operators.

[0162] In summary, through the above multiple conversion processes, the completely hardware-independent IR representation IR0 can be transformed through the TFUIR representation characterized by the fusion operator to the completely hardware-dependent IR representation IR1. This IR1 can then be handed over to the runtime compilation library to generate binary code that is executed on the corresponding intelligent processor.

[0163] Compilation optimization with variable input

[0164] Typically, the input data size of a neural network model is fixed, meaning that the data shape information known at compile time is consistent with what you know. In this case, the data shape of the instructions to be generated is known at compile time, so the opcode and address field in the instructions can be determined at compile time. To fully utilize hardware resources, various performance optimizations are provided at compile time, including pipeline layout. After processing by the runtime compilation library, binary code that can be directly recognized by intelligent computing hardware is generated. This code is then parsed by the driver and executed on the intelligent computing hardware. Therefore, the optimization of the fusion operator mentioned above is also performed at compile time.

[0165] Furthermore, when generating instructions at compile time, the generation logic expands all loops, describing the dependencies between instructions using a tree structure. During the actual instruction generation process, the tree structure is fully expanded. This approach comes at the cost of a large number of instructions and a significant storage burden, making it unsuitable for deployment on mobile or embedded devices with limited storage capacity.

[0166] However, in some computing scenarios, the input data size of the neural network model is not fixed, that is, variable. This form of variable input often leads to poor performance in the neural network inference stage, because it can generally be considered that the more specific the given input range, the better the performance optimization effect. If a set of corresponding instructions are generated for each input scale (for example, for different resolutions in image processing), the number of instructions will increase linearly, increasing the processing and storage burden. One known way is to divide the input range into intervals and perform inference calculations according to the upper limit of the interval. However, if each inference is calculated according to the upper limit, it will cause a waste of computing resources. Furthermore, since the input shape is unknown, the number of loops will also be unknown when loops are involved, making it impossible to expand the loop in the original way. Therefore, for the case where the input shape is variable, it is necessary to provide further optimization solutions to solve one or more of the aforementioned problems.

[0167] In the case of variable input shape, the shape and size of the input data during actual execution is unknown during compilation, that is, the specific shape of the input data of the instruction to be generated is unknown during compilation, and thus accurate parameters cannot be filled in the instruction domain. For example, the instruction domain parameters can only be filled according to the upper limit. In the embodiment of the present disclosure, by generating instructions based on the input shape data obtained at runtime, more accurate instruction domain parameters can be generated, thereby supporting better instruction execution performance. In other words, some steps of instruction generation can be adjusted to runtime. More specifically, the pipeline arrangement of instructions can be implemented in the code generation stage (codegen) of the compilation period. At runtime, accurate instruction parameters are generated based on the exact input data shape obtained, and the corresponding instruction domain is filled in, thereby generating instructions that can be executed on intelligent hardware. Furthermore, in the code generation stage (codegen) of the compilation period, loop expansion is no longer performed for the loop structure, but loop instructions are used instead, such as using the branch instruction supported by the hardware, which is encapsulated into a method that supports for loops, and the instruction pipeline is arranged on this basis.

[0168] Figure 8 The following is a flow chart showing an exemplary method for generating instructions at compile time by a computing device according to an embodiment of the present disclosure. The computing device may be, for example, Figure 2 The computing device 201. The instruction generation process is similar to the instruction generation process of the compiler, except that the input data shape is the exact information instead of the upper limit of the interval.

[0169] As shown in the figure, in step 810, according to the computation graph expressed by TFU IR, the shape of the operators in the computation graph is forward deduced to obtain complete shape information.

[0170] In forward derivation, you can call the forward derivation logic of each operator to obtain the complete shape information of each operator's input and output data, that is, the shape before any splitting. In this forward derivation, the accurate shape information of the input data obtained at runtime is used.

[0171] Specifically, forward derivation is typically performed using the starting operator in the computation graph as the entry operator. A starting operator is an operator in the computation graph that has no preceding operators, typically the first operator in the graph. When the starting operator is selected as the entry operator, shape derivation proceeds in a forward direction, following the direction of data flow in the computation graph, forward deducing the shape of the output data from the shape of the input data.

[0172] The forward derivation logic of an operator is associated with the mathematical operation logic of the operator and can be similarly determined based on the reverse derivation logic described above, which will not be repeated here.

[0173] Next, in step 820, based on the data splitting information and the complete shape information obtained by forward deduction, the memory access mode of the operators in the computational graph expressed by TFUIR is derived. In some implementations, the data splitting information here may be the data splitting information determined at the front end, for example, the data splitting information determined by automatic fusion technology during the second conversion process from IR0 to TFUIR. In other implementations, the data splitting information here may be data splitting information that has been readjusted based on the determined shape information of the input data obtained at runtime. This implementation will be described in detail later.

[0174] The process of memory access mode derivation is combined with the previous Figure 7 The described step 710 is similar, except that the data shape information used is the determined data shape information obtained during runtime, and therefore will not be described in detail here.

[0175] Through the above memory access mode derivation, we can obtain the memory access mode of the split blocks of input and output data of each operator in the graph.

[0176] Next, in step 830, based on the derived memory access pattern, address domain-related parameters of the instructions involved in the loop in the code logic of the computation graph are determined. The memory access pattern indicates the memory access address of the split blocks of input and output data of each operator. Therefore, the relevant parameters of the instruction can be determined based on this, such as the size of the split block processed by the instruction, the size of each dimension of the HWC, etc.

[0177] Next, in step 840 , the instructions in the loop are pipelined based on the instruction stream supported by the hardware.

[0178] In pipeline arrangement, instructions can be pipelined according to the configuration information of the operator fusion method determined in the previous automatic fusion process, such as whether to enable pipeline and how many levels of pipeline to use when enabling pipeline.

[0179] When arranging the pipeline, you can divide the stages into different streams based on the instruction stream types supported by the hardware. Instruction stream types can include, but are not limited to, IO streams (memory access instructions involving off-chip storage), move streams (memory access instructions between on-chip storage), compute streams (computational instructions), and so on. When dividing the stages, you can use the TFUIR generated by the front-end. Then, write loops in stages, inserting synchronization instructions such as barrier instructions or sync instructions when synchronization is required.

[0180] Finally, in step 850, the computation graph is compiled according to the address field related parameters of the instructions involved in the loop and the result of the instruction pipeline arrangement in the loop.

[0181] Optionally or additionally, in some embodiments, the data splitting information determined by the front end can be adjusted as appropriate so that the currently running instance can better adapt to the hardware, fully utilize the computing power of the hardware, improve processing efficiency, etc.

[0182] Figure 9 A flowchart of an exemplary method for generating instructions at compile time implemented by a computing device according to an embodiment of the present disclosure is shown. In this embodiment, whether the split information needs to be adjusted can be determined based on conditional judgment.

[0183] As shown in the figure, with Figure 8 Similarly, first in step 910, based on the computation graph expressed by the TFU IR, the shape of the operators in the graph is forward deduced to obtain complete shape information.

[0184] Next, in step 920 , based on the derived shape information, it is determined whether the data splitting information needs to be adjusted.

[0185] In some embodiments, the need to modify the data splitting information can be determined based on the splitting efficiency. Splitting efficiency can be evaluated from multiple perspectives. In some implementations, splitting efficiency can be defined by the utilization of the processing core. For example, the utilization of the processing core IPU efficiency It can be calculated according to the following formula:

[0186]

[0187] Among them, N o , H o , W o Respectively represent the original size of the output data in N, H, and W dimensions; Hipu-factor and W ipu-factor Respectively represent the split block size in H and W dimensions, IPU part Indicates the number of processing cores for parallel processing, Num cluster Indicates the number of computing clusters, Cluster part Indicates the number of computing clusters for parallel processing.

[0188] The numerator in the above formula represents the original data volume before splitting, which can be determined based on the shape information derived in step 910. The denominator in the above formula represents the data volume that needs to be processed after splitting according to the splitting scheme indicated by the front-end data splitting information.

[0189] If the splitting efficiency is lower than the specified threshold, it can be determined that the splitting scheme needs to be adjusted. At this time, the splitting scheme is adjusted to update the current data splitting information. It can be understood that the adjustment of the splitting scheme is based on the data splitting information provided by the front end, and the data splitting scheme is adjusted according to the current actual and determined data shape information. For example, for scenarios where data is variable, the data splitting information provided by the front end may be a splitting scheme determined based on the upper bound of a certain interval, and the actual data shape may be at the lower bound of the interval. Therefore, the original data splitting scheme will have more redundancy, resulting in low processing core utilization. In this regard, based on the actual data shape, the data splitting scheme is re-determined to improve the utilization of the processing core. The process of determining the data splitting scheme can be similar to that in the automatic fusion process, except that the data shape information used at this time is different from before.

[0190] For example, assuming that the information known by the front-end compile time is that the N dimension is variable in the range of 1 to 4, the splitting information generated by the compile time is for the case where the upper limit of the N dimension is 4, and the 4 batches of data are split into 4 computing clusters for execution. However, during actual runtime, the real batch of input data N=1. At this time, if the splitting scheme of the front-end compile time is still followed, the data of this batch will be allocated to 1 computing cluster for execution, and the remaining computing clusters will be idle. Obviously, this splitting method leads to low hardware utilization. Therefore, according to the embodiment of the present disclosure, during runtime, the data splitting scheme can be readjusted based on the determined data shape information obtained, for example, 1 batch of data can be split into 4 computing clusters according to the H and W dimensions for execution, thereby improving computing efficiency.

[0191] Optionally or additionally, in some embodiments, whether the data splitting information needs to be modified is determined based on the current hardware operating mode.

[0192] One indicator for measuring the hardware's processing performance on a model is throughput, which is the amount of input data that the hardware can process per second when running a model. Another indicator is latency, which is the average length of time it takes to process data. Throughput and latency are inversely proportional. Therefore, the hardware operating modes can be divided into Throughput mode and Latency mode. The Throughput mode mainly assesses the processing throughput in the case of large batches of data. The Latency mode mainly assesses the processing latency in the case of small batches of data. Based on the difference between the two modes, in some embodiments of the present disclosure, whether the data splitting information needs to be modified can be considered only when the current hardware operating mode is Latency mode.

[0193] Then, based on the updated data splitting information, the memory access mode of the operators in the graph is re-derived. The memory access mode derivation process is combined with the previous Figure 8 The above step 820 is similar and will not be described in detail here. Through the above memory access mode derivation, the memory access mode of the split blocks of input and output data of each operator in the graph can be obtained. If it is determined in step 920 that the data split information does not need to be adjusted, then jump directly to step 930 to derive the memory access mode of the operators in the computation graph based on the tensor data split information that meets the requirements.

[0194] Next, in step 940, based on the derived memory access mode, address domain related parameters of the instructions involved in the loop in the code logic of the computation graph are determined; in step 950, the instructions are pipelined based on the instruction stream supported by the intelligent processor; and in step 960, the computation graph is compiled according to the address domain related parameters of the instructions involved in the loop and the result of the instruction pipeline arrangement in the loop to obtain a binary file that can be recognized by the intelligent processor. Steps 940 to 960 are similar to Figure 8 Steps 830 to 850 are similar and will not be described in detail here.

[0195] Therefore, the above describes the runtime solution of the computational graph provided by the embodiment of the present disclosure. For scenarios with variable input shapes, it can use real and certain input data shape information to adjust instructions at runtime, so that the real scenario can be more adaptable to the hardware, fully utilize the computing power of the hardware, and improve processing efficiency.

[0196] The present disclosure also provides a processing device that can be used to compile a computation graph, including: a processor configured to execute program instructions; and a memory configured to store the program instructions, so that when the program instructions are loaded and executed by the processor, the processor executes the compilation method described in the embodiment of the present disclosure.

[0197] In the embodiments of the present disclosure, a computer-readable storage medium is also provided, wherein program instructions are stored. When the program instructions are loaded and executed by a processor, the processor executes the computation graph compilation method described in the embodiments of the present disclosure. In the embodiments of the present disclosure, a computer program product is also provided, including a computer program or instructions. When the computer program or instructions are executed by a processor, the computation graph compilation method described in the embodiments of the present disclosure is implemented.

[0198] The present disclosure also provides a chip, which may include the aforementioned processing device. Furthermore, the present disclosure also provides a board, which may include the aforementioned chip.

[0199] According to different application scenarios, the electronic devices or devices disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, PC devices, Internet of Things terminals, mobile terminals, mobile phones, driving recorders, navigators, sensors, cameras, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, automatic driving terminals, vehicles, household appliances, and / or medical equipment. The vehicles include airplanes, ships and / or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lights, gas stoves, and range hoods; the medical equipment includes magnetic resonance imaging (MRI), ultrasound machines and / or electrocardiographs. The electronic devices or devices disclosed herein may also be applied to the Internet, Internet of Things, data centers, energy, transportation, public administration, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, medical care and other fields. Furthermore, the electronic devices or devices disclosed herein may also be used in cloud, edge, terminal and other application scenarios related to artificial intelligence, big data and / or cloud computing. In one or more embodiments, electronic devices or apparatuses with high computing power according to the disclosed solution can be applied to cloud devices (such as cloud servers), while electronic devices or apparatuses with low power consumption can be applied to terminal devices and / or edge devices (such as smartphones or cameras). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or edge device are compatible with each other, so that according to the hardware information of the terminal device and / or edge device, appropriate hardware resources can be matched from the hardware resources of the cloud device to simulate the hardware resources of the terminal device and / or edge device, so as to complete the unified management, scheduling and collaborative work of end-to-end or cloud-edge-to-end.

[0200] It should be noted that, for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will understand that the scheme of the present disclosure is not limited by the order of the actions described. Therefore, based on the disclosure or teachings of the present disclosure, those skilled in the art will understand that some of the steps therein can be performed in other orders or simultaneously. Further, those skilled in the art will understand that the embodiments described in the present disclosure can be regarded as optional embodiments, that is, the actions or modules involved therein are not necessarily necessary for the implementation of one or more schemes of the present disclosure. In addition, depending on the different schemes, the description of some embodiments of the present disclosure also has different emphases. In view of this, those skilled in the art will understand that the parts that are not described in detail in a certain embodiment of the present disclosure may also refer to the relevant descriptions of other embodiments.

[0201] In terms of specific implementation, based on the disclosure and teachings of this disclosure, those skilled in the art can understand that several embodiments disclosed in this disclosure can also be implemented in other ways not disclosed herein. For example, with respect to the various units in the electronic device or device embodiments described above, this document divides them based on the consideration of logical functions, and there may be other ways of division in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. With respect to the connection relationship between different units or components, the connection discussed above in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0202] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network elements. In addition, according to actual needs, some or all of the units may be selected to achieve the purpose of the solution described in the embodiments of this disclosure. In addition, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically separately.

[0203] In some other implementation scenarios, the above-mentioned integrated unit can also be implemented in the form of hardware, that is, a specific hardware circuit, which may include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit may include but is not limited to physical devices, and the physical devices may include but are not limited to devices such as transistors or memristors. In view of this, the various devices described herein (such as computing devices or other processing devices) can be implemented by appropriate hardware processors, such as CPUs, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage unit or storage device can be any appropriate storage medium (including magnetic storage media or magneto-optical storage media, etc.), which can be, for example, resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high bandwidth memory (HBM), hybrid memory cube (HMC), ROM and RAM, etc.

[0204] Although a plurality of embodiments of the present disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art may conceive of many modifications, changes, and alternatives without departing from the ideas and spirit of the present disclosure. It should be understood that in practicing the present disclosure, various alternatives to the embodiments of the present disclosure described herein may be adopted. The appended claims are intended to define the scope of protection of the present disclosure and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A method for compiling a computational graph, wherein: The method comprises: Obtaining a computational graph to be compiled, expressed by a second intermediate representation, performing forward derivation of the shape, and obtaining complete shape information based on the forward derivation and tensor data splitting information; wherein the second intermediate representation is obtained by conversion based on the first intermediate representation, and the first intermediate representation is obtained by unified conversion based on a task model expressed in at least one front-end language; and the computational graph is a task executed on an intelligent processor; Using the complete shape information, determining whether it is necessary to adjust the tensor data splitting information; Determine the tensor data splitting information that meets the requirements according to the judgment result; Derivation of memory access modes for operators in the computation graph based on the tensor data splitting information that meets the requirements; Determining address domain-related parameters of instructions involved in a loop in the code logic of the computation graph based on the derived memory access pattern; Pipelining the instructions in the loop based on the instruction stream supported by the intelligent processor; The computation graph is compiled according to address domain related parameters of the instructions involved in the loop and the result of the instruction pipeline arrangement in the loop to obtain a binary file recognizable by the intelligent processor.

2. The method according to claim 1, wherein The memory access mode derivation indicates that: after a dimension of tensor data is split or divided, it is necessary to deduce the memory access address of each time / each data block of the associated operator in the computation graph for the input data and / or output data after the split according to the split data blocks.

3. The method according to claim 1, wherein The step of deducing a memory access mode for an operator in the computation graph based on the tensor data splitting information that meets the requirements includes: Taking the end operator in the computation graph as the entry operator, utilizing the splitting information of the output tensor data of the entry operator, and going against the direction of tensor data flow in the computation graph, reversely determine the memory access mode of the input tensor data of the entry operator from the splitting information of the output tensor data.

4. The method according to claim 1, wherein The steps to determine whether the tensor data splitting information needs to be adjusted include: Determine whether the tensor data splitting information needs to be adjusted based on the splitting efficiency of the tensor data.

5. The method according to claim 4, wherein: The split efficiency is defined by the utilization of the processing core, where the utilization of the processing core IPU efficiency Calculate according to the following formula: Among them, N o , H o , W o Respectively represent the original size of the output tensor data in N, H, and W dimensions; H ipu-factor and W ipu-factor Respectively represent the split block size in H and W dimensions, IPU part Indicates the number of processing cores of the intelligent processor for parallel processing, Num cluster Indicates the number of computing clusters of the intelligent processor. part Indicates the number of computing clusters processed in parallel by the intelligent processor.

6. The method of claim 1, wherein: The steps to determine whether the tensor data splitting information needs to be adjusted include: Whether the tensor data splitting information needs to be adjusted is determined according to the operating mode of the intelligent processor.

7. The method according to claim 6, wherein: The intelligent processor operation mode includes Throughput mode and Latency mode; wherein, Throughput mode represents a mode for measuring the processing throughput of the intelligent processor when the batch size of the tensor data dimension is greater than a threshold; Latency mode represents a mode for measuring the processing latency of the intelligent processor when the batch size of the tensor data dimension is less than a threshold.

8. A processing device for compiling a computation graph, comprising: a processor configured to execute program instructions; as well as A memory configured to store the program instructions, which, when loaded and executed by the processor, enables the processor to execute the method for compiling a computational graph according to any one of claims 1 to 7.

9. A computer-readable storage medium storing program instructions, which, when loaded and executed by a processor, causes the processor to execute the computation graph compilation method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program or instructions, which, when executed by a processor, implements the method for compiling a computational graph according to any one of claims 1 to 7.

Images