Method for compiling computational graph, and related product

Abstract

A method for compiling a computational graph, and a related product. The method comprises: acquiring a computational graph to be compiled that is expressed by a second intermediate representation, performing forward inference of a shape, and on the basis of the forward inference and tensor data splitting information, obtaining complete shape information; using the complete shape information to determine whether the tensor data splitting information needs to be adjusted; on the basis of a determination result, determining the tensor data splitting information that meets requirements; on the basis of the tensor data splitting information that meets the requirements, performing memory access pattern derivation on operators in the computational graph; on the basis of a derived memory access pattern, determining address-domain-related parameters of instructions involved in loops in code logic of the computational graph; performing pipeline scheduling on the instructions in the loops; and on the basis of the address-domain-related parameters of the instructions involved in the loops and a pipeline scheduling result of the instructions, compiling the computational graph, so as to obtain a binary file recognizable by an intelligent processor.

Description

Compilation method of a computation graph and related products Cross-reference to related applications

[0001] This application claims priority from the application filed on August 12, 2024, with the application number 2024111047749 and the title “Compilation method of a computation graph and related products”. TECHNICAL FIELD

[0002] The present disclosure generally relates to the field of intelligent computing, and more particularly to the field of neural networks. More specifically, the present disclosure relates to a compilation method of a computation graph, a computing device, a computer-readable storage medium, and a computer program product. BACKGROUND

[0003] In intelligent computing systems, for machine learning algorithm models of various machine learning applications, a deep learning framework is used as a main programming interface to build a corresponding machine learning / deep learning network. The deep learning framework can include various programming frameworks, such as Google’s TensorFlow, Caffe, Caffe2, MXNet, PyTorch, and the like. In these programming frameworks, a computation graph (or intermediate representation (IR) graph) is usually used to describe the computation process of the machine learning algorithm, all data in the computation graph is represented by tensors, and various operations are represented by operators.

[0004] For the IR graph, various graph-level optimizations can be performed. These optimizations can include, but are not limited to, constant folding, operator fusion, graph replacement, and the like. The optimized IR graph can be compiled into an abstract language representation, i.e., various instructions. These instructions can be executed by underlying machine learning hardware.

[0005] During the compilation from the computation graph to the underlying machine instruction code, it is usually necessary to introduce several fixed abstraction levels to convert the language intermediate representation into the intermediate representation supported by the target compilation framework in a step-by-step descending manner. On this multi-level compilation abstraction level, how to design and implement a general compilation technology route is an important challenge currently faced by the optimization compiler. SUMMARY

[0006] To at least solve one or more technical problems as mentioned above, the present disclosure proposes, in various aspects, a compilation scheme of a computation graph, which introduces a TFU IR level during compilation. This level can more effectively take into account the information of algorithm software and execution hardware by optimizing the computation graph from the perspective of operator fusion, thereby improving the optimization effect of the compiler. In other aspects, the present disclosure also provides a compilation scheme for converting from a higher level to the TFU IR level, and a compilation scheme for converting from the TFU IR level to a lower level.

[0007] In a first aspect, the disclosure discloses a compilation method of a computation graph, wherein the method comprises: obtaining a computation graph expressed by a second intermediate representation to be compiled, performing forward derivation of a shape, and obtaining complete shape information according to the forward derivation and tensor data splitting information; wherein the second intermediate representation is obtained by conversion based on a first intermediate representation, and the first intermediate representation is obtained by uniform conversion of a task model based on at least one front-end language representation; the computation graph is a task executed on an intelligent processor; using the complete shape information, it is determined whether the tensor data splitting information needs to be adjusted; according to the determination result, the tensor data splitting information meeting the requirements is determined; based on the tensor data splitting information meeting the requirements, memory access mode derivation is performed on operators in the computation graph; based on the derived memory access mode, address domain related parameters of instructions involved in a loop in the code logic of the computation graph are determined; based on an instruction stream supported by the intelligent processor, the instructions in the loop are arranged in a pipeline; and according to the address domain related parameters of the instructions involved in the loop and the result of the instruction pipeline arrangement in the loop, the computation graph is compiled to obtain a binary file recognizable by the intelligent processor.

[0008] In a second aspect, the disclosure provides a processing device for performing compilation on a computation graph, comprising: a processor configured to execute program instructions; and a memory configured to store the program instructions, which, when loaded and executed by the processor, cause the processor to perform the compilation method of the computation graph according to the first aspect of the disclosure.

[0009] In a third aspect, the disclosure provides a computer readable storage medium, wherein program instructions are stored, which, when loaded and executed by a processor, cause the processor to perform the compilation method of the computation graph according to the first aspect of the disclosure.

[0010] In a fourth aspect, the disclosure provides a computer program product comprising computer programs or instructions, which, when executed by a processor, implement the compilation method of the computation graph according to the first aspect of the disclosure.

[0011] In the compilation scheme of the computation graph as provided above, the embodiments of the disclosure introduce the concept of a second intermediate representation TFU IR level and a level-to-level conversion scheme associated with the level for the important optimization means-operator fusion in neural networks, thereby providing an efficient optimization and compilation scheme. BRIEF DESCRIPTION OF DRAWINGS

[0012] The above and other objects, features and advantages of the present disclosure exemplary embodiments will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0013] FIG. 1 shows a structural diagram of a board card according to an embodiment of the present disclosure;

[0014] FIG. 2 shows a structural diagram of a combined processing device according to an embodiment of the present disclosure;

[0015] FIG. 3a shows a schematic diagram of the internal structure of a single processor core of a single-core computing device according to an embodiment of the present disclosure;

[0016] FIG. 3b shows a simplified schematic diagram of the internal structure of a multi-core computing device according to an embodiment of the present disclosure;

[0017] FIG. 4 shows a schematic structure of a software stack;

[0018] FIG. 5 schematically shows a hierarchical structure of an optimizing compiler according to an embodiment of the present disclosure;

[0019] FIG. 6 schematically shows an exemplary flowchart of a second conversion process from IR0 to TFU IR implemented by a processing device according to an embodiment of the present disclosure;

[0020] FIG. 7 schematically shows an exemplary flowchart of a third conversion process from TFU IR to IR1 implemented by a processing device according to an embodiment of the present disclosure;

[0021] FIG. 8 shows an exemplary method flowchart of generating instructions at compile time implemented by a computing device according to some embodiments of the present disclosure;

[0022] FIG. 9 shows an exemplary method flowchart of generating instructions at compile time implemented by a computing device according to some other embodiments of the present disclosure. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of, rather than all of, the embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present disclosure.

[0024] It should be understood that the terms "first", "second", "third", and "fourth" and the like in the claims, specification, and drawings of this disclosure can be used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. The terms "including", "containing", "comprising", and the like when used herein are meant to be inclusive, but not restrictive, to the components, integers, steps, operations, elements, and / or groups thereof disclosed in the specification and / or claims so that other components, integers, steps, operations, elements, and / or groups can be added and / or deleted without departing from the scope of the disclosure.

[0025] It should also be understood that the terms used in the specification of the present disclosure are merely for the purpose of describing particular embodiments and are not intended to limit the present disclosure. As used in this specification and the claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items, and that the term "at least one of' is used to mean one, or, two or more, and that the term "one or more" is used to mean one or more than one. Moreover, it is understood that the terms "comprise", "comprising", "comprises", "include", "including", "includes", "contain", "containing", "contains", "characterized by" and variants thereof, when used in this specification and in the claims, which follow the co-pending U.S. Patent Application Serial No. 16 / 209, 1 10, are intended to specify the presence of stated features, integers, steps, or components but do not preclude the presence or addition of one or more other features, integers, steps, components, or groups thereof.

[0026] As used in this specification and claims, the terms "if' and "when" can, together with other phrases be used in the context of this specification to indicate a consequence of a preceding condition — that is, when the condition is met, then the consequence can occur.

[0027] A detailed description of specific embodiments of the disclosure follows.

[0028] Exemplary hardware architecture

[0029] Figure 1 shows a schematic diagram of a board card 10 according to an embodiment of the present disclosure. As shown in Figure 1, the board card 10 comprises a chip 101, which is a system on chip (SoC) or a system on a chip, integrating one or more combined processing devices, which is an artificial intelligence computing unit, to support various deep learning and machine learning algorithms, to meet the intelligent processing needs in complex scenarios in the fields of computer vision, speech, natural language processing, data mining, etc. In particular, deep learning technology is widely used in cloud intelligent applications, and a significant feature of cloud intelligent applications is the large amount of input data, which has high requirements for the storage capacity and computing capacity of the platform. The board card 10 of this embodiment is suitable for cloud intelligent applications, and has a large off-chip storage, on-chip storage and strong computing capacity.

[0030] The chip 101 is connected with an external device 103 through the external interface device 102. The external device 103 is, for example, a server, a computer, a camera, a display, a mouse, a keyboard, a network card or a wifi interface, etc. The data to be processed can be transmitted by the external device 103 to the chip 101 through the external interface device 102. The calculation result of the chip 101 can be transmitted back to the external device 103 through the external interface device 102. According to different application scenarios, the external interface device 102 can have different interface forms, such as PCIe interface, etc.

[0031] The board card 10 further comprises a storage device 104 for storing data, which comprises one or more storage units 105. The storage device 104 is connected and transmits data with the control device 106 and the chip 101 through a bus. The control device 106 in the board card 10 is configured to regulate the state of the chip 101. For this purpose, in one application scenario, the control device 106 can comprise a single-chip microcomputer (MCU).

[0032] Figure 2 is a structural diagram of the combined processing device in the chip 101 of this embodiment. As shown in Figure 2, the combined processing device 20 comprises a computing device 201, an interface device 202, a processing device 203 and a storage device 204.

[0033] The computing device 201 is configured to perform user-specified operations, mainly implemented as a single-core intelligent processor or a multi-core intelligent processor, to perform the calculation of deep learning or machine learning, which can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0034] The interface device 202 is used to transmit data and control instructions between the computing device 201 and the processing device 203. For example, the computing device 201 can obtain input data from the processing device 203 through the interface device 202 and write it into the storage device on the computing device 201. Further, the computing device 201 can obtain control instructions from the processing device 203 through the interface device 202 and write them into the control cache on the computing device 201. Alternatively or optionally, the interface device 202 can also read the data in the storage device of the computing device 201 and transmit it to the processing device 203.

[0035] The processing device 203 is a general processing device, which performs basic control including but not limited to data transfer, start and / or stop of the computing device 201, etc. Depending on the implementation, the processing device 203 can be one or more types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), or other general purpose and / or special purpose processors, including but not limited to a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc., and the number thereof can be determined according to actual needs. As mentioned above, only in terms of the computing device 201 of the present disclosure, it can be considered as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0036] The storage device 204 is used to store data to be processed, which can be a DRAM, a DDR memory, usually with a size of 16G or more, for saving data of the computing device 201 and / or the processing device 203.

[0037] When the computing device 201 runs a neural network, it is generally necessary to first compile the neural network using the processing device 203 to obtain an executable file, which contains device information, i.e., the executable file needs to be executed on which device in the heterogeneous computer system. After the executable file is assembled and linked, an executable program of the neural network can be obtained, and the executable program is stored in the storage device 204.

[0038] The processing device 203 can read the executable program from the storage location of the executable program, and obtain a plurality of tasks of the program according to the executable program. These tasks are distributed to the computing device 201 through the interface device 202 for execution, and finally the operation result is obtained.

[0039] FIG. 3a shows a schematic diagram of the internal structure of the processing core when the computing device 201 in FIG. 2 is a single-core device. The computing device 301 is used to process computer vision, speech, natural language, data mining, etc. input data, and the computing device 301 includes three modules: a control module 31 (also referred to as a controller), a computation module 32 (also referred to as a computer), and a storage module 33 (also referred to as a memory).

[0040] The control module 31 is used to coordinate and control the operation of the operation module 32 and the storage module 33 to complete the task of deep learning, which includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 is used to obtain instructions from the processing device 203, and the instruction decode unit 312 decodes the obtained instructions and sends the decoding results as control information to the operation module 32 and the storage module 33.

[0041] The operation module 32 includes a vector operation unit 321 and a matrix operation unit 322. The vector operation unit 321 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformation; the matrix operation unit 322 is responsible for the core calculation of the deep learning algorithm, i.e., matrix multiplication and convolution.

[0042] The storage module 33 is used to store or transfer related data, including a neuron storage unit (NRAM) 331, a weight storage unit (WRAM) 332, and a direct memory access module (DMA) 333. The NRAM 331 is used to store input neurons, output neurons, and intermediate results after calculation; the WRAM 332 is used to store the convolution kernel of the deep learning network, i.e., the weight; and the DMA 333 is connected to the DRAM 204 through the bus 34 and is responsible for data transfer between the computing device 301 and the DRAM 204. It should be noted that the NRAM and the WRAM here can be two storage areas divided in the logical storage space of the same memory, or two independent memories, which are not specifically limited here.

[0043] FIG. 3b shows a simplified schematic diagram of the internal structure of the multi-core computing device 201. The multi-core computing device can be abstracted by a hierarchical hardware model. As shown in the figure, the multi-core computing device can be abstracted into three levels, i.e., the chip level (Chip) 360, the processor cluster level (Cluster) 370, and the processor core level (Core) 380. The data transmission of the storage unit and the computing unit part are mainly involved in the present disclosure embodiment, so the relevant computing structure is briefly shown and introduced, and other parts are omitted.

[0044] At the chip level, each chip contains a local DDR storage, each processor chip serves as a computing and control unit, and each processor chip contains multiple multi-processors as computing units.

[0045] At the processor cluster level, each multi-processor includes multiple accelerator cores as control and computing units, and in addition, a shared storage SRAM as a storage unit.

[0046] At the processor core level, each accelerator core contains local storage and an array of local processing units. NFU refers to Neuron Function Unit, which is used to perform convolution calculation. The structure of a single processor core can be similar to the structure diagram of the single-core computing device shown in FIG. 3a, which will not be described in detail here.

[0047] In the multi-core computing device, the storage model includes card global memory, SRAM (shared memory) on the cluster, NRAM on the core, WRAM, and registers, etc. In order to obtain better performance, the data movement between the following storage levels of the card and the balance between memory and calculation can be explicitly controlled. The SRAM is contained in the memory processing unit MPU (Memory Process Unit Core, referred to as MPU, or Mem Core). The core refers to the intelligent processing core (Intelligent Process Unit Core, referred to as IPU Core or Core) in the multi-core computing device. One IPU Core contains NRAM, WRAM, NFU, etc. The cluster refers to the processor cluster or computing cluster. Generally, the multi-core computing device contains several clusters, and one cluster contains one Mem Core and N IPU Cores.

[0048] Exemplary software stack

[0049] FIG. 4 shows a schematic structure of a software stack 400. As shown, the software stack 400 includes a machine learning application 410, a deep learning framework 420, an IR graph 430, machine execution instructions 440, and machine learning hardware 450.

[0050] The machine learning application 410 provides corresponding machine learning algorithm models for different application scenarios. These algorithm models can be written in various high-level programming languages.

[0051] The deep learning framework 420 can include a variety of programming frameworks, such as Google’s TensorFlow, the deep learning framework Caffe, Caffe2, MXNet, PyTorch, etc. For the machine learning algorithm models of various machine learning applications 410, the deep learning framework 420 serves as a main programming interface to parse these algorithm models to build corresponding machine learning / deep learning networks. At the compilation stage, the deep learning framework 420 can provide trained algorithm models. The deep learning framework 420 can convert the built network into a graph-level intermediate representation (IR), referred to as an IR graph or a computation graph 430. The nodes in the IR graph represent operations or operators that can be applied on certain inputs and can generate outputs that are passed to other nodes. The edges in the IR graph represent the flow of data.

[0052] For the IR graph 430, various graph-level optimizations can be performed at the compilation stage. These optimizations can include, but are not limited to, constant folding, operator fusion, graph replacement, etc. The optimized IR graph can be compiled into an assembly-like file. This file is converted into machine language instructions, which are packaged into a format of a relocatable object program.

[0053] For the machine execution instructions 440, the relocatable object program is compiled into the machine execution instructions 440 by a runtime compilation library at the runtime stage. These instructions can be executed by underlying machine learning hardware 450. The machine learning hardware 450 can be, for example, a chip in the exemplary hardware architecture described above.

[0054] The compiler undertakes the task of converting the high-level language written, hardware-independent IR graph 430 to lay the foundation for the machine execution instructions 440 that can be efficiently executed on specific underlying hardware. During the conversion process, the compiler provides various optimization means so that the optimized program can be more efficiently executed on hardware at the execution stage.

[0055] Since the compiler needs to bridge the huge gap between the upper-level programming model and the underlying hardware, it is difficult to fully improve the performance of the program by using a single level of compilation abstraction. More and more compilers use a multi-level abstraction approach. On this multi-level compilation abstraction level, how to design and implement a general compilation technology route is an important challenge currently faced by optimized compilers.

[0056] Exemplary compilation architecture

[0057] In embodiments of the present disclosure, an optimized compiler is provided that divides the intermediate representation IR into three levels: IR0, TFUIR, and IR1, and performs different optimization processes with different focuses at different IR levels, thereby being able to convert hardware-independent programs into machine instructions that can be efficiently executed on specific hardware.

[0058] FIG. 5 illustrates an example of a hierarchy of an optimization compiler 500 according to embodiments of the present disclosure.

[0059] As shown, the compiler 500 can receive trained algorithm models 510 provided by various deep learning frameworks. The compiler 500 converts these algorithm models 510 programmed based on the deep learning frameworks into a unified computation graph representation through a first conversion (501), i.e., a first layer IR: IR0 520.

[0060] The first conversion is mainly used to set the types of operators based on different frameworks, and to convert the current numerous deep learning frameworks and task models written in various languages into an IR0-represented computation graph, so that subsequent model optimization and code generation can be implemented based on the unified computation graph. Since the first conversion focuses on converting various front-end languages into a unified form of computation graph, the IR0 only reflects the information of the algorithm model itself, i.e., is irrelevant to the specific hardware to be executed. The IR0 can be an intermediate representation defined based on the MLIR (multi-level IR) compiler framework.

[0061] An example of IR0 is given below, in the form of text as follows:

[0062] module{

[0063] func.foo(%arg0,%arg1,%arg2){

[0064] %0=mm.conv2d(%arg0,%arg1);

[0065] %1=mm.scale(%0,%arg2);

[0066] %2=mm.relu(%1);

[0067] return%2;}

[0068] }

[0069] In the above example, a function foo is described, which contains three parameters arg0, arg1 and arg2, and sequentially performs convolution operation (conv2d), scaling operation (scale) and activation operation (relu), and finally returns the operation result. As can be seen from the example, the IR0 only reflects the operation logic at the software algorithm level, and does not have hardware-related information.

[0070] Then, the compiler 500 converts the IR0 520 to a second layer of IR: TFU IR 530 through a second conversion (502). The second conversion mainly maps the operators expressed by the IR0 to the operators expressed by the TFU IR from the perspective of operator fusion, based on hardware fusion instructions, hardware-supported data copy behaviors, and other hardware resource information. For example: the conv operator of the IR0, after the second conversion, may correspond to the conv operator, fusion_conv operator, or fold_conv operator expressed by the TFU IR according to the scale parameter. The TFU IR operator 530 is an abstract representation based on the operator fusion scheme. Based on the expression of the TFU IR 530, optimization operation means in the operator fusion process such as pipeline unrolling and graph optimization are performed for the TFU-IR operator.

[0071] Operator fusion is a typical method of computing graph optimization, which can calculate multiple operators together in a single kernel without saving the intermediate results back to the global memory, thereby effectively reducing the input / output access in network model operation, that is, reducing IO memory access.

[0072] In the second conversion, an automatic fusion technique is adopted to automatically search the computing graph expressed by the IR0 based on a data-driven and performance evaluation model, and finally obtain the optimal operator fusion mode, and then perform IR conversion and optimization processing based on the fused operators.

[0073] Specifically, in the automatic fusion technique, various fusion feasibilities and various data splitting feasibilities in the computing graph expressed by the IR0 are traversed to obtain a list of operators in the computing graph with fusion possibilities and a scheduling strategy. In the scheduling strategy, information affecting the performance of operator fusion, such as the list of operators with fusion possibilities, data splitting methods, pipelining methods, weight residency, weight preloading, and other information, are uniformly defined as control data, thereby converting the search problem of the optimal operator fusion mode into the search problem of the optimal control data. In this way, only the operators connected by input / output on the computing graph are allowed to be fused. Further, in the automatic fusion process, a performance evaluation function of the fused operator can be given according to the control data of the fused operator based on specific hardware information. Based on the performance evaluation function, the fusion benefit of the fused operator can be calculated, so as to select the optimal fusion mode according to the fusion benefit. In this way, the optimal operator fusion mode under various networks can be automatically found.

[0074] As can be seen from the second conversion process, in addition to reflecting the logical operation information of the algorithm model, the TFU IR obtained by conversion also reflects the underlying hardware information due to the need to consider specific hardware information in the automatic fusion process. The granularity of the hardware information depends on the automatic fusion process. That is, the TFU IR introduces a part of hardware information based on operator fusion. Based on the operator fusion manner, in the process of executing a task by the artificial intelligence processor, the intermediate result is not saved back to the global memory, thereby effectively reducing the input / output access in network model operation, that is, reducing the IO access, and achieving the purpose of reducing the IO bottleneck.

[0075] An example of a TFU IR is given below, which is in the form of text as follows:

[0076] module{

[0077] func.foo(%arg0,%arg1,%arg2){

[0078] %0=tfu.fusion_conv(%arg0,%arg1,%arg2);

[0079] return%0;}

[0080] }

[0081] As can be seen, the example of the TFU IR continues the example of the preceding IR0, and fuses the three operators of the convolution operation (conv2d), the scaling operation (scale) and the activation operation (relu) into a fused convolution operator (fusion_conv).

[0082] Next, the compiler 500 converts the TFU IR 530 into a third layer IR: IR1 540 through a third conversion (503). The third conversion mainly optimizes the IR from the perspective of hardware execution, and the IR1 540 is a completely hardware-related abstract representation. The IR1 contains rich hardware information, and the underlying hardware can execute a task corresponding to the neural network model based on the hardware information.

[0083] In the third conversion, the IR is optimized based on information of specific execution hardware of the underlying layer, for example, adjusting the splitting scheme in the automatic fusion scheme determined in the foregoing, and then code generation (Codegen) is performed to obtain an abstract representation that can be provided to a runtime compilation library for further compilation and ultimately to generate machine-recognizable binary instructions. The conversion of the TFU IR expression to the IR1 expression explicitly indicates which hardware information is present in the TFU IR expression and which hardware information is present in the IR1 expression, similar to the form of stacking blocks. Since the positioning of the software stack itself is to fully use hardware resources, when positioning the software code problem, it is more convenient to locate the problem, and ultimately achieve more efficient problem solving.

[0084] Thus, the disclosure embodiments provide a compiler including three IR levels, wherein the intermediate TFU IR level is specially introduced for the important computation graph optimization means of operator fusion. The TFU IR level is based on an operator fusion scheme and performs more delicate computation graph optimization processing at the granularity of fused operators. More specifically, by converting the first level IR0 to the intermediate TFU IR level, hardware characteristics can be exhibited at a first granularity (for example, the granularity of operator fusion), and graph level optimization can be performed based on this granularity, and the graph level optimization belongs to optimization with hardware information, which is different from the graph level optimization in the IR0 stage. In addition, not all hardware information is included in the TFU IR expression, which is more friendly to programmers and more efficient when locating code problems. Then, after the code generation (Codegen) stage, the TFU IR level is converted to the third level IR1, which can more finely exhibit hardware characteristics. Thus, by this way of introducing hardware information layer by layer, a high-level, hardware-independent computation graph can be finally converted to a low-level, completely hardware-dependent computation graph and provided to a runtime compilation library to generate machine-recognizable binary instructions. In summary, the compilation scheme of the disclosure embodiments provides an effective performance optimization scheme for the acceleration processing of a neural network model on a hardware execution platform.

[0085] The conversion process between the various IR levels will be described below. The first conversion from the algorithm model provided by the deep learning framework to IR0 mainly focuses on the differences of various front-end languages, and various known conversion methods can be used, which are not limited by the disclosure embodiments and will not be described in detail.

[0086] Conversion from IR0 to TFUIR

[0087] As described above, in the second conversion from IR0 to TFU IR, the computation graph is mainly converted and expressed from the perspective of operator fusion.

[0088] FIG. 6 illustrates an example flowchart of a second conversion process from IR0 to TFU IR implemented by a processing device according to embodiments of the present disclosure. The processing device can be, for example, the processing device 203 of FIG. 2.

[0089] As shown, in step 610, an automatic fusion technique is first employed to automatically search for an optimal operator fusion manner from the computation graph represented by IR0. For details of the automatic fusion technique, please refer to Chinese Invention Patent Application CN202211372965.4 entitled “Operator Automatic Fusion Method for Computation Graph and Related Products”. Since the focus of the present disclosure is not on this, it will not be described here, only the part related to the present disclosure will be described.

[0090] In the automatic fusion technique, various fusion feasibilities and data splitting feasibilities in the computation graph are traversed, and various scheduling strategies (e.g., whether to enable weight residency, whether to enable weight preloading, etc.) are considered, and the optimal operator fusion manner is selected through a performance evaluation function. Therefore, after the automatic fusion processing, the preferred operator fusion manner is provided.

[0091] The operator fusion manner can include information on the following aspects: the operator sequence that can be fused and has the optimal performance; descriptive files or structured information about the fusion level; and data splitting information.

[0092] The operator sequence can be a sequence of operators directly extracted from the computation graph represented by IR0 according to the result of the automatic fusion processing. For example: the convolution operator (conv2d), the scaling operator (scale), and the activation operator (relu) given in the previous example. This operator sequence only provides a list of operators that can be fused, but does not reflect further fusion information, such as fusion level, etc.

[0093] The fusion hierarchy can be represented by a descriptive file or structured information of the fusion hierarchy. Since the operator fusion aims to reduce IO memory access, the fusion hierarchy can be determined according to the storage hierarchy of the hardware. As can be seen from the hardware architecture described in the foregoing in combination with FIG. 3b, the storage hierarchy can include three layers: the NRAM, WRAM and other memory on each processor core IPU in the processor core level 380 are abstracted as the L1 storage hierarchy, the SRAM memory on the MPU in the processor cluster level 370 is abstracted as the L2 storage hierarchy, and the global memory of the chip level 360 is the third storage hierarchy. The computation graph optimized by fusion can also be divided into a graph and a subgraph accordingly. When the network model optimized by fusion is executed in these hardware environments, the input and output of the graph are the global memory of the chip level 360, such as DDR, and the intermediate result resides on the L2 storage hierarchy. One graph corresponds to one fusion operator. A graph can include several subgraphs (fusion sub-operators). The input and output of each subgraph are the L2 storage hierarchy, and the intermediate result resides on the L1 storage hierarchy. One subgraph can include several operators. The input and output of each operator are the L1 storage hierarchy.

[0094] In some embodiments, the fusion hierarchy information of the fusion operator can be represented by the structured text description of the graph and the subgraph. For example, the following gives an example of a structured text description of a fusion hierarchy.

[0095] Graph1{

[0096] subgraph1{

[0097] Conv1

[0098] Add1

[0099] Relu1

[0100] }

[0101] subgraph2{

[0102] conv2

[0103] scale2

[0104] bn1

[0105] }

[0106] Graph2{

[0107] ...

[0108] }

[0109] In the above example, two graphs, graph1 and graph2, are shown, both of which take and output from chip-level global memory (e.g. DDR). That is, both graph1 and graph2 take data from DDR, perform computation internally, and store the final result back to DDR. In the above example, it is further shown that graph1 includes two subgraphs, subgraph1 and subgraph2, both of which take and output from L2 memory hierarchy. That is, both subgraph1 and subgraph2 take data from shared cache SRAM on MPU, perform computation internally, and store the final result back to SRAM. Each of the subgraphs includes several operators, which take and output from L1 memory hierarchy, i.e. NRAM, WRAM, etc. on IPU.

[0110] In some other embodiments, the fusion hierarchy information can be described directly at the IR0 level in the form of operator calls. In these embodiments, each of the above subgraphs can be considered as a kind of operator call, e.g. replaced by callop. Further, the operators included in the corresponding callop can be defined.

[0111] The data splitting information is used to represent the data splitting scheme in the operator fusion scheme. Specifically, the data splitting scheme includes parallel and serial data splitting between the memory levels, each of which involves the splitting dimension and the splitting block size of the data, etc.

[0112] For example, in the environment shown in FIG. 3b, which includes two memory hierarchies on chip, the data splitting scheme of the fusion operator includes parallel splitting of multiple L2 memory hierarchies of the fusion operator, serial splitting of a single L2 memory hierarchy of the fusion operator, parallel splitting of multiple L1 memory hierarchies of each fusion sub-operator, and serial splitting of a single L1 memory hierarchy of each fusion sub-operator.

[0113] The parallel splitting of multiple L2 memory hierarchies of the fusion operator involves the hardware information of the computing device running the computation graph, e.g. in the hardware environment shown in FIG. 3b, it involves the splitting between each cluster at the cluster level.

[0114] The serial splitting mode of the single L2 storage level of the fusion operator involves splitting of the input data, and the parameters mainly include the dimension of splitting. For example, when the HW (height and width) dimension of the input feature map (input data) does not exceed a first threshold, such as the entire feature map can be stored on a single L2 storage level, at this time it can be considered as "small map mode", and the splitting dimension is N (batch) dimension. For another example, when the HW dimension of the input feature map exceeds a second threshold, at this time it can be considered as "large map mode", and the splitting dimension can include N, H, and W three dimensions. It can be understood that the parameter setting of the serial splitting mode is only exemplary, and those skilled in the art can also construct more splitting modes, such as "medium map mode", "super large map mode", etc., and the embodiments of the present disclosure are not limited in this respect. In addition, the parameters of the serial splitting mode can also include the specific splitting block size of each splitting dimension.

[0115] The parallel splitting mode of the multiple L1 storage levels of each fusion sub-operator is mainly related to the dimension of the input feature map. These dimensions may, for example, include the N (batch) dimension, H (height) dimension, W (width) dimension, and C (channel) dimension of the input feature map. The specific splitting value of each dimension can be determined, which depends on, for example, the number of computing components in the hardware implementation.

[0116] Similarly, the serial splitting mode of the single L1 storage level of each fusion sub-operator also involves splitting of the input feature map, and the parameters mainly include the dimension of splitting and the splitting block size.

[0117] In summary, the data splitting information can include the following aspects:

[0118] a. Serial splitting or parallel splitting, which can be represented by the value of the parameter by-factor-or-by-part, such as: 1 represents serial, 0 represents parallel, and vice versa. The factor of serial splitting represents the size of each part after splitting, which is usually determined by the processing capability of the computing hardware. The part of parallel splitting represents the number of parts after splitting, which is usually determined by the number of processor cores in the parallel processing hardware.

[0119] b. Splitting dimension, which represents the dimension in which the data is split, which can be any dimension such as N, H, W, C, etc.

[0120] c. Splitting coefficient, which represents the value of the factor or part above, and has corresponding meanings according to serial or parallel splitting.

[0121] d. bind-axis, which is used to indicate the mapping or binding relationship between the parallel splitting scheme and the hardware. It can be understood that the parallel splitting scheme can be parallel split at different levels, such as an L2 level or an L1 level, and accordingly, these parallel splitting schemes need to be supported by parallel processing hardware at different levels. The bind-axis is used to indicate the corresponding parallel processing hardware dimension. For example, assuming that the parallel splitting part = 4, the splitting dimension is N dimension, and the bind-axis is dim-y, it means that the N dimension is split into 4 parts, and each part corresponds to a dim-y.

[0122] The dim-x, dim-y, and dim-z of the bind-axis can be defined according to different hardware architectures to correspond to the level dimensions of different hardware (computers). In an example, the task is issued in a job unit, and generally, a job represents an independent task that has no data dependency with each other, such as processing different data blocks, and each job has its own instruction file. Each job can have different hardware units on different execution hardware. For example, on execution hardware A, the job is in a processor cluster (see cluster in FIG. 3b) unit. On the same execution hardware, the job can also have different job types, and the number of hardware units included in different job types can be different. For example, the job type can include U1 (1 processor cluster), U2 (2 processor clusters), U4 (4 processor clusters), and the like. At this time, dim-x can represent the task type of the currently issued job, and more specifically, dim-x can indicate how many executable processor cores are in the current job. For example, taking the execution hardware in FIG. 3b as an example, 1 processor cluster includes 4 processor cores, then the U1 type of job corresponds to the dim-x value of 4, the U2 type of job corresponds to the dim-x value of 8, and the like. Dim-y can represent the number of jobs. Dim-z can represent a higher dimension, such as the number of kernels that need to be started.

[0123] Optionally or additionally, the operator fusion manner can also include some configuration information, such as whether to enable pipelining, whether to enable weight residency, and the like.

[0124] The pipelining manner of the IO / computation parallel of the fused operator or the fused sub-operator mainly involves whether to enable pipelining and the number of stages of pipelining when pipelining is enabled. For example, the configuration information can be set to not enable pipelining, and then the IO access and computation are not executed in parallel. For another example, the configuration information can be set to enable pipelining, and for some modes, LCS three-stage pipelining is enabled, and for other modes, LLCSS five-stage pipelining is enabled, where L represents load, C represents compute, and S represents store.

[0125] In general in neural network computation, the weights of each network layer are fixed, so the data that is resident is usually the weights. In this regard, data residency is also sometimes referred to as weight residency. Through data residency, data reload time can be effectively reduced, data reuse times can be increased, and thus the overall computation can be accelerated. Configuration information can be set to be non-resident or resident. When resident is selected, it can further include which model data (e.g., which operator weights) are resident on the L1 storage hierarchy and which model data are resident on the L2 storage hierarchy.

[0126] Continuing with FIG. 6, next in step 620, the IR0-level computation graph is converted into a TFU IR-level computation graph based on the information of the optimal operator fusion manner determined in the previous step. In the conversion process, the information of the optimal operator fusion manner determined in the previous step is used to perform various basic processing on the operators in the computation graph represented by IR0. Optionally or additionally, some optimization processing can also be performed.

[0127] The basic processing can include, but is not limited to, instruction-level replacement, memory scope addition, memory operation addition, etc.

[0128] The instruction-level replacement includes replacing a plurality of separate operator instructions in the IR0-level computation graph with a single operator instruction according to the information of the operator fusion manner, e.g., the descriptive file of the previous fusion level.

[0129] Specifically, in some implementations, based on the fusion operator instructions supported by the hardware that is going to execute the computation graph, the corresponding plurality of separate operator instructions are replaced with a single fusion operator instruction. For example: if the hardware supports a fusion instruction for convolution (e.g., fusing a convolution operator and the subsequent addition operator and relu operator in a convolution instruction, referred to as a convolution fusion instruction), and the determined operator fusion manner indicates that the three operators are fused together, then the three operator instructions in the IR0-level computation graph can be replaced with a single convolution fusion instruction. In some implementations, certain models of hardware can support multiple fusion operator instructions, such as a deep convolution fusion instruction, a fusion deconvolution instruction, a post-folded convolution fusion instruction, an instruction that combines several vector operations, etc.

[0130] In other implementations, if there is no fusion operator instruction corresponding to the fusion operator, then the plurality of operator instructions that need to be fused can be replaced with an operator call instruction callop, and the operators included in the corresponding callop are defined in the file. In some implementations, according to the structured text of the fusion level, one of the subgraphs is replaced with a callop, and it is defined after that which operators are included in the callop.

[0131] The addition of memory scopes includes adding attributes to the operator based on the definition of the operator itself to describe which scopes of memory the input and output data of the operator are stored in, respectively. At this stage, the granularity of the scope can be the memory scope divided according to the content of the data to be saved. For example, based on the hardware architecture of FIG. 3b, assuming that the convolution operator itself requires the input neurons to be stored on the NRAM, the input convolution kernel to be stored on the WRAM, and the output neurons to be stored on the NRAM, the memory scopes are added to the convolution operator according to the requirements of the convolution operator.

[0132] Further, information such as data type and data shape can also be described. The data type refers to whether the data is an integer type or a floating point type, and these data types can include, but are not limited to, int8, int16, int32, fp16, fp32, tf32, and the like. The data shape at this level refers to the complete shape of the data, which has not yet been split. For example, the complete shape of the input neurons of a convolution operator is 1x24x24x64, the data type is uint8, and the storage is on the NRAM. In some implementations, the above information can be described using memref, thereby indicating the layout of the tensor data in the memory space.

[0133] The addition of memory operations includes adding memory operations for moving data between different storage levels according to the information at the fusion level. As mentioned earlier, the input and output of the graph is the global memory of the chip level, such as DDR. One graph corresponds to one fused operator or kernel. The input and output of each subgraph is the L2 storage level, and the intermediate result resides on the L1 storage level. Therefore, between different graphs, memory operations need to be added to interact with the global memory (e.g., DDR); and between different subgraphs, memory operations need to be added to interact with the L2 storage level (e.g., shared storage SRAM). For example, a copy instruction from the L1 storage level to the L2 storage level is inserted after the output of subgraph 1 to store the output data back to the L2 storage level; and a copy instruction from the L2 storage level to the L1 storage level is inserted before the input of subsequent subgraph 2 to obtain the input data from the L2 storage level. For another example, a copy instruction from the L2 storage level to the chip-level global memory is inserted after the output of graph 1, and a copy instruction from the chip-level global memory to the L2 storage level is inserted before the input of subsequent graph 2.

[0134] Further, the added memory operations can further limit the memory source and / or destination of the operation according to the memory scope added to the operator. For example, for a fused convolution operator (assuming corresponding to a subgraph), before its input, a memory copy instruction from shared storage SRAM to NRAM can be inserted for the input neuron data; while for the convolution kernel, a memory copy instruction from shared storage SRAM to WRAM can be inserted.

[0135] Optionally or additionally, during the conversion of the IR0-level computation graph to the TFU IR-level computation graph, some optimization processing can also be performed. These optimization processing can include but are not limited to: Co-dimension splitting, alignment optimization, setting weight residency, inter-operator weight reuse, reuse activation table, marking shape inference entry, etc.

[0136] After the above conversion processing, in step 630, the converted TFU IR-level computation graph is output for subsequent processing.

[0137] As can be seen from the above conversion process, the IR0-level computation graph is completely hardware-independent, and simply describes the network-level computation from the software level. In the TFU IR-level computation graph, operator fusion is introduced, and hardware-related information is added from the perspective of operator fusion, such as hardware-supported fusion instructions, memory operations between hardware storage levels corresponding to the fused operator, etc. Further, in this conversion process, optimization processing can also be performed in units of fused operators, such as vector merging, memory operation merging, etc., thereby performing compilation optimization on the computation graph with fused operators as the granularity.

[0138] Conversion from TFU IR to IR1

[0139] After conversion from IR0 to TFU IR, it is further necessary to convert the TFU IR to IR1 through a third conversion. In this conversion process, shape inference, memory allocation, and then backend code generation need to be performed. The text of code generation is also IR1, so IR1 is a completely hardware-dependent abstract representation.

[0140] In this article, shape refers to the dimension information of data, including the number of dimensions, the size of each dimension, etc. In intelligent computing processing, the shape of the data will be reflected in the memory access address of the data. Therefore, in this context, the memory access mode refers to the memory access address of each data block obtained by splitting or cutting a certain dimension of the data according to the dimension splitting scheme.

[0141] In this context, shape inference, memory access pattern inference or memory access address inference refers to, after a certain dimension of data is split or sliced, the memory access address of each data block of the input data and / or output data associated with the operator in the computational graph needs to be inferred according to the split data block. Here, the associated operator refers to the operator in the computational graph that needs to directly or indirectly access the data, for example, needs to directly or indirectly use the data to perform calculation, or needs to directly or indirectly write the calculation result to the data. In addition, in the following description, unless otherwise specified, the data involved is tensor data.

[0142] FIG. 7 exemplarily shows an exemplary flowchart of a third conversion process from TFU IR to IR1 implemented by the processing device according to an embodiment of the present disclosure. The processing device can be, for example, the processing device 203 of FIG. 2.

[0143] As shown, in step 710, the memory access pattern of the operator in the TFU IR graph is inferred based on the data split information. The data split information here refers to the data split information determined according to the automatic fusion technique in the preceding second conversion process from IR0 to TFU IR, which can be referred to the foregoing description and will not be repeated here.

[0144] In some implementations, the data split information generally refers to the split information of the output data. Therefore, when performing the memory access pattern inference, the operator of the output data (i.e., the end operator in the computational graph) is taken as the entry operator, i.e., the starting point of the inference, to perform the memory access pattern inference. When the end operator is selected as the entry operator, the inference direction of the memory access pattern is reversed, i.e., the memory access pattern of the input data is inferred in the reverse direction against the data flow direction in the computational graph from the memory access pattern of the output data.

[0145] In this memory access pattern inference, the entry operator data is split into one or more data blocks according to the dimension split scheme of the data of the entry operator. Then, based on the memory access pattern of the entry operator data, the memory access pattern of the data associated with the entry operator data in the computational graph is inferred. When the entry operator data is split according to the dimension split scheme, the corresponding memory access pattern can be determined. The memory access pattern includes the memory access address of each data block into which the data set is split.

[0146] In some embodiments, the memory access pattern inference can include taking the end operator as the entry operator, starting from the entry operator, traversing each layer of operator in the computational graph from back to front, and performing reverse shape inference to obtain the memory access address of one or more input data blocks and output data blocks that need to be split for each layer of operator to meet the dimension split scheme of the entry operator.

[0147] The inventors found through research on deep learning algorithms that the network structure of a neural network is complex, but the basic component units thereof are relatively stable, mainly including convolution (conv), pooling, padding, vector operations, and the like. These basic operators account for the majority of the overall computation time. Further analysis of these basic operators reveals that they can be decomposed into some basic operations. In this document, these basic operations are referred to as atomic operations. Note that, since the processing of data in intelligent computing involves division by dimension, the atomic operations here are also determined in units of dimensions. In the embodiments of this disclosure, by analyzing the memory access patterns of the basic operators, a number of memory access pattern derivation formulas of atomic operations can be abstracted. Thus, the memory access patterns of various operator data can be derived based on these memory access pattern derivation formulas.

[0148] Specifically, in the traversal process of the above-mentioned computation graph, for each operator in the computation graph, the reverse shape derivation can include: decomposing the operation of the operator into a number of atomic operations in units of dimensions; and reversely deriving, according to the input-output memory address relationship of the atomic operations, i.e., the memory access pattern derivation formula, the memory address of the input data block required by the operator to compute one output data block.

[0149] In some embodiments, the atomic operations can include one or more of the following: a forward sliding window operation, such as used in convolution (conv), pooling, and the like, usually in the width (W) and height (H) dimensions; a forward padding operation, such as used in the padding operation of a convolution, pooling, and the like, or in a separate padding (pad) operator (with a negative pad size); a reverse sliding window operation, such as used in deconvolution, unpooling, and the like, usually also in the width (W) and height (H) dimensions; a reverse padding operation, such as used in deconvolution, or in a separate padding operator (with a negative pad size); a vector operation, used in various vector operations in the operator, such as addition, subtraction, multiplication, activation calculation, and the like; a broadcast operation, used in broadcast replication operations, similar to the broadcast operation of numpy on a certain dimension, such as in cycle add; and an aggregation operation (reduce), used to aggregate data, such as the accumulation operation on the Ci dimension in the convolution (conv) operator, reduce computation, and the like.

[0150] Based on the abstracted atomic operations described above, the memory access pattern derivation formula, i.e., the input-output memory address relationship of the atomic operations, can be derived according to the specific operation mode thereof. In this way, when traversing the computation graph, the operation of the current operator can be decomposed into a number of atomic operations in units of dimensions, and then the memory access pattern derivation formula of the atomic operations can be derived to obtain the memory access pattern of the data of the current operator.

[0151] Then, in step 720, memory allocation is performed based on the derived memory access pattern. The derived memory access pattern indicates the sizes of the split chunks of the input and output data of each operator, i.e., indicates the memory sizes required to be occupied by each split chunk. Based on this information, memory can be allocated for the split chunks. In a specific implementation, the memory allocation is performed on the corresponding memory scopes based on the memory scopes added for the operators in the process of converting to the TFU IR. The memory allocation information may, for example, include the memory scope, the starting position (the first address) in the memory scope, and the allocated space size (the size of the split chunk).

[0152] Finally, in step 730, based on the result of the memory allocation, a code generation process is performed based on the TFU IR to compile and optimize to obtain a binary file recognizable by the intelligent processor.

[0153] The above describes an exemplary implementation flow of the third conversion process. As can be seen from the above conversion process, the computation graph at the IR1 level is completely hardware-related, describing the execution process of the computation graph from the level of hardware instructions, such as the specific execution implementation of the operators, the flow of instructions, and the like. In addition, the implementation of the operators is based on the computation graph at the TFU IR level, i.e., at the granularity of the fused operators.

[0154] In summary, through the above-mentioned multiple conversion processes, the completely hardware-independent IR representation IR0 can be converted to the completely hardware-dependent IR representation IR1 via the TFU IR representation featuring the fused operators. In turn, the IR1 can be handed over to the runtime compilation library to generate binary code for execution on the corresponding intelligent processor.

[0155] Input-variable compilation optimization

[0156] Generally, the input data size of a neural network model is fixed, i.e., the data shape information is consistent at the compilation period. In this case, the data shape of the instructions to be generated is known at the compilation period, so the opcode and address field in the instructions can be determined at the compilation period. In order to fully utilize hardware resources, various performance optimizations are provided at the compilation period, including pipeline arrangement, etc., and the runtime compilation library is processed to generate binary code that can be directly recognized on the intelligent computing hardware, which is handed over to the driver for analysis and execution on the intelligent computing hardware. Therefore, the aforementioned optimization of the fused operators is also performed at the compilation period.

[0157] In addition, when generating instructions at the compilation period, the generation logic is to expand all loops and describe the dependency relationship between instructions through a tree structure. In the actual instruction generation process, the tree structure is completely expanded. The cost of this processing manner is that the number of instructions is very large, and the storage burden is very large, which is not conducive to deployment on mobile or embedded devices with limited storage capacity.

[0158] However, in some computing scenarios, the input data size of the neural network model is not fixed, i.e., variable. Such variable input often leads to poor performance in the neural network inference phase, because it is generally believed that the more specific the given input range is, the better the performance optimization effect is. If a set of corresponding instructions is generated for each input size (for example, for different resolutions in image processing), the number of instructions will increase linearly, increasing the processing and storage burden. One known way is to divide the input range into intervals and perform inference calculation according to the upper limit of the interval. However, if the calculation is performed according to the upper limit each time, it will cause waste of computing resources. Further, due to the unknown input shape, when a loop is involved, the number of loops is also unknown, so the loop cannot be unfolded in the original way. Therefore, for the case of variable input shape, further optimization schemes need to be provided to solve one or more of the foregoing problems.

[0159] In the case of variable input shape, the shape size of the input data during actual execution is not known during compilation, i.e., the compiler does not know the specific input data shape of the generated instructions, so it cannot fill the accurate parameters in the instruction domain, for example, it can only fill the instruction domain parameters according to the upper limit. In the embodiments of the present disclosure, by generating instructions based on the input shape data obtained at runtime, more accurate instruction domain parameters can be generated, thereby supporting better instruction execution performance. In other words, part of the instruction generation steps can be adjusted to the runtime. More specifically, the instruction flow arrangement can be implemented in the code generation stage (codegen) during the compilation period, and accurate instruction parameters can be generated according to the obtained exact input data shape at runtime, filled into the corresponding instruction domain, thereby generating instructions that can be executed on smart hardware. Further, in the code generation stage (codegen) during the compilation period, the loop is no longer unfolded, but a loop instruction is used instead, for example, a branch instruction supported by hardware is used, which is encapsulated to support a for loop, and the instruction pipeline is arranged on this basis.

[0160] FIG. 8 shows an exemplary method flowchart of generating instructions by a computing device during the compilation period according to embodiments of the present disclosure. The computing device may, for example, be the computing device 201 of FIG. 2. The process of generating instructions is similar to the processing done by the instruction generation process of the compiler, the difference being that the input data shape at this time is exact information, not the upper limit of the interval.

[0161] As shown, in step 810, according to the computation graph expressed by the TFU IR, the shape of the operators in the computation graph is forward-derived to obtain complete shape information.

[0162] In the forward pass, the forward pass logic of each operator can be invoked to obtain the full shape information of the input and output data of each operator, i.e. the shape without any splitting. In this forward pass, the accurate shape information of the input data obtained at runtime is used.

[0163] In particular, the forward pass is generally performed with the starting operator in the computation graph as the entry operator. The starting operator refers to an operator in the computation graph without a predecessor operator, which is usually the first operator in the computation graph. When the starting operator is selected as the entry operator, the shape derivation direction is forward, i.e. along the direction of data flow in the computation graph, the shape of the output data is derived from the shape of the input data forwardly.

[0164] The forward pass logic of an operator is associated with the mathematical operation logic of the operator, which can be determined similarly based on the backward pass logic described above and will not be repeated here.

[0165] Next, in step 820, the memory access pattern of the operators in the computation graph expressed in the TFU IR is derived based on the data splitting information and the full shape information obtained in the forward pass. In some implementations, the data splitting information here can be the data splitting information determined in the front end, e.g. the data splitting information determined according to the automatic fusion technique in the second conversion process from IR0 to TFU IR. In other implementations, the data splitting information here can be the data splitting information adjusted again according to the determined shape information of the input data obtained at runtime, which will be described in detail later.

[0166] The process of the memory access pattern derivation is similar to step 710 described above in connection with FIG. 7, with the difference that the data shape information used is the determined shape information of the input data obtained at runtime, which will not be described in detail here.

[0167] Through the above memory access pattern derivation, the memory access pattern of the split tiles of the input and output data of each operator in the graph can be obtained.

[0168] Next, in step 830, based on the derived memory access pattern, the address domain related parameters of the instructions involved in the loop in the code logic of the computation graph are determined. The memory access pattern indicates the memory access address of the split tiles of the input and output data of each operator, and thus the related parameters of the instructions can be determined based thereon, e.g. the size of the split tile processed by the instructions, the size of each dimension of HWC, etc.

[0169] Next, in step 840, the instructions in the loop are pipelined based on the instruction stream supported by the hardware.

[0170] In pipelining, the instructions can be pipelined according to the configuration information of the operator fusion mode determined in the previous automatic fusion process, such as whether to enable pipelining, and the number of stages of pipelining when pipelining is enabled.

[0171] In pipelining, the instructions can be pipelined according to the configuration information of the operator fusion mode determined in the previous automatic fusion process, such as whether to enable pipelining, and the number of stages of pipelining when pipelining is enabled.

[0172] Finally, in step 850, the computation graph is compiled according to the address domain related parameters of the instructions involved in the loop and the results of the instruction pipelining in the loop.

[0173] Optionally or additionally, in some embodiments, the data splitting information determined by the front end can be adjusted as appropriate to enable the currently running instance to better adapt to the hardware, fully utilize the computing power of the hardware, improve processing efficiency, etc.

[0174] Figure 9 shows an exemplary method flowchart for generating instructions at compile time implemented by a computing device according to embodiments of the present disclosure. In this embodiment, it can be determined whether the splitting information needs to be adjusted according to the conditional judgment.

[0175] As shown, similar to Figure 8, first in step 910, according to the computation graph expressed by the TFU IR, the shape of the operators in the graph is forward derived to obtain complete shape information.

[0176] Next, in step 920, based on the derived shape information, it is determined whether the data splitting information needs to be adjusted.

[0177] In some embodiments, the data splitting information can be modified according to the splitting efficiency. The splitting efficiency can be evaluated from multiple perspectives. In some implementations, the splitting efficiency can be defined by the utilization rate of the processing core. For example, the utilization rate of the processing core IPU efficiency The utilization rate of the processing core IPU

[0178] where N o , H o , W o represent the original size of the output data in the N, H, and W dimensions respectively; H ipu-factor and W ipu-factor represent the splitting block size in the H and W dimensions respectively, IPUpart Num represents the number of processing cores for parallel processing. cluster Indicates the number of clusters, Cluster part This represents the number of computation clusters processed in parallel.

[0179] In the above formula, the numerator represents the original data volume before splitting, which can be determined, for example, based on the shape information derived in step 910. The denominator represents the amount of data that needs to be processed after splitting according to the splitting scheme indicated by the front-end data splitting information.

[0180] If the splitting efficiency falls below a specified threshold, it's determined that the splitting scheme needs adjustment. In this case, the splitting scheme is adjusted to update the current data splitting information. This adjustment uses the data splitting information provided by the front end as a reference, adjusting the scheme based on the actual, defined data shape. For example, in scenarios with variable data, the front end might provide a splitting scheme based on the upper bound of a certain interval, while the actual data shape might be at the lower bound of that interval. Therefore, the original splitting scheme would have significant redundancy, leading to low processing core utilization. To address this, a new splitting scheme is determined based on the actual data shape to improve processing core utilization. The process of determining the splitting scheme is similar to that in automatic fusion, except that the data shape information used is different.

[0181] For example, assuming the front-end compiler knows that the N dimension is variable within the range of 1 to 4, the splitting information generated during compilation is for the case where the upper bound of the N dimension is 4, splitting the data into 4 batches and executing them on 4 computing clusters. However, in actual runtime, the actual input data batch N=1. In this case, if the splitting scheme of the front-end compiler is still followed, this 1 batch of data will be allocated to 1 computing cluster for execution, and the other computing clusters will be idle. Obviously, this splitting method leads to low hardware utilization. Therefore, according to the embodiments disclosed herein, during runtime, based on the obtained determined data shape information, the data splitting scheme can be readjusted, for example, splitting 1 batch of data into 4 computing clusters according to the H and W dimensions for execution, thereby improving computational efficiency.

[0182] Optionally or additionally, in some embodiments, it is determined whether the data splitting information needs to be modified based on the current hardware operating mode.

[0183] One metric to measure the processing performance of a hardware to a model is Throughput, i.e. the amount of input data that the hardware can process per second when running a certain model. Another metric is Latency, i.e. the average length of time to process data. Throughput and Latency are inversely related. Therefore, the hardware running mode can be divided into Throughput mode and Latency mode. Throughput mode mainly tests the processing throughput in the case of large batch data. Latency mode mainly tests the processing latency in the case of small batch data. Based on the difference between the two modes, in some embodiments of the present disclosure, whether the data split information needs to be modified can be considered only when the current hardware running mode is Latency mode.

[0184] Then, based on the updated data split information, the memory access mode of the operators in the graph is re-derived. The process of memory access mode derivation is similar to step 820 described above in combination with FIG. 8, and will not be described in detail here. Through the above memory access mode derivation, the memory access mode of the split blocks of the input and output data of each operator in the graph can be obtained. If it is determined in step 920 that the data split information does not need to be adjusted, then directly jump to step 930 to perform memory access mode derivation on the operators in the computation graph based on the required tensor data split information.

[0185] Then, in step 940, based on the derived memory access mode, the address domain related parameters of the instructions involved in the loop in the code logic of the computation graph are determined; in step 950, based on the instruction stream supported by the intelligent processor, the instructions are pipelined; and in step 960, the computation graph is compiled according to the address domain related parameters of the instructions involved in the loop and the results of the instruction pipelining in the loop, to obtain a binary file that can be recognized by the intelligent processor. Steps 940 to 960 are similar to steps 830 to 850 in FIG. 8, and will not be described in detail here.

[0186] Thus, the above describes the runtime scheme of the computation graph provided by the embodiments of the present disclosure, which can use real and determined input data shape information to adjust the instructions in real time for the scenario of variable input shape, so that the real scenario can better adapt to the hardware, fully utilize the computing power of the hardware, and improve the processing efficiency.

[0187] The present disclosure also provides a processing device which can be used to perform compilation on a computation graph, comprising: a processor configured to execute program instructions; and a memory configured to store the program instructions, which when loaded and executed by the processor, cause the processor to perform the compilation method described in the embodiments of the present disclosure.

[0188] In the embodiments of the present disclosure, a computer readable storage medium is also provided, in which program instructions are stored, and when the program instructions are loaded and executed by a processor, the processor performs the compilation method of the computational graph described in the embodiments of the present disclosure. In the embodiments of the present disclosure, a computer program product is also provided, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the compilation method of the computational graph described in the embodiments of the present disclosure is implemented.

[0189] The embodiments of the present disclosure also provide a chip, which can include the aforementioned processing device. Further, the present disclosure also provides a board card, which can include the aforementioned chip.

[0190] According to different application scenarios, the electronic device or apparatus of the present disclosure can include a server, a cloud server, a server cluster, a data processing device, a robot, a computer, a printer, a scanner, a tablet computer, a smart terminal, a PC device, an Internet of Things terminal, a mobile terminal, a mobile phone, a vehicle recording device, a navigation device, a sensor, a camera, a camera, a video camera, a projector, a watch, a headset, a mobile storage, a wearable device, a visual terminal, an autonomous driving terminal, a vehicle, a household appliance, and / or a medical device. The vehicle includes an airplane, a ship and / or a vehicle; the household appliance includes a television, an air conditioner, a microwave oven, a refrigerator, an electric rice cooker, a humidifier, a washing machine, an electric lamp, a gas stove, an exhaust hood; the medical device includes a nuclear magnetic resonance instrument, a B-ultrasound instrument and / or an electrocardiograph. The electronic device or apparatus of the present disclosure can also be applied to the fields of Internet, Internet of Things, data center, energy, transportation, public management, manufacturing, education, power grid, telecommunications, finance, retail, construction site, medical treatment, etc. Further, the electronic device or apparatus of the present disclosure can also be used in cloud, edge, terminal and other application scenarios related to artificial intelligence, big data and / or cloud computing. In one or more embodiments, the electronic device or apparatus with high computing power according to the present disclosure scheme can be applied to a cloud device (such as a cloud server), and the electronic device or apparatus with small power consumption can be applied to a terminal device and / or an edge device (such as a smart phone or a camera). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or the edge device are compatible with each other, so that according to the hardware information of the terminal device and / or the edge device, appropriate hardware resources can be matched from the hardware resources of the cloud device to simulate the hardware resources of the terminal device and / or the edge device, so as to complete the unified management, scheduling and collaborative work of end-cloud integration or cloud-edge integration.

[0191] It should be noted that, for the purpose of clarity, the disclosure describes some methods and embodiments thereof as a series of acts and / or combinations thereof, but those skilled in the art will understand that the present disclosure is not limited to the order of the acts described. Those skilled in the art will understand and appreciate that some steps of the methods can be decided to be executed in other orders or at the same time with other steps. Further, those skilled in the art will understand and appreciate that some of the embodiments described in the disclosure can be considered optional, i.e., the acts or modules involved therein are not necessarily essential for the implementation of one or more of the aspects of the present disclosure. In addition, the disclosure describes some embodiments with different focuses according to different aspects. In view of this, those skilled in the art will understand that the parts not described in detail in some embodiments of the disclosure can also be seen from the relevant description of other embodiments.

[0192] In terms of specific implementation, based on the disclosure and teachings of the present disclosure, those skilled in the art can understand that the embodiments disclosed in the present disclosure can also be implemented in other ways not disclosed herein. For example, as for each unit in the electronic device or apparatus embodiments described above, the units are divided herein on the basis of logical functions, and other division manners can also be used in actual implementation. For another example, a plurality of units or components can be combined or integrated into another system, or some features or functions of the units or components can be selectively disabled. As for the connection relationship between different units or components, the connection discussed above in conjunction with the drawings can be a direct or indirect coupling between the units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0193] In the present disclosure, the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units. The aforementioned components or units can be located in the same place or distributed on a plurality of network units. In addition, according to actual needs, some or all of the units can be selected to achieve the purpose of the aspects described in the embodiments of the present disclosure. In addition, in some scenarios, a plurality of units in the embodiments of the present disclosure can be integrated into one unit or each unit physically exists separately.

[0194] In some other implementation scenarios, the above-mentioned integrated units can also be implemented in the form of hardware, i.e., specific hardware circuits, which can include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit can include but is not limited to physical devices, and the physical devices can include but are not limited to transistors or memristors, etc. In view of this, various apparatuses (e.g., computing apparatuses or other processing apparatuses) described herein can be implemented by appropriate hardware processors, such as CPUs, GPUs, FPGAs, DSPs, ASICs, etc. Further, the aforementioned storage units or storage devices can be any appropriate storage medium (including magnetic storage media or magneto-optical storage media, etc.), which can be, for example, resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM, etc.

[0195] While the present disclosure has been illustrated and described with reference to various embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details can be made therein without departing from the spirit and scope of the disclosure. It should be understood that various alternatives to the embodiments of the disclosure described herein can be employed in practicing the disclosure. The appended claims are intended to cover such alternatives and equivalents.

Claims

1. A method of compiling a computational graph, wherein, The method comprises: obtaining a computation graph expressed by a second intermediate representation to be compiled, performing forward derivation of a shape, and obtaining complete shape information according to the forward derivation and tensor data splitting information; wherein the second intermediate representation is obtained by conversion based on a first intermediate representation, the first intermediate representation is obtained by uniform conversion of a task model represented by at least one front-end language; and the computation graph is a task executed on an intelligent processor; using the complete shape information to determine whether tensor data splitting information needs to be adjusted; determining required tensor data splitting information according to a result of the determination; performing memory access mode derivation on operators in the computation graph based on the required tensor data splitting information; based on the derived memory access mode, determining address domain related parameters of instructions involved in a loop in code logic of the computation graph; based on an instruction stream supported by the intelligent processor, performing pipeline arrangement on the instructions in the loop; performing compilation on the computation graph according to the address domain related parameters of the instructions involved in the loop and a result of the pipeline arrangement of the instructions in the loop, to obtain a binary file recognizable by the intelligent processor.

2. The method of claim 1, wherein, The memory access mode derivation means that, after a certain dimension of tensor data is split or cut, the memory access address of each data block of input data and / or output data of an associated operator in the computation graph needs to be derived according to the split data blocks.

3. The method of claim 1, wherein, The step of performing memory access mode derivation on the operators in the computation graph based on the required tensor data splitting information comprises: using an end operator in the computation graph as an entry operator, using splitting information of output tensor data of the entry operator, and determining the memory access mode of input tensor data of the entry operator in a reverse direction from the splitting information of the output tensor data, against the direction of tensor data flow in the computation graph.

4. The method of claim 1, wherein, The step of determining whether the tensor data splitting information needs to be adjusted comprises: determining whether the tensor data splitting information needs to be adjusted according to the splitting efficiency of the tensor data.

5. The method of claim 4, wherein, The splitting efficiency is defined by the utilization of processing cores, wherein the utilization of processing cores IPU efficiency The calculation is performed according to the following formula: wherein, N o , H o , W o respectively represent the original size of the output tensor data in N, H, W dimensions; H ipu-factor and W ipu-factor respectively represent the split block size in H and W dimensions, IPU part represents the number of processing cores of the smart processor parallel processing, Num cluster represents the number of computing clusters of the smart processor, Cluster part represents the number of computing clusters of the smart processor parallel processing.

6. The method of claim 1, wherein, The step of determining whether the tensor data splitting information needs to be adjusted comprises: determining whether the tensor data splitting information needs to be adjusted according to the intelligent processor running mode.

7. The method of claim 6, wherein, The intelligent processor running mode comprises a Throughput mode and a Latency mode; wherein the Throughput mode means a mode for measuring the processing throughput of the intelligent processor when the dimension batch of tensor data is greater than a threshold; and the Latency mode means a mode for measuring the processing latency of the intelligent processor when the dimension batch of tensor data is less than the threshold.

8. A processing device for performing compilation on a computation graph, comprising: a processor configured to execute program instructions; and a memory configured to store the program instructions, which, when loaded and executed by the processor, cause the processor to perform the computation graph compilation method according to any one of claims 1-7.

9. A computer readable storage medium, having stored therein program instructions, which when loaded and executed by a processor, cause the processor to perform the method of compiling a computational graph according to any one of claims 1-7.

10. A computer program product, comprising a computer program or instructions, which when executed by a processor, implement the method of compiling a computational graph according to any one of claims 1-7.

Images