Computing graph compiling method and related product

Abstract

The invention discloses a compilation method of a computational graph and a related product. The method comprises the steps of obtaining a to-be-compiled computational graph expressed by a second intermediate representation, performing forward derivation of a shape, and obtaining complete shape information according to the forward derivation and tensor data splitting information; judging whether tensor data splitting information needs to be adjusted or not by utilizing the complete shape information; tensor data splitting information meeting requirements is determined according to a judgment result; performing memory access mode derivation on operators in the computational graph based on tensor data splitting information meeting requirements; based on the deduced memory access mode, determining address field related parameters of instructions involved in circulation in code logic of the computational graph; performing pipeline arrangement on the instructions in the circulation; and compiling the calculation graph according to the address field related parameters of the instruction involved in the loop and an instruction pipeline arrangement result to obtain a binary file which can be identified by the intelligent processor.

Description

Technical Field

[0001] The present disclosure generally relates to the field of intelligent computing, and more particularly to the field of neural networks. More specifically, the present disclosure relates to a method for compiling a computational graph, a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] In intelligent computing systems, for machine learning algorithm models of various machine learning applications, deep learning frameworks are used as the main programming interface to build corresponding machine learning / deep learning networks. Deep learning frameworks can include a variety of programming frameworks, such as Google's tensor flow graph TensorFlow, deep learning frameworks Caffe, Caffe2, MXNet, PyTorch, etc. In these programming frameworks, computational graphs (or intermediate representation IR graphs) are usually used to describe the computational process of machine learning algorithms, tensors are used to represent all data in the computational graphs, and operators are used to represent various operations.

[0003] Various graph-level optimizations can be performed on the IR graph. These optimizations may include, but are not limited to, constant folding, operator fusion, graph substitution, etc. The optimized IR graph can be compiled into an abstract language representation, i.e., various instructions. These instructions can be executed by the underlying machine learning hardware.

[0004] During the compilation from the computation graph to the underlying machine instruction code, it is usually necessary to introduce several fixed levels of abstraction to convert the intermediate representation of the language into the intermediate representation supported by the target compilation framework in a step-by-step manner. At this multi-level compilation abstraction level, how to design and implement a general compilation technology route is an important challenge facing current optimizing compilers. Summary of the invention

[0005] In order to at least solve one or more of the technical problems mentioned above, the present disclosure proposes a compilation scheme for a computational graph in multiple aspects, which introduces a TFUIR level during compilation, which optimizes the computational graph from the perspective of operator fusion, and can more effectively take into account the information of the algorithm software and the execution hardware, thereby improving the compiler optimization effect. In other aspects, the present disclosure also provides a compilation scheme for converting from a higher level to the TFUIR level, and a compilation scheme for converting from the TFUIR level to a lower level.

[0006] In a first aspect, the present disclosure discloses a method for compiling a computation graph, wherein the method comprises:

[0007] Obtaining a computational graph to be compiled and expressed by a second intermediate representation, performing forward derivation of the shape, and obtaining complete shape information according to the forward derivation and tensor data splitting information; wherein the second intermediate representation is obtained by conversion based on the first intermediate representation, and the first intermediate representation is obtained by unified conversion based on a task model expressed by at least one front-end language; and the computational graph is a task executed on an intelligent processor;

[0008] Using the complete shape information, determining whether the tensor data splitting information needs to be adjusted;

[0009] Determine the tensor data splitting information that meets the requirements according to the judgment result;

[0010] Derivation of memory access modes for operators in the computation graph based on the tensor data splitting information that meets the requirements;

[0011] Based on the derived memory access mode, determining address domain related parameters of instructions involved in a loop in the code logic of the computation graph;

[0012] Pipelining the instructions in the loop based on the instruction stream supported by the intelligent processor;

[0013] The calculation graph is compiled according to address domain related parameters of the instructions involved in the loop and the result of the instruction pipeline arrangement in the loop to obtain a binary file recognizable by the intelligent processor.

[0014] In a second aspect, the present disclosure provides a processing device for performing compilation on a computation graph, comprising:

[0015] a processor configured to execute program instructions; and

[0016] A memory configured to store the program instructions, which, when loaded and executed by the processor, enables the processor to execute the method for compiling the computational graph according to the first aspect of the present disclosure.

[0017] In a third aspect, the present disclosure provides a computer-readable storage medium having program instructions stored therein, which, when loaded and executed by a processor, causes the processor to execute the method for compiling the computational graph according to the first aspect of the present disclosure.

[0018] In a fourth aspect, the present disclosure provides a computer program product, including a computer program or an instruction, which, when executed by a processor, implements the method for compiling the computational graph according to the first aspect of the present disclosure.

[0019] In the compilation scheme of the computational graph provided above, the disclosed embodiment specifically introduces the concept of the second intermediate expression TFUIR level and the inter-level conversion scheme associated with this level, targeting operator fusion, an important optimization method in neural networks, thereby providing an efficient optimization compilation scheme. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] By reading the detailed description below with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0021] Figure 1 A structural diagram of a board according to an embodiment of the present disclosure is shown;

[0022] Figure 2 A structural diagram showing a combined processing device according to an embodiment of the present disclosure;

[0023] Figure 3a A schematic diagram showing the internal structure of a single processor core of a single-core computing device according to an embodiment of the present disclosure;

[0024] Figure 3b A simplified schematic diagram showing the internal structure of a multi-core computing device according to an embodiment of the present disclosure;

[0025] Figure 4 shows the schematic structure of the software stack;

[0026] Figure 5 The hierarchical structure of the optimization compiler of the embodiment of the present disclosure is exemplarily shown;

[0027] Figure 6 An exemplary flow chart illustrating a second conversion process from IR0 to TFUIR implemented by a processing device according to an embodiment of the present disclosure;

[0028] Figure 7 An exemplary flow chart exemplarily illustrates a third conversion process from TFUIR to IR1 implemented by a processing device according to an embodiment of the present disclosure;

[0029] Figure 8 A flowchart of an exemplary method for generating instructions at compile time implemented by a computing device according to some embodiments of the present disclosure is shown;

[0030] Fig. 9 A flowchart of an exemplary method for generating instructions at compile time and implemented by a computing device according to other embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0032] It should be understood that the terms "first", "second", "third", and "fourth" etc. that may appear in the claims, specifications, and drawings of the present disclosure are used to distinguish different objects rather than to describe a specific order. The terms "include" and "comprise" used in the specifications and claims of the present disclosure indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their collections.

[0033] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.

[0034] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context.

[0035] The specific implementation of the present disclosure is described in detail below with reference to the accompanying drawings.

[0036] Exemplary Hardware Architecture

[0037] Figure 1 FIG. 1 is a schematic diagram showing the structure of a board 10 according to an embodiment of the present disclosure. Figure 1As shown, the board 10 includes a chip 101, which is a system-on-chip (SoC), or system on chip, integrated with one or more combined processing devices, which are artificial intelligence computing units that support various deep learning and machine learning algorithms to meet the intelligent processing needs in complex scenarios in the fields of computer vision, speech, natural language processing, data mining, etc. In particular, deep learning technology is widely used in the field of cloud intelligence. A notable feature of cloud intelligence applications is the large amount of input data, which places high demands on the storage capacity and computing power of the platform. The board 10 of this embodiment is suitable for cloud intelligence applications and has huge off-chip storage, on-chip storage, and powerful computing power.

[0038] The chip 101 is connected to an external device 103 via an external interface device 102. The external device 103 is, for example, a server, a computer, a camera, a display, a mouse, a keyboard, a network card, or a wifi interface. The data to be processed can be transmitted from the external device 103 to the chip 101 via the external interface device 102. The calculation result of the chip 101 can be transmitted back to the external device 103 via the external interface device 102. According to different application scenarios, the external interface device 102 can have different interface forms, such as a PCIe interface.

[0039] The board 10 also includes a storage device 104 for storing data, which includes one or more storage units 105. The storage device 104 is connected to the control device 106 and the chip 101 through a bus and transmits data. The control device 106 in the board 10 is configured to control the state of the chip 101. To this end, in an application scenario, the control device 106 may include a microcontroller (MCU).

[0040] Figure 2 1 is a block diagram showing the combined processing device in the chip 101 of this embodiment. Figure 2 As shown in , the combined processing device 20 includes a computing device 201 , an interface device 202 , a processing device 203 and a storage device 204 .

[0041] The computing device 201 is configured to execute user-specified operations, and is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0042] The interface device 202 is used to transmit data and control instructions between the computing device 201 and the processing device 203. For example, the computing device 201 can obtain input data from the processing device 203 via the interface device 202 and write it into the storage device on the computing device 201 chip. Further, the computing device 201 can obtain control instructions from the processing device 203 via the interface device 202 and write them into the control cache on the computing device 201 chip. Alternatively or optionally, the interface device 202 can also read data in the storage device of the computing device 201 and transmit it to the processing device 203.

[0043] The processing device 203, as a general processing device, performs basic controls including but not limited to data handling, starting and / or stopping the computing device 201. Depending on the implementation, the processing device 203 can be a central processing unit (CPU), a graphics processing unit (GPU) or one or more types of processors in other general and / or special processors, these processors include but are not limited to digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, only with respect to the computing device 201 disclosed in the present invention, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are integrated and considered together, the two are regarded as forming a heterogeneous multi-core structure.

[0044] The storage device 204 is used to store data to be processed, which may be DRAM or DDR memory, and is usually 16G or larger in size, and is used to store data of the computing device 201 and / or the processing device 203 .

[0045] When the computing device 201 runs the neural network, it is generally necessary to first compile the neural network using the processing device 203 to obtain an executable file, which contains device information, that is, which device in the heterogeneous computer system the executable file needs to be executed on. After the executable file is assembled and linked, an executable program of the neural network can be obtained, and the executable program is stored in the storage device 204.

[0046] The processing device 203 can read the executable program from the storage location of the executable program and obtain multiple tasks of the program according to the executable program. These tasks are distributed to the computing device 201 for execution via the interface device 202, and finally obtain the operation result.

[0047] Figure 3a Shows Figure 2 The internal structure diagram of the processing core when the computing device 201 in FIG. 2 is a single-core device. The computing device 301 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 301 includes three modules: a control module 31 (also called a controller), an operation module 32 (also called an operator), and a storage module 33 (also called a memory).

[0048] The control module 31 is used to coordinate and control the operation of the operation module 32 and the storage module 33 to complete the deep learning task, and includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 is used to obtain instructions from the processing device 203, and the instruction decode unit 312 decodes the obtained instructions and sends the decoding results to the operation module 32 and the storage module 33 as control information.

[0049] The operation module 32 includes a vector operation unit 321 and a matrix operation unit 322. The vector operation unit 321 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformation; the matrix operation unit 322 is responsible for the core calculation of the deep learning algorithm, namely: matrix multiplication and convolution.

[0050] The storage module 33 is used to store or transfer relevant data, including a neuron RAM (NRAM) 331, a weight RAM (WRAM) 332, and a direct memory access module (DMA) 333. NRAM 331 is used to store input neurons, output neurons, and intermediate results after calculation; WRAM 332 is used to store the convolution kernel of the deep learning network, that is, the weight; DMA 333 is connected to DRAM 204 via bus 34, and is responsible for data transfer between the computing device 301 and DRAM 204. It should be noted that the NRAM and WRAM here can be two storage areas formed by dividing the same memory in the logical storage space, or they can be two independent memories, which are not specifically limited here.

[0051] Figure 3bThe figure shows a simplified schematic diagram of the internal structure of the computing device 201 as a multi-core. The multi-core computing device can be abstracted using a hierarchical hardware model. As shown in the figure, the multi-core computing device can be abstracted into three levels, namely, the chip level (Chip) 360, the processor cluster level (Cluster) 370 and the processor core level (Core) 380. The disclosed embodiment mainly involves the data transmission and computing unit part of the storage unit, so the drawings and descriptions briefly show and introduce the relevant computing structure, and omit other parts.

[0052] At the chip level, each chip contains local DDR storage, each processor chip serves as a computing and control unit, and each processor chip contains multiple multiprocessors as computing units.

[0053] At the processor cluster level, each multiprocessor includes multiple accelerator cores as control and computing units, and a shared memory SRAM as a storage unit.

[0054] At the processor core level, each accelerator core contains local storage and a local processing unit array. NFU refers to the Neuron Function Unit, which is used for convolution calculations. The structure of a single processor core can be similar to Figure 3a The structural diagram of the single-core computing device shown is not described in detail here.

[0055] In this multi-core computing device, the storage model includes the global memory of the board, SRAM (shared memory) on the Cluster, NRAM, WRAM and registers on the Core, etc. In order to obtain better performance, the data movement between the storage levels below the Card and the balance between memory access / computation can be explicitly controlled. SRAM is contained in the storage processing unit MPU (Memory Process Unit Core, referred to as MPU, or Mem Core). Core refers to the intelligent processing core (Intelligent Process Unit Core, referred to as IPU Core or Core) in the multi-core computing device. 1 IPU Core contains NRAM, WRAM, NFU, etc. Cluster refers to a processor cluster or a computing cluster. Usually a multi-core computing device contains several Clusters, and a Cluster contains 1 Mem Core + N IPU Cores.

[0056] Exemplary Software Stack

[0057] Figure 4 The schematic structure of the software stack 400 is shown. As shown in the figure, the software stack 400 includes a machine learning application 410, a deep learning framework 420, an IR graph 430, a machine execution instruction 440, and a machine learning hardware 450.

[0058] The machine learning application 410 provides corresponding machine learning algorithm models for different application scenarios. These algorithm models can be written in various high-level programming languages.

[0059] The deep learning framework 420 may include a variety of programming frameworks, such as Google's tensor flow graph TensorFlow, deep learning frameworks Caffe, Caffe2, MXNet, PyTorch, etc. For the machine learning algorithm models of various machine learning applications 410, the deep learning framework 420 is used as the main programming interface to parse these algorithm models to build corresponding machine learning / deep learning networks. During the compilation period, the deep learning framework 420 can provide a trained algorithm model. The deep learning framework 420 can convert the constructed network into a graph-level intermediate representation (IR), referred to as an IR graph or computational graph 430. The nodes in the IR graph represent operations or operators, which can be applied to a certain input and can generate outputs passed to other nodes. The edges in the IR graph represent the flow of data.

[0060] Various graph-level optimizations can be performed on the IR graph 430 during compilation. These optimizations may include, but are not limited to, constant folding, operator fusion, graph substitution, etc. The optimized IR graph can be compiled into an assembler-like file. This file is converted into machine language instructions, and these instructions are packaged into a format that can be used as a relocatable target program.

[0061] For the machine execution instructions 440, at runtime, the relocatable target program is just-in-time compiled into machine execution instructions 440 by the runtime compilation library. These instructions can be executed by the underlying machine learning hardware 450. The machine learning hardware 450 can be, for example, a chip in the exemplary hardware architecture described above.

[0062] The compiler is responsible for converting the hardware-independent IR graph 430 written in a high-level language into machine execution instructions 440 that can be efficiently executed on specific underlying hardware. During the conversion process, the compiler provides various optimization methods so that the optimized program can be executed more efficiently on the hardware during the execution phase.

[0063] Since compilers need to bridge the huge gap between the upper-level programming model and the underlying hardware, it is difficult to fully improve program performance by using a single layer of compiler abstraction. More and more compilers are using a multi-level abstraction approach. At this multi-level level of compiler abstraction, how to design and implement a general compilation technology route is an important challenge facing current optimization compilers.

[0064] Example Compilation Architecture

[0065] In an embodiment of the present disclosure, an optimizing compiler is provided, which divides the intermediate representation IR into three levels: IR0, TFUIR and IR1, and performs optimization processing with different focuses at different IR levels, so that hardware-independent programs can be converted into machine instructions that can be efficiently executed on specific hardware.

[0066] Figure 5 The hierarchical structure of the optimizing compiler 500 according to the embodiment of the present disclosure is exemplified.

[0067] As shown in the figure, the compiler 500 can receive trained algorithm models 510 provided by various deep learning frameworks. The compiler 500 converts these algorithm models 510 programmed based on the deep learning framework into a unified computational graph representation, that is, the first layer IR: IR0 520 through the first conversion (501).

[0068] The first conversion is mainly used to set the type of operator based on different frameworks, and to convert the current numerous deep learning frameworks to uniformly convert task models written in various languages ​​into a computational graph represented by IR0, so that subsequent model optimization and code generation can be implemented based on this unified computational graph. Since the first conversion focuses on processing the conversion of various front-end languages ​​to a unified form of computational graph, IR0 only reflects the information of the algorithm model itself, which is irrelevant to the specific hardware to be executed. IR0 can be an intermediate representation defined based on the MLIR (multi-level IR) compiler framework.

[0069] An example of IR0 is given below, which has the following textual form:

[0070] module{

[0071] func.foo(%arg0,%arg1,%arg2){

[0072] %0=mm.conv2d(%arg0,%arg1);

[0073] %1=mm.scale(%0,%arg2);

[0074] %2 = mm.relu(%1);

[0075] return %2;}

[0076] }

[0077] In the above example, the function foo is described, which contains three parameters arg0, arg1 and arg2, performs convolution operation (conv2d), scaling operation (scale) and activation operation (relu) in sequence, and finally returns the operation result. From this example, it can be seen that IR0 only reflects the operation logic at the software algorithm level, without hardware-related information.

[0078] Next, the compiler 500 converts IR0 520 into the second-level IR: TFUIR 530 through the second conversion (502). The second conversion mainly maps the operator expressed by IR0 to the operator expressed by TFU IR from the perspective of operator fusion, based on hardware resource information such as hardware fusion instructions and hardware-supported data copy behaviors. For example: the conv operator of IR0, after the second conversion, may correspond to the conv operator, fusion_conv operator or fold_conv operator expressed by TFU IR according to the scale parameter. The TFUIR operator 530 is an abstract representation based on the operator fusion solution. Based on the expression of TFU IR530, optimization operation means in the operator fusion process such as drainage and graph optimization are carried out for the TFU-IR operator.

[0079] Operator fusion is a typical computational graph optimization method that can calculate multiple operators together in a single kernel without saving the intermediate results back to the global memory, thereby effectively reducing input / output access in network model operations, that is, reducing IO memory access.

[0080] In the second conversion, automatic fusion technology is used to automatically search the computational graph expressed by IR0 based on data-driven and performance evaluation models, and finally obtain the optimal operator fusion method, and then perform IR conversion and optimization processing based on the fused operators.

[0081] Specifically, in the automatic fusion technology, various fusion feasibility and data splitting feasibility in the calculation graph represented by IR0 are traversed to obtain the operator list and scheduling strategy in the calculation graph with fusion possibility. In the scheduling strategy, the information that affects the operator fusion performance, such as the operator list of fusion possibility, data splitting method, pipeline method, weight retention, weight preloading and other information, is uniformly defined as control data, thereby converting the search problem of the operator fusion method with the best performance into the search problem of the optimal control data. In this way, as long as the input / output is connected on the calculation graph, the operator is allowed to be fused. Furthermore, in the automatic fusion process, the performance evaluation function of the fusion operator can be given according to the control data of the fusion operator for specific hardware information. Based on the performance evaluation function, the fusion benefit of the fusion operator can be calculated, so as to select the optimal fusion method according to the fusion benefit. In this way, the better operator fusion method under various networks can be automatically found.

[0082] From the second conversion process, it can be seen that the converted TFUIR not only reflects the logical operation information of the algorithm model, but also reflects the underlying hardware information because specific hardware information needs to be considered in the automatic fusion process. The granularity of the hardware information depends on the automatic fusion process. In other words, TFU IR introduces some hardware information based on operator fusion. Based on the operator fusion method, the calculation graph expressed by IR0 is converted to the calculation graph expressed by TFUIR. During the execution of the task by the artificial intelligence processor, the intermediate results are not saved back to the global memory, thereby effectively reducing the input / output access in the network model operation, that is, reducing IO memory access, and achieving the purpose of reducing IO bottlenecks.

[0083] An example of a TFUIR is given below, and its text form is as follows:

[0084] module{

[0085] func.foo(%arg0,%arg1,%arg2){

[0086] %0=tfu.fusion_conv(%arg0,%arg1,%arg2);

[0087] return %0;}

[0088] }

[0089] It can be seen that this TFUIR example continues the previous IR0 example, fusing the three operators of convolution operation (conv2d), scaling operation (scale) and activation operation (relu) into a fused convolution operator (fusion_conv).

[0090] Next, the compiler 500 converts TFUIR 530 into a third-level IR: IR1 540 through a third conversion (503). The third conversion mainly optimizes IR from the perspective of hardware execution, and IR1 540 is a completely hardware-related abstract representation. IR1 contains rich hardware information, and the underlying hardware can perform tasks corresponding to the neural network model based on this.

[0091] In the third conversion, based on the information of the underlying specific execution hardware, the IR is optimized, such as adjusting the splitting scheme in the automatic fusion scheme determined earlier, and then performing code generation (Codegen) to obtain an abstract representation that can be provided to the runtime compilation library for further compilation and ultimately generating machine-recognizable binary instructions. The TFU IR expression is converted to the IR1 expression to clarify which hardware information appears in the TFU IR expression and which hardware information appears in the IR1 expression, similar to the form of stacking building blocks. Since the positioning of the software stack itself is to make full use of hardware resources, when locating software code problems, it can be more convenient to locate the problem, and ultimately achieve a more efficient solution to the problem.

[0092] Therefore, the disclosed embodiment provides a compiler including three IR levels, wherein the middle TFUIR level is specifically introduced for the important calculation graph optimization means of operator fusion. The TFUIR level is based on the operator fusion scheme, and performs more delicate calculation graph optimization processing with the fusion operator as the granularity. More specifically, by converting the first level IR0 to the middle TFUIR level, the hardware characteristics can be displayed according to the first granularity (such as the granularity of operator fusion), and the graph level optimization is performed based on this granularity, and the graph level optimization belongs to the optimization with hardware information, which is different from the graph level optimization in the IR0 stage. In addition, the TFU IR expression does not contain all the hardware information, which is more friendly to programmers and more efficient in locating code problems. Then, after the code generation (Codegen) stage, the TFUIR level is converted to the third level IR1, which can display the hardware characteristics in a more fine-grained manner. Therefore, by introducing hardware information layer by layer in this way, the high-level, hardware-independent calculation graph can be finally converted into a low-level, completely hardware-related calculation graph, and provided to the runtime compilation library to generate machine-recognized binary instructions. In summary, the compilation scheme of the disclosed embodiment provides an effective performance optimization solution for accelerating the processing of neural network models on a hardware execution platform.

[0093] The conversion process between each IR level will be described below. The first conversion from the algorithm model provided by the deep learning framework to IR0 mainly focuses on the differences between various front-end languages, and various known conversion methods can be used. This disclosure embodiment does not limit this and does not expand the description.

[0094] Conversion from IR0 to TFUIR

[0095] As mentioned above, in the second conversion from IR0 to TFUIR, the computational graph is converted and expressed mainly from the perspective of operator fusion.

[0096] Figure 6FIG. 1 is an exemplary flow chart showing a second conversion process from IR0 to TFUIR implemented by a processing device according to an embodiment of the present disclosure. The processing device may be, for example, Figure 2 The processing device 203.

[0097] As shown in the figure, in step 610, the automatic fusion technology is first used to automatically search for the optimal operator fusion method from the calculation graph represented by IR0. For the content of the automatic fusion technology, reference can be made to the Chinese invention patent application CN202211372965.4 entitled "Automatic Fusion Method of Operators in Calculation Graphs and Related Products". Since the focus of the disclosed embodiment is not on this, this article will not be described in detail, and only the parts related to the disclosed embodiment will be described.

[0098] In the automatic fusion technology, various fusion feasibility and data splitting feasibility in the calculation graph are traversed, and various scheduling strategies (for example, pipeline mode, whether to enable weight retention, whether to enable weight preloading, etc.) are considered, and the optimal operator fusion method is selected through the performance evaluation function. Therefore, after the automatic fusion process, the preferred operator fusion method will be provided.

[0099] The operator fusion method can include the following information: the operator sequence that can be fused and has the best performance; descriptive files or structured information about the fusion level; and data splitting information.

[0100] The operator sequence can be a sequence of operators directly extracted from the computational graph represented by IR0 based on the result of automatic fusion processing. For example, the convolution operator (conv2d), scaling operator (scale), and activation operator (relu) given in the previous example. This operator sequence only provides a list of operators that can be fused one by one, without reflecting further fusion information, such as fusion level, etc.

[0101] The fusion level can be represented by a descriptive file or structured information of the fusion level. Since operator fusion aims to reduce IO memory access, the fusion level can be determined according to the storage level of the hardware. Figure 3bIt can be seen from the described hardware architecture that the storage level can include three layers: the NRAM, WRAM and other memories on the IPU of each processor core in the processor core level 380 are abstracted as the L1 storage level, the SRAM memory on the MPU in the processor cluster level 370 is abstracted as the L2 storage level, and the global memory of the chip level 360 is the third storage level. The fusion-optimized computational graph can also be divided into graphs and subgraphs accordingly. When the fusion-optimized network model is executed in these hardware environments, the input and output of the graph are the global memory of the chip level 360, such as DDR, and the intermediate results reside in the L2 storage level. One graph corresponds to one fusion operator. A graph can include several subgraphs (fusion sub-operators). The input and output of each subgraph are the L2 storage level, and the intermediate results reside in the L1 storage level. A subgraph can include several operators. The input and output of each operator are the L1 storage level.

[0102] In some embodiments, the fusion level information of the fusion operator can be represented by a structured text description of a graph and a subgraph. For example, an example of a structured text description of a fusion level is given below.

[0103] Graph1{

[0104] subgraph1{

[0105] Conv1

[0106] Add1

[0107] Relu1

[0108] }

[0109] subgraph2{

[0110] conv2

[0111] scale2

[0112] bn1

[0113] }

[0114] Graph2{ ...

[0116] }

[0117] In the above example, two graphs are shown: graph1 and graph2, whose input and output are both chip-level global memory (such as DDR). That is, graph1 and graph2 both fetch data from DDR, perform internal operations, and then store the final results back to DDR. The above example further shows that graph1 includes two subgraphs: subgraph1 and subgraph2, whose input and output are both L2 storage levels. That is, subgraph1 and subgraph2 both fetch data from the shared cache SRAM on the MPU, perform internal operations, and then store the final results back to SRAM. Each subgraph includes several operators, and the input and output of these operators are L1 storage levels, that is, NRAM, WRAM, etc. on the IPU.

[0118] In other embodiments, the fusion level information can be directly described at the IR0 level in the form of operator calls. In these embodiments, each of the above subgraphs can be considered as an operator call, for example, replaced by callop. Further, the operators included in the corresponding callop can be defined.

[0119] The data splitting information is used to represent the data splitting scheme in the operator fusion scheme. Specifically, the data splitting scheme includes parallel and serial data splitting between various levels of storage, and each splitting involves the splitting dimension and splitting block size of the data.

[0120] For example, in Figure 3b In the environment shown in the figure where there are two storage levels on the chip, the data splitting scheme of the fusion operator includes a parallel splitting method of multiple L2 storage levels of the fusion operator, a serial splitting method of a single L2 storage level of the fusion operator, a parallel splitting method of multiple L1 storage levels of each fusion sub-operator, and a serial splitting method of a single L1 storage level of each fusion sub-operator.

[0121] The parallel splitting method of multiple L2 storage levels of the fusion operator involves the hardware information of the computing device running the computation graph, such as Figure 3b In the hardware environment shown, the splitting between clusters is involved at the processor cluster level.

[0122] The serial splitting method of a single L2 storage level of the fusion operator involves the splitting of the input data, and the parameters mainly include the dimensions of the splitting. For example, when the HW (height and width) dimension of the input feature map (input data) does not exceed the first threshold, such as when the entire feature map can be stored on a single L2 storage level, it can be considered as a "small image mode" at this time, and its splitting dimension is N (batch) dimension. For another example, when the HW dimension of the input feature map exceeds the second threshold, it can be considered as a "large image mode" at this time, and its splitting dimensions may include three dimensions: N, H, and W. It can be understood that the parameter setting of the serial splitting method is only exemplary, and those skilled in the art can also construct more splitting methods, such as "medium image mode", "super large image mode", etc., and the disclosed embodiments are not limited in this regard. In addition, the parameters of the serial splitting method can also include the specific splitting block size of each splitting dimension.

[0123] The parallel splitting mode of multiple L1 storage levels of each fusion sub-operator is mainly related to the dimension of the input feature map. These dimensions may include, for example, the N (batch) dimension, H (height) dimension, W (width) dimension, and C (channel) dimension of the input feature map. The specific split value of each dimension may be determined, which depends on, for example, the number of computing components in the hardware implementation.

[0124] Similarly, the serial splitting method of a single L1 storage level of each fusion sub-operator also involves the splitting of the input feature map, and the parameters mainly include the splitting dimension and the splitting block size.

[0125] In summary, data splitting information can include the following aspects:

[0126] a. Whether to split the data in series or in parallel can be indicated by the value of the parameter by-factor-or-by-part, for example, 1 indicates series, 0 indicates parallel, and vice versa. The factor of the series split indicates the size of each part, which usually depends on the processing power of the computing hardware. The part of the parallel split indicates the number of parts, which usually depends on the number of processor cores in the parallel processing hardware.

[0127] b. Split dimension: This parameter indicates the dimension on which the data is split. It can be any dimension such as N, H, W, C, etc.

[0128] c. Split coefficient: This parameter indicates the value of the factor or part above, and has corresponding meanings depending on serial or parallel splitting.

[0129] d. Bind-axis: This parameter is used to indicate the mapping or binding relationship between the parallel splitting scheme and the hardware. It can be understood that the parallel splitting scheme can be split in parallel at different levels, such as the L2 level or the L1 level. Accordingly, these parallel splitting schemes require the support of parallel processing hardware at different levels. The binding axis is used to indicate the corresponding parallel processing hardware dimension. For example, assuming that the parallel split part=4, the splitting dimension is N dimension, and the binding axis is dim-y, it means that the N dimension is split into 4 parts, each corresponding to a dim-y.

[0130] The dim-x, dim-y, and dim-z of the binding axis can be defined as hierarchical dimensions corresponding to different hardware (computing units) according to different hardware architectures. In one example, tasks are issued in units of jobs. Usually, jobs represent independent tasks that have no data dependencies on each other, such as processing different data blocks. Each job has its own instruction file. Each job can have different hardware units on different execution hardware. For example, on execution hardware A, a job is issued in units of processor clusters (see Figure 3b On the same execution hardware, a job can also have different job types, and the number of hardware units contained in different job types can be different. For example, job types can include U1 (1 processor cluster), U2 (2 processor clusters), U4 (4 processor clusters), and so on. At this time, dim-x can represent the task type of the currently issued job. More specifically, dim-x can indicate the number of executable processing cores in the current job. For example, Figure 3b For example, if a processor cluster includes 4 processing cores, then the corresponding dim-x value for U1 type jobs is 4, and the corresponding dim-x value for U2 type jobs is 8, and so on. dim-y can represent the number of jobs. dim-z can represent higher dimensions, such as the number of kernels that need to be started.

[0131] Optionally or additionally, the operator fusion method may also include some configuration information, such as whether to enable pipelining, whether to enable weight retention, and so on.

[0132] The IO / computation parallel pipelining mode of a fusion operator or a fusion sub-operator mainly involves whether to enable pipelining and how many levels of pipelining to use when pipelining is enabled. For example, the configuration information can be set to not enable pipelining, so that IO access and computation are not executed in parallel. For another example, the configuration information can be set to enable pipelining, and for some modes, the LCS three-level pipelining is enabled, and for other modes, the LLCS five-level pipelining is enabled, where L stands for load, C stands for compute, and S stands for store.

[0133] Usually in neural network calculations, the weights of each network layer are fixed, so the resident data are usually weights. In this regard, data residency is sometimes also called weight residency. Through data residency, the time for repeated data loading can be effectively reduced, the number of data reuses can be increased, and the overall calculation can be accelerated. The configuration information can be set to non-resident or resident. When choosing to reside, it can further include which model data (such as the weights of which operators) reside on the L1 storage level and which model data reside on the L2 storage level.

[0134] continue Figure 6 Then, in step 620, based on the information of the optimal operator fusion method determined in the previous step, the computation graph at the IR0 level is converted to the computation graph at the TFUIR level. During the conversion process, various basic processing is performed on the operators in the computation graph represented by IR0 based on the information of the optimal operator fusion method determined in the previous step. Optionally or additionally, some optimization processing can also be performed.

[0135] Basic processing may include, but is not limited to, replacement of instruction level, addition of memory scope, addition of memory operation, etc.

[0136] The replacement at the instruction level includes replacing multiple operator instructions separated in the calculation graph of the IR0 level with a single operator instruction according to the information of the operator fusion method, such as the descriptive file of the previous fusion level.

[0137] Specifically, in some implementations, based on the fused operator instructions supported by the hardware that will execute the computation graph, the corresponding multiple separate operator instructions are replaced with a single fused operator instruction. For example: if the hardware supports convolution fusion instructions (for example, the convolution operator and the subsequent addition operator and relu operator are fused in one convolution instruction, which is called a convolution fusion instruction), and the determined operator fusion method indicates that the three operators are fused together, then the three operator instructions in the computation graph at the IR0 level can be replaced with a single convolution fusion instruction. In some implementations, certain models of hardware can support multiple fused operator instructions, such as deep convolution fusion instructions, fused deconvolution instructions, folded convolution post-fusion instructions, instructions for merging several vector operations, and the like.

[0138] In other implementations, if there is no fusion operator instruction corresponding to the fusion operator, the multiple operator instructions to be fused can be replaced with an operator call instruction callop, and the operators included in the corresponding callop are defined in the file. In some implementations, according to the structured text of the fusion level, one of the subgraphs is replaced with a callop, and then which operators the callop contains are defined.

[0139] Adding memory scopes involves adding attributes to operators based on the operator definition to describe the memory scopes where the operator's input and output data are stored. At this stage, the granularity of the scope can be the memory scope divided according to the data content to be saved. For example, based on Figure 3b Hardware architecture, assuming that the convolution operator itself requires that the input neurons be stored on NRAM, the input convolution kernels be stored on WRAM, and the output neurons be stored on NRAM, then add these memory scopes to the convolution operator according to the requirements of the convolution operator.

[0140] Furthermore, information such as data type and data shape can also be described. Data type refers to whether the data is integer or floating point type. These data types may include but are not limited to int8, int16, int32, fp16, fp32, tf32, etc. The data shape at this level refers to the complete shape of the data, which has not yet been split. For example, the complete shape of the input neuron of a convolution operator is 1×24×24×64, the data type is uint8, and it is stored on NRAM. In some implementations, memref can be used to describe the above information, thereby indicating the layout of the tensor data in the memory space.

[0141] The addition of memory operations includes: adding memory operations for moving data between different storage levels based on the information of the fusion level. As mentioned earlier, the input and output of the graph are chip-level global memories, such as DDR. One graph corresponds to a fusion operator or kernel. The input and output of each subgraph are the L2 storage level, and the intermediate results reside on the L1 storage level. Therefore, between different graphs, it is necessary to add memory operations that interact with global memory (such as DDR); and between different subgraphs, it is necessary to add memory operations that interact with the L2 storage level (for example, shared storage SRAM). For example, in a subgraph Figure 1 A copy instruction from the L1 storage level to the L2 storage level is inserted after the output of the L1 storage level to store the output data back to the L2 storage level; and the following sub- Figure 2 A copy instruction from the L2 storage level to the L1 storage level is inserted before the input of to obtain the input data from the L2 storage level. Figure 1 A copy instruction from the L2 storage level to the chip-level global memory is inserted after the output of Figure 2 A copy instruction from chip-level global memory to L2 storage hierarchy is inserted before the input of

[0142] Furthermore, these added memory operations can further limit the memory source and / or destination of their operations according to the memory scope previously added to the operator. For example, for a fused convolution operator (assuming it corresponds to a subgraph), before its input, a memory copy instruction from the shared storage SRAM to the NRAM can be inserted for the input neuron data; and for the convolution kernel, a memory copy instruction from the shared storage SRAM to the WRAM can be inserted.

[0143] Optionally or additionally, during the conversion of the IR0-level computation graph to the TFUIR-level computation graph, some optimization processes may be performed, including but not limited to: Co dimension splitting, alignment optimization, setting weight retention, weight reuse between operators, reusing activation tables, marking the entrance of shape derivation, etc.

[0144] After the above conversion process, in step 630, the converted TFUIR-level calculation graph is output for subsequent processing.

[0145] From the above conversion process, it can be seen that the calculation graph at the IR0 level is completely unrelated to the hardware and simply describes the calculation at the network level from the software level. In the calculation graph at the TFUIR level, operator fusion is introduced, and hardware-related information is added from the perspective of operator fusion, such as fusion instructions supported by the hardware, memory operations between hardware storage levels corresponding to the fusion operator, etc. Furthermore, in this conversion process, optimization processing can also be performed on a fusion operator basis, such as vector merging, memory operation merging, etc., thereby compiling and optimizing the calculation graph on a fusion operator basis.

[0146] Conversion from TFUIR to IR1

[0147] After converting from IR0 to TFUIR, it is necessary to further convert TFUIR to IR1 through the third conversion. During this conversion process, shape deduction, memory allocation, and then back-end code generation are required. The text of the code generation is also IR1, so IR1 is a completely hardware-related abstract representation.

[0148] In this article, shape refers to the dimensional information of data, including the number of dimensions, the size of each dimension, etc. In intelligent computing processing, the shape of data is reflected in the memory access address of the data. Therefore, in this context, the memory access mode refers to the memory access address for accessing each data block after a certain dimension of the data is split or divided according to the dimension splitting scheme.

[0149] In this article, shape derivation, memory access mode derivation or memory access address derivation means that when a dimension of the data is split or divided, it is necessary to derive the memory access address of each time / each data block of the associated operator in the calculation graph for the input data and / or output data after the split according to the split data blocks. The associated operator here refers to the operator in the calculation graph that needs to directly or indirectly access the data, for example, it is necessary to directly or indirectly use the data to perform calculations, or it is necessary to directly or indirectly write the calculation results to the data. In addition, in the following description, unless otherwise specified, the data involved are all tensor data.

[0150] Figure 7 FIG. 1 is an exemplary flow chart showing a third conversion process from TFUIR to IR1 implemented by a processing device according to an embodiment of the present disclosure. The processing device may be, for example, Figure 2 The processing device 203.

[0151] As shown in the figure, in step 710, the memory access mode of the operator in the TFUIR graph is derived based on the data splitting information. The data splitting information here refers to the data splitting information determined according to the automatic fusion technology in the previous second conversion process from IR0 to TFUIR, which can be referred to the previous description and will not be repeated here.

[0152] In some implementations, data splitting information usually refers to splitting information of output data. Therefore, when performing memory access mode derivation, the operator of the output data (that is, the end operator in the computation graph) is used as the entry operator, that is, the starting point of derivation, to perform memory access mode derivation. When the end operator is selected as the entry operator, the derivation direction of the memory access mode is reversed, that is, the memory access mode of the input data is reversely deduced from the memory access mode of the output data in the opposite direction of the data flow in the computation graph.

[0153] In this memory access mode derivation, the entry operator data is split into one or more data blocks according to the dimension splitting scheme of the entry operator data. Then, based on the memory access mode of the entry operator data, the memory access mode of the data associated with the entry operator data in the calculation graph is derived. When the entry operator data is split according to the dimension splitting scheme, its corresponding memory access mode can also be determined. The memory access mode includes the memory access address of each data block into which the data set is split.

[0154] In some embodiments, memory access mode derivation may include: taking the end operator as the entry operator, starting from the entry operator, traversing each layer of operators in the computational graph from back to front, and performing reverse shape derivation to obtain the memory access addresses of one or more input data blocks and output data blocks that each layer of operators needs to split to meet the dimensional splitting scheme of the entry operator.

[0155] The inventors discovered through studying deep learning algorithms that although the network structure of a neural network is complex, its basic components are relatively stable, mainly including operators such as convolution (conv), pooling (pool), padding (pad), and vector operations. These basic operators take up most of the time for the overall calculation. Further analysis of these basic operators revealed that they can be decomposed into some basic operations. In this article, these basic operations are referred to as atomic operations. Note that since the processing of data in intelligent computing involves division by dimension, the atomic operations here are also determined in units of dimension. In the disclosed embodiments, by analyzing the memory access mode of the basic operators, memory access mode derivation formulas for several kinds of atomic operations can be abstracted. Thus, the memory access mode derivation of various operator data can be performed based on these memory access mode derivation formulas.

[0156] Specifically, in the traversal process of the above-mentioned computational graph, for each layer of operators in the computational graph, reverse shape deduction may include: decomposing the operation of the operator into several atomic operations by dimension; and based on the input-output memory access address relationship of the atomic operation, that is, the memory access mode deduction formula, reversely deriving the memory access address of the input data block required for the operator to calculate an output data block.

[0157] In some embodiments, the atomic operation may include one or more of the following: a forward sliding window operation, such as used in operators such as convolution (conv) and pooling (pool), usually in the width (W) and height (H) dimensions; a forward padding operation, such as used in padding operations of operators such as convolution and pooling, or in an independent padding (pad) operator (pad size is a negative number); a reverse sliding window operation, such as used in operators such as deconv and unpooling (unpool), usually also in the width (W) and height (H) dimensions; a reverse padding operation, such as used in deconv, or in an independent padding operator (pad size is a negative number); a vector operation, used for various vector operations in operators, such as addition, subtraction, multiplication, activation calculations, etc.; a broadcast operation, used for broadcast copy operations, similar to numpy's broadcast operations on a certain dimension, such as in a cycle add; and an aggregation operation (reduce), used to aggregate data, such as the accumulation operation on the Ci dimension in the convolution (conv) operator, reduce calculations, etc.

[0158] Based on the above abstracted atomic operations, we can derive the memory access mode derivation formula according to their specific operation mode, that is, the input and output memory access address relationship of the atomic operation. In this way, when traversing the computation graph, we can decompose the operation of the current operator into several atomic operations by dimension, and then deduce them according to the memory access mode derivation formula of the atomic operation to obtain the memory access mode of the current operator's data.

[0159] Next, in step 720, memory allocation is performed based on the derived memory access mode. The derived memory access mode indicates the size of each split block of the input and output data of each operator, that is, the memory size required to be occupied by each split block. Based on this information, memory can be allocated for the split block. In a specific implementation, memory allocation is also performed on the corresponding memory scope based on the memory scope added to the operator during the conversion to TFUIR. The memory allocation information may include, for example, the memory scope, the starting position (first address) in the memory scope, and the allocated space size (split block size).

[0160] Finally, in step 730, according to the result of memory allocation, the code generation process is performed based on TFUIR, and the compilation and optimization are performed to obtain a binary file recognizable by the intelligent processor.

[0161] The above describes an exemplary implementation flow of the third conversion process. From the above conversion process, it can be seen that the calculation graph at the IR1 level is completely related to the hardware, and the execution process of the calculation graph is described from the hardware instruction level, such as the specific execution implementation of the operator, the pipeline of instructions, etc. In addition, the implementation of the operator is based on the calculation graph at the TFUIR level, that is, the granularity is the fusion operator.

[0162] In summary, through the above multiple conversion processes, the completely hardware-independent IR representation IR0 can be converted into a completely hardware-dependent IR representation IR1 through the TFUIR representation characterized by the fusion operator. Then, the IR1 can be handed over to the runtime compilation library to generate binary code executed on the corresponding intelligent processor.

[0163] Compilation optimization with variable input

[0164] Normally, the input data size of the neural network model is fixed, that is, the data shape information known at compile time is consistent with that at compile time. In this case, the data shape of the instruction to be generated is known at compile time, so the opcode and address field in the instruction can be determined at compile time. In order to make full use of hardware resources, various performance optimizations are provided at compile time, including pipeline arrangement, etc., which are processed by the runtime compilation library to generate binary code that can be directly recognized on the intelligent computing hardware, and then handed over to the driver for parsing and execution on the intelligent computing hardware. Therefore, the optimization of the fusion operator mentioned above is also performed at compile time.

[0165] In addition, when generating instructions at compile time, the generation logic is to expand all loops and describe the dependencies between instructions through a tree structure. In the actual instruction generation process, the tree structure is fully expanded. The cost of this processing method is that the number of instructions is very large, the storage burden is very large, and it is not conducive to deployment on mobile or embedded devices with limited storage capacity.

[0166] However, in some computing scenarios, the input data size of the neural network model is not fixed, that is, variable. This form of variable input often leads to poor performance in the neural network reasoning stage, because it can generally be considered that the more specific the given input range, the better the performance optimization effect. If a set of corresponding instructions is generated for each input scale (for example, for different resolutions in image processing), the number of instructions will increase linearly, increasing the processing and storage burden. One known method is to divide the input range into intervals and perform reasoning calculations according to the upper limit of the interval. However, if each reasoning is calculated according to the upper limit, it will cause a waste of computing resources. Furthermore, since the input shape is unknown, the number of loops will also be unknown when loops are involved, so that the loop cannot be expanded in the original way. Therefore, for the case where the input shape is variable, it is necessary to provide further optimization solutions to solve one or more of the aforementioned problems.

[0167] In the case of variable input shape, the shape size of the input data during actual execution is unknown during compilation, that is, the input data shape of the instruction to be generated is unknown during compilation, so that accurate parameters cannot be filled in the instruction domain, for example, the instruction domain parameters can only be filled according to the upper limit. In the disclosed embodiment, by generating instructions based on the input shape data obtained at runtime, more accurate instruction domain parameters can be generated, thereby supporting better instruction execution performance. In other words, some steps of instruction generation can be adjusted to runtime. More specifically, the pipeline arrangement of instructions can be implemented in the code generation stage (codegen) of the compilation period, and accurate instruction parameters are generated according to the exact input data shape obtained at runtime, and the corresponding instruction domain is filled, thereby generating instructions that can be executed on intelligent hardware. Furthermore, in the code generation stage (codegen) of the compilation period, loop expansion is no longer performed for the loop structure, but loop instructions are used instead, such as using the branch instruction supported by the hardware, which is encapsulated into a way to support for loops such as for loops, and the instruction pipeline is arranged on this basis.

[0168] Figure 8 The following is a flowchart of an exemplary method for generating instructions at compile time by a computing device according to an embodiment of the present disclosure. The computing device may be, for example, Figure 2 The computing device 201. The process of generating this instruction is similar to the process of generating the instruction of the compiler, except that the input data shape is the exact information instead of the upper limit of the interval.

[0169] As shown in the figure, in step 810, according to the computation graph expressed by TFU IR, the shape of the operators in the computation graph is forward deduced to obtain complete shape information.

[0170] In the forward derivation, the forward derivation logic of each operator can be called to obtain the complete shape information of the input data and output data of each operator, that is, the shape without any splitting. In this forward derivation, the accurate shape information of the input data obtained at runtime is used.

[0171] Specifically, the starting operator in the computation graph is generally used as the entry operator for forward deduction. The starting operator refers to an operator in the computation graph that has no preceding operator, which is usually the first operator in the computation graph. When the starting operator is selected as the entry operator, the shape deduction direction is forward, that is, the shape of the output data is forward deduced from the shape of the input data along the direction of data flow in the computation graph.

[0172] The forward derivation logic of an operator is associated with the mathematical operation logic of the operator, and can be similarly determined based on the reverse derivation logic described above, which will not be repeated here.

[0173] Next, in step 820, based on the data splitting information and the complete shape information obtained by forward deduction, the memory access mode of the operator in the computational graph expressed by TFUIR is derived. In some implementations, the data splitting information here may be the data splitting information determined at the front end, for example: the data splitting information determined according to the automatic fusion technology in the second conversion process from IR0 to TFUIR. In other implementations, the data splitting information here may be the data splitting information that is readjusted according to the determined shape information of the input data obtained at runtime, and this implementation will be described in detail later.

[0174] The process of memory access mode derivation is combined with the previous Figure 7 The described step 710 is similar, except that the data shape information used is the determined data shape information obtained during runtime, so it will not be described in detail here.

[0175] Through the above memory access mode derivation, we can obtain the memory access mode of the split blocks of input and output data of each operator in the graph.

[0176] Next, in step 830, based on the derived memory access mode, address domain related parameters of the instructions involved in the loop in the code logic of the computation graph are determined. The memory access mode indicates the memory access address of the split blocks of input and output data of each operator, so the relevant parameters of the instruction can be determined based on this, such as the size of the split block processed by the instruction, the size of each dimension of the HWC, and so on.

[0177] Next, in step 840, the instructions in the loop are pipelined based on the instruction stream supported by the hardware.

[0178] In pipeline arrangement, instructions may be pipelined according to the configuration information of the operator fusion method determined in the previous automatic fusion process, such as whether to enable pipeline and how many levels of pipeline to use when enabling pipeline.

[0179] When arranging the pipeline, you can divide the stages according to the instruction stream types supported by the hardware. Instruction stream types can include, but are not limited to, IO streams (memory access instructions involving off-chip storage), move streams (memory access instructions between on-chip storage), compute streams (computational instructions), and so on. When dividing the stages, you can use the TFUIR generated by the front end. Then write the loop in stages, and insert synchronization instructions, such as barrier instructions or sync instructions, when synchronization is needed.

[0180] Finally, in step 850, the computation graph is compiled according to address domain related parameters of the instructions involved in the loop and the result of the instruction pipeline arrangement in the loop.

[0181] Optionally or additionally, in some embodiments, the data splitting information determined by the front end can be adjusted as appropriate so that the currently running instance can better adapt to the hardware, fully utilize the computing power of the hardware, improve processing efficiency, etc.

[0182] Fig. 9 The flowchart of an exemplary method for generating instructions at compile time implemented by a computing device according to an embodiment of the present disclosure is shown. In this embodiment, it can be determined whether the split information needs to be adjusted based on conditional judgment.

[0183] As shown in the figure, Figure 8 Similarly, first in step 910, according to the computational graph expressed by the TFU IR, the shape of the operators in the graph is forward deduced to obtain complete shape information.

[0184] Next, in step 920, based on the derived shape information, it is determined whether the data splitting information needs to be adjusted.

[0185] In some embodiments, it may be determined whether the data splitting information needs to be modified based on the splitting efficiency. The splitting efficiency may be evaluated from multiple perspectives. In some implementations, the splitting efficiency may be defined by the utilization of the processing core. For example, the utilization of the processing core IPU efficiency It can be calculated according to the following formula:

[0186]

[0187] Among them, N o , H o , W o Respectively represent the original size of the output data in N, H, and W dimensions; Hipu-factor and W ipu-factor Respectively represent the split block size in H and W dimensions, IPU part Indicates the number of processing cores for parallel processing, Num cluster Indicates the number of computing clusters, Cluster part Indicates the number of computational clusters for parallel processing.

[0188] The information in the numerator in the above formula represents the original data volume before splitting, which can be determined, for example, according to the shape information derived in step 910. The information in the denominator in the above formula represents the data volume to be processed after splitting according to the splitting scheme indicated by the front-end data splitting information.

[0189] If the splitting efficiency is lower than the specified threshold, it can be determined that the splitting scheme needs to be adjusted. At this time, the splitting scheme is adjusted to update the current data splitting information. It can be understood that the adjustment of the splitting scheme is based on the data splitting information provided by the front end as a reference, and the data splitting scheme is adjusted according to the current actual and determined data shape information. For example, for scenarios where data is variable, the data splitting information provided by the front end may be a splitting scheme determined based on the upper bound of a certain interval, and the actual data shape may be at the lower bound of the interval. Therefore, the original data splitting scheme will have more redundancy, resulting in low utilization of the processing core. In this regard, based on the actual data shape, the data splitting scheme is re-determined to improve the utilization of the processing core. The process of determining the data splitting scheme can be similar to that in the automatic fusion process, except that the data shape information used at this time is different from before.

[0190] For example, assuming that the information known by the front-end compile time is that the N dimension is variable in the range of 1 to 4, the splitting information generated during the compilation period is for the case where the upper limit of the N dimension is 4, and 4 batches of data are split into 4 computing clusters for execution. However, during actual runtime, the real batch of input data N=1. At this time, if the splitting scheme of the front-end compile time is still followed, the data of this batch will be allocated to 1 computing cluster for execution, and the remaining computing clusters will be idle. Obviously, this splitting method leads to low hardware utilization. Therefore, according to the disclosed embodiment, during runtime, the data splitting scheme can be readjusted based on the determined data shape information obtained, for example, 1 batch of data can be split into 4 computing clusters according to the H and W dimensions for execution, thereby improving computing efficiency.

[0191] Optionally or additionally, in some embodiments, whether the data splitting information needs to be modified is determined based on the current hardware operating mode.

[0192] One indicator to measure the hardware's processing performance for a model is throughput, which is the amount of input data that the hardware can process per second when running a model. Another indicator is latency, which is the average length of time it takes to process data. Throughput and latency are inversely proportional. Therefore, the hardware operating modes can be divided into Throughput mode and Latency mode. The Throughput mode mainly assesses the processing throughput in large batches of data. The Latency mode mainly assesses the processing latency in small batches of data. Based on the difference between the two modes, in some embodiments of the present disclosure, it may be considered whether the data splitting information needs to be modified only when the current hardware operating mode is Latency mode.

[0193] Next, based on the updated data splitting information, the memory access mode of the operators in the graph is re-derived. The memory access mode derivation process is combined with the previous Figure 8 The step 820 described is similar and will not be described in detail here. Through the above memory access mode derivation, the memory access mode of the split blocks of input and output data of each operator in the graph can be obtained. If it is determined in step 920 that there is no need to adjust the data splitting information, jump directly to step 930 to derive the memory access mode of the operators in the computational graph based on the tensor data splitting information that meets the requirements.

[0194] Next, in step 940, based on the derived memory access mode, address domain related parameters of the instructions involved in the loop in the code logic of the computation graph are determined; in step 950, the instructions are pipelined based on the instruction stream supported by the intelligent processor; and in step 960, the computation graph is compiled according to the address domain related parameters of the instructions involved in the loop and the result of the pipeline arrangement of the instructions in the loop to obtain a binary file recognizable by the intelligent processor. Steps 940 to 960 are similar to Figure 8 Steps 830 to 850 are similar and will not be described in detail here.

[0195] Therefore, the above describes the runtime solution of the computational graph provided by the disclosed embodiment. For scenarios with variable input shapes, the real and definite input data shape information can be used to adjust the instructions at runtime, so that the real scenario can be more adaptable to the hardware, fully utilize the computing power of the hardware, and improve processing efficiency.

[0196] The present disclosure also provides a processing device, which can be used to perform compilation on a computational graph, including: a processor, which is configured to execute program instructions; and a memory, which is configured to store the program instructions, so that when the program instructions are loaded and executed by the processor, the processor executes the compilation method described in the embodiment of the present disclosure.

[0197] In the disclosed embodiment, a computer-readable storage medium is also provided, in which program instructions are stored, and when the program instructions are loaded and executed by a processor, the processor executes the method for compiling the computational graph described in the disclosed embodiment. In the disclosed embodiment, a computer program product is also provided, including a computer program or instruction, which, when executed by a processor, implements the method for compiling the computational graph described in the disclosed embodiment.

[0198] The present disclosure also provides a chip, which may include the aforementioned processing device. Furthermore, the present disclosure also provides a board, which may include the aforementioned chip.

[0199] According to different application scenarios, the electronic equipment or device disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, driving recorders, navigators, sensors, cameras, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, automatic driving terminals, transportation, household appliances, and / or medical equipment. The transportation includes airplanes, ships and / or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lights, gas stoves, and range hoods; the medical equipment includes magnetic resonance imaging, ultrasound machines and / or electrocardiographs. The electronic equipment or device disclosed herein may also be applied to the Internet, IoT, data centers, energy, transportation, public administration, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and medical fields. Further, the electronic equipment or device disclosed herein may also be used in cloud, edge, and terminal applications related to artificial intelligence, big data, and / or cloud computing. In one or more embodiments, electronic devices or devices with high computing power according to the disclosed solution can be applied to cloud devices (such as cloud servers), while electronic devices or devices with low power consumption can be applied to terminal devices and / or edge devices (such as smart phones or cameras). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or edge device are compatible with each other, so that according to the hardware information of the terminal device and / or edge device, appropriate hardware resources can be matched from the hardware resources of the cloud device to simulate the hardware resources of the terminal device and / or edge device, so as to complete the unified management, scheduling and collaborative work of end-to-end or cloud-edge-to-end.

[0200] It should be noted that, for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will appreciate that the scheme of the present disclosure is not limited by the order of the actions described. Therefore, based on the disclosure or teaching of the present disclosure, those skilled in the art will appreciate that some of the steps therein may be performed in other orders or simultaneously. Further, those skilled in the art will appreciate that the embodiments described in the present disclosure may be regarded as optional embodiments, i.e., the actions or modules involved therein may not necessarily be necessary for the implementation of one or some of the schemes of the present disclosure. In addition, depending on the different schemes, the present disclosure also has different focuses on the description of some embodiments. In view of this, those skilled in the art may appreciate the parts that are not described in detail in a certain embodiment of the present disclosure, and may also refer to the relevant descriptions of other embodiments.

[0201] In terms of specific implementation, based on the disclosure and teaching of this disclosure, those skilled in the art can understand that several embodiments disclosed in this disclosure can also be implemented by other methods not disclosed herein. For example, with respect to the various units in the electronic device or device embodiments described above, this article divides them on the basis of considering logical functions, and there may be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. In terms of the connection relationship between different units or components, the connection discussed in the above text in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0202] In the present disclosure, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed on multiple network units. In addition, according to actual needs, some or all of the units may be selected to achieve the purpose of the scheme described in the embodiments of the present disclosure. In addition, in some scenarios, multiple units in the embodiments of the present disclosure may be integrated into one unit or each unit may exist physically separately.

[0203] In some other implementation scenarios, the above-mentioned integrated unit can also be implemented in the form of hardware, that is, a specific hardware circuit, which may include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit may include but is not limited to physical devices, and the physical devices may include but are not limited to devices such as transistors or memristors. In view of this, the various devices described herein (such as computing devices or other processing devices) can be implemented by appropriate hardware processors, such as CPU, GPU, FPGA, DSP and ASIC, etc. Further, the aforementioned storage unit or storage device can be any appropriate storage medium (including magnetic storage medium or magneto-optical storage medium, etc.), which can be, for example, a variable resistive memory (Resistive Random Access Memory, RRAM), a dynamic random access memory (Dynamic Random Access Memory, DRAM), a static random access memory (Static Random Access Memory, SRAM), an enhanced dynamic random access memory (Enhanced Dynamic Random Access Memory, EDRAM), a high bandwidth memory (High Bandwidth Memory, HBM), a hybrid memory cube (Hybrid Memory Cube, HMC), ROM and RAM, etc.

[0204] Although multiple embodiments of the present disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art may think of many changes, modifications, and alternatives without departing from the thought and spirit of the present disclosure. It should be understood that in the process of practicing the present disclosure, various alternatives to the embodiments of the present disclosure described herein may be adopted. The attached claims are intended to define the scope of protection of the present disclosure, and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A method for compiling a computational graph, in, The method comprises: Obtaining a computational graph to be compiled and expressed by a second intermediate representation, performing forward derivation of the shape, and obtaining complete shape information according to the forward derivation and tensor data splitting information; wherein the second intermediate representation is obtained by conversion based on the first intermediate representation, and the first intermediate representation is obtained by unified conversion based on a task model expressed by at least one front-end language; and the computational graph is a task executed on an intelligent processor; Using the complete shape information, determining whether the tensor data splitting information needs to be adjusted; Determine the tensor data splitting information that meets the requirements according to the judgment result; Derivation of memory access modes for operators in the computation graph based on the tensor data splitting information that meets the requirements; Based on the derived memory access mode, determining address domain related parameters of instructions involved in a loop in the code logic of the computation graph; Pipelining the instructions in the loop based on the instruction stream supported by the intelligent processor; The calculation graph is compiled according to address domain related parameters of the instructions involved in the loop and the result of the instruction pipeline arrangement in the loop to obtain a binary file recognizable by the intelligent processor.

2. The method according to claim 1, in, The memory access mode derivation indicates that: when a certain dimension of tensor data is split or divided, it is necessary to derive the memory access address of each time / each data block of the associated operator in the computation graph for the input data and / or output data after the split according to the split data blocks.

3. The method according to claim 1, in, The step of deriving a memory access mode for an operator in the computational graph based on the tensor data splitting information that meets the requirements includes: Taking the end operator in the computation graph as the entry operator, utilizing the splitting information of the output tensor data of the entry operator, and going against the direction of tensor data flow in the computation graph, reversely determine the memory access mode of the input tensor data of the entry operator from the splitting information of the output tensor data.

4. The method according to claim 1, in, The steps to determine whether the tensor data splitting information needs to be adjusted include: Whether the tensor data splitting information needs to be adjusted is determined based on the splitting efficiency of the tensor data.

5. The method according to claim 4, in, The split efficiency is defined by the utilization of the processing core, where the utilization of the processing core IPU efficiency Calculate according to the following formula: Among them, N o , H o , W o Respectively represent the original size of the output tensor data in N, H, and W dimensions; H ipu-factor and W ipu-factor Respectively represent the split block size in H and W dimensions, IPU part Indicates the number of processing cores of the intelligent processor for parallel processing, Num cluster Indicates the number of computing clusters of the intelligent processor. part Indicates the number of computing clusters processed in parallel by the intelligent processor.

6. The method according to claim 1, in, The steps to determine whether the tensor data splitting information needs to be adjusted include: Whether the tensor data splitting information needs to be adjusted is determined according to the operating mode of the intelligent processor.

7. The method according to claim 6, in, The intelligent processor operation modes include Throughput mode and Latency mode; wherein, Throughput mode represents a mode for measuring the processing throughput of the intelligent processor when the batch of tensor data dimension is greater than a threshold; Latency mode represents a mode for measuring the processing latency of the intelligent processor when the batch of tensor data dimension is less than a threshold.

8. A processing device for compiling a computation graph, include: a processor configured to execute program instructions; as well as A memory configured to store the program instructions, and when the program instructions are loaded and executed by the processor, the processor executes the method for compiling a computational graph according to any one of claims 1 to 7.

9. A computer-readable storage medium storing program instructions, which, when loaded and executed by a processor, enables the processor to execute the method for compiling a computational graph according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program or instructions, which, when executed by a processor, implements the method for compiling a computational graph according to any one of claims 1 to 7.

Images