Method, readable storage medium and compiler for optimizing loops

Abstract

The present invention relates to a method, readable storage medium and compiler for optimizing a description of a loop of a neural network operator at compile time, wherein the computing device of the present invention is comprised in an integrated circuit device comprising a general purpose interconnect interface and other processing devices. The computing device interacts with the other processing devices to jointly perform a user-specified computation operation. The integrated circuit device can further comprise a storage device connected to the computing device and the other processing devices, respectively, for data storage of the computing device and the other processing devices.

Description

Technical Field

[0001] This invention generally relates to the field of neural networks. More specifically, this invention relates to a method for optimizing loops describing neural network operators during compilation, a readable storage medium, and a compiler. Background Technology

[0002] When implementing operators in neural networks, developers can utilize various computational methods to achieve the same results. Different methods can lead to differences in code locality and parallelism. TVM is an open-source deep learning compiler that generates efficient machine code for different hardware backends by providing graph-level and operator-level optimizations.

[0003] TVM defines operators based on tensor expressions and provides a series of scheduling primitives to perform iterative transformations on the operator code. Operators described using tensor expressions are translated by TVM into TVM intermediate representations (IRs), which are tree-like, high-level representations that easily describe iterative computations. TVM implements a series of optimized passes of analysis and transformations on the intermediate representations, adjusting the operators to ultimately generate efficient code for the target hardware platform.

[0004] Important operators in deep learning applications, such as convolution, matrix multiplication, and vector addition, are often generated by TVM compilers into multi-layered nested loops in their intermediate representations. These programs, after loop transformations, produce complex expressions related to loop iteration variables. Existing non-TVM compilers optimize loop invariant extraction by storing computation results in physical registers to avoid recalculation on each use. However, the high-level intermediate representations of existing TVM compilers are unaware of physical registers. Blindly referencing the loop invariant extraction methods of non-TVM compilers introduces unnecessary register pressure, ultimately degrading operator performance. Therefore, directly applying the loop invariant extraction algorithm from existing non-TVM compilers to TVM compilers does not yield ideal optimization results.

[0005] Secondly, in the process of detecting loop invariants, the order of operands in the expression affects the discovery of invariants. Existing optimization algorithms have limited ability to detect complex nested conditional expressions that frequently appear in deep learning operators; in other words, their optimization level is not good.

[0006] Finally, the operator code generated by the TVM compiler still needs to undergo compilation and optimization by the target platform compiler before it can generate executable code for the target hardware. The target platform compiler may perform other optimizations at lower-level intermediate representations. These two different levels of optimization often conflict, reducing the effectiveness of the loop invariant outfetching algorithm.

[0007] Therefore, a technique for optimizing the loops describing neural network operators during compiler execution is urgently needed. Summary of the Invention

[0008] To at least partially solve the technical problems mentioned in the background art, the present invention provides a method for optimizing loops describing neural network operators during compilation, a readable storage medium, and a compiler.

[0009] In one aspect, the present invention discloses a method for optimizing loops describing neural network operators during compiler execution, comprising: traversing all statements in the program, determining whether the expression of each statement is loop-invariant, wherein the expression is associated with an initial variable; if not, marking the expression as "non-promotable"; replacing each expression not marked as "non-promotable" with different promoted variables respectively; and inserting assignment statements for all promoted variables before the loop.

[0010] In another aspect, the present invention discloses a computer-readable storage medium storing computer program code that, when compiled, optimizes loops describing neural network operators, and executes the aforementioned method when the computer program code is run by a processing device.

[0011] In another aspect, the present invention discloses a compiler that iterates through all statements in a program at compile time to optimize loops, comprising: a judgment module, a marking module, a substitution module, and an insertion module. The judgment module determines whether the expression of each statement is loop-invariant, wherein the expression is associated with an initial variable; the marking module marks the expression as "non-loop-invariant" if the expression of each statement is not loop-invariant; the substitution module replaces each expression not marked as "non-loop-invariant" with different looped variables; and the insertion module inserts assignment statements for all looped variables before the loop.

[0012] This invention proposes a loop invariant extraction scheme for high-level intermediate representations in deep learning compilers. By extracting profitable loop invariants using a cost model, the contradiction between eliminating redundant computation and register allocation is resolved in a simple and easily implemented way. Secondly, before detecting loop invariants, this invention can normalize the expression by adjusting the operand combination order and transforming the form of the conditional expression, providing a better foundation for loop invariant extraction optimization and subsequent optimizations. Finally, this invention, considering the specific characteristics of TVM intermediate representations and the target platform compiler, provides special handling for inner loops with relatively few branches and iterations, resolving optimization conflicts at different levels of intermediate representations. Attached Figure Description

[0013] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts. Wherein:

[0014] Figure 1 This is a structural diagram of the board card according to an embodiment of the present invention;

[0015] Figure 2 This is a structural diagram illustrating an integrated circuit device according to an embodiment of the present invention;

[0016] Figure 3 This illustrates the compilation process of TVM;

[0017] Figure 4 This diagram illustrates the overall architecture and key compilation flowchart of an embodiment of the present invention in TVM;

[0018] Figure 5 This is a flowchart illustrating the optimized description of the loop of the neural network operator according to an embodiment of the present invention;

[0019] Figure 6 This is another flowchart illustrating an embodiment of the present invention that optimizes the loop describing the neural network operator;

[0020] Figure 7 This illustrates the overall architecture diagram and key compilation flowchart of another embodiment of the present invention in TVM;

[0021] Figure 8 It shows the abstract syntax tree corresponding to the exemplary expression during the compilation process;

[0022] Figure 9 It shows the abstract syntax tree corresponding to the rewritten expression;

[0023] Figure 10 This is a flowchart illustrating an expression in a statement that standardizes the description of a neural network operator according to another embodiment of the present invention;

[0024] Figure 11 This shows the syntax tree corresponding to the example conditional expression;

[0025] Figure 12 It shows the syntax tree corresponding to the merged conditional expression;

[0026] Figure 13 This is a schematic diagram illustrating a compiler according to another embodiment of the present invention; and

[0027] Figure 14 This is a schematic diagram illustrating a compiler according to another embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] It should be understood that the terms "first," "second," "third," and "fourth," etc., in the claims, specification, and drawings of this invention are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" used in the specification and claims of this invention indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0030] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this specification and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0031] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection."

[0032] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0033] Controlling AI chips to execute complex neural networks generates a large number of instructions. Developers often cannot meet the requirements of generalization when writing instructions, so a compiler is needed to accelerate neural networks. This compiler can be a neural network compiler. The compiler generates a sequence of hardware-executable instructions based on the graph structure and operator structure of the neural network. Compilers can be broadly divided into front-end and back-end. The front-end mainly optimizes and transforms the network structure described by deep learning frameworks such as TensorFlow, Caffe, and PyTorch into an intermediate representation. The back-end optimizes and transforms the intermediate representation into a hardware-executable program. The main challenge of the front-end is designing a universal intermediate representation that can interface with different deep learning frameworks. More complex compiler front-ends also include many hardware-independent optimization strategies, such as constant folding and operator fusion. The back-end is hardware-dependent. Different hardware has specific design structures, especially specially designed AI chips. Due to the uniqueness of the chip, the back-end needs to be optimized for these design structures and memory allocations for different networks. This hardware can be a circuit board or a chip.

[0034] Figure 1 A schematic diagram of the structure of a board 10 according to an embodiment of the present invention is shown. Figure 1 As shown, board 10 includes chip 101, which is a system-on-chip (SoC) integrating one or more combined processing units. These combined processing units are artificial intelligence computing units used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in cloud intelligence. A significant characteristic of cloud intelligence applications is the large volume of input data, placing high demands on the platform's storage and computing capabilities. Board 10 in this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and powerful computing capabilities.

[0035] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as a PCIe interface.

[0036] The board 10 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 106 and the chip 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the chip 101. Therefore, in one application scenario, the controller 106 may include a microcontroller (MCU).

[0037] Figure 2 This is a structural diagram illustrating the combined processing device in chip 101 of this embodiment. (As shown) Figure 2 As shown, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203, and off-chip memory 204.

[0038] The computing device 201 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0039] Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them to on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from storage device of computing device 201 and transmit it to processing device 203.

[0040] The processing device 203, as a general-purpose processing device, performs basic controls including but not limited to data transfer and starting / stopping the computing device 201. Depending on the implementation, the processing device 203 may be one or more types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), or other general-purpose and / or special-purpose processors. These processors include, but are not limited to, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, the computing device 201 of this invention can be considered as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

[0041] The off-chip memory 204 is used to store data to be processed. It is DDR memory, typically 16G or larger, and is used to store data of computing device 201 and / or processing device 203.

[0042] To effectively utilize the aforementioned hardware, this embodiment optimizes the existing TVM architecture. TVM provides a complete compiler architecture solution, customizing the intermediate representation and abstracting the computation process (including operators, variables, etc.) to generate complete code and execution framework.

[0043] Figure 3 The compilation flow of TVM is illustrated. Model 301 refers to the series of code controlling the aforementioned hardware generated after the developer completes the programming. Front-end import 302 is the process by which the front-end component extracts neural networks from various deep learning frameworks, especially tensor expressions, and transforms them into intermediate representations of TVM. The so-called tensor expressions here are domain-specific languages ​​designed by TVM to describe tensor operations based on Halide's idea of ​​separating operator description and scheduling optimization. This language defines the input-output relationships between tensors and provides a series of scheduling primitives such as loop splitting, loop merging, and loop blocking. The combination of these scheduling primitives specifies the specific strategy for computing the output tensor.

[0044] The intermediate representation module 303 generates intermediate representations of relays, which are high-level abstract representations used to construct a data flow graph containing multiple nodes. The intermediate representation of a relay is described by `relay::function`, which describes the entire graph structure, has input and output variables, and internally contains the computation sequence. The intermediate representation module 303 optimizes and transforms the network graph structure based on the relay passes function 304, such as constant folding, useless code elimination, layout transformation, and scale factor folding, to improve hardware operating efficiency.

[0045] The intermediate representation generated at this point mainly includes two types of nodes: expression nodes and statement nodes. Expression nodes contain variables and constants, array data loading, function calls, arithmetic and logical expressions, conditional expressions, etc.; statement nodes contain sequence, loop, branch, assignment, etc. TVM traverses and modifies the nodes using the relay pass function 304. At this point, the front-end task of the TVM compiler is complete.

[0046] Next, the backend program is executed. Primitive-function scheduling 305 calls primitives related to the aforementioned nodes and searches for the optimal optimization space through automatic TVM function 306. Automatic TVM function 306 is an automatic optimization algorithm whose main idea is to achieve network optimization based on reinforcement learning. It collects information from computing device 201 on how to perform the same task in different ways to continuously update optimization variables.

[0047] The intermediate representation module 307, based on the underlying intermediate representation pass (TIR passes) function 308, optimizes and transforms the operator structure, mostly handling dimensionality reduction, such as expanding multidimensional paths into one-dimensional pointer paths and modifying function entry points to be runtime-callable. The underlying intermediate representation pass function 308 is a representation structure closer to the hardware, which can optimize instructions such as tensor computation operators and load / store operations. If developers have custom instructions, they are usually defined in the underlying intermediate representation pass function 308.

[0048] Target translation 309 converts the intermediate representation generated by intermediate representation module 307 into an executable program, which is encapsulated by runtime module 310, making it easier to load, export, and execute in the runtime environment of computing device 201. At this point, the backend tasks of the TVM compiler are complete.

[0049] Specifically, this embodiment incorporates an optimized compiler designed by the developer into the underlying intermediate representation transfer function 308, particularly optimizing the loops describing neural network operators. Under certain conditions, and considering the characteristics of the intermediate representation module 307 and the target platform compiler, this embodiment reduces redundant computations by hoisting loop invariants, thereby improving the runtime performance of the target translation result 309.

[0050] The following example defines a tensor expression describing a fixed-size matrix multiplication operator:

[0051] M=1024; K=1024; N=1024; bn=32

[0052] A=te.placeholder((M,K),name="A")

[0053] B=te.placeholder((K,N),name="B")

[0054] k = te.reduce_axis((0,K),"k")

[0055] C=te.compute((M,N),lambda x,y:te.sum(A[x,k]*B[k,y],axis=k),name="C")

[0056] s = te.create_schedule(C.op)

[0057] Variables A and B are defined as two-dimensional matrices of size 1024×1024, and k is defined as the cyclic axis for summation and reduction during matrix multiplication. Matrix C is calculated from matrices A and B. After the above operator description is compiled by the intermediate representation module 307 of the TVM deep learning compiler, the following intermediate representation code can be obtained:

[0058]

[0059] In existing technologies, to implement the innermost loop in the code above, TVM uses functions such as tile, split, and reorder to perform loop-blocking, loop-split, and loop-reordering on the matrix multiplication operator, thereby obtaining the following intermediate representation:

[0060] xo,yo,xi,yi=s[C].tile(C.op.axis[0],C.op.axis[1],bn,bn)

[0061] (k,)=s[C].op.reduce_axis ko,ki=s[C].split(k,factor=32)

[0062] s[C].reorder(xo,yo,ko,ki,xi,yi)

[0063] The transformed operator description, after being translated by Target Translation 309, yields the following intermediate representation code:

[0064]

[0065]

[0066] After undergoing a loop transformation, operators achieve better parallelism and data locality, and can fully utilize the characteristics of the underlying hardware. However, this transformation changes the array expression from a simple expression to a semantically equivalent but more complex one, adding unnecessary scalar operations. For example, in the intermediate representation code obtained after the loop transformation of the matrix multiplication operator above, for the innermost loop, the loop iteration variable is yi, and its expression xo*32768+xi*1024+yo*32 remains unchanged in each iteration of the innermost loop, making the calculations after the first iteration redundant.

[0067] To address this technical problem, this embodiment modifies the machine learning operator-level optimization module of TVM by moving redundant calculations where the values ​​remain unchanged multiple times within a loop outside the loop, thereby reducing additional redundant operations and generating operators with higher performance. Figure 4 This diagram illustrates the overall architecture and key compilation flowchart of this embodiment in TVM.

[0068] In step 401, the ScheduleOps function is executed to translate the tensor expression of the operator into an intermediate representation. In step 402, the StorageFlattern function is executed to flatten the multidimensional array of tensors into a one-dimensional array. In step 403, the InjectDoubleBuffer function is executed to maximize hardware utilization and optimize the pipeline. In step 404, the StorageRewrite function is executed to calculate memory reuse and optimize memory-related variables. In step 405, the LICM (loop invariant code motion) function is executed, using the loop invariant hoisting algorithm of this embodiment to hoist variables that remain unchanged within the loop to the outside of the loop. In step 406, the UnrollLoop function is executed to unroll the loop to reduce the number of loops. In step 407, the HoistIfThenElse function is executed to move inner conditional branch statements to outer loops as much as possible to reduce the number of calculations.

[0069] Step 405 is the key operation for optimizing the loop. The loop nodes in the intermediate representation of TVM contain the nested loop relationship and the corresponding iteration variables. In this embodiment, it is recursively determined whether a certain expression remains unchanged in the loop. If so, it means that the expression has redundant calculations in the loop. Therefore, the expression is taken out of the loop and only needs to be executed once outside the loop. Figure 5 The flowchart of the loop describing the neural network operator in step 405 is further shown in this embodiment.

[0070] In step 501, the underlying intermediate representation transfer function 308 is used to traverse all statements in the program, classifying and discussing them according to their possible syntactic forms, and determining whether the expression of each statement is loop-invariant. These expressions are all associated with variables. For ease of distinction, the variables before loop optimization are called initial variables, and the variables after loop optimization are called hoisted variables. Apart from the difference in name, they are essentially the same intermediate representation variables and have no difference.

[0071] In this embodiment, an expression is considered cyclically invariant if the value constituting the expression satisfies any of the following criteria: the initial variable is a constant, including immutable program parameters or immediate values; the assignment of the variable in the expression occurs outside the loop; and the variable is used to only one fixed value, and the expression for that fixed value is also cyclically invariant. In other words, this step determines whether the value constituting the expression satisfies any of the above criteria.

[0072] If the expression is not loop-invariant, it means that the initial variable associated with the expression will change within the loop and cannot be extracted from the loop. In this case, step 502 is executed, marking the expression as "cannot be extracted." Specifically, this embodiment identifies invariant expressions within the loop and marks those expressions whose calculation results may change in multiple iterations of the loop. The remaining unmarked expressions are the loop invariants. This embodiment does not mark or process loop-invariant expressions at this stage.

[0073] In other cases, in step 501, when it is determined that the statement is an assignment to an initial variable through an expression, such as initial variable x = e, and the expression e is marked as "not to be moved out", the initial variable x is not a loop invariant, that is, the initial variable x defined by the assignment statement is regarded as "not to be moved out", then in step 502, the initial variable x in the expression is marked as "not to be moved out".

[0074] In other cases, in step 501, when the judgment statement is a loop statement and the expression is not a loop invariant, then in step 502, the initial variable in the expression is marked as "cannot be moved out".

[0075] Regardless of whether the expression is a loop that does not change, step 503 is executed to determine whether the iteration has been completed. If not, the process returns to step 501 to evaluate the next expression.

[0076] After traversing all statements, step 504 is executed, replacing each expression not marked as "non-promotable" with a different hoisted variable. In this step, expressions in the statements not marked as "non-hoisted" are considered loop invariants. For this purpose, this embodiment generates a new variable, the hoisted variable, rewrites the loop-invariant expressions in the statements using the hoisted variable, and adds the hoisted variables of all identified loop invariants to the loop invariant set for subsequent processing.

[0077] In step 505, all assignment statements for hoisted variables are inserted before the loop. Since expressions not marked as "non-hoisted" do not change within the loop, repeating their evaluation within the loop is meaningless and wastes resources. In this step, the assignment statements for the hoisted variables corresponding to these expressions are moved outside the loop. Preferably, the assignment statements for the hoisted variables corresponding to these expressions are inserted before the loop.

[0078] In step 506, the hoisted expressions are deleted. Since expressions not marked as "non-hoisted" have been rewritten as hoisted variables and inserted before the loop, the corresponding expressions in the loop are deleted to avoid duplicate execution.

[0079] In step 507, an intermediate representation is generated based on the assignment statements of all hoisted variables. The intermediate representation module 307 generates an intermediate representation based on the assignment statements of all hoisted variables and other statements.

[0080] In step 508, translation is performed based on the intermediate representation. Target translation 309 translates the intermediate representation generated in step 507 to generate an executable program for computing device 201.

[0081] In step 509, runtime module 310 encapsulates the executable program.

[0082] Since the loop invariant has been moved outside the loop, when the computing device 201 performs deep learning tasks according to the encapsulated executable program, such as performing intelligent needs in fields like computer vision, speech, natural language processing, and data mining, it will no longer repeatedly execute the loop-invariant expression in the loop, thus completing the task more efficiently.

[0083] Another embodiment of the present invention also modifies the machine learning operator-level optimization module of TVM. By moving redundant calculations in which the values ​​remain unchanged multiple times within a loop outside the loop, additional redundant operations are reduced, generating higher-performance operators, and essentially performing the same operation. Figure 3 and Figure 4 The process differs from the previous embodiment in that this embodiment introduces the concept of a benefit threshold, calling a cost function to evaluate the expression extraction, and extraction will only be performed if the benefit is high enough. Figure 6 The flowchart shown is a process for optimizing the loop describing the neural network operator in step 405 of this embodiment.

[0084] In step 601, the underlying intermediate representation transfer function 308 is used to traverse all statements in the program, classify and discuss them according to the possible syntactic forms of the statements, and determine whether the expression of each statement is a loop invariant.

[0085] In this embodiment, an expression is considered cyclically invariant if the value constituting the expression satisfies any of the following conditions: the initial variable is a constant, including immutable program parameters or immediate values; the assignment of the variable in the expression is outside the loop; the variable is used only once, and the expression for that constant value is also cyclically invariant.

[0086] If the expression is not loop-invariant, it means that the initial variable associated with the expression will be changed within the loop and cannot be taken out of the loop. In this case, step 602 is executed, and the expression is marked as "cannot be taken out".

[0087] Regardless of whether the expression is a loop that does not change, step 603 is executed to determine whether the iteration has been completed. If not, the process returns to step 601 to evaluate the next expression.

[0088] After traversing all statements, step 604 is executed, replacing each expression not marked as "non-promotable" with a different hoisted variable. This embodiment generates hoisted variables, rewrites loop-invariant expressions in the statements with hoisted variables, and adds the hoisted variables of all identified loop invariants to the loop invariant set for subsequent processing.

[0089] In step 605, a cost function is invoked to calculate the cost of each expression not marked as "non-promotable". In this embodiment, the cost function cost(e) is invoked to calculate the cost of outsourcing expression e. The cost function cost(e) takes expression e as input, calculates and returns the outsourcing cost of e. The reason for introducing the cost function cost(e) in this embodiment is that the high-level operator code generated by TVM needs to be compiled by the target platform compiler to generate the executable program on computing device 201. Since the target platform compiler will also perform a series of optimizations on the program at a lower-level intermediate representation, in order to avoid interference between the two different levels of optimization, the cost function cost(e) is used to calculate and select more efficient loop invariants for outsourcing.

[0090] The cost function cost(e) can be defined as follows:

[0091]

[0092] When e is a constant or variable, its cost is defined as 0; when e is a compound expression, its cost is recursively defined on its various subexpressions a, plus the cost of its operator ⊕.

[0093] In expression For example, according to its definition, its cost function for:

[0094]

[0095] Different operators ⊕ have different costs based on their required clock cycles. Generally, the addition instruction requires fewer clock cycles, so in this example, cost(+) = 1. The division instruction requires more clock cycles, so in this example, cost(÷) = 3. Since x, 1024, and y are constants or variables, cost(x), cost(1024), and cost(y) are all 0. The cost function can be obtained in this step.

[0096] It is important to note that the cost function cost(e) can be adjusted arbitrarily to match the actual situation of different hardware platforms. For example, for GPUs, the compiler on the platform usually chooses to expand loops with a very small iteration range and perform other optimizations on this basis. Based on this characteristic, the cost value of the expression in the loop with a very small iteration range can be set to 0, thereby avoiding interference between two different levels of optimization.

[0097] In step 606, it is determined whether the cost value is greater than or equal to a threshold. In this embodiment, the threshold is set to 1, which is whether the cost value is greater than or equal to 1. By comparing it with the preset threshold, it is evaluated whether extracting the expression produces a sufficiently large benefit.

[0098] If the cost is not greater than or equal to the threshold, it indicates insufficient benefit and is not worthwhile to extract the expression. Therefore, step 607 is executed to abandon the extraction attempt, remove the extracted variable from the set of loop invariants, restore the replaced expression, and mark the expression as "cannot be extracted". For example, although the constant is also loop invariant, its cost is zero, which is less than the threshold. Extracting the constant would not be beneficial, so the constant will not be extracted in this step.

[0099] If the cost is greater than or equal to the threshold, the compiler considers it a sufficient benefit, and if expression hoisting has no side effects, the expression can be hoisted. In this case, step 608 is executed, inserting the assignment statements for all hoisted variables before the loop. For example... The cost is 4, which is greater than the threshold of 1. It is beneficial to hoist the expression, so the assignment statement of its hoisted variable is inserted before the loop.

[0100] In step 609, the lofted expressions are deleted. In step 610, an intermediate representation is generated based on the assignment statements of all lofted variables. In step 611, the intermediate representation is translated and transformed to generate an executable program. In step 612, the executable program is encapsulated.

[0101] Since efficient loop invariants are moved out of the loop while inefficient loop invariants remain unchanged, the computing device 201 can perform deep learning tasks more efficiently when executing the encapsulated executable program.

[0102] Another embodiment of the present invention also modifies the machine learning operator-level optimization module of TVM. Redundant calculations where the value remains unchanged multiple times within a loop are moved outside the loop, thereby reducing additional redundant computations. Unlike the previous two embodiments, this embodiment introduces the concept of normalization, recombining expressions and merging conditional expressions and branches, before using the loop invariant hoisting algorithm of the present invention to hoist variables that remain unchanged within the loop.

[0103] Figure 7 This diagram illustrates the overall architecture and key compilation flowchart of this embodiment in TVM.

[0104] In step 701, the ScheduleOps function is executed to translate the tensor expression of the operator into an intermediate representation. In step 702, the StorageFlattern function is executed to flatten the multidimensional array of tensors into a one-dimensional array. In step 703, the InjectDoubleBuffer function is executed to maximize hardware utilization and optimize the pipeline. In step 704, the StorageRewrite function is executed to calculate memory reuse and optimize memory-related variables.

[0105] In step 705, the Normalize function is executed, which is called normalization. By performing semantic-preserving transformations on statements and expressions, the normalized statements are more conducive to subsequent loop invariant hoisting or optimization by the target platform compiler.

[0106] When the compiler seeks to identify loop invariants in an expression, the order of operands is crucial. Different operand orders not only affect whether certain loop-invariant expressions can be detected but also determine whether certain branch optimizations in the compiler can be triggered. Therefore, before executing the LICM function, this embodiment normalizes the expression in this step, using the associativity of operators to perform transformations such as reassociation, so that the updated operand order is conducive to detecting loop-invariant expressions and triggering branch optimizations.

[0107] The most important operations in normalization are expression reassociation and merging of conditional expressions and branches. Expression reassociation refers to using specific algebraic laws, such as associativity, commutativity, and distributivity, to divide an expression into constant parts, loop-invariant parts, and variable parts. Although expression reassociation itself does not improve performance, since whether a subexpression in an expression is a loop-invariant expression largely depends on the order of operations and associativity of the operands, expression reassociation can significantly improve subsequent loop invariant outfetching optimizations.

[0108] To illustrate the normalization process more clearly, the following expression is given as an example:

[0109] ((((3<i)&&(3<j))&&(i<56))&&(j<56))

[0110] Where i is the loop iteration variable. Based on the execution priority determined by the parentheses in the above expression, the corresponding abstract syntax tree during compilation is as follows: Figure 8As shown. One task of normalization is to make appropriate adjustments so that the subexpression is as invariant as possible to facilitate its hoisting in subsequent steps. For example, the compiler in this embodiment can rewrite the above expression after normalization as follows:

[0111] (((3<i)&&(i<56))&&((3<j)&&(j<56)))

[0112] The abstract syntax tree corresponding to the rewritten expression is as follows: Figure 9 As shown. This expression has the same computational result as the unnormalized expression, but from... Figure 9 It can be seen that the normalized expression yields a more ideal loop-invariant expression ((3<j)&&(j<56)).

[0113] Figure 10 This embodiment shows a flowchart of the expression in the statement that normalizes the description of the neural network operator in step 705.

[0114] In step 1001, it is determined whether the statement includes a conditional expression and whether the condition is true or false corresponds to the same return value. Conditional expressions have complex structures, so they are processed first. Each conditional expression produces a corresponding return value for both the truth and falsehood of the condition. In nested conditional expressions, multiple conditions may have the same return value. In this case, this embodiment merges these conditions with the same return value, making the loop invariants in the transformed expression more complete.

[0115] For example, the following expression is a nested conditional expression, where only i is the loop iteration variable:

[0116] ((3<i)&&(3<j))? (((i<56)&&(j<56))?1:0):0

[0117] This conditional expression includes two levels of conditional judgment. The first level returns 1 if both i < 56 and j < 56 are true, otherwise it returns 0. The second level returns the value of the first level if both 3 < i and 3 < j are true, otherwise it returns 0. The syntax tree for this conditional expression is as follows: Figure 11 As shown, based on this syntax tree, the candidate loop invariants that the TVM compiler can determine are 3 < j and j < 56.

[0118] In the example above, the compiler determines that the statement does indeed include a conditional expression. When both conditions (i < 56) && (j < 56) and (3 < i) && (3 < j) are true, it returns the same value, 1. Furthermore, when both conditions (i < 56) && (j < 56) and (3 < i) && (3 < j) are false, it also returns the same value, 0. Under this premise, step 1002 is executed, calling the function `condCollapse(e)` to optimize the expression `e` that can have its conditions merged. That is, it merges the conditions with the same return value and returns the optimized result to update the conditional expression. The merged expression is as follows:

[0119] ((((3<i)&&(i<56))&&((3<j)&&(j<56)))?1:0)

[0120] Its corresponding syntax tree is as follows Figure 12 As shown. The merged expression produces a better candidate loop-invariant expression ((3<j)&&(j<56)), and the forms of ((3<i)&&(i<56)) and ((3<j)&&(j<56)) are more conducive to implementing peephole optimizations related to Boolean expressions. Peephole optimization is a local optimization method where the compiler only focuses on the code already generated in the sliding window, and improves the performance of the code in the sliding window through instruction transformation, taking into account the instruction characteristics of computing device 201.

[0121] If the statement in step 1001 does not include a conditional expression or the condition (true or false) does not correspond to the same return value, or if step 1002 is executed and then step 1003 is executed to determine whether the statement satisfies the associativity law, then the statement satisfies the associativity law. If the statement satisfies the associativity law, it means that moving the relative position of the expression within the statement does not change the result of the statement's operation, and the statement can be normalized. More specifically, this embodiment iterates through the statement to find whether there exists an operator ⊕ that satisfies the associativity law. The operator ⊕ is merely a symbolic designation used to represent all types of operators that satisfy the associativity law.

[0122] If a statement does not satisfy the associativity law, it cannot be normalized. Execute step 1004 to end the normalization process for that statement.

[0123] If the statement satisfies the associative law, then step 1005 is executed, setting the level of the expression according to the type of each expression. If the expression has been merged in step 1002, then this step sets the level according to the type of the merged conditional expression.

[0124] Specifically, this embodiment calls `deconstruct(e, ⊕)` to convert the expressions connected by the operator `⊕` into an ordered sequence of expressions `L`, assigning a level to each expression. This embodiment divides expressions into three levels based on their type: constants are the highest level, nested loops are the middle level, and other types are the lowest level. Higher levels are more beneficial for subsequently identifying expressions with invariant loops and triggering branch optimizations. Based on the aforementioned rules, this step first determines whether the expression is a constant; if it is, its level is set to level one. If it is not a constant, it further determines whether the expression contains nested loops; if it does, its level is set to level two. If it does not contain nested loops, meaning the expression is neither a constant nor contains nested loops, its level is set to level three regardless of its type.

[0125] Because nested loops can have multiple levels, variables in the outer loop are more easily hoisted than those in the inner loop. Therefore, if the expression includes nested loops, and these nested loops contain N loops (N greater than 1), this step, while setting the level of the expression containing nested loops to the second level, further sets the child levels based on the loop hierarchy. Specifically: the child level of the outermost loop is set as the first child level; the child level of the next outermost loop is set as the second child level; and so on, setting the child levels of other loops until the child level of the innermost loop is set as the Nth child level. In other words, the outermost loop has the highest child level, and the innermost loop has the lowest. Similarly, a higher child level is more conducive to subsequently identifying loop-invariant expressions and triggering branch optimizations.

[0126] For example, a statement might contain three expressions: one a constant, another containing a nested loop, and the last of some other type, with three nested loops. This step first determines if the expressions are constants and assigns them a first-level priority. Next, it checks for nested loops and assigns them a second-level priority. Since there are three levels of nested loops, this step further assigns the outermost loop's child to the first-level priority; the second outermost loop's child to the second-level priority; and the innermost loop's child to the third-level priority. Finally, the last expression is assigned a third-level priority.

[0127] In some cases, an expression may include multiple subexpressions, each with a different level. This embodiment sets the level of the expression based on the lowest level among all subexpressions. Specifically, if an expression contains any subexpression that is not a constant and does not contain nested loops, then the expression must contain other types of subexpressions, and this embodiment sets the level of the expression to level three. If all subexpressions in an expression are constants or contain nested loops, this embodiment sets the level of the expression to level two based on the level of the nested loops.

[0128] In step 1006, each expression is sorted based on its rank. In this embodiment, the function stableSort(L) is called to sort the sequence of expressions L with rank markings from highest to lowest according to their rank and sub-rank, and the sorted sequence L is returned.

[0129] The sorting method is as follows: all expressions in the statement are arranged in the order of their level 1, level 2, and level 3, and all expressions in the level 2 are arranged in the order of their child level 1, level 2, and so on up to level N.

[0130] In the example above, the priority order of the expressions is: constant (level 1) > outermost loop (level 2, first sub-level) > second outermost loop (level 2, second sub-level) > innermost loop (level 2, third sub-level) > others (level 3). Based on the priority order of the expressions, this embodiment can further consider reorganizing these levels and sub-levels to obtain a more intuitive sorting method and facilitate recording the priority order. For example, the expression for constant can be adjusted to level 0, the expression for outermost loop to level 1, the expression for second outermost loop to level 2, the expression for innermost loop to level 3, and the other expressions to level 4. In this way, the sorting can be intuitively performed according to the level number.

[0131] In step 1007, expressions with the same level or sublevel are concatenated using operators to generate sibling expressions. This embodiment calls the function `groupExpressions(L, ⊕)` to concatenate expressions with the same level or sublevel in the sorted sequence `L` using the operator `⊕` to form a new expression, and returns the transformed sequence `L`. For example, if a statement contains multiple constant expressions, since the level of these constant expressions is all set to first level, these constant expressions will be concatenated together using the operator `⊕` to form a new expression.

[0132] In step 1008, all sibling expressions are concatenated using operators based on hierarchical or child order to return the updated statement. This embodiment calls the function `construct(L, ⊕)`, which uses the operator `⊕` to sequentially concatenate all expressions in sequence `L` and returns the transformed final result `e`. Since normalization is performed under associative conditions, the transformed final result `e` has an equivalent computational result to the original sequence `L`.

[0133] At this point, the standardized operations of this embodiment have been completed. Next, this embodiment will continue to compile based on the updated statements.

[0134] Back Figure 7 In step 706, the LICM function is executed. Based on the normalized statements, the loop invariant hoisting algorithm of this invention hoists variables that are invariant within the loop to the outside of the loop. Since this embodiment has normalized the statements, the order of operands in this step is beneficial for identifying loop-invariant expressions and triggering branch optimization. In step 707, the UnrollLoop function is executed to unroll the loop, reducing the number of loops. In step 708, the HoistIfThenElse function is executed to move inner conditional branch statements to outer loops as much as possible, reducing the number of calculations. At this point, this embodiment completes the operator-level optimization operation.

[0135] This embodiment optimizes memory read / write operations at a lower-level intermediate representation. The normalization in this embodiment focuses on specific transformations of different variables and conditional expressions, which is beneficial for subsequent loop invariant hoisting optimization and control flow optimization.

[0136] Another embodiment of the present invention is a compiler for normalizing expressions in statements, used to implement, for example... Figure 10 The method shown is a normalized description of expressions in statements that describe neural network operators. Figure 13 The diagram shows a compiler for this embodiment. The compiler includes a judgment module 1301, a merging module 1302, a setting module 1303, a sorting module 1304, and a connection module 1305. Each module transmits information through a bus 1306.

[0137] The judgment module 1301 is used to determine whether a statement includes a conditional expression and whether the condition is true or false corresponds to the same return value. A conditional expression produces a corresponding return value for both the truth and falsehood of the judgment condition. In nested conditional expressions, multiple judgment conditions may correspond to the same return value.

[0138] The merging module 1302 is used to merge judgment conditions with the same return value and return the optimized result to update the condition expression.

[0139] Next, the judgment module 1301 further determines whether the statement satisfies the associativity law. If the statement satisfies the associativity law, it means that moving the relative positions of the expressions within the statement will not change the result of the statement's operation, and the statement can be normalized. More specifically, the judgment module 1301 iterates through the statement to find whether there exists an operator ⊕ that satisfies the associativity law. If the expression has been merged by the merging module 1302, the judgment module 1301 will determine whether the statement satisfies the associativity law based on the type of the merged conditional expression.

[0140] If a statement does not satisfy the associativity law, normalization cannot be performed, and the normalization process for that statement ends. If a statement satisfies the associativity law, the setting module 1303 sets the level of each expression according to its type. Specifically, the setting module 1303 converts expressions connected by the operator ⊕ into an ordered sequence of expressions L, and assigns a level to each expression. In this embodiment, the setting module 1303 divides expressions into three levels according to their type: constants are the highest level, nested loops are the middle level, and other types are the lowest level. A higher level is more conducive to subsequently discovering expressions with invariant loops and triggering branch optimizations.

[0141] Because nested loops can have multiple levels, variables in the outer loop are more easily hoisted than those in the inner loop. Therefore, if the expression includes nested loops, and these nested loops contain N loops (N greater than 1), then module 1303, while setting the level of the expression containing nested loops to the second level, will further set the child levels according to the loop hierarchy. Specifically, the child level of the outermost loop is set as the first child level; the child level of the next outermost loop is set as the second child level; and so on, setting the child levels of other loops until the child level of the innermost loop is set as the Nth child level.

[0142] In some cases, an expression may include multiple sub-expressions, each with a different level. The setting module 1303 sets the lowest level among all sub-expressions to the level of the expression. Specifically, if an expression contains any sub-expression that is not a constant and does not contain nested loops, then the expression must contain other types of sub-expressions, and the setting module 1303 sets the level of that expression to level three. If all sub-expressions in an expression are constants or contain nested loops, the setting module 1303 sets the level of the expression to level two, based on the level of the nested loops.

[0143] The sorting module 1304 sorts each expression based on its level in the statement, sorts the sequence of expressions L with marked levels from highest to lowest, and returns the sorted sequence L.

[0144] The sorting method is as follows: all expressions in the statement are arranged in the order of their level 1, level 2, and level 3, and all expressions in the level 2 are arranged in the order of their child level 1, level 2, and so on up to level N.

[0145] The joiner module 1305 is used to connect expressions of the same level or sublevel using operators to generate sibling expressions. More specifically, the joiner module 1305 connects expressions of the same level or sublevel in a sorted sequence L using the operator ⊕ to form a new expression and returns the transformed sequence L. The joiner module 1305 also connects all sibling expressions based on their level or sublevel order using operators to return an updated statement; that is, it sequentially connects all expressions in sequence L using the operator ⊕ and returns the final transformed result e.

[0146] Since normalization is performed under the associative law, the transformed final result e has an equivalent operational result to the original sequence L. At this point, the normalization operation in this embodiment has been completed.

[0147] Another embodiment of the present invention is a compiler that iterates through all statements in a program at compile time to optimize loops, in order to achieve, as Figure 5 or Figure 6 The method shown is an optimized description of the loop of a neural network operator. Figure 14 The diagram shows a compiler for this embodiment. The compiler includes a judgment module 1401, a tagging module 1402, a replacement module 1403, a calculation module 1404, an insertion module 1405, and a generation module 1406. Each module transmits information through a bus 1407.

[0148] The judgment module 1401 is used to traverse all statements in the program, classify and discuss them according to their possible syntactic forms, and determine whether the expression of each statement is a loop invariant.

[0149] In this embodiment, an expression is considered cyclically invariant if the value constituting the expression satisfies any of the following conditions: the initial variable is a constant, including immutable program parameters or immediate values; the assignment of the variable in the expression occurs outside the loop; and the variable is used to only one fixed value, and the expression for that fixed value is also cyclically invariant. In other words, the judgment module 1401 determines whether the value constituting the expression satisfies any of the above conditions.

[0150] If the expression is not loop-invariant, it means that the initial variable associated with the expression will be changed within the loop and cannot be taken out of the loop. In this case, the marking module 1402 marks the expression as "cannot be taken out".

[0151] Regardless of whether the expression is a loop that does not change, the judgment module 1401 will then check whether the traversal has been completed. If not, it will continue to judge the next expression.

[0152] After traversing all statements, the replacement module 1403 replaces each expression not marked as "non-promotable" with different hoisted variables. The replacement module 1403 generates hoisted variables, rewrites the loop-invariant expressions in the statements with hoisted variables, and adds the hoisted variables of all identified loop invariants to the loop invariant set for subsequent processing.

[0153] The calculation module 1404 is used to calculate the cost of each expression not marked as "non-promotable". The calculation module 1404 calculates the cost of lifting out the expression e based on the cost function cost(e), which takes the expression e as input, calculates and returns the cost of lifting out e. The cost function cost(e) can be defined, for example, as follows:

[0154]

[0155] When e is a constant or variable, its cost is defined as 0; when e is a compound expression, its cost is recursively defined on its various subexpressions a, plus the cost of its operator ⊕. Different operators ⊕ have different costs depending on the clock cycles they require, and the cost function cost(e) can be arbitrarily adjusted to match the actual situation of different hardware platforms.

[0156] The judgment module 1401 then determines whether the cost value is greater than or equal to a threshold. In this embodiment, the threshold can be set to any number. By comparing it with a preset threshold, the module evaluates whether extracting the expression produces a sufficiently large benefit.

[0157] If the cost is not greater than or equal to the threshold, it indicates that the benefit is insufficient and it is not worthwhile to extract the expression. Therefore, the replacement module 1403 abandons the attempt to extract the expression, removes the extracted variable from the set of loop invariants, restores the replaced expression, and marks the expression as "cannot be extracted".

[0158] If the cost is greater than or equal to the threshold, the compiler considers it to have sufficient benefit, and if the expression hoisting has no side effects, the expression can be hoisted. At this time, the insertion module 1405 inserts the assignment statements of all hoisted variables before the loop and deletes the hoisted expression.

[0159] Module 1406 generates an intermediate representation based on the assignment statements of all outsourced variables. The generated intermediate representation is then used for... Figure 3 The target translator 309 performs translation and conversion, and then the runtime module 310 encapsulates it into an executable program.

[0160] Because efficient loop invariants are moved out of the loop, the computing device 201 can perform deep learning tasks more efficiently when executing the encapsulated executable program.

[0161] Another embodiment of the present invention is a computer-readable storage medium storing computer program code that optimizes the loops describing neural network operators during compilation. When the computer program code is run by a processor, it executes the methods described in the preceding embodiments. In some implementation scenarios, the integrated units described above can be implemented as software program modules. If implemented as software program modules and sold or used as independent products, the integrated units can be stored in a computer-readable memory. Based on this, when the solution of the present invention is embodied in the form of a software product (e.g., a computer-readable storage medium), the software product can be stored in a memory, which may include several instructions to cause a computer device (e.g., a personal computer, a server, or a network device) to execute some or all of the steps of the methods described in the embodiments of the present invention. The aforementioned memory may include, but is not limited to, various media capable of storing program code, such as USB flash drives, flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0162] This invention reduces redundant computations by moving redundant calculations that keep the value constant within a loop outside the loop, thereby generating higher-performance operators. Currently, a typical characteristic of various AI-specific processors is that many components are designed for running computationally intensive programs, exhibiting high computational parallelism. To fully utilize these hardware characteristics, the target platform compiler of this invention chooses to unroll inner loops. The latency of some computations is compensated for by the instruction pipeline; therefore, the redundant computations of unrolling inner loops do not affect runtime but rather reduce physical register pressure. Since AI-specific processors focus on parallel computing, branch optimization tasks need to be handled by the compiler. Therefore, the outsourcing of Boolean expressions and conditional branch statements requires careful handling, and the form of conditional expressions may be transformed when necessary to make the generated code more conducive to optimization by the underlying compiler.

[0163] In summary, this invention proposes a loop invariant extraction scheme for deep learning compilers. This scheme takes into account the characteristics of TVM intermediate representation and target platform compilers, and reduces redundant computation by extracting loop invariants, thereby improving the performance of TVM generation operators.

[0164] Depending on the application scenario, the electronic devices or apparatus of the present invention may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus of the present invention can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus of the present invention can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal computing. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present invention can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0165] It should be noted that, for the sake of brevity, this invention describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solution of this invention is not limited to the order of the described actions. Therefore, based on the disclosure or teachings of this invention, those skilled in the art will understand that some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this invention can be considered as optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions of this invention. In addition, depending on the solution, the description of some embodiments of this invention also has different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this invention can also refer to the relevant descriptions of other embodiments.

[0166] In terms of specific implementation, based on the disclosure and teachings of this invention, those skilled in the art will understand that the several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or device embodiments described above, this document has divided them based on logical functions, but in actual implementation, there may be other ways of division. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationship between different units or components, the connection discussed above in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection utilizing an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0167] In this invention, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same position or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this invention. Additionally, in some scenarios, multiple units in the embodiments of this invention may be integrated into one unit or each unit may exist physically independently.

[0168] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0169] The foregoing can be better understood in accordance with the following terms:

[0170] Clause A1. A method for optimizing a loop describing a neural network operator during compiler execution, comprising: traversing all statements in the program and determining whether the expression of each statement is loop-invariant, wherein the expression is associated with an initial variable; if not, marking the expression as "non-moveable"; replacing each expression not marked as "non-moveable" with a different hoisted variable; and inserting assignment statements for all hoisted variables before the loop.

[0171] Clause A2, the method according to Clause A1, wherein the expression is cyclically invariant when the initial variable is a constant.

[0172] Clause A3, the method according to Clause A1, wherein the expression is loop-invariant when the initial variable is assigned a value outside the loop.

[0173] Clause A4. The method according to Clause A1, wherein when the use of the initial variable is constant and the constant expression is cyclically invariant, the expression is cyclically invariant.

[0174] Clause A5. According to the method described in Clause A1, wherein when the statement is an assignment to the initial variable through the expression and the expression is marked as "not movable out", and the initial variable is not cyclically invariant, then in the marking step, the initial variable in the expression is marked as "not movable out".

[0175] Clause A6. According to the method described in Clause A1, where the expression is not loop-invariant when the statement is a loop statement, the initial variable in the expression is marked as "not movable out" in the marking step.

[0176] Clause A7, the method described in Clause A1, wherein the program is machine learning operator code generated by the TVM compiler.

[0177] Clause A8, the method described in Clause A7, wherein all statements in the program are traversed via an underlying intermediate representation pass function.

[0178] Clause A9. The method described in Clause A7 further includes: calling a cost function to calculate the cost value of each expression not marked as "non-promotable"; determining whether the cost value is greater than or equal to a threshold; if the cost value is greater than or equal to the threshold, performing the insertion step; and if the cost value is not greater than or equal to the threshold, marking the expression as "non-promotable".

[0179] Clause A10, the method according to Clause A9, wherein the threshold is 1.

[0180] Clause A11. The method described in Clause A1 further includes: generating an intermediate representation based on the assignment statements of all the external variables.

[0181] Clause A12. The method described in Clause A1 further includes: determining whether the statement satisfies the associative law; if the statement satisfies the associative law, setting the level of the expression according to the type of each expression; and sorting each expression based on the levels of all expressions in the statement; wherein the step of determining whether the expression of each statement is a loop invariant is based on the sorted expressions.

[0182] Clause A13, the method described in Clause A12 further includes: determining whether the statement includes a conditional expression and whether the condition is true or false corresponds to the same return value; and if the statement includes a conditional expression and whether the condition is true or false corresponds to the same return value, merging the judgment conditions with the same return value to update the conditional expression; wherein the setting step sets the level according to the type of the updated conditional expression.

[0183] Clause A14. The method according to Clause A12, wherein the setting step includes: determining whether the expression is a constant; and if the expression is a constant, setting the level to level one.

[0184] Clause A15, the method according to Clause A14, wherein the setting step includes: determining whether the expression contains a nested loop; and if the expression contains a nested loop, setting the level to level two.

[0185] Clause A16. The method according to Clause A15, wherein the nested loop comprises N loops, and the step of setting the level to the second level comprises: setting the child of the outermost loop as the first child; setting the child of the next outermost loop as the second child; and setting the child of each other loop in this manner until the child of the innermost loop is set as the Nth child.

[0186] Clause A17, the method according to Clause A16, wherein the setting step includes: setting the level to level three if the expression is not a constant and does not contain nested loops.

[0187] Clause A18, the method according to Clause A17, wherein the expression comprises multiple sub-expressions, the setting step comprising: setting the level to level three if any sub-expression is not a constant and does not contain nested loops.

[0188] Clause A19. The method according to Clause A17, wherein the expression comprises multiple sub-expressions, the setting step comprises: if all sub-expressions are constants and include one of nested loops, and any sub-expression includes a nested loop, setting the level to level two.

[0189] Clause A20, the method according to Clause A17, wherein in the sorting step, all expressions in the statement are arranged in the order of the first level, the second level, and the third level.

[0190] Clause A21. The method according to Clause A20, wherein in the sorting step, all expressions of the second level are arranged in the order of the first sub-level, the second sub-level, and so on, up to the Nth sub-level.

[0191] Clause A22, the method described in Clause A21, further includes: concatenating expressions of the same level or sublevel with operators to generate sibling expressions; concatenating all sibling expressions with operators based on the order of level or sublevel to return an updated statement; and compiling based on the updated statement.

[0192] Clause A23. A computer-readable storage medium having stored thereon computer program code that, when compiled, optimizes a loop describing a neural network operator, and when the computer program code is run by a processing device, performs the method described in any one of Clauses A1 to A22.

[0193] Item A24. A compiler that iterates through all statements in a program at compile time to optimize loops, comprising: a judgment module for determining whether an expression of each statement is loop-invariant, wherein the expression is associated with an initial variable; a marking module for marking the expression as "non-loop-invariant" if the expression of each statement is not loop-invariant; a replacement module for replacing each expression not marked as "non-loop-invariant" with a different looped variable; and an insertion module for inserting assignment statements of all looped variables before the loop.

[0194] The embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for optimizing loops describing neural network operators at compile time, comprising: Iterate through all statements in the program and determine whether the expression in each statement is a loop invariant, wherein the expression is associated with the initial variable; If not, mark the expression as "cannot be extracted"; Replace each expression that is not marked as "cannot be moved out" with a different moved-out variable; as well as Call the cost function to calculate the cost of each expression that is not marked as "cannot be extracted"; Determine whether the cost value is greater than or equal to the threshold; If the value of the loss is greater than or equal to the threshold, insert the assignment statements of all outgoing variables before the loop.

2. The method according to claim 1, wherein when the initial variable is a constant, the expression is cyclically invariant.

3. The method according to claim 1, wherein the expression is loop-invariant when the initial variable is assigned a value outside the loop.

4. The method according to claim 1, wherein when the use of the initial variable reaches only one constant value, and the expression of the constant value is also cyclically invariant, the expression is cyclically invariant.

5. The method according to claim 1, wherein when the statement is an assignment to the initial variable through the expression, and the expression is marked as "not movable out", and the initial variable is not cyclically invariant, then in the marking step, the initial variable in the expression is marked as "not movable out".

6. The method according to claim 1, wherein when the statement is a loop statement, and the expression is not loop-invariant, the initial variable in the expression is marked as "not movable out" in the marking step.

7. The method according to claim 1, wherein the program is machine learning operator code generated by a TVM compiler.

8. The method according to claim 7, wherein all statements in the program are traversed through an underlying intermediate representation pass function.

9. The method according to claim 7, further comprising: If the value of the expression is not greater than or equal to the threshold, the expression is marked as "cannot be extracted".

10. The method of claim 9, wherein the threshold is 1.

11. The method according to claim 1, further comprising: An intermediate representation is generated based on the assignment statements of all the external variables.

12. The method according to claim 1, further comprising: Determine whether the statement satisfies the associative law; If the statement satisfies the associative law, the level of the expression is set according to the type of each expression; as well as Sort each expression based on the rank of all expressions in the statement; The step of determining whether the expression of each statement is a loop invariant is based on the sorted expressions.

13. The method of claim 12, further comprising: Determine whether the statement includes a conditional expression and whether the condition is true or false corresponds to the same return value; as well as If the statement includes a conditional expression and the condition is true or false, the same return value is returned. Conditions with the same return value are merged to update the conditional expression. The setting step sets the level according to the type of the updated conditional expression.

14. The method of claim 12, wherein the setting step comprises: Determine whether the expression is a constant; as well as If the expression is a constant, the level is set to level one.

15. The method of claim 14, wherein the setting step comprises: Determine whether the expression contains nested loops; as well as If the expression includes nested loops, the level is set to level two.

16. The method of claim 15, wherein the nested loop comprises N loops, and the step of setting the level to the second level comprises: Set the child of the outermost loop as the first child; Define the child of the outermost loop as the second child; as well as Set the child elements of each other loop in this way, until the child element of the innermost loop is set as the Nth child element.

17. The method of claim 16, wherein the setting step comprises: If the expression is not a constant and does not contain nested loops, the level is set to level three.

18. The method of claim 17, wherein the expression comprises a plurality of sub-expressions, and the setting step comprises: If any subexpression is not a constant and does not contain nested loops, the level is set to level three.

19. The method of claim 17, wherein the expression comprises a plurality of sub-expressions, and the setting step comprises: If all subexpressions are constants and include one of nested loops, and any subexpression includes a nested loop, then the level is set to Level 2.

20. The method of claim 17, wherein in the sorting step, all expressions in the statement are arranged in the order of the first level, the second level, and the third level.

21. The method of claim 20, wherein in the sorting step, all expressions of the second level are arranged in the order of the sublevel being the first sublevel, the second sublevel, and so on up to the Nth sublevel.

22. The method of claim 21, further comprising: Connect expressions of the same level or sublevel with operators to generate sibling expressions; Connect all sibling expressions with operators based on their level or child order to return the updated statement; as well as Compile based on the updated statements.

23. A computer-readable storage medium storing computer program code that, when compiled, optimizes loops describing neural network operators, and when the computer program code is run by a processing device, performs the method according to any one of claims 1 to 22.

24. A compiler that iterates through all statements in a program at compile time to optimize loops, comprising: The judgment module is used to determine whether the expression in each statement is loop-invariant, wherein the expression is associated with the initial variable; A flag module is used to mark an expression as "not takeout" if the expression in each statement is not a loop invariant. The computation module calculates the cost of each expression that is not marked as "non-extractable" based on the cost function; The judgment module is also used to determine whether the cost value is greater than or equal to the threshold. The replacement module is used to replace each expression that is not marked as "cannot be moved out" with different outward variables; as well as An insertion module is used to insert all assignment statements of the outgoing variables before the loop when the value of the substitution is greater than or equal to the threshold.

Images