Operator fusion method and apparatus, electronic device, and storage medium

Abstract

An operator fusion method and device, electronic equipment and storage medium. The operator fusion method comprises: writing a first graph layer representation corresponding to at least one first operator and a second graph layer representation corresponding to at least one second operator into a target kernel function, wherein the first operator and the second operator are called by a device function in the target kernel function; obtaining a first intermediate representation corresponding to the at least one first operator; obtaining a second intermediate representation corresponding to the at least one second operator; and based on the first intermediate representation and the second intermediate representation, fusing the at least one first operator and the at least one second operator to obtain a fused target kernel function. The operator fusion method can support operator fusion of any graph topology, different categories and different implementation modes (automatic generation or handwriting), thereby greatly reducing the overhead of kernel function startup, intermediate temporary buffer, data relocation and other computing resources, improving computing efficiency and enhancing hardware performance.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to an operator fusion method and apparatus, an electronic device, and a storage medium. BACKGROUND

[0002] In the field of artificial intelligence (AI), deep neural networks (DNNs) have become the basis of the most advanced technologies and the core driving force of many applications. As the application scenarios of deep learning continue to generalize, deep learning computing tasks also need to be deployed on deep neural networks. Due to the deep layers and complex structure of deep learning networks, the number of generated operators is large, which brings huge consumption of computing resources and time.

[0003] Operator fusion is a method for improving the execution efficiency of deep learning computing tasks. Before executing a computing task, multiple operators in a neural network that meet certain conditions or rules can be fused to form a fused operator. By fusing multiple operators, memory reuse can be achieved, and the utilization of computing resources such as central processing units (CPUs), graphics processing units (GPUs), compilers, and registers can be improved. SUMMARY

[0004] At least one embodiment of the present disclosure provides an operator fusion method, which comprises: writing a first graph layer representation corresponding to at least one first operator and a second graph layer representation corresponding to at least one second operator into a target kernel function, wherein the at least one first operator and the at least one second operator are called by a device function in the target kernel function; obtaining a first intermediate representation corresponding to the at least one first operator; obtaining a second intermediate representation corresponding to the at least one second operator; and fusing the at least one first operator and the at least one second operator based on the first intermediate representation and the second intermediate representation to obtain a fused target kernel function, wherein the device function is a sub-function of the target kernel function.

[0005] For example, in the operator fusion method provided by at least one embodiment of the present disclosure, the at least one first operator includes a fused operator, a non-fused operator, or a handwritten operator, and the non-fused operator includes an automatically generated operator.

[0006] For example, in the operator fusion method provided in at least one embodiment of the present disclosure, the at least one second operator includes a fusion operator, a non-fusion operator, or a handwritten operator, and the non-fusion operator includes an automatically generated operator.

[0007] For example, in the operator fusion method provided in at least one embodiment of the present disclosure, in response to the at least one first operator including a fusion operator, before the first graph layer representation corresponding to the at least one first operator and the second graph layer representation corresponding to the at least one second operator are written into the target kernel function, the method further includes: performing first-level fusion on at least one third operator by pattern matching to obtain the at least one first operator, wherein the at least one third operator includes an automatically generated operator.

[0008] For example, in the operator fusion method provided in at least one embodiment of the present disclosure, writing the first graph layer representation corresponding to the at least one first operator and the second graph layer representation corresponding to the at least one second operator into the target kernel function includes: rewriting the first graph layer representation to call the at least one first operator by the device function manner; rewriting the second graph layer representation to call the at least one second operator by the device function manner; calling a first device function corresponding to the at least one first operator and a second device function corresponding to the at least one second operator in the target kernel function, and performing second-level fusion on the at least one first operator and the at least one second operator by the inline function manner.

[0009] For example, in the operator fusion method provided in at least one embodiment of the present disclosure, before the second device function corresponding to the at least one second operator is called in the target kernel function, the method further includes: performing first function declaration on the at least one second operator.

[0010] For example, in the operator fusion method provided in at least one embodiment of the present disclosure, the input of the second device function includes at least one output of the first device function, or at least one input of the first device function includes the output of the second device function.

[0011] For example, in the operator fusion method provided in at least one embodiment of the present disclosure, obtaining the first intermediate representation corresponding to the at least one first operator includes: assembling the first device function into the first intermediate representation.

[0012] For example, in the operator fusion method provided in at least one embodiment of the present disclosure, obtaining the second intermediate representation corresponding to the at least one second operator includes: assembling the second device function into the second intermediate representation.

[0013] For example, in the operator fusion method provided in at least one embodiment of the present disclosure, assembling the second device function into the second intermediate representation includes: setting the attribute of the second device function so that the function inlining can be performed; and performing the second function declaration and the second function call on the at least one second operator to obtain the second intermediate representation.

[0014] For example, in the operator fusion method provided in at least one embodiment of the present disclosure, based on the first intermediate representation and the second intermediate representation, the at least one first operator and the at least one second operator are fused to obtain the fused target kernel function, including: rewriting the second intermediate representation; and adding the rewritten second intermediate representation to the first intermediate representation by the inline function method to obtain the fused target kernel function, so as to perform the third-level fusion on the at least one first operator and the at least one second operator.

[0015] For example, in the operator fusion method provided in at least one embodiment of the present disclosure, rewriting the second intermediate representation includes: linking the second intermediate representation to replace the second function declaration of the at least one second operator in the second intermediate representation with the function definition; and performing the function inlining on the linked second intermediate representation to inline the second function call of the at least one second operator in the second intermediate representation into the corresponding function implementation.

[0016] At least one embodiment of the present disclosure also provides an operator fusion device, which includes: a writing module configured to write a first graph layer representation corresponding to at least one first operator and a second graph layer representation corresponding to at least one second operator into a target kernel function, wherein the at least one first operator and the at least one second operator are called by a device function in the target kernel function; an obtaining module configured to obtain a first intermediate representation corresponding to the at least one first operator, and obtain a second intermediate representation corresponding to the at least one second operator; and a fusion module configured to fuse the at least one first operator and the at least one second operator based on the first intermediate representation and the second intermediate representation to obtain a fused target kernel function.

[0017] For example, in the operator fusion device provided in at least one embodiment of the present disclosure, in response to the at least one first operator including a fusion operator, the fusion module is further configured to perform the first-level fusion on at least one third operator by the pattern matching method to obtain the at least one first operator, wherein the at least one third operator includes an automatically generated operator.

[0018] For example, in the operator fusion device provided in at least one embodiment of the present disclosure, the writing module is further configured to: rewrite the first graph representation to call the at least one first operator in the device function manner; rewrite the second graph representation to call the at least one second operator in the device function manner; and call the first device function corresponding to the at least one first operator and the second device function corresponding to the at least one second operator in the target kernel function, to perform second-level fusion on the at least one first operator and the at least one second operator.

[0019] For example, in the operator fusion device provided in at least one embodiment of the present disclosure, the fusion module is further configured to: rewrite the second intermediate representation; and add the rewritten second intermediate representation to the first intermediate representation in the inline function manner, to obtain the fused target kernel function, to perform third-level fusion on the at least one first operator and the at least one second operator.

[0020] At least one embodiment of the present disclosure further provides an electronic device. The electronic device includes a processor, a memory including one or more computer program modules, wherein the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules are used to implement the operator fusion method provided in any embodiment of the present disclosure.

[0021] At least one embodiment of the present disclosure further provides a storage medium storing non-transitory computer readable instructions, which, when executed by a computer, implement the operator fusion method provided in any embodiment of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings described in the following description only relate to some embodiments of the present disclosure, but not limit the present disclosure.

[0023] Figure 1 A schematic diagram of an operator fusion manner;

[0024] Figure 2 An exemplary flowchart of the operator fusion method provided in at least one embodiment of the present disclosure;

[0025] Figure 3 A schematic diagram of one example of the operator fusion method provided in at least one embodiment of the present disclosure;

[0026] Figure 4 A schematic diagram of one example of the operator fusion method provided in at least one embodiment of the present disclosure; Figure 2 An exemplary flowchart of one example of step S10 in the method;

[0027] Figure 5 A schematic diagram of another example of the operator fusion method provided for at least one embodiment of the present disclosure;

[0028] Figure 6 For Figure 2 An exemplary flowchart of one example of step S40 in the method;

[0029] Figure 7 A schematic diagram of another example of the operator fusion method provided for at least one embodiment of the present disclosure;

[0030] Figure 8 An exemplary flowchart of another example of the operator fusion method provided for at least one embodiment of the present disclosure;

[0031] Figure 9 A schematic block diagram of an operator fusion apparatus provided for at least one embodiment of the present disclosure;

[0032] Figure 10 A schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure;

[0033] Figure 11 A schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure; and

[0034] Figure 12 A schematic diagram of a storage medium provided for at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] In order to make the objects, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions of the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure. Based on the described embodiments of the present disclosure, all other embodiments obtained by those of ordinary skill in the art without any creative work fall within the scope of protection of the present disclosure.

[0036] Unless otherwise defined, technical terms or scientific terms used in the present disclosure shall have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. The terms "first", "second", and similar terms do not denote any order, quantity, or importance, but are used to distinguish different components. Also, the terms "one", "a", or "the" do not denote a quantity of particular mentioned items, but indicate the presence of at least one of the items. The terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process or method that comprises several steps does not require all of the steps to be present. The terms "connected", "coupled", or any other terminology of a similar meaning are not limited to a direct connection or coupling, but can include an indirect connection or coupling such as through an intermediate circuit. The terms "upper", "lower", "left", "right", and the like are used only to indicate relative positional relationship, and when the absolute position of the described object is changed, the relative positional relationship can also be changed accordingly.

[0037] The present disclosure is described below through several specific embodiments. In order to keep the following description of the embodiments of the present disclosure clear and concise, the detailed description of known functions and known components can be omitted. When any component of the embodiments of the present disclosure appears in more than one figure, the component is denoted by the same or similar reference signs in each figure.

[0038] A deep learning computing task is composed of multiple computing units, each of which can be referred to as an operator (Op). In a neural network model, an operator corresponds to the computing logic of a layer or a node. For example, a convolution computing task in a convolution layer can be an operator; a weight summation computing task in a fully-connected layer (FC layer) can also be an operator. In the implementation of a neural network, an operator can be implemented in a software manner (for example, through a computer program) or in a hardware manner (for example, through a circuit).

[0039] Deep learning computing tasks need to be deployed on different computing devices and hardware architectures, and the actual deployment or training scene also has more aggressive requirements for performance, such as customizing the computing code for hardware characteristics. These requirements have been difficult to meet in general AI frameworks. The compilation and optimization of deep learning computing tasks using an AI compiler can meet the hardware performance requirements of the actual deployment or training scene of the computing task. The AI compiler translates and optimizes the current deep learning computing task through one or more layers of intermediate representation (IR), and finally converts it into an executable file on the target hardware. For example, specifically, the front end of the AI compiler mainly operates based on the graph intermediate representation (Graph IR). The graph IR saves the knowledge / information of the compilation process in the graph, and the corresponding operators are described by operating on the objects (nodes, edges, lists, and trees) in the graph. Further, the graph IR can be unfolded downward to the middle end of the AI compiler, and the operators corresponding to the graph IR can be implemented as tensor intermediate representation (Tensor IR); that is, the graph IR can be converted into a tensor IR that is closer to the executable code of the target hardware. Further, the tensor IR is unfolded downward to the back end of the AI compiler, and the operators corresponding to the tensor IR can be implemented as low-level virtual machine intermediate representation (Low Level Virtual Machine, LLVM IR); the back end of the AI compiler optimizes the LLVM IR, and can generate an executable file that can be deployed on the target hardware for execution.

[0040] Because deep learning computing tasks are often programmed and expressed in the form of DSL (Domain Specific Language) in AI frameworks, this itself makes the optimization and execution of deep learning computing tasks naturally conform to the compilation and optimization process of traditional computer languages. The AI compiler can automatically generate operators to complete the computing task, but in some scenarios, the automatically generated operators may not achieve the ultimate performance of the handwritten operators. Therefore, the AI compiler will also call handwritten operators by calling operator libraries (such as cudnn, mkldnn, etc.) or directly calling operators written by programmers, so as to implement complex computing tasks.

[0041] To improve the computing efficiency, the optimization of the AI compiler can use operator fusion technology; operator fusion mainly occurs at the graph layer, and the kernel layer can also use the operator fusion strategy to provide fused operators to the AI framework. For example, the graph-level operator fusion technology uses pattern matching to decide the fusion of subgraphs, and the pattern matching rule is to decide whether the code generation of the fused operator is supported. However, the pattern matching method can only automatically fuse the generated operators; when the AI compiler calls the operator library or other handwritten operators, the handwritten operators need to be called through a separate kernel function, that is, the pattern matching does not support the fusion of handwritten operators and automatically generated operators.

[0042] In addition, the fusion mode supported by the pattern matching method is also limited, that is, the pattern matching does not support the fusion between some categories of automatically generated operators. For example, when using the pattern matching method for operator fusion, the fusion mode needs to be defined first (that is, the target operator to be fused is defined), and then the fusion algorithm corresponding to the fusion mode can be adapted. For example, the automatically generated operators include different categories, such as complex operator categories (such as conv operators, relu operators, etc.), reduce categories (such as reduce_add operators, etc.), broadcast categories (such as broadcast_add operators, etc.), and elementwise categories (such as elementwise_add operators, etc.); among the above four example categories, three fusion modes can be matched: the operators of the broadcast category and the elementwise category can be fused by pattern matching, the operators of the complex operator category and the broadcast category and the elementwise category can be fused by pattern matching, and the operators of the reduce category and the broadcast category and the elementwise category can be fused by pattern matching; for the above four example categories, the pattern matching may not support other fusion modes in addition to the above three fusion modes.

[0043] For example, for a neural network model with a relatively complex functional structure, it may be necessary to define tens of fusion modes or even hundreds of fusion modes; in addition, when the hardware platform running the neural network changes, the fusion mode will also change, and the change of the fusion mode also leads to the need to redevelop and compile the corresponding fusion algorithm, thereby making the operator fusion step occupy more computing resources and debugging costs.

[0044] Figure 1 A schematic diagram of an operator fusion method.

[0045] For example, as shown in FIG. 1, the operator fusion method includes the following steps: Figure 1As shown in Example 1-1, operators A, C, D, and E are automatically generated operators, while operator B is a handwritten operator. The output of operator A is the input of operator B, the output of operator B is the input of operator C, the output of operator C is the input of operator D, and the output of operator D is the input of operator E. Since pattern matching does not support the fusion of automatically generated operators (A, C, D, or E) and handwritten operator B, after pattern matching fusion, operators A and B are each grouped independently. While operators C and D belong to fusion patterns that can be performed using pattern matching, operator E does not belong to fusion patterns that can be performed using pattern matching with operators C and D. After pattern matching fusion, as shown... Figure 1 As shown by the dashed lines used for operator grouping, operators C and D are fused into one group, while operator E is in a separate group. Therefore, the pattern matching fusion in Example 1-1 cannot achieve complete fusion of operator AE.

[0046] For example, such as Figure 1 As shown in Example 1-2, operators A, C, D, and E are automatically generated operators, while operator B is a handwritten operator. The output of operator A is the input of operators B and C, the outputs of operators B and C are the input of operator D, and the output of operator D is the input of operator E. Since pattern matching does not support the fusion of automatically generated operators (A, C, D, or E) and handwritten operator B, and the output of operator A requires simultaneous input of operators B and C (i.e., operators A and C cannot be fused using pattern matching), and the outputs of operators B and C require simultaneous input of operator D (i.e., operators C and D cannot be fused using pattern matching), after pattern matching fusion, as shown... Figure 1 As shown by the dashed lines used for operator grouping, only operators D and E are fused into one group, while operators A, B, and C are each in an independent group. Therefore, the pattern matching fusion in Example 1-2 also fails to achieve complete fusion of operator AE.

[0047] At least one embodiment of this disclosure provides an operator fusion method, which includes: writing a first layer representation corresponding to at least one first operator and a second layer representation corresponding to at least one second operator into a target kernel function, wherein the first and second operators are invoked in the target kernel function via device functions; obtaining a first intermediate representation corresponding to at least one first operator; obtaining a second intermediate representation corresponding to at least one second operator; and fusing the at least one first operator and the at least one second operator based on the first and second intermediate representations to obtain a fused target kernel function.

[0048] At least one embodiment of this disclosure also provides an operator fusion apparatus, an electronic device, and a storage medium for implementing the operator fusion method of the above embodiments.

[0049] The method, device, electronic equipment and storage medium provided by at least one embodiment of the present disclosure can support the fusion of operators of different topologies, different categories and different implementation manners (automatically generated or handwritten) by calling the device function in the target kernel function and combining the first operator and the second operator in the form of the intermediate representation, thereby greatly reducing the overhead of computing resources such as kernel launch, intermediate temporary buffer and data movement, improving the computing efficiency and enhancing the hardware performance of the AI compiler.

[0050] In the following, at least one embodiment of the present disclosure will be explained in detail with reference to the accompanying drawings. It should be noted that the same reference numerals in different drawings will be used to refer to the same elements that have been described.

[0051] Figure 2 An exemplary flowchart of the operator fusion method provided by at least one embodiment of the present disclosure.

[0052] For example, as shown in Figure 2 The operator fusion method provided by at least one embodiment of the present disclosure can include the following steps S10-S40.

[0053] Step S10: writing a first graph layer representation corresponding to at least one first operator and a second graph layer representation corresponding to at least one second operator into a target kernel function;

[0054] Step S20: obtaining a first intermediate representation corresponding to the at least one first operator;

[0055] Step S30: obtaining a second intermediate representation corresponding to the at least one second operator;

[0056] Step S40: fusing the at least one first operator and the at least one second operator based on the first intermediate representation and the second intermediate representation to obtain a fused target kernel function.

[0057] For example, the first operator or the second operator can include different types corresponding to different types of deep learning computing models or different execution processes of the same computing model; the type of the operator may, for example, include the operator types after the deep learning network Resnet, Bert and Yolo network are disassembled, such as tensor type, vector type or scalar type, etc. For example, the operator of the tensor type can be used to execute the convolution algorithm, the batch normalization (BN) algorithm, the pooling algorithm; the operator of the vector type can be used to execute the matrix multiplication operation; the operator of the scalar type can be used to execute the activation function, the cross-entropy, the loss function, etc. The embodiments of the present disclosure do not limit this.

[0058] For example, in step S10, the number of the first operators can be one or more, which can include a fusion operator, a non-fusion operator or a handwritten operator, and the non-fusion operator can include an automatically generated operator; the number of the second operators can be one or more, which can include a fusion operator, a non-fusion operator or a handwritten operator, and the non-fusion operator can include an automatically generated operator. For example, the first operator and the second operator respectively correspond to a kernel function that can be used to perform a specific operation; the source program corresponding to the kernel function can be written by a computer language, such as a high-level language, an assembly language or a machine language, and embodiments of the present disclosure do not limit this. In some examples, the kernel functions corresponding to the first operator and the second operator are processed and generated by an AI compiler; and the CPU calls the kernel functions generated by the AI compiler and provides the kernel functions and input data to the device end, and then the device end executes the kernel functions to process the input data; the CPU controls the device end to copy the processing result back to the host end, thereby completing the computing task.

[0059] In some examples, in response to the at least one first operator including a fusion operator, before step S10, the operator fusion method can further include: performing first-level fusion on at least one third operator by a mode matching manner to obtain the at least one first operator. For example, the number of the third operators can be one or more, which can include an automatically generated operator.

[0060] For example, the first-level fusion can be, for example Figure 1 the mode matching fusion shown in FIG. 1, the third operators include operator A, operator C, operator D and operator E, and the second operator includes operator B. Taking example 1-1 of FIG. 1 as an example, operator A, operator C, operator D and operator E are automatically generated operators, and operator B is a handwritten operator. For example, operator A, operator C, operator D and operator E are fused by the mode matching manner to obtain the first operator including operator A, operator C+D and operator E, wherein operator C+D is a fusion operator, and operator A and operator E are non-fusion operators. Figure 1 For example, the first-level fusion can be, for example Figure 1 the mode matching fusion shown in FIG. 1, the third operators include operator A, operator C, operator D and operator E, and the second operator includes operator B. Taking example 1-1 of FIG. 1 as an example, operator A, operator C, operator D and operator E are automatically generated operators, and operator B is a handwritten operator. For example, operator A, operator C, operator D and operator E are fused by the mode matching manner to obtain the first operator including operator A, operator C+D and operator E, wherein operator C+D is a fusion operator, and operator A and operator E are non-fusion operators. Figure 1 the description of FIG. 1, and details are not described herein.

[0061] It should be noted that the first operator, the second operator and the third operator can also be selected according to actual needs, and embodiments of the present disclosure do not limit this.

[0062] For example, in step S10, the first operator and the second operator are invoked by device function in the target kernel function. In some examples, the device function is a sub-function of the target kernel function. For example, the first graph representation and the second graph representation are written into the target kernel function, and then are invoked as sub-functions in the target kernel function; or the first graph representation and the second graph representation are assembled down to the tensor level, and then the device function (for example, tensor IR) of the tensor level is invoked as a sub-function in the target kernel function.

[0063] For example, the first graph representation and the second graph representation belong to the intermediate representation of the graph level (i.e., graph IR); after being written into the target kernel function, the first graph representation and the second graph representation are assembled down from the graph level to the intermediate representation of the tensor level (i.e., tensor IR). For example, the tensor IR corresponding to the first operator and the second operator is invoked as a device function of the tensor level in the target kernel function. For example, the tensor IR corresponding to the first operator is the first device function, and the tensor IR corresponding to the second operator is the second device function; in step S10, the second-level fusion of the first operator and the second operator is implemented by invoking the first device function and the second device function in the target kernel function.

[0064] For example, in steps S20 and S30, the first device function and the second device function are assembled down from the tensor level to the LLVM IR. In some examples, step S20 includes assembling the first device function into a first intermediate representation, and step S30 includes assembling the second device function into a second intermediate representation, where the first intermediate representation and the second intermediate representation are both LLVM IR.

[0065] For example, in step S40, the first intermediate representation and the second intermediate representation are further rewritten and fused at the LLVM IR level to implement the third-level fusion of the first operator and the second operator, so as to obtain the fused target kernel function; that is, through the first-level fusion (for example, fusion of the graph IR), the second-level fusion (for example, fusion of the tensor IR), and the third-level fusion (for example, fusion of the LLVM IR), the fusion of the first operator and the second operator is implemented, so that the fused operator can be obtained.

[0066] In the operator fusion method provided in at least one embodiment of the present disclosure, the fusion operations of the first operator and the second operator in steps S10-S40 are all implemented in one target kernel function, without the need for additional compilation outside the target kernel function, and each intermediate result of the entire calculation process can be written on a register or other shared memory, thereby greatly saving the computing resources and improving the computing efficiency.

[0067] In some examples, further, the fused target kernel function can be further optimized (e.g., llvmpass optimization) and hardware resource allocation is performed, so as to generate an executable file (e.g., machine executable code) which can be deployed on the target hardware to execute. Further, during the process of running the generated executable file, i.e., during the runtime, the kernel function corresponding to the first operator and the kernel function corresponding to the second operator share the same memory address (e.g., usharp ID), and through performing corresponding data reading and writing, the running result of the fused operator can be finally obtained.

[0068] The operator fusion method provided by at least one embodiment of the present disclosure can further perform operator fusion at a level lower than the graph level, such as the tensor level or the LLVM IR level, compared with the operator fusion manner of the graph level, so that operator fusion of different categories and different implementation manners can be implemented. For example, when the first operator is a fused or non-fused automatically generated operator and the second operator is a handwritten operator, the fusion of the handwritten operator and the automatically generated operator can be implemented through the operations in steps S10-S40; or when the first operator or the second operator includes operators of different categories, the fusion of operators of different categories can be implemented through the operations in steps S10-S40.

[0069] It should be noted that the graph level representation, device function, intermediate representation, etc. corresponding to the first operator and the second operator are not limited to the graph IR, the tensor IR and the LLVM IR, and other specific implementation forms can also be selected according to actual needs, and embodiments of the present disclosure do not limit this.

[0070] Figure 3 A schematic diagram of one example of the operator fusion method provided by at least one embodiment of the present disclosure is shown in FIG. 2. Figure 3 A specific example of the operator fusion method is shown in FIG. 2. Figure 2

[0071] For example, as shown in FIG. 2, example 2-1 corresponds to example 1-1 of FIG. 1, and example 2-2 corresponds to example 1-2 of FIG. 1, that is, the operators A, C, D and E are automatically generated operators, and the operator B is a handwritten operator, and the graph topology type (i.e., the input-output relationship between operators, etc.) of the operators is also consistent with that in example 1, which will not be repeated here. Figure 3 Figure 1 Figure 1 Figure 1

[0072] For example, the first-level fusion in example 2-1 and example 2-2 can be the pattern matching fusion in example 1-1, and the first-level fusion can obtain the result as shown in FIG. 1. Figure 1 Figure 3 ​​​​​​The plurality of groups shown: for example 2-1, the fusion operator C+D, the non-fusion operators A, E are all first operators, and the handwritten operator B is a second operator; for example 2-2, the fusion operator D+E, the non-fusion operators A, C are all first operators, and the handwritten operator B is a second operator.

[0073] For example, as shown in Figure 3 In step S10 of Figure 2 , 3 first graph representations respectively corresponding to 3 first operators (C+D, A, E in example 2-1, or D+E, A, C in example 2-2) and a second graph representation corresponding to the second operator B are written into the target kernel function; by calling first device functions and second device functions respectively corresponding to the first operator and the second operator in the target kernel function, the second-level fusion of the first operator and the second operator is realized. In Figure 2 steps S20 and S30, the first device function and the second device function are respectively assembled into a first intermediate representation and a second intermediate representation. In Figure 2 step S40, based on the first intermediate representation and the second intermediate representation, the third-level fusion of the first operator and the second operator is realized to obtain the fused target kernel function.

[0074] For example, in example 2-1, even if the operators (A, C, D, or E) are automatically generated operators and the operator B is a handwritten operator, and the category of operator E and operators C, D does not belong to the fusion mode that can be matched by pattern matching, after the operation of steps S10-S40, complete fusion of operators A-E can be realized.

[0075] For example, in example 2-2, even if the operators (A, C, D, or E) are automatically generated operators and the operator B is a handwritten operator, and there is an input-output relationship between the operators that cannot be fused by pattern matching, after the operation of steps S10-S40, complete fusion of operators A-E can be realized.

[0076] It should be noted that the first-level fusion, second-level fusion, and third-level fusion of operators A-E as shown in Figure 3 are only an example, and embodiments of the present disclosure do not limit the number and type of operators, the type of operator graph topology, the specific fusion mode (for example, the first-level fusion can be skipped, and the second-level fusion and the third-level fusion can be directly performed), and the like.

[0077] The operator fusion method provided by at least one embodiment of the present disclosure can support fusion of operators of any graph topology, different categories and different implementation manners (automatic generation or handwriting) by calling device functions in a target kernel function and combining the form of intermediate representation to fuse the first operator and the second operator, thereby greatly reducing the overhead of calculation resources such as kernel function startup, intermediate temporary buffer and data movement, improving the calculation efficiency, and improving the hardware performance of the AI compiler.

[0078] Figure 4 For Figure 2 An example of the example flowchart of step S10.

[0079] For example, as Figure 4 shown, Figure 2 Step S10 can further include steps S101-S103.

[0080] Step S101: rewriting the first graph representation to call at least one first operator by means of a device function;

[0081] Step S102: rewriting the second graph representation to call at least one second operator by means of a device function;

[0082] Step S103: calling the first device function corresponding to the at least one first operator and the second device function corresponding to the at least one second operator in the target kernel function, and performing second-level fusion of the at least one first operator and the at least one second operator by means of a function inline.

[0083] For example, in steps S101 and S102, the first graph representation and the second graph representation are assembled from the graph layer to the tensor IR, and the target kernel function can call the first operator and the second operator by means of a device function; in step S103, when the first device function and the second device function are called in the target kernel function, the second-level fusion can be implemented by means of a function inline. For example, the function inline (also sometimes referred to as an online function or a compile-time expansion function) is a programming language structure used to suggest the AI compiler to perform inline expansion (or online expansion) on some target functions, that is, to suggest the compiler to insert and replace the function body of the target function in each context where the target function is called, thereby saving the additional overhead caused by each call to the target function, and the embodiments of the present disclosure do not limit the specific form of the function inline.

[0084] For example, when the first device function and the second device function are called in the target kernel function, the input of the second device function can include at least one output of the first device function, or at least one input of the first device function can include at least one output of the second device function, so as to realize the second-level fusion of the at least one first operator and the at least one second operator.

[0085] In some examples, before the second device function corresponding to the second operator is called in the target kernel function, the operator fusion method provided by at least one embodiment of the present disclosure further includes: performing a first function declaration on the at least one second operator. For example, when the second operator is a handwritten operator, the code of the handwritten operator is not included in the automatically generated code, and the kernel function corresponding to the second operator needs to be called from the outside to be fused with the first operator (i.e., the automatically generated operator). When the kernel function corresponding to the second operator is called from the outside, first, the kernel function corresponding to the second operator needs to be declared as an external function (e.g., called a first function declaration), and then the external function is called at a specific position of the target kernel function (e.g., called a first function call), so as to realize the calling of the second device function.

[0086] Figure 5 Another example of the operator fusion method provided by at least one embodiment of the present disclosure is shown in the schematic diagram. For example, Figure 5 For Figure 4 A specific example of the operator fusion method is shown.

[0087] For example, as Figure 5 shown, taking example 2-2 in Figure 3 , in steps S101 and S102 of Figure 4 , the first graph layer representation of the first operator A, C, D+E is assembled downward to the first device function, and the second graph layer representation of the second operator B is assembled downward to the second device function, and the obtained target kernel function is fused_kernel(). In the target kernel function fused_kernel(), the first device function fused_A(), fused_C() and fused_D_E() are called, and the second device function fused_B() is called.

[0088] For example, as Figure 5As shown, the input of the target kernel function fused_kernel() is input_tensor; since the second operator B is a handwritten operator, in the target kernel function, first, the first function declaration of the second operator B is needed, that is, func_extern_B_decl(arg_out, arg_in), the arg_out part is the output of the second device function, and the arg_in part is the input of the second device function. Further, the first device function fused_A() is called to input input_tensor into the operator A; then the first function call of the second operator B is performed, that is, Call_extern_B(B_out, A_out), and the output A_out of the first device function fused_A() is taken as the input of the second device function fused_B(), to obtain the output B_out of the second device function fused_B(); then the first device function fused_C() is called, and the output A_out of the first device function fused_A() is taken as the input of the first device function fused_C(). Further, the first device function fused_D_E() is called, and the output C_out of the first device function fused_C() and the output B_out of the second device function fused_B() are taken as the inputs of the first device function fused_D_E(), to obtain the output DE_out of the first device function fused_D_E(), thereby realizing the automatic generation of the fused operator code, and realizing the second-level fusion of the operators A-E.

[0089] For example, as shown in Figure 5 In the process of calling multiple first device functions and second device functions in the target kernel function fused_kernel(), the inline function mode (for example, compute inline, compute at, etc.) is mostly used, and the specific implementation mode of the inline function can refer to the description in the art, which will not be repeated here.

[0090] In some examples, Figure 2 The step S20 can further include: assembling the first device function into a first intermediate representation, Figure 2 The step S30 can further include: assembling the second device function into a second intermediate representation, that is, the first device function and the second device function are respectively assembled from the tensor layer downward to the LLVM IR.

[0091] In some examples, when the second operator is a handwritten operator, a kernel function corresponding to the second operator needs to be called from outside to obtain the second device function, that is, the second device function includes the attribute of the external function, and cannot be inlined at the LLVM IR level as the device function corresponding to the automatically generated operator. Therefore, in the process of assembling the second device function into the second intermediate representation, the attribute of the second device function needs to be set first so that it can be inlined, for example, the attribute of the second device function can be set to always inline, and the like, and embodiments of the present disclosure do not limit this; further, at the LLVM IR level, the second operator also needs to be declared and called a second function to inline the external function into the corresponding LLVM IR, thereby obtaining the second intermediate representation.

[0092] Figure 6 For Figure 2 An example of the example flowchart of step S40.

[0093] For example, as Figure 6 shown, Figure 2 Step S40 can further include steps S401-S402:

[0094] Step S401: rewriting the second intermediate representation;

[0095] Step S402: adding the rewritten second intermediate representation to the first intermediate representation by inline function to obtain the fused target kernel function, to perform third-level fusion on the at least one first operator and the at least one second operator.

[0096] In some examples, step S401 can include: linking the second intermediate representation to replace the second function declaration of the second operator in the second intermediate representation with a function definition; and inlining the linked second intermediate representation to inline the second function call of the second operator in the second intermediate representation into the corresponding function implementation.

[0097] For example, since the kernel function corresponding to the second operator needs to be called externally when the second operator is a handwritten operator, the second function declaration and the second function call are made to the second operator at the LLVM IR level, and there is some redundant code about the external function definition, calling and other steps in the preliminarily generated second intermediate representation. Therefore, in step S401, first, the second intermediate representation is linked (llvm-link), and the second function declaration in the code is replaced by the corresponding function definition; then the function inline is performed on the linked second intermediate representation, and the second function call in the code is inlined into the corresponding function implementation. Specifically, the llvm-link and the function inline can be simply illustrated by the following pseudo codes (1)-(3): (1)

[0099] Func A(int a, int b) / / part that needs llvm-link

[0100] Func B(int a, int b) {

[0101] A(a, b);

[0102] } (2)

[0104] / / After llvm-link (1), it becomes:

[0105] Func A(int a, int b) { b = a;}

[0106] Func B(int a, int b) {

[0107] A(a, b); / / part that needs function inline

[0108] } (3)

[0110] / / After function inline (2), it becomes:

[0111] Func A(int a, int b) { b = a;}

[0112] Func B(int a, int b) {

[0113] b = a;

[0114] }

[0115] For example, according to the above pseudo code (1)-(3), after the code (1) is linked (llvm-link), the second function declaration Func A(int a, int b) is replaced by the corresponding function definition Func A(int a, int b) {b=a;}; after the code (2) is function inlined (function inline), the second function call A(a, b) is inlined as the corresponding function implementation b=a.

[0116] It should be noted that the above pseudo code (1)-(3) is only an example for explaining the linking (llvm-link) and the function inlining (function inline), and in the operator fusion method provided by at least one embodiment of the present disclosure, the specific code of the linking and the function inlining needs to be set according to actual needs, and the embodiments of the present disclosure do not limit this.

[0117] It should be noted that the linking and the function inlining are only two exemplary ways of rewriting the second intermediate representation, and other rewriting ways can also be selected according to actual needs, and the embodiments of the present disclosure do not limit this.

[0118] For example, in step S402, the second intermediate representation after the rewriting such as the linking and the function inlining is added to the first intermediate representation by an inline function way (including but not limited to compute inline, compute at, function inline, etc.), to obtain the fused target kernel function at the LLVM IR level, so as to realize the third-level fusion of the at least one first operator and the at least one second operator.

[0119] Figure 7 A schematic diagram of another example of the operator fusion method provided by at least one embodiment of the present disclosure. For example, Figure 7 A specific example of the operator fusion method is shown in Figure 6

[0120] For example, as shown in Figure 7 Figure 3 For example, as shown in Figure 7 ​​The left side is a target kernel function fused_kernel() at the tensor level, which includes first device functions fused_A(), fused_C() and fused_D_E() corresponding to the first operators A, C, D+E and a second device function fused_B() corresponding to the second operator B; the target kernel function at the tensor level is assembled downward to obtain a target kernel function fused_kernel() at the LLVM IR level, which includes first intermediate representations LLVM IR corresponding to the first operators A, C, D+E and a second intermediate representation LLVM IR corresponding to the second operator B. For example, since the second operator B is a handwritten operator, the kernel function corresponding to the second operator needs to be called from the outside, and the second intermediate representation of the second operator B can be a kernel function corresponding to the second operator B through Figure 6 The second intermediate representation after rewriting in step S401.

[0121] For example, as shown in FIG. 4, in step S401, first, the second intermediate representation LLVM IR of the second operator B is linked (llvm-link), and the second function declaration in the code is replaced by the corresponding function definition; then the second intermediate representation of the second operator B after linking is inlined (function inline), and the second function call in the code is inlined into the corresponding function implementation. Figure 7 Figure 6 For example, as shown in FIG. 4, in step S402, the second intermediate representation of the rewritten second operator B is added to the first intermediate representation of the first operators A, C, D+E by means of inline function to obtain a fused target kernel function fused_kernel() at the LLVM IR level; in the target kernel function fused_kernel(), the first intermediate representation corresponding to the first operators A, C, D+E and the second intermediate representation corresponding to the second operator B can be called, thereby realizing the third-level fusion of the first operators A, C, D+E and the second operator B.

[0122] For example, as shown in FIG. 4, in step S402, the second intermediate representation of the rewritten second operator B is added to the first intermediate representation of the first operators A, C, D+E by means of inline function to obtain a fused target kernel function fused_kernel() at the LLVM IR level; in the target kernel function fused_kernel(), the first intermediate representation corresponding to the first operators A, C, D+E and the second intermediate representation corresponding to the second operator B can be called, thereby realizing the third-level fusion of the first operators A, C, D+E and the second operator B. Figure 7 Figure 6 For example, as shown in FIG. 4, in the process of calling multiple first intermediate representations and second intermediate representations in the target kernel function fused_kernel() at the LLVM IR level, the inline function mode (for example, computeinline, compute at, etc.) is mainly used (in particular, for the second operator B as a handwritten operator, the modes of llvm-link and function inline are used), and the specific implementation mode of the inline function can refer to the description in the art, which will not be repeated here.

[0123] For example, as shown in FIG. 4, in the process of calling multiple first intermediate representations and second intermediate representations in the target kernel function fused_kernel() at the LLVM IR level, the inline function mode (for example, computeinline, compute at, etc.) is mainly used (in particular, for the second operator B as a handwritten operator, the modes of llvm-link and function inline are used), and the specific implementation mode of the inline function can refer to the description in the art, which will not be repeated here. Figure 7

[0124] ​​​Figure 8 Another example flowchart of the operator fusion method provided for at least one embodiment of the present disclosure. For example, Figure 8 For Figure 2 One specific example of the operator fusion method shown.

[0125] For example, as Figure 8 shown, taking operator fusion when an AI compiler compiles a source program for performing a deep learning computing task as an example, the AI compiler first acquires a subgraph topology corresponding to the source program of the deep learning computing task (for example, traversing in an operator structure graph of a neural network to find a required subgraph topology) or a neural network computing model, and then selects a corresponding operator for the computing model, which may, for example, include a first operator, a second operator, or a third operator, wherein the first operator and the third operator may include automatically generated operators, and the second operator may include a handwritten operator; further, the first operator, the second operator, or the third operator is registered, and the category and implementation (for example, an automatically generated manner or a handwritten operator, and a complex operator class, a reduce class, a broadcast class, and an elementwise class in an automatically generated operator) of the operator are identified during the operator registration, and then a mode matching operator fusion manner is defined (for example, the mode matching only supports a fusion mode of a specific category operator).

[0126] For example, as Figure 8 shown, further, if the operator category that can be matched by the mode is identified during the operator registration, the third operator can be fused by the first level fusion through the mode matching manner to obtain the first operator. For example, the first level fusion is an operation based on a graph layer IR in a front end of the AI compiler, the graph layer IR corresponding to the first operator is a first graph layer representation, and the graph layer IR corresponding to the second operator is a second graph layer representation; the first level fusion may, for example, be the mode matching fusion shown in Figure 1 , and a specific mode matching operator fusion process is described in Figures 1-3 , which is not repeated here.

[0127] For example, as Figure 8 shown, further, the first graph layer representation corresponding to the first operator and the second graph layer representation corresponding to the second operator are written into a target kernel function; the first graph layer representation and the second graph layer representation are respectively unfolded from the graph layer to a middle end of the AI compiler, and the graph layer IR is assembled downward into a tensor IR, wherein the tensor IR corresponding to the first operator and the second operator is called as a device function in the target kernel function; by calling the first device function and the second device function in the target kernel function, the second level fusion of the first operator and the second operator is realized; that is, step S10 in Figure 2 is performed.

[0128] For example, further, the first device function and the second device function are respectively unfolded from the tensor level down to the backend of the AI compiler, the tensor IR is assembled down to the LLVM IR, so as to obtain the first intermediate representation corresponding to the at least one first operator, and obtain the second intermediate representation corresponding to the second operator; that is, steps S20 and S30 in Figure 2 are performed.

[0129] For example, as shown in Figure 8 , further, the first intermediate representation and the second intermediate representation are further rewritten and fused at the LLVM IR level, the third level fusion of the first operator and the second operator is implemented to obtain the fused target kernel function; that is, step S40 in Figure 2 is performed.

[0130] In summary, as shown in Figure 8 , through the above-mentioned first level fusion (for example, fusion of the graph IR), the second level fusion (for example, fusion of the tensor IR) and the third level fusion (for example, fusion of the LLVM IR), the fusion of the first operator and the second operator is realized, so as to obtain the fused operator. For example, the specific implementation of steps S10-S40 is described in the foregoing description, which will not be repeated here.

[0131] For example, as shown in Figure 8 , further, at the backend of the AI compiler, the fused LLVM IR level target kernel function can be subjected to llvm pass optimization and hardware resource allocation, so as to generate an executable file (for example, machine executable code) that can be deployed on the target hardware for execution. For example, the llvm pass optimization can include dead code elimination, peephole optimization, elimination of redundant instructions such as load / store, and hardware resource allocation such as register allocation; the optimization of the fused target kernel function is not limited to llvm pass optimization and hardware resource allocation, and other options can also be selected according to actual needs, and the embodiments of the present disclosure are not limited thereto; the specific implementation of the optimization (llvm pass optimization, hardware resource allocation, etc.) of the fused target kernel function can refer to the description in the art, which will not be repeated here.

[0132] For example, as shown in Figure 8 , further, the generated executable file can be run in the AI compiler; during the program running process, the kernel function corresponding to the first operator and the kernel function corresponding to the second operator share the same memory address (for example, usharp ID), and through corresponding data reading and writing, the running result of the fused operator can be finally obtained.

[0133] The operator fusion method provided by at least one embodiment of the present disclosure can support fusion of operators of different categories and different implementation manners (automatic generation or handwriting) by calling device functions in a target kernel function and combining the forms of intermediate representations, thereby greatly reducing the overhead of computing resources such as kernel function startup, intermediate temporary buffers, data movement, and improving the computing efficiency and hardware performance such as AI compiler.

[0134] Figure 9 A schematic block diagram of the operator fusion device provided by at least one embodiment of the present disclosure is shown.

[0135] For example, as shown in Figure 9 The operator fusion device 200 includes a writing module 210, an obtaining module 220, and a fusion module 230.

[0136] For example, the writing module 210 is configured to write a first graph layer representation corresponding to at least one first operator and a second graph layer representation corresponding to at least one second operator into a target kernel function. For example, the first operator and the second operator are called by a device function in the target kernel function, and in some examples, the device function is a sub-function of the target kernel function. That is, the writing module 210 can be configured to perform, for example, the step S10 shown in Figure 2

[0137] For example, the obtaining module 220 is configured to obtain a first intermediate representation corresponding to at least one first operator and a second intermediate representation corresponding to at least one second operator. That is, the obtaining module 220 can be configured to perform, for example, the steps S20 and S30 shown in Figure 2

[0138] For example, the fusion module 230 is configured to fuse at least one first operator and at least one second operator based on the first intermediate representation and the second intermediate representation to obtain a fused target kernel function. That is, the fusion module 230 can be configured to perform, for example, the step S40 shown in Figure 2

[0139] For example, in some examples, the number of first operators can be one or more, and the first operators can include fusion operators, non-fusion operators, or handwritten operators, and the non-fusion operators can include automatically generated operators. The number of second operators can be one or more, and the second operators can include fusion operators, non-fusion operators, or handwritten operators, and the non-fusion operators can include automatically generated operators.

[0140] ​​​For example, in some examples, in response to the first operator including a fusion operator, the fusion module 230 is further configured to perform first-level fusion on the at least one third operator to obtain the at least one first operator by means of pattern matching. In some examples, the number of the third operators can be one or more, and the third operator can include an automatically generated operator.

[0141] For example, in some examples, the writing module 210 is further configured to: rewrite the first layer representation to invoke the at least one first operator by means of a device function; rewrite the second layer representation to invoke the at least one second operator by means of a device function; and invoke a first device function corresponding to the at least one first operator and a second device function corresponding to the at least one second operator in the target kernel function to perform second-level fusion on the at least one first operator and the at least one second operator.

[0142] For example, in some examples, before the second device function corresponding to the second operator is invoked in the target kernel function, the writing module 210 is further configured to: perform first function declaration on the at least one second operator. For example, in some examples, the input of the second device function includes at least one output of the first device function, or at least one input of the first device function includes the output of the second device function.

[0143] For example, in some examples, the obtaining module 220 is further configured to assemble the first device function into a first intermediate representation, and assemble the second device function into a second intermediate representation. For example, in order to assemble the second device function into the second intermediate representation, the obtaining module 220 can be further configured to: set the attribute of the second device function so that it can be functionally inlined; and perform second function declaration and second function call on the at least one second operator to obtain the second intermediate representation.

[0144] For example, in some examples, the fusion module 230 is further configured to: rewrite the second intermediate representation; and add the rewritten second intermediate representation to the first intermediate representation by means of inline function to obtain a fused target kernel function to perform third-level fusion on the at least one first operator and the at least one second operator.

[0145] For example, in order to rewrite the second intermediate representation, the fusion module 230 can be further configured to: link the second intermediate representation to replace the second function declaration of the at least one second operator in the second intermediate representation with a function definition; and perform function inlining on the linked second intermediate representation to inline the second function call of the at least one second operator in the second intermediate representation into the corresponding function implementation.

[0146] For example, in some examples, the writing module 210 is further configured to: rewrite the first layer representation to invoke the at least one first operator by means of a device function; rewrite the second layer representation to invoke the at least one second operator by means of a device function; and invoke a first device function corresponding to the at least one first operator and a second device function corresponding to the at least one second operator in the target kernel function to perform second-level fusion on the at least one first operator and the at least one second operator. Figure 2The details of the contents involved in the operation of the operator fusion device 200 have been described in the process of the operator fusion method shown, and thus the details are not described again for the sake of brevity, and the relevant details can be referred to the description above. Figures 2-8

[0147] It should be noted that Figure 9 The above modules in the operator fusion device 200 can be respectively configured as software, hardware, firmware or any combination of the above for performing specific functions. For example, the modules can correspond to dedicated integrated circuits, can correspond to pure software codes, and can correspond to software and hardware combined modules. As an example, the device described with reference to Figure 9 may be a PC computer, a tablet device, a personal digital assistant, a smart phone, a web application or other devices capable of executing program instructions, but is not limited thereto.

[0148] In addition, although the operator fusion device 200 is divided into modules for performing respective processes in the above description, it is clear to those skilled in the art that the processes performed by the modules can also be performed without any specific module division in the device or without clear demarcation between the modules. In addition, the operator fusion device 200 described above with reference to Figure 9 is not limited to including the above-described modules, but can also add some other modules (for example, reading modules, control modules, etc.) as needed, or the above modules can be combined.

[0149] At least one embodiment of the present disclosure also provides an electronic device, which includes a processor and a memory; the memory includes one or more computer program modules; the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules include a method for implementing the operator fusion method provided by the embodiments of the present disclosure described above. For example, the processor can be a single-core processor or a multi-core processor.

[0150] Figure 10 A schematic block diagram of an electronic device provided by at least one embodiment of the present disclosure.

[0151] For example, as Figure 10 shown, the electronic device 300 includes a processor 310 and a memory 320. For example, the memory 320 is used to store non-transitory computer readable instructions (for example, one or more computer program modules). The processor 310 is used to run the non-transitory computer readable instructions, which can perform one or more steps according to the operator fusion method described above when the non-transitory computer readable instructions are run by the processor 310. The memory 320 and the processor 310 can be interconnected by a bus system and / or other forms of connection mechanism (not shown). ​

[0152] For example, the processor 310 can be a central processing unit (CPU), a graphics processing unit (GPU), a general purpose graphics processing unit (GPGPU), a digital signal processor (DSP), or other forms of processing units with operator fusion capability and / or program execution capability, such as a field programmable gate array (FPGA), etc.; for example, the central processing unit (CPU) can be of X86, RISC-V, or ARM architecture, etc. The processor 310 can be a general purpose processor or a special purpose processor, and can control other components in the electronic device 300 to perform desired functions.

[0153] For example, the memory 320 can include any combination of one or more computer program products, which can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. For example, the volatile memory can include random access memory (RAM), cache memory, etc. For example, the non-volatile memory can include read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer program modules can be stored on the computer-readable storage medium, and the processor 310 can run the one or more computer program modules to implement various functions of the electronic device 300. Various application programs and various data used and / or generated by the application programs, etc. can also be stored in the computer-readable storage medium.

[0154] It should be noted that in the embodiments of the present disclosure, the specific functions and technical effects of the electronic device 300 can refer to the description of the operator fusion method provided by the at least one embodiment of the present disclosure, which will not be repeated here.

[0155] Figure 11 Another schematic block diagram of an electronic device provided by at least one embodiment of the present disclosure.

[0156] For example, as shown in FIG. 4, the electronic device 400 is adapted to implement the operator fusion method provided by the embodiments of the present disclosure. It should be noted that, Figure 11 The electronic device 400 shown is only an example, which does not bring any limitation to the functions and use range of the embodiments of the present disclosure. Figure 11

[0157] For example, as shown in FIG. 4, the electronic device 400 is adapted to implement the operator fusion method provided by the embodiments of the present disclosure. It should be noted that, Figure 11 ​As shown, the electronic device 400 can include a processing apparatus (e.g., a central processor, a graphics processor, etc.) 41, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 42 or loaded into a random access memory (RAM) 43 from a storage apparatus 48. Various programs and data required by the electronic device 400 to operate are also stored in the RAM 43. The processing apparatus 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44. Generally, the following apparatuses can be connected to the I / O interface 45: input apparatuses 46 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output apparatuses 47 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage apparatus 48 including, for example, a magnetic tape, a hard disk, etc.; and a communication apparatus 49. The communication apparatus 49 can allow the electronic device 400 to communicate with other electronic devices wirelessly or through wires to exchange data.

[0158] Although Figure 11 The electronic device 400 is shown with various apparatuses, it should be understood that all of the illustrated apparatuses are not required, and that the electronic device 400 can alternatively include more or less apparatuses.

[0159] For detailed explanations and technical effects of the electronic device 400, reference can be made to the relevant descriptions above regarding the operator fusion method, which will not be repeated here.

[0160] Figure 12 A schematic diagram of a storage medium provided for at least one embodiment of the present disclosure.

[0161] For example, as Figure 12 shown, the storage medium 500 stores non-transitory computer-readable instructions 510. For example, when the non-transitory computer-readable instructions 510 are executed by a computer, one or more steps according to the operator fusion method described above are performed.

[0162] For example, the storage medium 500 can be applied in the electronic device 300 as Figure 10 shown. For example, the storage medium 500 can be the memory 320 in the electronic device 300. For example, the relevant descriptions regarding the storage medium 500 can refer to the corresponding descriptions of the memory 320 in the electronic device 300 as Figure 10 shown, which will not be repeated here.

[0163] For the present disclosure, the following points need to be explained:

[0164] (1) In the drawings of the embodiments of the present disclosure, only the structures related to the embodiments of the present disclosure are involved, and other structures can refer to the usual designs.

[0165] (2) In the case of no conflict, the features in the same embodiment and different embodiments of the disclosure can be combined with each other.

[0166] The above is only a specific embodiment of the disclosure, but the protection scope of the disclosure is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the disclosure, which should be covered within the protection scope of the disclosure. Therefore, the protection scope of the disclosure should be subject to the protection scope of the claims.

Claims

1. An operator fusion method, comprising: writing a first graph layer representation corresponding to at least one first operator and a second graph layer representation corresponding to at least one second operator into a target kernel function, wherein the at least one first operator and the at least one second operator are invoked by a device function manner in the target kernel function; obtaining a first intermediate representation corresponding to the at least one first operator; obtaining a second intermediate representation corresponding to the at least one second operator; fusing the at least one first operator and the at least one second operator based on the first intermediate representation and the second intermediate representation to obtain a fused target kernel function, wherein a kernel function corresponding to the first operator and a kernel function corresponding to the second operator share a same register or a shared memory address, and the device function is a sub-function of the target kernel function; wherein the writing of the first graph layer representation corresponding to the at least one first operator and the second graph layer representation corresponding to the at least one second operator into the target kernel function comprises: rewriting the first graph layer representation to invoke the at least one first operator by the device function manner; rewriting the second graph layer representation to invoke the at least one second operator by the device function manner; invoking a first device function corresponding to the at least one first operator and a second device function corresponding to the at least one second operator in the target kernel function and performing a second level fusion of the at least one first operator and the at least one second operator by an inline function manner; wherein the first device function and the second device function are respectively assembled into the first intermediate representation and the second intermediate representation; wherein the fusing of the at least one first operator and the at least one second operator based on the first intermediate representation and the second intermediate representation to obtain the fused target kernel function comprises: rewriting the second intermediate representation; adding the rewritten second intermediate representation into the first intermediate representation by the inline function manner to obtain the fused target kernel function to perform a third level fusion of the at least one first operator and the at least one second operator.

2. The operator fusion method of claim 1, wherein, The at least one first operator comprises a fusion operator, a non-fusion operator or a handwritten operator, and the non-fusion operator comprises an automatically generated operator.

3. The operator fusion method of claim 1, wherein, The at least one second operator comprises a fusion operator, a non-fusion operator or a handwritten operator, and the non-fusion operator comprises an automatically generated operator.

4. The operator fusion method of claim 1, wherein, in response to the at least one first operator comprising a fusion operator, before the writing of the first graph layer representation corresponding to the at least one first operator and the second graph layer representation corresponding to the at least one second operator into the target kernel function, the method further comprises: performing a first level fusion of at least one third operator by a pattern matching manner to obtain the at least one first operator, wherein the at least one third operator comprises an automatically generated operator.

5. The operator fusion method of claim 1, wherein, before the invoking of the second device function corresponding to the at least one second operator in the target kernel function, the method further comprises: performing a first function declaration on the at least one second operator.

6. The operator fusion method of claim 1, wherein, the input of the second device function comprises at least one output of the first device function, or at least one input of the first device function comprises an output of the second device function.

7. The operator fusion method of claim 1, wherein, compiling the second device function into the second intermediate representation, comprising: setting a property of the second device function to enable function inlining; performing second function declaration and second function invocation on the at least one second operator to obtain the second intermediate representation.

8. The operator fusion method of claim 1, wherein, rewriting the second intermediate representation, comprising: linking the second intermediate representation to replace the second function declaration of the at least one second operator in the second intermediate representation with a function definition; performing function inlining on the linked second intermediate representation to inline the second function invocation of the at least one second operator in the second intermediate representation into a corresponding function implementation. 9.An operator fusion apparatus, comprising: a writing module configured to write a first graph layer representation corresponding to at least one first operator and a second graph layer representation corresponding to at least one second operator into a target kernel function, wherein the at least one first operator and the at least one second operator are invoked by means of device functions in the target kernel function; an obtaining module configured to obtain a first intermediate representation corresponding to the at least one first operator, and obtain a second intermediate representation corresponding to the at least one second operator; a fusion module configured to fuse the at least one first operator and the at least one second operator based on the first intermediate representation and the second intermediate representation to obtain a fused target kernel function; wherein a kernel function corresponding to the first operator shares a same register or a shared memory address with a kernel function corresponding to the second operator, and the device function is a sub-function of the target kernel function; wherein the writing module is further configured to: rewrite the first graph layer representation to invoke the at least one first operator by means of the device functions; rewrite the second graph layer representation to invoke the at least one second operator by means of the device functions; invoke a first device function corresponding to the at least one first operator and a second device function corresponding to the at least one second operator in the target kernel function, and perform second-level fusion of the at least one first operator and the at least one second operator by means of inline functions; wherein the first device function and the second device function are compiled into the first intermediate representation and the second intermediate representation respectively; wherein the fusion module is further configured to: rewrite the second intermediate representation; add the rewritten second intermediate representation into the first intermediate representation by means of inline functions to obtain the fused target kernel function to perform third-level fusion of the at least one first operator and the at least one second operator.

10. The operator fusion device of claim 9, wherein, in response to the at least one first operator comprising a fusion operator, the fusion module is further configured to perform first-level fusion of at least one third operator by means of pattern matching to obtain the at least one first operator, wherein the at least one third operator comprises an automatically generated operator. 11.An electronic device, comprising: a processor; a memory including one or more computer program modules; wherein the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules are configured to implement the operator fusion method of any one of claims 1-8.

12. A storage medium storing non-transitory computer-readable instructions that, when executed by a computer, implement the operator fusion method of any one of claims 1-8.

Images