Automatic operator fusion method and related products for computational graphs

Abstract

The present disclosure discloses a method for automatic fusion of operators of a computational graph, a computing device, and related products. The computing device may be included in a combination processing device, which may also include an interface device and other processing devices. The computing device interacts with other processing devices to jointly complete the computing operations specified by the user. The combination processing device may also include a storage device, which is respectively connected to the computing device and the other processing devices and is used to store data of the computing device and the other processing devices. The solution disclosed in the present disclosure provides an automatic fusion solution for operators of a computational graph, which can automatically search for an optimized operator fusion solution without manual analysis, thereby improving the efficiency of fusion analysis.

Description

Technical Field

[0001] The present disclosure generally relates to the field of intelligent computing, and more particularly to the field of neural networks. More specifically, the present disclosure relates to a method for automatically fusing operators in a computational graph, a computing device, and a computer-readable storage medium. Background Art

[0002] A neural network is a system of multiple neurons connected according to specific rules. It generally consists of four layers: input layer, convolution layer, pooling layer, and fully connected layer. With technological advancements, the number of neural network layers has increased. For example, in the classic VGG architecture, VGG-A has 11 weight layers, VGG-B has 13 weight layers, VGG-C has 16 weight layers, VGG-D has 16 weight layers, and VGG-E has 19 weight layers. Convolutional layers and fully connected layers generally refer to weight layers. Some neural networks have hundreds of layers. Furthermore, as the number of layers increases, the number of parameters in a neural network increases exponentially. For example, AlexNet has 60 million parameters involved in its computation. These multiple layers and parameters require extensive on-chip and off-chip input / output accesses, which consumes significant resources and delays computation time.

[0003] In intelligent computing systems, programming frameworks encapsulate common operations in neural network model algorithms into operators, such as convolution and pooling, for programmers to directly call. TensorFlow and PyTorch are currently popular deep learning frameworks. These frameworks typically use computational graphs to describe the computational process of machine learning algorithms, using tensors to represent all data in the computational graph and operators to represent various operations.

[0004] One mechanism for reducing input / output accesses in neural network model operations is to fuse operators within the computational graph. However, current operator fusion solutions rely primarily on manual analysis, which is extremely cumbersome and inefficient when the neural network is large and the number of operators is large. Therefore, an automatic operator fusion solution is urgently needed. Summary of the Invention

[0005] In order to at least partially solve one or more technical problems mentioned in the background technology, the present disclosure provides an automatic operator fusion solution for a computational graph, which automatically searches for the optimal operator fusion method based on data-driven and performance evaluation models.

[0006] In a first aspect, the present disclosure discloses a method for automatic fusion of operators of a computational graph implemented by a processing device, comprising: the processing device determines control data that affects the fusion performance of operators; the processing device generates different control data samples according to the control data for hardware information of the computational device to execute the computational graph; the processing device applies the control data samples to calculate the fusion benefits of candidate fusion subgraphs corresponding to different control data samples; and the processing device determines the optimal fusion method according to the fusion benefits, compiles and optimizes the computational graph based on the optimal fusion method, and obtains a corresponding binary instruction sequence to be allocated to the computing device to execute the task corresponding to the computational graph.

[0007] In a second aspect, the present disclosure provides a processing device for performing automatic fusion processing of operators of a computational graph, comprising a processor configured to execute program instructions; and a memory configured to store the program instructions. When the program instructions are loaded and executed by the processor, the processor executes the method for automatic fusion of operators of the computational graph according to the first aspect of the present disclosure.

[0008] In a third aspect, the present disclosure provides a computer-readable storage medium having program instructions stored therein. When the program instructions are loaded and executed by a processor, the processor executes the method for automatic fusion of operators of a computational graph according to the first aspect of the present disclosure.

[0009] In a fourth aspect, the present disclosure provides a computer program product, comprising a computer program or instructions, which, when executed by a processor, implements the method for automatic fusion of operators of a computational graph described in the first aspect of the present disclosure.

[0010] Through the automatic operator fusion scheme provided above, the embodiments of the present disclosure provide an optimization scheme for a computational graph, which uniformly defines the information affecting the operator fusion performance as control data, thereby converting the search problem for the optimal operator fusion scheme into a search problem for the optimal control data, and by providing a suitable performance evaluation model, it is possible to automatically search for the optimal operator fusion method. Some embodiments of the present disclosure also provide a variety of search algorithms to adapt to different network models. Other embodiments of the present disclosure also provide a weight retention scheme to maximize the benefits of weight retention. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts.

[0012] Figure 1A structural diagram of a board according to an embodiment of the present disclosure is shown;

[0013] Figure 2 A structural diagram showing a combined processing device according to an embodiment of the present disclosure;

[0014] Figure 3a A schematic diagram showing the internal structure of a single-core computing device according to an embodiment of the present disclosure;

[0015] Figure 3b A schematic diagram showing the internal structure of a multi-core computing device according to an embodiment of the present disclosure;

[0016] Figure 4 An exemplary system architecture to which the automatic operator fusion solution of the disclosed embodiment can be applied is shown;

[0017] Figure 5 An exemplary flow chart of an automatic operator fusion method according to an embodiment of the present disclosure is shown;

[0018] Figure 6 A schematic flow chart illustrating an automatic operator fusion method at the L1 storage level according to an embodiment of the present disclosure is shown;

[0019] Figure 7 An example of adjacent operator fusion search is shown;

[0020] Figure 8 An example of block-structured fusion search is shown;

[0021] Figure 9 A schematic flow chart illustrating an automatic operator fusion method at the L2 storage layer according to an embodiment of the present disclosure is shown;

[0022] Figure 10a A flowchart of an exemplary method for solving the weight retention problem according to an embodiment of the present disclosure is shown;

[0023] Figure 10b An exemplary traversal process for solving the weight retention problem according to an embodiment of the present disclosure is shown;

[0024] Figure 11 A schematic block diagram exemplarily illustrating a policy scheduling process according to an embodiment of the present disclosure; and

[0025] Figure 12 A block diagram illustrating a hardware configuration of a computing device that can implement various aspects of the disclosed embodiments. DETAILED DESCRIPTION

[0026] The following will clearly and completely describe the technical solutions in the embodiments of this disclosure in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this disclosure, not all of them. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this disclosure.

[0027] It should be understood that the terms "first," "second," "third," and "fourth," etc. in the claims, specification, and drawings of the present disclosure are used to distinguish different objects rather than to describe a specific order. The terms "include" and "comprising" used in the specification and claims of the present disclosure indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0028] It should also be understood that the terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the disclosure. As used in this disclosure and the claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" as used in this disclosure and the claims refers to any and all possible combinations of one or more of the associated listed items, including and including these combinations.

[0029] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context.

[0030] The specific embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0031] Exemplary hardware environment

[0032] Figure 1 FIG. 1 is a schematic diagram showing the structure of a board 10 according to an embodiment of the present disclosure. Figure 1As shown, board 10 includes chip 101, which is a system-on-chip (SoC), or system-on-chip, integrated with one or more combined processing devices. The combined processing device is an artificial intelligence computing unit that supports various deep learning and machine learning algorithms to meet the intelligent processing needs in complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in the field of cloud intelligence. A notable feature of cloud-based intelligent applications is the large amount of input data, which places high demands on the platform's storage and computing capabilities. Board 10 of this embodiment is suitable for cloud-based intelligent applications and has extensive off-chip storage, on-chip storage, and powerful computing capabilities.

[0033] Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. Calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface formats, such as a PCIe interface.

[0034] Board 10 also includes a memory device 104 for storing data, which includes one or more storage units 105. Memory device 104 is connected to a control device 106 and chip 101 via a bus for data transmission. Control device 106 in board 10 is configured to control the state of chip 101. To this end, in one application scenario, control device 106 may include a microcontroller (MCU).

[0035] Figure 2 FIG. 1 is a block diagram showing the combined processing device in the chip 101 of this embodiment. Figure 2 As shown in , the combined processing device 20 includes a computing device 201 , an interface device 202 , a processing device 203 and a storage device 204 .

[0036] The computing device 201 is configured to perform user-specified operations and is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

[0037] Interface device 202 is used to transmit data and control instructions between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to a storage device on-chip of computing device 201. Furthermore, computing device 201 can obtain control instructions from processing device 203 via interface device 202 and write them to a control cache on-chip of computing device 201. Alternatively or optionally, interface device 202 can also read data from the storage device of computing device 201 and transmit it to processing device 203.

[0038] The processing device 203 is a general processing device that performs basic controls including but not limited to data handling, starting and / or stopping the computing device 201. Depending on the implementation, the processing device 203 can be a central processing unit (CPU), a graphics processing unit (GPU) or one or more types of processors in other general and / or special processors, which include but are not limited to digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, with respect to the computing device 201 disclosed herein, it can be regarded as having a single-core structure or a homogeneous multi-core structure. However, when the computing device 201 and the processing device 203 are integrated and considered together, the two are regarded as forming a heterogeneous multi-core structure.

[0039] The storage device 204 is used to store data to be processed, which may be DRAM or DDR memory, and is typically 16G or larger in size, for storing data of the computing device 201 and / or the processing device 203 .

[0040] When computing device 201 runs a neural network, it typically first compiles the neural network using processing device 203 to produce an executable file. This executable file contains device information, indicating which device in the heterogeneous computer system the executable file should be executed on. After the executable files are assembled and linked, the neural network executable program is generated and stored in storage device 204.

[0041] The processing device 203 can read the executable program from its storage location and obtain multiple tasks of the program according to the executable program. These tasks are distributed to the computing device 201 for execution via the interface device 202 to finally obtain the calculation results.

[0042] Figure 3a The figure shows the internal structure of the processing core when the computing device 201 is a single-core device. The computing device 301 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 301 includes three modules: a control module 31, a calculation module 32, and a storage module 33.

[0043] The control module 31 coordinates and controls the operations of the computing module 32 and the storage module 33 to complete deep learning tasks. It includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 retrieves instructions from the processing device 203, while the instruction decode unit 312 decodes the retrieved instructions and sends the decoded results as control information to the computing module 32 and the storage module 33.

[0044] The operation module 32 includes a vector operation unit 321 and a matrix operation unit 322. The vector operation unit 321 is used to perform vector operations, supporting complex operations such as vector multiplication, addition, and nonlinear transformations. The matrix operation unit 322 is responsible for the core calculations of the deep learning algorithm, namely matrix multiplication and convolution.

[0045] The storage module 33 is used to store or transfer relevant data and includes a neuron RAM (NRAM) 331, a weight RAM (WRAM) 332, and a direct memory access (DMA) module 333. NRAM 331 stores input neurons, output neurons, and intermediate computational results; WRAM 332 stores the convolution kernels (i.e., weights) of the deep learning network; and DMA 333, which connects to DRAM 204 via bus 34, transfers data between the computing device 301 and DRAM 204.

[0046] Figure 3bThe figure shows a simplified schematic diagram of the internal structure of a multi-core computing device 201. A multi-core computing device can be abstracted using a hierarchical hardware model. As shown in the figure, the multi-core computing device can be abstracted into three levels: chip level (Chip) 360, processor cluster level (Cluster) 370, and processor core level (Core) 380. The disclosed embodiments mainly involve data transmission of storage units and computing units. Therefore, the figures and description briefly illustrate and introduce the relevant computing structures, omitting other parts.

[0047] At the chip level, each chip contains local DDR storage, each processor chip serves as a computing and control unit, and each processor chip contains multiple multiprocessors as computing units.

[0048] At the processor cluster level, each multiprocessor includes multiple accelerator cores as control and computing units, and a shared memory SRAM as a storage unit.

[0049] At the processor core level, each accelerator core contains local storage and an array of local processing units. NFU stands for Neuron Function Unit, which is used for convolutional computations.

[0050] In this multi-core computing device, the storage model includes the global memory of the board, SRAM (shared memory) on the Cluster, NRAM, WRAM and registers on the Core, etc. In order to obtain better performance, the data movement between the storage levels below the Card and the balance between memory access / computation can be explicitly controlled. SRAM is contained in the memory processing unit MPU (Memory Process Unit Core, abbreviated as MPU, or Mem Core). Core refers to the intelligent processing core (Intelligent Process Unit Core, abbreviated as IPU Core or Core) in the multi-core computing device. 1 IPU Core contains NRAM, WRAM, NFU, etc. Cluster refers to a processor cluster or a computing cluster. Usually a multi-core computing device contains several Clusters, and a Cluster contains 1 Mem Core + N IPU Cores.

[0051] Exemplary operator fusion scheme

[0052] Regarding the terms "node" and "operator (OP)" mentioned in this disclosure, it should be noted that the term "operator" is from the computer computing level (or from the software level or algorithm level); while the term "node" is a more figurative term (from the graphics level or a more intuitive level). In terms of what they refer to, the terms "operator" and "node" actually refer to the same thing. That is, in this disclosure, it can be considered that the terms "operator" and "node" have the same meaning and can be used interchangeably, but are described from different perspectives.

[0053] Generally, there are a large number of operators in a neural network. When the computing device 201 executes the operation logic of these operators, it usually executes the following steps in a loop:

[0054] The operation result of the previous operator is read from the off-chip cache (such as the storage device 204), the operation task of the current operator is executed based on the operation result of the previous operator, and after the operation task of the current operator is executed, the operation result of the operation task is written to the off-chip cache (such as the storage device 204).

[0055] Therefore, when a computing device performs a neural network computing task, it needs to exchange data after completing each related operation of an operator. This not only reduces data processing efficiency but also occupies inter-chip communication resources.

[0056] As mentioned in the background technology, a typical optimization of the computational graph is operator fusion, which is to calculate multiple operators together in a single kernel without saving the intermediate results back to the global memory, thereby effectively reducing the input / output access in the network model operation.

[0057] One known operator fusion method uses the topological sequence of operators in a computational graph to find fusible operators. A topological sequence is obtained by topologically sorting a directed acyclic graph (DAG), arranging all vertices in the DAG into a linear sequence. In graph theory, a topological sort is a linear sequence of all vertices in a DAG that must satisfy the following two conditions:

[0058] 1. Each vertex appears once and only once.

[0059] 2. If there is a path from vertex A to vertex B, then vertex A appears before vertex B in the sequence.

[0060] Typically, such linear sequences are called topologically ordered sequences, or simply topological sequences. Specifically, a topological sequence is an order established based on the interdependencies between operators in a computational graph. In this fusion method, operators are sorted according to the topological order, and only adjacent operators are allowed to fuse. However, this approach prevents some interconnected operators in the computational graph (operators with input-output relationships) from being fused due to non-adjacent topological order, reducing the possibility of fusion.

[0061] Another operator fusion method is to find the operator fusion method in the computation graph based on some predetermined rules. However, this rule is only effective for specific network models and has poor universality.

[0062] In the disclosed embodiment, various fusion feasibility scenarios and various data splitting feasibility scenarios in the computation graph are traversed to obtain a list of operators in the computation graph with fusion possibility and a scheduling strategy. In the scheduling strategy, information that affects the performance of operator fusion, such as the operator list of fusion possibility, data splitting method, pipeline method, weight residence, weight preloading, etc., is uniformly defined as control data, thereby converting the search problem for the operator fusion method with the best performance into a search problem for the optimal control data.

[0063] In this way, the fusion of operators is not limited to topological order adjacency, as long as the operators with connected inputs / outputs on the computational graph are allowed to be fused. On the other hand, the disclosed embodiment can provide a performance evaluation function of the fusion operator based on the control data of the fusion operator for specific hardware information. Based on the performance evaluation function, the fusion benefit of the fusion operator can be calculated, and the optimal fusion method can be selected based on the fusion benefit. According to the disclosed embodiment, the time evaluation is relatively accurate, the search space is sufficient, and a better operator fusion method can be found under various networks.

[0064] Figure 4 An exemplary system architecture to which the automatic operator fusion solution of the disclosed embodiment can be applied is shown. As shown in the figure, the system architecture includes a graph optimization module 410, an automatic fusion module 420, a policy scheduling module 430, and a code generator module 440.

[0065] The graph optimization module 410 can be used to expand the computational graph of the network model and perform various optimizations that do not involve underlying hardware information, including but not limited to pruning, constant folding, arithmetic simplification, layout optimization, etc.

[0066] The main function of the automatic fusion module 420 is to interact with the policy scheduling module 430 through a pre-defined search algorithm, provide the optimized computational graphs of various network models to the graph optimization module 410, and search for the operator fusion method with the best performance for each network model. The code generator 440 is then called to compile these fused operators into binary kernel functions and replace the fused operator subgraphs with a single fused operator on the computational graph.

[0067] The main function of the policy scheduling module 430 is to traverse various fusion possibilities and various split possibilities in the calculation graph, obtain the subgraphs that may be fused in the calculation graph and the scheduling strategy; split the operation operations involved in the subgraph in terms of dimension; carry out address deduction based on the split result, and determine the on-chip space that the subgraph needs to occupy during the operation process. The capacity of the on-chip space is Figure 3b The feasibility of fusion is determined by comparing the capacity of the on-chip space of the computing device shown, and the control information corresponding to the feasibility of fusion is fed back to the automatic fusion module 420.

[0068] The main function of the code generator 440 is to generate a corresponding kernel when the automatic fusion module 420 selects a more optimal subgraph. The kernel is a CPU function called at runtime.

[0069] The specific functions and interaction methods of the automatic fusion module 420 and the policy scheduling module 430 will be described in detail later. However, those skilled in the art will understand that the module division in the above figure is based on the consideration of logical functions, and other splitting methods can also be used in actual implementation. For example, the automatic fusion module and the policy scheduling module 430 can be combined into a single module, or the functions of the automatic fusion module and the policy scheduling module 430 can be split into more modules. The disclosed embodiments are not limited in this respect.

[0070] In the embodiment of the present disclosure, the network model after fusion optimization is executed in the hardware environment described above. Figure 3b, the NRAM, WRAM and other memories on the IPU of each processor core in the processor core level 380 can be abstracted as the L1 storage level, and the SRAM memory on the MPU in the processor cluster level 370 can be abstracted as the L2 storage level. The on-chip space mentioned above can refer to the space of the L1 storage level and / or the space of the L2 storage level. When the fused and optimized network model is executed in these hardware environments, the input and output of the fusion operator are the global memory of the chip level 360, such as DDR, and the intermediate results reside in the L2 storage level; the fusion operator can also be divided into several sub-operators, including fusion sub-operators, and the input and output of each sub-operator are the L2 storage level, and the intermediate results reside in the L1 storage level. It can be understood that there is parallel and serial data splitting between each level of storage, and each splitting involves the data splitting dimension and splitting block size, etc.

[0071] To elaborate further, the first case is when the input / output data of a batch is too large to be fully loaded into the L2 storage layer. In this case, the IO / computation parallel pipeline process of the neural network operators is divided into five stages, including:

[0072] The first stage is to load the input data from DDR to the L2 memory level.

[0073] The second stage is to load the input data on the L2 storage level into the L1 storage level, where the L1 storage level is NRAM.

[0074] The third stage is to perform operations based on the input data on the NRAM, obtain output data, and store the output data on the NRAM.

[0075] The fourth stage is to move the output data from NRAM to the L2 storage level for storage.

[0076] The fifth stage is: moving the output data from the L2 storage level to the DDR for storage.

[0077] The second case is when the on-chip space required for a batch of input / output data is less than or equal to the on-chip space on the MPU. In this case, the neural network operator's IO / computation parallel pipeline process includes two layers of three-stage pipelines. The three stages of the outer pipeline are:

[0078] Phase 1: Load input data from DDR to the L2 memory level.

[0079] Phase 2: Perform operations based on the input data on the L2 storage layer, obtain output data, and store the output data on the L2 storage layer.

[0080] The third stage: load the output data from the L2 memory level to the DDR.

[0081] The three stages of the inner flow method include:

[0082] The first stage: load the input data from the L2 storage level to the L1 storage level, which is NRAM at this time.

[0083] The second stage: perform operations based on the input data of NRAM, obtain output data, and store the output data on NRAM.

[0084] The third stage: load the output data from NRAM to the L2 storage level.

[0085] In the second case, it can be seen from the above two-layer pipeline method that the intermediate data of the operation is stored as intermediate data in the L2 storage layer before being transferred to the DDR as output data.

[0086] In some embodiments of the present disclosure, given that the intermediate results of the fused operators in a computation graph reside in the L2 storage layer, and the intermediate results of the fused sub-operators within the fused operator reside in the L1 storage layer, the operator fusion methods may include fusion at the L1 storage layer and fusion at the L2 storage layer. Furthermore, the optimal fusion method can be searched for in the following order: first, the optimal L1 fusion method is searched for and determined for fusion at the L1 storage layer; then, based on the optimal L1 fusion method, the optimal L2 fusion method is further searched for and determined for fusion at the L2 storage layer. In this case, the control data for the L2 fusion method includes the control data for the L1 fusion. If the optimal L1 fusion method is searched first, the control data for the optimal L1 fusion can be reused when searching for the optimal L2 fusion, significantly reducing the search space for L2 fusion and accelerating the search. Alternatively, the optimal L2 fusion method can be searched first, which is equivalent to searching the search space for L2 fusion before searching the search space for L1 fusion, without any reuse. Because the solution space for control data related to L1 fusion is large, this order is less efficient in obtaining the optimal fusion method. A person skilled in the art, while understanding the essence of the technical solution of the present application, may produce other deformations or transformations based on the technical solution of the present application. However, as long as the functions implemented and the technical effects achieved are similar to those of the present application, they should all fall within the scope of protection of the present application.

[0087] Figure 5 An exemplary flow chart of an automatic operator fusion method implemented by a processing device according to an embodiment of the present disclosure is shown.

[0088] As shown in the figure, in step 510, the processing device determines control data that affects the performance of operator fusion.

[0089] In some embodiments, the main control data of the fusion operator include one or more of the following: an operator list of each sub-operator in the fusion operator, a parallel splitting method of multiple L2 storage levels of the fusion operator, a serial splitting method of a single L2 storage level of the fusion operator, an IO / computational parallel pipeline method of the fusion operator, a parallel splitting method of multiple L1 storage levels of each fusion sub-operator, a serial splitting method of a single L1 storage level of each fusion sub-operator, an IO / computational parallel pipeline method of each fusion sub-operator, which model data resides on the L1 storage level, which model data resides on the L2 storage level, whether the fusion operator enables weight preloading, etc.

[0090] Different control data can be determined in different ways. The operator list of each sub-operator in the fusion operator can be determined directly based on the calculation graph. The parallel splitting method of multiple L2 storage levels of the fusion operator involves the hardware information of the computing device running the calculation graph, such as Figure 3b In the hardware environment shown, the splitting between clusters is involved at the processor cluster level.

[0091] The serial splitting method for a single L2 storage level of a fusion operator involves splitting the input feature map, with parameters primarily including the dimensions of the split. For example, when the HW (height and width) dimensions of the input feature map do not exceed a first threshold, such as when the entire feature map can be stored on a single L2 storage level, it can be considered "small image mode," with the splitting dimensions being N (batch). For another example, when the HW dimensions of the input feature map exceed a second threshold, it can be considered "large image mode," with the splitting dimensions including N, H, and W. It will be appreciated that the parameter settings for the serial splitting method are merely exemplary, and those skilled in the art may also devise additional splitting methods, such as "medium image mode" and "extra-large image mode," and the presently disclosed embodiments are not limited in this respect. Furthermore, the parameters for the serial splitting method may also include the specific block size for each splitting dimension. This parameter may be set when determining the control data or when subsequently generating control data samples for search. In some embodiments of the present disclosure, given that the impact of the size of the specific split blocks is localized, the specific split values ​​for each splitting dimension may be subsequently searched and determined by the policy scheduling module 430.

[0092] The IO / compute parallel pipelining method for fusion operators mainly involves whether pipelining is enabled, and when it is enabled, the number of pipelining stages used. For example, the control data can be set to disable pipelining, so that IO access and computation are not performed in parallel. For another example, the control data can be set to enable pipelining, and for small graph mode, the LCS three-stage pipelining is enabled, and for large graph mode, the LLCS five-stage pipelining is enabled, where L stands for load, C stands for compute, and S stands for store.

[0093] The parallel splitting method of multiple L1 storage levels of each fusion operator is mainly related to the dimension of the input feature map. Searches can be performed in each dimension to traverse various dimensional splitting methods. These dimensions can include, for example, the N (batch) dimension, H (height) dimension, W (width) dimension, and C (channel) dimension of the input feature map. The specific split value of each dimension can be determined, which depends on the number of computing components in the hardware implementation.

[0094] Similarly, the serial splitting of a single L1 storage level of each fusion sub-operator also involves the splitting of the input feature map, with parameters mainly including the split dimension and the split block size. In some embodiments, considering the locality of the influence of this control data, it can also be subsequently searched and determined by the policy scheduling module 430.

[0095] The IO / computational parallel pipeline mode of each fusion sub-operator is similar to the IO / computational parallel pipeline mode of the fusion operator described above, and can also include not enabling pipeline, enabling pipeline, and dividing pipeline into several levels when enabling pipeline, which will not be detailed here.

[0096] Regarding the control data of data residency, usually in neural network calculations, the weights of each network layer are fixed, so the resident data are usually weights. In this regard, in some embodiments, data residency is sometimes also referred to as weight residency. Through data residency, the time for repeated data loading can be effectively reduced, the number of data reuses can be increased, and the overall calculation can be accelerated. This control data can be set to non-resident or resident. When choosing to reside, it can be further determined which model data (such as the weights of which operators) reside on the L1 storage level and which model data reside on the L2 storage level. The specific determination method will be described later.

[0097] Similar to data residency, the control parameter for whether to enable weight preloading for the fusion operator can be set to either no preloading or preloading. When preloading is selected, the corresponding algorithm is called to evaluate and decide whether to support weight preloading or only support preloading of some weights.

[0098] The determination of the above control data will be described later in conjunction with specific implementations.

[0099] continue Figure 5 In step 520, different control data samples are generated based on the determined control data and the hardware information of the computing device to be executed on the computation graph. As previously mentioned, the optimization of operator fusion is closely related to the specific hardware architecture executing the computation graph. Different hardware architectures will have different control data search spaces. Therefore, it is necessary to generate control data samples based on the determined control data in combination with the specified hardware information to fully search the control data value space.

[0100] For example, assuming that the serial splitting method of a single L2 storage layer of the fusion operator is a large graph mode, and its splitting dimensions can include three dimensions: N, H, and W, then according to the specified hardware information, such as the capacity size of the L2 storage layer, control data samples of various combined splits in the three dimensions of N, H, and W can be generated to traverse possible serial splitting methods.

[0101] Next, in step 530 , these control data samples are applied to calculate the fusion gains of the candidate fusion subgraphs corresponding to different control data samples.

[0102] In some embodiments, the generated control data samples can be applied to the candidate fusion subgraphs by the policy scheduling module 430. Combined with the previously specified hardware information, the feasibility of the fusion solution can be determined and the runtime of the candidate fusion subgraph on the hardware architecture can be estimated. Based on this performance evaluation model, the fusion benefits of different fusion methods can be calculated.

[0103] In some embodiments, the fusion benefit can be expressed as the following formula:

[0104] Fusion benefit = the sum of the shortest execution time of each operator before fusion - the shortest execution time of the operator after fusion

[0105] Based on the above fusion benefit model, the fusion operators can be traversed according to the search algorithm to find the optimal operator fusion method.

[0106] That is, in step 540, the optimal fusion method is determined based on the fusion benefit. The computation graph can then be compiled and optimized based on this optimal fusion method to obtain a corresponding binary instruction sequence, which is then distributed to the computing device to execute the task corresponding to the computation graph. It can be understood that the fusion benefit of the optimal fusion method is the positive and maximum value among all possible fusion benefits.

[0107] Therefore, the automatic fusion solution provided by the embodiment of the present disclosure is described above. By defining control data related to fusion performance and constructing a performance evaluation model, it is possible to search for the optimal control data to achieve the operator fusion method with the best performance.

[0108] As mentioned above, the fusion of operators can include L1 storage layer fusion and L2 storage layer fusion. Regardless of the storage layer fusion, you can follow Figure 5 The automatic fusion method described above performs a fusion search to find the optimal fusion method for the corresponding storage layer. In some embodiments, the optimal L1 fusion method is first searched and determined for fusion at the L1 storage layer; then, based on the optimal L1 fusion method, the optimal L2 fusion method is further searched and determined for fusion at the L2 storage layer.

[0109] Figure 6 A schematic flowchart of an automatic operator fusion method at the L1 storage level according to an embodiment of the present disclosure is shown.

[0110] As shown in the figure, L1 fusion control data of the L1 storage layer is determined in step 610. When searching for operator fusion methods of the L1 storage layer, only the control data that affects the operator fusion performance of the L1 storage layer needs to be determined without considering all the control data listed above.

[0111] Specifically, the L1 fusion control data may include one or more of the following: a parallel splitting method of the fusion operator between multiple L2 storage levels; a serial splitting method of the fusion operator in a single L2 storage level; a parallel pipeline method of the IO and calculation of the fusion operator; a parallel splitting method of the fusion sub-operator between multiple L1 storage levels; and a parallel pipeline method of the IO and calculation of the fusion sub-operator.

[0112] It can be understood that operator fusion at the L1 storage level is equivalent to the fusion of operators within a subgraph within a computation graph, or fused sub-operators. Therefore, operator fusion at the L1 storage level focuses only on subgraph-level control data, not on higher-level control data. For example, the L1 storage level does not focus on weight optimization, specifically whether weights are resident or preloaded. Therefore, these control data are not included in L1 fusion control data.

[0113] In addition, in some embodiments, considering that the serial splitting method of a single L1 storage layer of each fusion sub-operator only affects the performance of the fusion operator of the L1 storage layer itself, it can be searched and determined by the policy scheduling module 430 and is not included in the above L1 fusion control data.

[0114] Next, in step 620, potential fusion operators are searched.

[0115] In some embodiments, when searching for potential fusion operators, a combination of multiple search methods can be used to adapt to the different structures of various network models. These search methods include, but are not limited to, adjacent operator fusion search, block structure fusion search, etc. The combination of search methods can also be diverse. In some implementations, adjacent operator fusion search can be applied first, then block structure fusion search can be applied, and finally adjacent operator fusion search can be applied again. In other implementations, block structure fusion search can be applied first, then adjacent operator fusion search can be applied. Various search methods will be described in detail later.

[0116] Next, in step 630 , for each potential fusion operator, the L1 fusion control data sample generated based on the L1 fusion control data is applied to calculate the fusion benefit of the fusion mode corresponding to the L1 fusion control data sample.

[0117] In the automatic operator fusion method at the L1 storage level, L1 fusion control data samples are generated according to the determined L1 fusion control data for the specified hardware information.

[0118] For example, assuming that the parallel splitting method of the fusion sub-operator in the L1 fusion control data between multiple L1 storage levels is set to split in the N and C dimensions, then according to the number of computing components in the hardware, for example, there are 4 processor cores, L1 fusion control data samples with various combinations of splits in the N and C dimensions can be generated to traverse possible parallel splitting methods.

[0119] For another example, the aforementioned serial splitting method of a single L1 storage level of each fusion sub-operator can be searched and determined by the policy scheduling module 430. In this case, various serial splitting methods can be traversed based on, for example, the storage capacity of a single L1 storage level and the dimensionality information of the input feature map to generate corresponding L1 fusion control data samples.

[0120] Finally, in step 640, the optimal L1 fusion method is determined based on the fusion gain. Simply put, the fusion method that yields the largest, positive fusion gain is the optimal L1 fusion method. Specifically, the comparison targets for the fusion gain vary slightly depending on the search process, and will be described later in conjunction with specific search methods.

[0121] Therefore, by defining L1 fusion control data related to fusion of the L1 storage level and generating L1 fusion control data samples accordingly, an optimal L1 fusion method can be searched and determined through a search method.

[0122] In the adjacent operator fusion search, we backtrack from the output operator of the computation graph toward the input operator and try to fuse adjacent operators according to the following algorithm to find the two adjacent nodes with the maximum fusion benefit:

[0123] Fuse the current operator with its adjacent operators in the backtracking direction and find the first operator with the largest and positive fusion benefit.

[0124] Fuse the first operator with its adjacent operators in the backtracking direction and find the second operator with the maximum positive fusion benefit.

[0125] When the second operator is consistent with the current operator, the current operator is merged with the first operator;

[0126] Repeat the above steps until no operators can be fused.

[0127] Figure 7 An example of adjacent operator fusion search is shown. The computation graph shown in the figure includes multiple operators A, B, C, D, and E, which are sequentially connected in the backtracking direction from the output operator to the input operator. In other words, the search direction is the backtracking direction from operator A to operator E.

[0128] First, for the current operator A, we can fuse it with its forward and backward adjacent operators in the backtracking direction. At this point, there is only the forward adjacent operator B, so we try to fuse operators A and B to find the first operator with the largest and positive fusion benefit, which in this example is operator B.

[0129] Next, fuse the first operator (operator B) with its forward and backward adjacent operators in the backtracking direction. At this point, operator B has operator C as its forward adjacent operator and operator A as its backward adjacent operator. Therefore, operator B is fused with operator C and operator A, respectively, to find the second operator with the maximum positive fusion benefit. In this example, assume that the second operator found is operator A.

[0130] Determine whether the second operator is the same as the current operator. In the above example, both the second operator and the current operator are operator A. Therefore, we can attempt to fuse operator A and operator B. In this case, the fused operator that combines operators A and B can be called operator A'. In other words, when two operators are each other's preferred fusion operators, they can be fused. Otherwise, the fusion is not performed.

[0131] The above search steps can be tried repeatedly until no operators can be fused. For example, in the above example, after fusing operators A and B, you can continue searching in the backtracking direction, taking operator C as the current operator, and repeating the above steps. If the preferred fusion operator for operator C is operator A', and the preferred fusion operator for operator D is operator E, then operators D and C will not be fused, and the search will continue to operator E. Assuming that the preferred fusion operator for operator E is operator D, operators E and D will be fused, called operator D'. Thus, a fusion search is performed on the entire computation graph. The solid-line boxes in the figure represent confirmed fusions, and the dotted-line boxes represent fusion attempts.

[0132] Because operator fusion occurred during this fusion search, meaning the computation graph changed, for example, to include operators A', C, and D', another fusion search is required for the modified graph. For example, consider whether operators A', C, and D' can be fused. If operator C can be fused with operator D', the computation graph has changed, and the fusion search must be repeated according to the above steps until the graph no longer changes.

[0133] Through this adjacent operator fusion search method, the fusion of operators is not limited to topological order adjacency. As long as the operators are connected in input / output on the computational graph, fusion is allowed, thereby exploring as many potential operator fusion methods as possible to maximize the fusion optimization effect.

[0134] As can be understood from the fusion benefit formula described above, calculating the fusion benefit requires comparing the execution time difference before and after fusion. Therefore, for each operator in the unfused computational graph, the L1 fusion control data sample must be applied as described above to determine its execution time. This approach is equivalent to treating a single operator as a fusion operator or subgraph, applying various possible control data samples to it, and solving accordingly. This process can also reveal certain operators that may not support the current fusion method. These operators can then be marked so that they can be avoided in subsequent fusion processes to avoid unnecessary attempts.

[0135] In block-structured fusion search, a fusion search is performed on a multi-branch, integrated block structure within a network model. A block structure refers to a subnetwork within a neural network model that has only one input and one output, but contains multiple branches. This means that some layers of the subnetwork have multiple inputs or outputs. Examples include the resblock structure of ResNet50 and the block structure of Inception_V3. Fusion of this block structure can significantly reduce I / O throughput. However, because it requires more than two operators to construct, adjacent operator fusion search strategies often fail to optimize it. Therefore, a fusion search targeting block structures is required.

[0136] When trying to fuse a structure that combines multiple branches into one, if you try to fuse all permutations and combinations and select the one with the best performance, you can get the optimal solution. However, when the number of branches is large, a longer compilation time is required. Therefore, in the operator fusion at the L1 storage level, one simplified process is to give priority to fusing the ones with short execution time, that is, to perform the fusion search in the order of priority of fusing the branches with short execution time in the block structure, so that the execution time of the fused sub-operators at the L1 storage level is more uniform. Since the calculation time / execution time of the operator is generally proportional to the loading time of the weight, this fusion method is conducive to the parallel flow of weight IO and calculation. When the execution time of the sub-operators is relatively uniform, the loading time of the weights can be effectively masked in the parallel flow of IO and calculation, thereby avoiding IO bottlenecks and better achieving the purpose of weight preloading.

[0137] Specifically, in the operator fusion search at the L1 storage level, the following algorithm can be used to perform fusion search on each branch of the block structure to find a branch fusion method that is successful and has a positive fusion benefit:

[0138] Sort the branches in the block structure according to the above priority order, that is, the order of the operator execution time of each branch in the block structure from small to large;

[0139] Similarly, each branch is traversed according to the priority order to try to merge them, so as to find a branch fusion method that is successful and has a positive fusion benefit.

[0140] Figure 8 An example of a block structure fusion search is shown in FIG. 810 in a computation graph, which includes four branches 801 , 802 , 803 , and 804 , each of which includes several operators.

[0141] First, sort the branches in ascending order based on the cumulative execution time of the operators on each branch. In this example, let's assume the execution time of the four branches is sorted as follows: 801 < 802 < 803 < 804.

[0142] Next, the branches are attempted to be merged according to the above sorting order. For example, first, based on branch 801, an attempt is made to merge it with branch 802.

[0143] Then, for this fusion attempt, the L1 fusion control data sample generated based on the L1 fusion control data can be applied to calculate the fusion benefit of the fusion method corresponding to the L1 fusion control data sample. Figure 4If the fusion fails (the policy scheduling module 430 returns a success or failure result) or the fusion benefit is negative, the branch 802 is not fused, and other branches are tried until all branches are tried.

[0144] For example, assuming that the fusion attempt of branch 801 and branch 802 fails, continue to try to merge branch 801 and branch 803. If the fusion still fails, try to merge branch 801 and branch 804. If the fusion still fails, start based on branch 802 and try the fusion combination with branch 803 and branch 804, and so on until all branches are tried.

[0145] Assuming that the fusion attempt of branch 801 and branch 802 is successful and the fusion benefit is positive, the fusion of branches 801 and 802 is retained, and on this basis, the fusion attempt of the next branch 803 is continued.

[0146] Therefore, the above article provides a simplified fusion search method for the block structure in the neural network model, which does not need to try all branch permutations and combinations, thus saving search time. In addition, by performing fusion attempts in order of execution time from small to large, it is beneficial to fully utilize the flow of weight IO and calculation, solving the IO bottleneck problem. Moreover, the above-mentioned block structure fusion search method can support both the situation where all branches can be fully fused and the situation where only some branches can be fused, so it has a wide range of adaptability.

[0147] After searching and determining a preferred L1 fusion method for the L1 storage layer according to the aforementioned method, based on the preferred L1 fusion method, a preferred L2 fusion method may be further searched and determined for fusion of the L2 storage layer.

[0148] Figure 9 The schematic flow chart of the automatic fusion method of the L2 storage layer according to the embodiment of the present disclosure is shown. Similar to the fusion method of the L1 storage layer, the fusion method of the L2 storage layer also needs to determine the fusion control data, search for the fusion operator, and apply the fusion control data sample. Therefore, in the following description, only the differences between the two are described, and the rest of the content can be referred to in conjunction with the above. Figure 5-Figure 8 Description.

[0149] As shown in the figure, in step 910, the L2 fusion control data for the L2 storage layer is determined. When searching for the operator fusion method for the L2 storage layer, since the operator fusion method for the L1 storage layer has already been determined, such as the inner layer parallel splitting and inner layer pipelining, only the control data that affects the operator fusion performance of the L2 storage layer, that is, the outer layer control data, needs to be determined, without considering all the control data listed above. This can also greatly reduce the possibility of fusion attempts, that is, reduce the search space, and thus improve the search speed.

[0150] Specifically, the L2 fusion control data may include one or more of the following: the parallel splitting method of the fusion operator between multiple L2 storage levels; the serial splitting method of the fusion operator in a single L2 storage level; the parallel pipeline method of the IO and calculation of the fusion operator; whether to enable weight preloading; and whether to enable weight residence.

[0151] Operator fusion at the L2 storage level primarily considers weight optimization, namely, whether weight preloading is enabled, whether weight retention is enabled, etc. For example, if weight retention is enabled, it consumes storage space, thus affecting the depth of fusion.

[0152] When weight preloading is enabled in the L2 fusion control data, in response to determining to enable weight preloading, a corresponding weight preloading algorithm is called to determine whether the weight storage space meets the weight preloading requirement.

[0153] When weight preloading is enabled, a ping-pong pipeline is typically used to improve preloading efficiency. In a ping-pong pipeline, at least two storage areas are typically configured. For example, the memory space of a memory is divided into at least two storage areas to support data access between one storage area and, for example, an external storage unit, while data access between another storage area and, for example, a processing core is performed simultaneously. These two storage areas can be referred to as the ping-pong storage space and the pong storage space.

[0154] Therefore, the constraint for weight preloading is that the maximum value of the cumulative storage requirements of the weights of any adjacent sub-operators in the computation graph cannot exceed the weight storage space. If it does not exceed, it can be determined that the weight storage space meets the weight preloading requirements. For example, suppose the computation graph includes four adjacent sub-operators, namely A, B, C, and D, and their weight storage requirements are 100k, 200k, 600k, and 800k respectively. Then the cumulative storage requirements of the weights of any adjacent sub-operators are: A+B=300k, B+C=800k, C+D=1400k. Assuming that the allocated weight storage space is 1M, weight preloading cannot be supported for the current computation graph because C+D=1400k>1M.

[0155] When the weight storage space does not meet the weight preloading requirements, the sub-operators in the computation graph can be split until the weight preloading requirements are met or the sub-operators can no longer be split. Sub-operators usually contain multiple operators, so they can be split. For example, continuing with the above example, sub-operator D can be split into sub-operators D1 and D2, with corresponding weight storage requirements of 300k and 500k respectively. In this case, the C+D item becomes C+D1=900k and D1+D2=800k, both of which are less than the weight storage space of 1M. Therefore, the split computation graph can support weight preloading.

[0156] It is understandable that if the weight preloading requirement is still not met after the split, the result of weight preloading failure will be returned. In this case, the control data can only be set to not enable weight preloading.

[0157] When weight residency is enabled in the L2 fusion control data, in response to determining to enable weight residency, a corresponding weight residency algorithm is called to sequentially determine the optimal residency scheme of the weight in the L1 storage layer and the optimal residency scheme in the L2 storage layer.

[0158] When the available storage space is fixed, the weight of an operator can be used to calculate the benefit of retaining the operator. In this case, the retention problem can be transformed into a classic knapsack problem.

[0159] The knapsack problem is an NP-complete combinatorial optimization problem. It can be described as: given a set of items, each with its own weight and price, how can one choose the items that maximize the total price within a limited total weight? The name of the problem comes from the question of how to choose the most appropriate items to place in a given knapsack. It can also be described as a decision problem: can the total value reach V while the total weight does not exceed W?

[0160] For weight retention, the problem can be described as maximizing the retention benefit while ensuring that the total storage size of the weights does not exceed the resident storage space. However, in the weight retention problem, the size of the backpack will change because non-resident weights also require storage space, so the size of the backpack depends on the size of the non-resident weights.

[0161] Figure 10a A flowchart of an exemplary method for solving the weight retention problem according to an embodiment of the present disclosure is shown.

[0162] As shown in the figure, in step 1010, the candidate weights can be arranged in descending order according to the required storage space. For example, assuming there are n candidate weights, they can be arranged in descending order according to the required storage space.w1 >C w2 >C w3 …>C wn , where w1 to wn represent n candidate weights, C w1 to C wn Represents the storage space required for the corresponding candidate weight.

[0163] Next, in step 1020, candidate weights are traversed in sequence according to the above order. During the traversal, for each candidate weight, two cases are classified: resident and non-resident, and the optimal resident mode is solved when non-resident.

[0164] First, for the first candidate weight w1 that occupies the largest space, that is, the current weight w1, a choice is made between resident and non-resident. Here, it is assumed that the size of the resident storage space provided initially is C0. On the one hand, when the current weight is selected to be non-resident, sufficient storage space needs to be reserved for the non-resident weight to be loaded when the operation is performed, so the available resident storage space becomes C1 = C0-C w1 At this point, the weight retention solution is converted into a knapsack problem. The knapsack size is C1, and the weights that can be retained are w2 to wn. Based on the solution to the knapsack problem, the optimal retention method under the current conditions can be found. It is assumed that the optimal retention method is S1. Since C has been reserved before, w1 Therefore, even if the weights that are not stored in the optimal storage method S1 obtained by solving the knapsack problem are not stored, there is enough storage space for loading during the calculation, because the storage space required for any subsequent weight is less than C w1 , so the reserved space can be reused. On the other hand, when the current weight is selected to reside, the available resident storage space is updated to C1 = C0-C w1 .

[0165] Then, traverse to the next weight w2. Similarly, you can choose to reside or not reside. When you choose not to reside, the available resident storage space becomes C2 = C1-C w2 Similarly, the weight retention solution is transformed into a knapsack problem. The knapsack size becomes C2, and the weights that can be retained are w3 to wn. Based on the solution to the knapsack problem, the optimal retention method under the current conditions can be found. Let's assume that the optimal retention method is S2. When the current weight w2 is selected for retention, the available retention storage space is updated to C2 = C1 - C w2 .

[0166] Similarly, the remaining weights are processed one by one. After all the traversals are completed, n optimal residence methods obtained by solving the knapsack problem can be obtained, that is, n optimal residence methods S1, S2, ..., Sn.

[0167] Finally, in step 1030, the optimal retention method is selected from the n optimal retention methods obtained during the traversal process as the final solution. As mentioned earlier, the benefits of weight retention can be considered as value. Each optimal retention method has a corresponding value. The optimal retention method with the highest value is selected as the optimal solution for weight retention.

[0168] Figure 10b The exemplary traversal process for solving the weight retention problem according to the embodiment of the present disclosure is shown. In this example, still according to Figure 10a The scenarios of the embodiments are assumed.

[0169] As shown in the figure, first traverse the candidate weight w1, and when w1 is not resident, construct the knapsack problem P1. In the knapsack problem P1, the knapsack size is the current available resident storage space minus the storage space required by the currently traversed candidate weight, that is, C1 = C0-C w1 The items that can be placed in the backpack, or their weights, are the candidate weights that follow the currently traversed candidate weight, and w2 through wn. The value of each weight is the corresponding retention benefit. The optimal retention method S1 can be found by using various knapsack problem solutions.

[0170] When w1 is resident, update the currently available resident storage space to C1 = C0-C w1 , then proceeds to the next candidate weight w2. If w2 does not persist, a knapsack problem P2 is constructed and solved to obtain the optimal persisting method S2. Thus, when traversing each candidate weight, a branch is formed to solve the corresponding knapsack problem and obtain the optimal persisting method.

[0171] Finally, the best one is selected from the n best residence modes obtained as the final solution.

[0172] The above method can be used when determining the optimal residency scheme for weights in the L1 storage layer, and can also be used when determining the optimal residency scheme for weights in the L2 storage layer. For example, when not all weights can reside in the L1 storage layer, the optimal residency scheme for the L1 storage layer can be determined first to maximize the benefits of L1 weight residency. At this time, the candidate weights can be all weights. After the optimal residency scheme for the L1 storage layer is determined, the weights that are not resident in the L1 storage layer can be considered for residency in the L2 storage layer. At this time, the candidate weights are those remaining weights that are not resident in the L1 storage layer.

[0173] It is understood that although the solution for determining weight retention is described in the context of automatic operator fusion, the solution can also be used independently of this context or applied to other scenarios. For example, when executing a neural network model on a specific hardware architecture, the weight retention method provided in the embodiments of this disclosure can be used to design an optimal retention scheme to maximize retention benefits.

[0174] return Figure 9 After the L2 fusion control data is determined, in step 920, potential fusion operators are searched.

[0175] Similar to the fusion search at the L1 storage layer, the fusion search at the L2 storage layer can also use a combination of multiple search methods. These search methods also include, but are not limited to, adjacent operator fusion search and block-structured fusion search. There are also various combinations of search methods. In the fusion search at the L2 storage layer, the adjacent operator fusion search method is similar to that of the L1 storage layer and will not be further described here. However, the block-structured fusion search is slightly different.

[0176] At the L2 storage level, the purpose of sorting the branches of the block structure is to maximize the number of branches that are fused, thereby saving I / O overhead. The key factor that most affects L2 storage level fusion is L2 memory usage. Therefore, branches with low L2 memory overhead in the same batch are fused first.

[0177] Specifically, when searching for the optimal L2 fusion method, the fusion search is performed in the order of priority: branches with a large number of batches in the block structure are fused first, and branches with a small L2 storage space are fused first when the number of batches is the same. Therefore, after sorting the branches in the block structure according to this priority order, we can still refer to the previous combination Figure 8 The search method described is used to search for possible branch fusion methods.

[0178] Next, in step 930 , for each potential fusion operator, the L2 fusion control data sample generated based on the L2 fusion control data is applied to calculate the fusion benefit of the fusion method corresponding to the L2 fusion control data sample.

[0179] In the automatic operator fusion method at the L2 storage level, L2 fusion control data samples are generated according to the determined L2 fusion control data for the specified hardware information.

[0180] For example, assuming that the serial splitting mode of the fusion operator in the L2 fusion control data on a single L2 storage layer is set to large graph mode, L2 fusion control data samples with various combinations of splits in the dimensions N, H, and W can be generated to traverse possible serial splitting modes.

[0181] Finally, in step 940, the optimal L2 fusion method is determined based on the fusion benefit. Similarly, the fusion method corresponding to the maximum and positive fusion benefit is the optimal L2 fusion method.

[0182] Therefore, by defining L2 fusion control data related to fusion of the L2 storage layer and generating L2 fusion control data samples accordingly, an optimal L2 fusion method can be searched and determined through a search method.

[0183] In the above description, for each fusion operator, the control data samples are applied to calculate the fusion benefits of the fusion methods corresponding to different control data samples. In some embodiments, this step can be performed by Figure 4 The strategy scheduling module 430 is used to execute the calculation. Generally, for the same computation result, there may be several different computation methods (corresponding to the control data samples and fusion methods in the disclosed embodiments). These different computation methods will result in different locality and performance. Therefore, in the disclosed embodiments, controlling how the computation is performed is called "scheduling." Scheduling is a set of primitives used to describe operations such as loop splitting, parallelization, and merging.

[0184] When applying control data samples to each fusion operator, the policy scheduling module 430 determines the feasibility of the searched candidate fusion methods and their corresponding performance parameters (such as space usage and runtime). Specifically, the policy scheduling module 430 mainly involves the following functions: address derivation, memory allocation, and runtime evaluation.

[0185] Figure 11 A schematic block diagram exemplarily illustrates a policy scheduling process according to an embodiment of the present disclosure.

[0186] As shown in the figure, the fusion engine 1110 can search the input computation graph, search for possible operator fusion schemes (i.e., fusion subgraphs), and also search for possible scheduling strategies, which are then output as candidate fusion schemes. The scheduling strategies here can correspond to, for example, the control data samples described above, including various strategies such as pipeline splitting, parallel splitting, serial splitting, data residency, etc. Therefore, it can be understood that the fusion engine 1110 here includes the above Figure 4 The functions of the automatic fusion module 420 and some functions of the policy scheduling module 430 are described. In this embodiment, the functions are re-divided from the perspective of function for the convenience of description. Those skilled in the art will understand that the division of these functions can be in various forms, and the embodiments of this disclosure are not limited to a specific functional division form.

[0187] In some embodiments, the scheduling policy may provide several fixed modes. These fixed modes are, for example, preferred scheduling methods for specific intelligent computing hardware. Thus, depending on the specific hardware configuration, a corresponding scheduling policy may be selected from these limited fixed modes.

[0188] In one example, a scheduling strategy mode is a large-graph mode, which can be divided into five levels, and the N, H, and W dimensions of the data are all split during the process of loading from the external storage circuit to the shared storage circuit. In another example, a scheduling strategy mode is a small-graph mode, which can be divided into two layers and three levels, and the H and W dimensions of the data are not split during the process of loading from the external storage circuit to the shared storage circuit, but are split again when the data is loaded from the shared storage circuit to the storage circuit of the processing core. In yet another example, a scheduling strategy mode is a mode without shared memory, which can be divided into one layer and three levels, for example.

[0189] The validity verification module 1120 can perform validity verification on each candidate fusion solution output by the fusion engine 1110 to confirm whether the candidate fusion solution is feasible. The validity verification mainly includes spatial verification.

[0190] Because computational graph operations involve various splits, such as serial, parallel, and pipeline splits, ensuring sufficient on-chip space is crucial. Therefore, a series of algorithms are required for address derivation and allocation to confirm whether a given hardware configuration can accommodate the data for the candidate fusion solution, thus completing space verification.

[0191] When performing spatial verification, operator information in the computation graph needs to be used, and the operator information can be provided by the operator definition module 1130 .

[0192] In some embodiments, the operator definition module 1130 can provide definitions for several types of operators, including but not limited to: convolution (Conv), pooling (Pool), elementwise, broadcast, etc. Various combinations of operators in these categories can basically represent the algorithms of most neural network models. Each operator category can include several operators. For example, the elementwise class can include commonly used activation functions (such as ReLU, GELU), ScalarMultiply (multiplying each element of a Tensor by a scalar); the convolution class can include depthwise convolution and dconv deconvolution; the pool class can include maximum pooling (maxpool), average pooling (avergepool), etc.

[0193] Furthermore, based on the operational characteristics of different dimensions, operators can be classified according to the derivation method of the operator dimensions. These dimensional derivation methods may include, but are not limited to, sliding window, addpad, element-wise, broadcast, no split, etc. For example, for the shape derivation of the convolution operator, its N dimension can be element-wise split, the H and W dimensions can be sliding window split, and the C dimension can be no split.

[0194] The definition of an operator may include the following information: description of iterations, dimensions and / or loops, and specific calculations / algorithms.

[0195] Different types of operators may have corresponding address derivation formulas according to the derivation categories of each dimension, so that the validity verification module 1120 may quickly perform space verification based on the information provided by the operator definition module 1130 .

[0196] The derivation formula actually describes the operator's requirements for input and output addresses. For example, for a vector addition operator, the derivation formula states that the i-th input address corresponds to the i-th output address, which is a one-to-one correspondence. Therefore, when deducing addresses, assuming that the output addresses are from i to i+N, it can be deduced that the corresponding input addresses are also from i to i+N, thus confirming the storage space required by the operator.

[0197] After receiving candidate fusion solutions (including fusion subgraphs and scheduling policies) from the fusion engine 1110 and operator information from the operator definition module 1130, the validity verification module 1120 performs address derivation and memory allocation for the entire computation, thereby verifying the feasibility of the candidate fusion solution. The validity verification module 1120 can also analyze the execution time of the fusion subgraphs. These test results are returned to the fusion engine 1110, which then determines the final fusion solution.

[0198] After confirming the final fusion plan, the fusion engine can send the fusion subgraph and the corresponding scheduling information to the code generator module (see Figure 4 The code generator module 440) performs the actual instruction generation.

[0199] In the above embodiment, by fixing the scheduling strategy to a limited number of preferred modes, the automatic fusion search process can be accelerated and the detection of the validity verification module can be simplified. Furthermore, by explicitly defining the types of operators, for example, by classifying them according to the derivation method of the dimension, the validity verification module can quickly perform address derivation based on this prior information, thereby simplifying the verification process and speeding up the verification process.

[0200] The inventors applied the automatic fusion solution of the disclosed embodiment to more than ten typical visual models that have been disclosed, and the performance of the fusion method obtained is comparable to the optimal fusion method of manual analysis.

[0201] The present disclosure also provides a processing device that can be used to implement an automatic operator fusion method for a computational graph.

[0202] Figure 12 1 is a block diagram showing a hardware configuration of a processing device 1200 that can implement various solutions of the disclosed embodiments. As shown in the figure, the processing device 1200 may include a processor 1210 and a memory 1220. Figure 12 In the processing device 1200, only the components related to this embodiment are shown. Therefore, it is obvious to those skilled in the art that the processing device 1200 may also include components related to Figure 12 The components shown in the figure are different from common components, such as displays.

[0203] The processing device 1200 may correspond to a computing device having various processing functions, such as a computing device for compiling a computation graph. For example, the processing device 1200 may be implemented as various types of devices, such as a personal computer (PC), a server device, a mobile device, etc.

[0204] The processor 1210 is configured to execute program instructions to control all functions of the processing device 1200. For example, the processor 1210 controls all functions of the processing device 1200 by executing a program stored in the memory 1220 on the processing device 1200. The processor 1210 can be implemented by a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), an artificial intelligence processor chip (IPU), etc. provided in the processing device 1200. However, the present disclosure is not limited thereto.

[0205] Memory 1220 is hardware used to store various data processed by processing device 1200. For example, memory 1220 can store processed data and data to be processed by processing device 1200. Memory 1220 can also store data processed or to be processed by processor 1210, such as computational graphs before and after automatic fusion. Furthermore, memory 1220 can store program instructions, such as applications and drivers, to be driven by processing device 1200. For example, memory 1220 can store various programs related to the automatic fusion algorithm for computational graphs to be executed by processor 1210. Memory 1220 can be DRAM, but the present disclosure is not limited thereto. Memory 1220 can include at least one of volatile memory and non-volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FRAM), and the like. The volatile memory may include dynamic RAM (DRAM), static RAM (SRAM), synchronous DRAM (SDRAM), PRAM, MRAM, RRAM, ferroelectric RAM (FeRAM), etc. In an embodiment, the memory 1220 may include at least one of a hard disk drive (HDD), a solid-state drive (SSD), a high-density flash memory (CF), a secure digital (SD) card, a micro secure digital (Micro-SD) card, a mini secure digital (Mini-SD) card, an extreme digital (xD) card, a cache, or a memory stick.

[0206] In summary, the specific functions implemented by the memory 1220 and the processor 1210 of the processing device 1200 provided in the embodiments of this specification can be interpreted in comparison with the aforementioned embodiments in this specification, and can achieve the technical effects of the aforementioned embodiments, so they will not be repeated here.

[0207] In an embodiment of the present disclosure, a computer-readable storage medium is also provided, in which program instructions are stored. When the program instructions are loaded and executed by a processor, the processor executes the automatic operator fusion method of the computation graph described in the embodiment of the present disclosure.

[0208] In an embodiment of the present disclosure, a computer program product is also provided, including a computer program or instruction. When the computer program or instruction is executed by a processor, the method for automatic fusion of operators of the computational graph described in the embodiment of the present disclosure is implemented.

[0209] According to different application scenarios, the electronic devices or devices disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablet computers, smart terminals, PC devices, Internet of Things terminals, mobile terminals, mobile phones, driving recorders, navigators, sensors, cameras, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, automatic driving terminals, vehicles, household appliances, and / or medical equipment. The vehicles include airplanes, ships and / or vehicles; the household appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, electric lights, gas stoves, and range hoods; the medical equipment includes magnetic resonance imaging (MRI), ultrasound machines and / or electrocardiographs. The electronic devices or devices disclosed herein may also be applied to the Internet, Internet of Things, data centers, energy, transportation, public administration, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, medical care and other fields. Furthermore, the electronic devices or devices disclosed herein may also be used in cloud, edge, terminal and other application scenarios related to artificial intelligence, big data and / or cloud computing. In one or more embodiments, electronic devices or apparatuses with high computing power according to the disclosed solution can be applied to cloud devices (such as cloud servers), while electronic devices or apparatuses with low power consumption can be applied to terminal devices and / or edge devices (such as smartphones or cameras). In one or more embodiments, the hardware information of the cloud device and the hardware information of the terminal device and / or edge device are compatible with each other, so that according to the hardware information of the terminal device and / or edge device, appropriate hardware resources can be matched from the hardware resources of the cloud device to simulate the hardware resources of the terminal device and / or edge device, so as to complete the unified management, scheduling and collaborative work of end-to-end or cloud-edge-to-end.

[0210] It should be noted that, for the purpose of simplicity, the present disclosure describes some methods and embodiments thereof as a series of actions and combinations thereof, but those skilled in the art will understand that the scheme of the present disclosure is not limited by the order of the actions described. Therefore, based on the disclosure or teachings of the present disclosure, those skilled in the art will understand that some of the steps therein can be performed in other orders or simultaneously. Further, those skilled in the art will understand that the embodiments described in the present disclosure can be regarded as optional embodiments, that is, the actions or modules involved therein are not necessarily necessary for the implementation of one or more schemes of the present disclosure. In addition, depending on the different schemes, the description of some embodiments of the present disclosure also has different emphases. In view of this, those skilled in the art will understand that the parts that are not described in detail in a certain embodiment of the present disclosure may also refer to the relevant descriptions of other embodiments.

[0211] In terms of specific implementation, based on the disclosure and teachings of this disclosure, those skilled in the art can understand that several embodiments disclosed in this disclosure can also be implemented in other ways not disclosed herein. For example, with respect to the various units in the electronic device or device embodiments described above, this document divides them based on the consideration of logical functions, and there may be other ways of division in actual implementation. For another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. With respect to the connection relationship between different units or components, the connection discussed above in conjunction with the accompanying drawings can be a direct or indirect coupling between units or components. In some scenarios, the aforementioned direct or indirect coupling involves a communication connection using an interface, wherein the communication interface can support electrical, optical, acoustic, magnetic or other forms of signal transmission.

[0212] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network elements. In addition, according to actual needs, some or all of the units may be selected to achieve the purpose of the solution described in the embodiments of this disclosure. In addition, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically separately.

[0213] In some other implementation scenarios, the above-mentioned integrated unit can also be implemented in the form of hardware, that is, a specific hardware circuit, which may include digital circuits and / or analog circuits, etc. The physical implementation of the hardware structure of the circuit may include but is not limited to physical devices, and the physical devices may include but are not limited to devices such as transistors or memristors. In view of this, the various devices described herein (such as computing devices or other processing devices) can be implemented by appropriate hardware processors, such as CPUs, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage unit or storage device can be any appropriate storage medium (including magnetic storage media or magneto-optical storage media, etc.), which can be, for example, resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high bandwidth memory (HBM), hybrid memory cube (HMC), ROM and RAM, etc.

[0214] The above is a detailed introduction to the embodiments of the present disclosure. Specific examples are used herein to illustrate the principles and implementation methods of the present disclosure. The description of the above embodiments is only used to help understand the method and core ideas of the present disclosure. At the same time, for those skilled in the art, based on the ideas of the present disclosure, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present disclosure.

Claims

1. A method for automatically fusing operators of a computational graph implemented using a processing device, comprising: The processing device determines control data that affects operator fusion performance; The processing device generates different control data samples according to the control data for hardware information of a computing device to execute the computation graph; The processing device applies the control data samples to calculate fusion gains of candidate fusion subgraphs corresponding to different control data samples; as well as The processing device determines an optimal fusion method according to the fusion benefit, compiles and optimizes the computation graph based on the optimal fusion method, obtains a corresponding binary instruction sequence, and distributes it to the computing device to execute the task corresponding to the computation graph; The hardware information includes a processor cluster, a processor core, an L1 storage layer, and an L2 storage layer. The intermediate results of the fused operator in the computation graph reside in the L2 storage layer, and the intermediate results of the fused sub-operators in the fused operator reside in the L1 storage layer. The fusion method includes fusion at the L1 storage layer and fusion at the L2 storage layer. The method further includes: For L1 storage layer fusion, search and determine the optimal L1 fusion method; as well as Based on the optimal L1 fusion method, we further search for and determine the optimal L2 fusion method for the L2 storage layer fusion.

2. The method according to claim 1, wherein searching and determining the optimal L1 fusion method comprises: Determine L1 fusion control data for the L1 storage layer, where the L1 fusion control data includes one or more of the following: The parallel splitting of fusion operators across multiple L2 storage layers; Serial splitting of fusion operators in a single L2 storage layer; Parallel pipeline mode of IO and computation of fusion operators; The parallel splitting of fusion sub-operators across multiple L1 storage levels; as well as Parallel pipeline mode of IO and computation of fusion sub-operators.

3. The method according to claim 2, wherein searching and determining the optimal L2 fusion method comprises: Determine L2 fusion control data for the L2 storage layer, where the L2 fusion control data includes one or more of the following: The parallel splitting of fusion operators across multiple L2 storage layers; Serial splitting of fusion operators in a single L2 storage layer; Parallel pipeline mode of IO and computation of fusion operators; Whether to enable weight preloading; and Whether to enable weight retention.

4. The method according to claim 3, wherein searching and determining the optimal L1 fusion mode or L2 fusion mode further comprises: Search for potential fusion operators; For each potential fusion operator, a fusion control data sample generated based on the corresponding fusion control data is applied to calculate a fusion benefit of the fusion mode corresponding to the fusion control data sample.

5. The method according to any one of claims 3 to 4, wherein searching and determining the optimal L1 fusion method or L2 fusion method further comprises combining one or more of the following to perform a fusion search on operators in the computation graph: Adjacent operator fusion search; and Block-structured fusion search.

6. The method according to claim 5, wherein the adjacent operator fusion search comprises: Backtracking from the output operator of the computation graph toward the input operator; Fuse the current operator with its adjacent operators in the backtracking direction and find the first operator with the largest and positive fusion benefit. Fuse the first operator with its adjacent operators in the backtracking direction, and find the second operator with the maximum positive fusion benefit; When the second operator is consistent with the current operator, fusing the current operator with the first operator; Repeat the above steps until no operators can be fused.

7. The method according to claim 5, wherein the block structure fusion search comprises: When searching to determine the optimal L1 fusion method, the fusion search is performed in the order of priority where branches with shorter execution times in the block structure are fused first; or When searching to determine the optimal L2 fusion method, the fusion search is performed in the order of priority: branches with a larger batch number in the block structure are fused first, and branches with a smaller L2 storage space are fused first when the batch number is the same.

8. The method according to claim 7, wherein the block-structured fusion search further comprises: sorting the branches in the block structure according to the priority order; Traverse each branch according to the priority order to merge them and find a branch fusion method that is successful and has a positive fusion benefit.

9. The method according to claim 5, wherein searching and determining the optimal L2 fusion method further comprises: In response to determining to enable weight preloading, determining whether the weight storage space meets the weight preloading requirement; as well as When the weight storage space does not meet the weight preloading requirement, the sub-operators in the computation graph are split until the weight preloading requirement is met or the splitting cannot be performed.

10. The method according to claim 9, wherein determining whether the weight storage space meets the weight preloading requirement comprises: It is determined whether the maximum value of the accumulated storage requirements of the weights of any adjacent sub-operators in the calculation graph exceeds the weight storage space; if not, it is determined that the weight storage space meets the weight preloading requirement.

11. The method according to claim 5, wherein searching and determining the optimal L2 fusion method further comprises: In response to determining that weight residency is enabled, an optimal residency scheme for the weight in the L1 storage level and an optimal residency scheme for the weight in the L2 storage level are sequentially determined.

12. The method according to claim 11, wherein determining the optimal residency scheme of the weight at the L1 storage level or the optimal residency scheme at the L2 storage level comprises: Arrange the candidate weights in descending order of storage space; Traverse the candidate weights in the order described, and for each candidate weight, solve the optimal residence method when the current candidate weight does not reside according to the knapsack problem; as well as The best one is selected from the obtained optimal residence modes as the optimal residence solution.

13. The method according to claim 12, wherein the backpack size in the backpack problem is the current available resident storage space minus the storage space required for the currently traversed candidate weights, the weights that can be placed in the backpack are the candidate weights arranged after the currently traversed candidate weights, and the value of each weight is the corresponding resident benefit.

14. The method of claim 1 , wherein generating different control data samples according to the control data comprises: Control data samples are generated according to a fixed pattern, wherein the fixed pattern includes any one of the following: a large-image pattern, a small-image pattern, and a pattern without shared memory, and each pattern defines a different strategy in one or more of the following aspects: pipeline splitting, parallel splitting, serial splitting, and data residency.

15. The method of claim 1 , wherein applying the control data sample comprises: For the candidate fusion subgraph, one or more of the following is performed according to each control parameter in the control data sample: address derivation, memory allocation, and runtime estimation. 16 . The method according to claim 15 , wherein when performing address derivation, address derivation is performed on the candidate fused subgraph based on predefined operator information. 17 . The method according to claim 16 , wherein the predefined operator information includes a dimension derivation method, and the dimension derivation method includes one or more of the following categories: sliding window, padding, element-by-element, broadcasting, and no splitting.

18. The method according to claim 1, wherein the fusion benefit is determined based on at least a difference between a cumulative value of the shortest execution time of each operator before fusion and a shortest execution time of the operator after fusion.

19. A processing device for optimizing a computation graph or performing data processing, comprising: a processor configured to execute program instructions; as well as A memory configured to store the program instructions, which, when loaded and executed by the processor, enables the processor to execute the method for automatic fusion of operators of a computational graph implemented by a processing device according to any one of claims 1 to 18.

20. A computer-readable storage medium storing program instructions, which, when loaded and executed by a processor, causes the processor to execute the method for automatic operator fusion of a computational graph implemented by a processing device according to any one of claims 1 to 18.

21. A computer program product, comprising a computer program or instructions, which, when executed by a processor, implements the method for automatic operator fusion of a computational graph implemented by a processing device as described in any one of claims 1 to 18.

Images