Method and apparatus for optimizing deep learning computational graph

Abstract

Apparatus and methods for optimizing a deep learning computational graph are provided herein. The method comprises the following steps: obtaining a deep learning calculation graph comprising a calculation-intensive operator and a memory-intensive operator; fusing the memory-intensive operator into the calculation-intensive operator to generate a new calculation graph; dividing the new calculation graph into sub calculation graphs; and fusing the compute-intensive operators in each of the sub-compute graphs to generate an optimized compute graph. Other embodiments may also be disclosed and claimed.

Description

Technical Field

[0001] Embodiments described herein relate generally to deep learning (DL) networks, and more particularly to methods and apparatus for optimizing deep learning computation graphs. Background Art

[0002] Nowadays, deep neural network (DNN) models have become deeper and more complex, with hundreds or even more layers. To obtain a suitable DNN, the deep learning computation graph generated from its corresponding intermediate representation (IR) should be optimized. However, for such complex DNNs, optimization is generally computationally expensive and time-consuming, and may cause large cache pressure. Summary of the Invention

[0003] Aspects of the present disclosure provide a method for optimizing a deep learning computation graph, comprising: obtaining a deep learning computation graph including compute-intensive operators and memory-intensive operators; fusing the memory-intensive operators into the compute-intensive operators to generate a new computation graph; dividing the new computation graph into sub-computation graphs; and fusing the compute-intensive operators in each of the sub-computation graphs to generate an optimized computation graph.

[0004] Another aspect of the present disclosure provides a device for optimizing a deep learning computation graph, comprising: an interface circuit module; and a processor circuit module coupled to the interface circuit module and configured to: obtain a deep learning computation graph comprising compute-intensive operators and memory-intensive operators; fuse the memory-intensive operators into the compute-intensive operators to generate a new computation graph; divide the new computation graph into sub-computation graphs; and fuse the compute-intensive operators in each of the sub-computation graphs to generate an optimized computation graph.

[0005] Another aspect of the present disclosure provides a computer-readable medium having instructions stored thereon, which, when executed by a processor, causes the processor to: obtain a deep learning computation graph including compute-intensive operators and memory-intensive operators; fuse the memory-intensive operators into the compute-intensive operators to generate a new computation graph; divide the new computation graph into sub-computation graphs; and fuse the compute-intensive operators in each of the sub-computation graphs to generate an optimized computation graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Embodiments of the present disclosure will be described, by way of example and not limitation, with reference to the figures of the accompanying drawings in which like reference numerals refer to similar elements and in which:

[0007] Figure 1 A flowchart illustrating an example of a method for optimizing a deep learning computation graph according to an embodiment of the present application is illustrated;

[0008] Figure 2 A flowchart illustrating another example of a method for optimizing a deep learning computation graph according to an embodiment of the present application is illustrated;

[0009] Figure 3 A flowchart illustrating another example of a method for optimizing a deep learning computation graph according to an embodiment of the present application is illustrated;

[0010] Figure 4 A flowchart illustrating another example of a method for optimizing a deep learning computation graph according to an embodiment of the present application is illustrated;

[0011] Figure 5 A schematic diagram illustrating an example of batch and sample division according to an embodiment of the present application;

[0012] Figure 6 A schematic diagram illustrating another example of sample partitioning according to an embodiment of the present application;

[0013] Figure 7 A block diagram illustrating an example of a device for optimizing a deep learning computation graph according to an embodiment of the present application is illustrated;

[0014] Figure 8 is a block diagram illustrating components capable of reading instructions from a machine-readable or computer-readable medium and performing any one or more of the methodologies discussed herein, according to some example embodiments; and

[0015] Figure 9 is a block diagram of an example processor platform according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0016] The various aspects of the illustrative embodiments will be described using terms commonly used by those skilled in the art to convey the essence of the present disclosure to other persons skilled in the art. However, it will be apparent to those skilled in the art that many alternative embodiments may be practiced using portions of the described aspects. For purposes of explanation, specific numbers, materials, and configurations have been set forth to provide a thorough understanding of the illustrative embodiments. However, it will be apparent to those skilled in the art that alternative embodiments may be practiced without the specific details. In other instances, well-known features may have been omitted or simplified to avoid blurring the illustrative embodiments.

[0017] Furthermore, various operations will be described as multiple discrete operations in sequence in a manner that is most helpful for understanding the illustrative embodiments; however, the order of description should not be construed as implying that these operations are necessarily order-dependent. In particular, these operations do not need to be performed in the order presented.

[0018] The phrases "in an embodiment," "in one embodiment," and "in some embodiments" are used repeatedly herein. The phrases generally do not refer to the same embodiment; however, they may. The terms "including," "having," and "comprising" are synonymous unless the context dictates otherwise. The phrases "A or B" and "A / B" mean "(A), (B), or (A and B)."

[0019] To obtain a proper deep neural network (DNN) from a deep learning framework, the following processing stages are generally performed: a) Graph construction phase, where a computational graph is constructed based on information from the deep learning framework via its intermediate representation (IR); b) a compilation phase, where the computation graph is transformed (e.g., optimized) and the IR is optimized and lowered to a hardware-specific IR; and c) Code generation phase, where binary code or equivalent representation is generated based on the optimized IR.

[0020] During the compilation phase, the computation graph is optimized, for example, by operator fusion.

[0021] Traditionally, operator fusion is usually performed by applying fixed-pattern methods or polyhedron-based loop fusion. However, fixed-pattern methods are limited by the specific operators they fix and cannot be used universally. Polyhedron-based loop fusion may miss potential fusion opportunities due to the lack of operator-level information.

[0022] Furthermore, for complex DNNs with a large number of layers, the above methods may be very inefficient and they may cause large cache pressure.

[0023] Figure 1 A flowchart illustrating an example of a method for optimizing a deep learning computation graph according to an embodiment of the present application is illustrated.

[0024] The methods herein may be performed by any suitable device, such as a deep learning compiler or a processor.

[0025] The deep learning computation graph herein can be a deep learning computation graph for any DNN model (such as a convolutional neural network (CNN) model with a large batch size), for inference (e.g., RN50 throughput in MLPerf) or for training (e.g., with Deep Learning Recommendation Mode (DLRM)). The deep learning computation graph can be a computation graph built by a deep learning compiler as described above.

[0026] refer to Figure 1 , in box S110, a deep learning computation graph including computation-intensive operators and memory-intensive operators is obtained.

[0027] For example, a compute-intensive operator in a deep learning computation graph can be a convolution or matrix multiplication. A memory-intensive operator can be an element-wise, binary, or memory move. Generally speaking, for a deep learning computation graph used for a CNN model, a compute-intensive operator can be followed by one or more memory-intensive operators.

[0028] In block S120 , the memory-intensive operators are fused into the computation-intensive operators to generate a new computation graph.

[0029] In an embodiment, one or more sequential memory-intensive operators may be fused into a preceding or succeeding compute-intensive operator. In this manner, a new computation graph may be generated. The new computation graph may include multiple layers, and each layer may include multiple compute-intensive operators. Thus, the new computation graph only includes compute-intensive operators.

[0030] In block S130 , the new computation graph is partitioned into sub-computation graphs.

[0031] For example, each sub-computation graph can include one or more layers of the new computation graph.

[0032] In an embodiment, the new computation graph may be partitioned into sub-computation graphs based on the output properties of each layer of the new computation graph and the platform capacity of the platform on which the new computation graph is to be executed.

[0033] For example, the output attribute of each layer may be an output attribute indicating the buffer capacity that should be allocated to this layer for operator fusion.

[0034] In an embodiment, each of the compute-intensive operators of the new computation graph may output an output activation, and the output activations of the compute-intensive operators of each layer form an output batch. The output attributes may include the size of the output batch of each layer and / or the size of the output activations of each compute-intensive operator of the new computation graph.

[0035] As for platform capacity, in an embodiment, the new computation graph may be executed on a central processing unit (CPU), and then the platform capacity may include the size of a data cache unit (DCU) and a middle layer unit (MLC) of the CPU.

[0036] The following will be about Figure 2 and Figure 3 An embodiment of partitioning the sub-computation graph based on output attributes and platform capacity in block S130 is further described.

[0037] In block S140 , computationally intensive operators are fused in each of the sub-computation graphs to generate an optimized computation graph.

[0038] For example, after fusing computation-intensive operators in each sub-computation graph, all computation-intensive operators will form an optimized computation graph.

[0039] By dividing the computation graph into sub-graphs and fusing operators in each sub-graph, the efficiency and cache pressure of deep learning computation graphs can be improved and optimized.

[0040] Figure 2 A flowchart of another example of a method for optimizing a deep learning computation graph according to an embodiment of the present application is illustrated.

[0041] exist Figure 2 , blocks S110, S120, and S140 are similar to Figure 1 , and Figure 2 and Figure 1 The difference is: Figure 1 The block S130 in the Figure 2 Detailed description of blocks S131, S132 and S133 in FIG.

[0042] refer to Figure 2 ,In block S131, based on the output characteristics of each layer, a partitioning parameter is obtained with the help of a heuristic rule.

[0043] In an embodiment, the partitioning parameters may include a batch partition number (x) and a spatial partition number (y). The batch partition number may correspond to the number of sub-batches into which the output batch is to be partitioned, and the spatial partition number may correspond to the number of sub-activations into which the output activation is to be partitioned. The partitioning of sub-batches and sub-activations will be further described below.

[0044] In block S132, based on the output attributes of each layer, the buffer size (a) allocated for the output batch of the layer and the weight of the layer is obtained. i ).

[0045] The buffer size for each layer may be estimated by means of any estimation method.

[0046] In block S133 , the layers are sequentially partitioned into sub-computation graphs based on the partitioning parameters, buffer size, and platform capacity in the topological order of the new computation graph.

[0047] About the following Figure 3 An embodiment of dividing the layers in block S133 is described.

[0048] Figure 3 A flowchart of another example of a method for optimizing a deep learning computation graph according to an embodiment of the present application is illustrated.

[0049] exist Figure 3, blocks S110, S120, S130, and S140 are similar to Figure 2 , and Figure 3 and Figure 2 The difference is: Figure 2 The block S133 in the Figure 3 Detailed description of blocks S133-1 and S133-2 in FIG.

[0050] Reference Figure 3 In block S133-1, based on the partitioning parameters (x and y) and the buffer size (a i ) to obtain the reduced buffer size (a) for each layer R_i ).

[0051] In an embodiment, the reduced buffer size may be expressed by the following equation (1):

[0052] In equation (1), a R_i Indicates the reduced buffer size for layer i of the new computation graph in topological order, w i Indicates the weight of the i-th layer, a i indicates the buffer size of the i-th layer, x indicates the number of batch divisions, y indicates the number of spatial divisions, and a R_i 、i、w i 、a i Each of , x, and y is greater than 0.

[0053] In block S133 - 2 , one or more consecutive layers that satisfy a predetermined condition are divided into sub-computation graphs.

[0054] In an embodiment, in box S133-2, one or more consecutive layers are divided into sub-computation graphs, for which the sum of the reduced buffer sizes of the one or more consecutive layers is less than or equal to the platform capacity, and the sum of the reduced buffer sizes of the one or more consecutive layers and the subsequent layer is greater than the platform capacity.

[0055] In other words, the reduced buffer size can be accumulated layer by layer starting from the first layer of the new computation graph, and once the above predetermined conditions are met, the accumulation will be re-executed starting from the latter layer.

[0056] In an embodiment, for each sub-computation graph, the following formula (2) is satisfied:

[0057] In equation (2), N indicates the starting layer of the sub-computation graph, M indicates the last layer of the sub-computation graph, T indicates a threshold value set based on the CPU, L1 indicates the size of the DCU, L2 indicates the size of the MLC, N, M, L1 and L2 are greater than 0, and T is greater than 0 and less than or equal to 1 (for example, T ranges from 0.9 to 0.95).

[0058] After completing the partitioning of the sub-computation graph, the fusion of computationally intensive operators can be performed. Figure 4 An example of fusion is shown.

[0059] Figure 4 A flowchart of another example of a method for optimizing a deep learning computation graph according to an embodiment of the present application is illustrated.

[0060] exist Figure 4 , blocks S110, S120, and S130 are similar to Figure 2 , and Figure 4 and Figure 2 The difference is: Figure 2 The block S140 in the Figure 4 Detailed description of blocks S141 and S142 in FIG.

[0061] refer to Figure 4 ,In block S141, the output batch of each layer of the sub-computation graph is divided into sub-batches by the batch split number.

[0062] For example, if the batch size is determined to be 2, the output batch of each layer will be divided into 2 sub-batches.

[0063] In block S142 , the output activation of each compute-intensive operator of the sub-computation graph is partitioned into sub-activations by a spatial partitioning number.

[0064] For example, if the number of spatial partitions is determined to be 3, the output activation of each computationally intensive operator will be divided into 3 sub-activations.

[0065] In an embodiment, each output activation may include one or more output samples (and each sample may include one or more channels).

[0066] Under this condition, the division in block S142 may be performed by dividing each output sample of the output activation into subsamples along the height direction of the sample by a spatial division number.

[0067] About the following Figure 5 and Figure 6 Embodiments of dividing into sub-batches and sub-activations are further described.

[0068] In block S143 , computationally intensive operators are fused in the sub-computation graph based on the sub-batches and sub-activations.

[0069] The fusion of computationally intensive operators in block S143 may be achieved by means of any suitable fusion solution.

[0070] Table 1 below provides example machine-readable language for implementing the fusion in block S140 . Table 1

[0071] It should be understood that the fusion in block S140 may be implemented by any other machine-readable language that can implement fusion as described above.

[0072] Figure 5 A schematic diagram illustrating an example of batch and sample division according to an embodiment of the present application is shown.

[0073] Figure 5 An output batch including 4 samples is shown, namely sample 1, sample 2, sample 3, and sample 4. Each sample includes 4 channels. For example, if the computation graph is used to process an image with 4 channels (e.g., red, green, blue, and write channels), each sample may include 4 corresponding channels.

[0074] exist Figure 5 , the X-axis and Y-axis directions indicated by arrows are respectively a batch direction for dividing batches and a height direction for dividing samples.

[0075] exist Figure 5 In the embodiment shown in , the batch division number x is 2, and the spatial division number y is 3. The output batch is divided into two sub-batches along the batch direction X, as indicated by line L3, one of the sub-batches includes samples 1 and 2, and the other sub-batches includes samples 3 and 4. Each of samples 1, 2, 3, and 4 is divided into three sub-samples along the height direction Y, as indicated by lines L1 and L2.

[0076] This partitioning of samples may be suitable in the case where corresponding computationally intensive operators of two consecutive layers have kernels with the same dimensions, such as 1×1 convolution kernels (e.g., with stride=1).

[0077] In the case where the corresponding computationally intensive operators of two consecutive layers have kernels with different dimensions, the partitioning of samples of the current layer can depend on the output samples of the next layer. Figure 6 An example of this is shown in .

[0078] Figure 6 A schematic diagram illustrating another example of sample partitioning according to an embodiment of the present application is shown.

[0079] exist Figure 6 In the embodiment shown in , samples c1 and c2 are samples output from the current layer, and samples f1 and f2 are samples output from the next layer. The number of spatial divisions y is 2. The kernel of the computationally intensive operator corresponding to sample c1 or c2 is a 1×1 convolution kernel (e.g., with a stride of 1), and the kernel of the computationally intensive operator corresponding to sample f1 or f2 is a 3×3 convolution kernel (e.g., with a stride of 1).

[0080] In this case, in order to obtain two subsamples of sample f1 or f2 (as indicated by line L4), each subsample divided from the corresponding sample c1 or c2 should include four rows among the rows shown in sample c1 or c2. That is, for sample c1 or c2, the first four rows can be divided into subsamples (as indicated by line L5), and the last four rows can be divided into another subsample (as indicated by line L6), which means that the middle two rows will be reused by the two subsamples.

[0081] It should be understood that the above division of output batches and samples and the dimensions of the core of the above computationally intensive operator are provided only as examples, and the batches and samples may be divided in any other way, and the core of the computationally intensive operator may have other dimensions according to actual requirements.

[0082] According to the method for optimizing a deep learning computation graph according to an embodiment of the present application, the optimization efficiency and cache pressure can be improved by dividing the computation graph into sub-computation graphs and fusing operators in each sub-computation graph respectively.

[0083] Figure 7 A block diagram illustrating an example of a device 700 for optimizing a deep learning computation graph according to an embodiment of the present application is illustrated.

[0084] refer to Figure 7 According to an embodiment of the present application, a device 700 for optimizing a deep learning computation graph includes a processor circuit module 710 and an interface circuit module 720 coupled to each other.

[0085] The processor circuit module 710 may be a deep learning compiler or any other processor. The processor circuit module 710 is configured to: obtain a deep learning computation graph including computation-intensive operators and memory-intensive operators; fuse the memory-intensive operators into the computation-intensive operators to generate a new computation graph; partition the new computation graph into sub-computation graphs; and fuse the computation-intensive operators in each of the sub-computation graphs to generate an optimized computation graph.

[0086] In one embodiment, the processor circuit module 710 may be configured to fuse memory-intensive operators into compute-intensive operators by fusing one or more sequential memory-intensive operators into a preceding or succeeding compute-intensive operator to generate a new computation graph.

[0087] In an embodiment, the new computation graph may include multiple layers, and each of the layers may include multiple computationally intensive operators.

[0088] In an embodiment, the new computation graph may be partitioned into sub-computation graphs based on the output properties of each layer of the new computation graph and the platform capacity of the platform on which the new computation graph is to be executed.

[0089] In an embodiment, each of the compute-intensive operators of the new computation graph outputs an output activation, and the output activations of the compute-intensive operators of each layer form an output batch. The output attributes may include the size of the output batch of each layer of the new computation graph and / or the size of the output activation of each compute-intensive operator.

[0090] In an embodiment, the platform capacity may include the size of a data cache unit (DCU) and the size of a middle layer cell (MLC) of a central processing unit (CPU) on which a new computation graph is to be executed.

[0091] In an embodiment, the processor circuit module 710 can be configured to divide the new computation graph into sub-computation graphs in the following manner: based on the output characteristics of each layer, obtain a division parameter with the aid of a heuristic rule; based on the output characteristics of each layer, obtain the buffer size of the layer assigned to the output batch and weight of the layer; and based on the division parameter, the buffer size and the platform capacity, sequentially divide the layers into sub-computation graphs according to the topological order of the new computation graph.

[0092] In an embodiment, the partitioning parameters may include a batch partitioning number and a spatial partitioning number. The batch partitioning number may correspond to the number of sub-batches into which the output batch is to be partitioned, and the spatial partitioning number may correspond to the number of sub-activations into which the output activation is to be partitioned.

[0093] In an embodiment, the processor circuit module 710 can be configured to sequentially divide the layers into sub-computation graphs in the topological order of the new computation graph based on the division parameters, buffer size and platform capacity in the following manner: obtaining the reduced buffer size of the layer based on the buffer size and division parameters of each layer; and dividing one or more consecutive layers into sub-computation graphs, for which the sum of the reduced buffer sizes of the one or more consecutive layers is less than or equal to the platform capacity, and the sum of the reduced buffer sizes of the one or more consecutive layers and the subsequent layer is greater than the platform capacity.

[0094] In an embodiment, the reduced buffer size may be represented by the above equation (1), and each partitioned sub-computation graph may satisfy the above equation (2).

[0095] In an embodiment, the processor circuit module 710 can be configured to fuse compute-intensive operators in each sub-computation graph in the sub-computation graph in the following manner: for each sub-computation graph, divide the output batch of each layer of the sub-computation graph into sub-batches by a batch partitioning number; divide the output activation of each compute-intensive operator of the sub-computation graph into sub-activations by a spatial partitioning number; and fuse the compute-intensive operators in the sub-computation graph based on the sub-batches and sub-activations.

[0096] In an embodiment, the processor circuit module 710 can be configured to divide the output activation of each computationally intensive operator of the sub-computation graph into sub-activations by a spatial division number in the following manner: each output sample of the output activation is divided into sub-samples along the height direction of the sample by a spatial division number.

[0097] The details of the operations performed by the processor circuit module 710 of the apparatus 700 for optimizing a deep learning computation graph may be referred to Figures 1 to 6 The above embodiments are shown in , which will not be repeated in this article.

[0098] According to the device for optimizing deep learning computation graphs according to the embodiments of the present application, the optimization efficiency and cache pressure can be improved by dividing the computation graph into sub-computation graphs and fusing operators in each sub-computation graph.

[0099] Furthermore, a computer-readable medium is provided. The computer-readable medium stores instructions. When executed by a processor, the instructions cause the processor to: obtain a deep learning computation graph including compute-intensive operators and memory-intensive operators; fuse the memory-intensive operators into the compute-intensive operators to generate a new computation graph; partition the new computation graph into sub-computation graphs; and fuse the compute-intensive operators in each of the sub-computation graphs to generate an optimized computation graph.

[0100] For example, when executed by a processor, the instructions may cause the processor to perform the Figures 1 to 6 The operations described will not be repeated in this article.

[0101] Figure 8 is a block diagram illustrating components capable of reading instructions from a machine-readable or computer-readable medium (e.g., a non-transitory machine-readable storage medium) and performing any one or more of the methodologies discussed herein, according to some example embodiments. Specifically, Figure 8A schematic diagram of hardware resources 800 is shown, comprising one or more processors (or processor cores) 810, one or more memory / storage devices 820, and one or more communication resources 830, each of which may be communicatively coupled via a bus 840. For embodiments in which node virtualization (e.g., NFV) is utilized, a hypervisor 802 may be executed to provide an execution environment for one or more network slices / subslices to utilize the hardware resources 800.

[0102] Processor 810 may include, for example, processor 812 and processor 814, which may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a vision processing unit (VPU), a field programmable gate array (FPGA), or any suitable combination thereof.

[0103] The memory / storage device 820 may include main memory, disk storage, or any suitable combination thereof. The memory / storage device 820 may include, but is not limited to, any type of volatile or non-volatile memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, solid-state memory, etc.

[0104] The communication resources 830 may include interconnect or network interface components or other suitable means for communicating with one or more peripheral devices 804 or one or more databases 806 via the network 808. For example, the communication resources 830 may include a wired communication component (e.g., for coupling via a universal serial bus (USB)), a cellular communication component, an NFC component, Components (e.g. Low energy), components and other communication components.

[0105] The instructions 850 may include software, a program, an application, an applet, an application, or other executable code for causing at least any one of the processors 810 to perform any one or more of the methodologies discussed herein. The instructions 850 may reside, in whole or in part, within at least one of the processor 810 (e.g., within a cache memory of the processor), the memory / storage device 820, or any suitable combination thereof. Furthermore, any portion of the instructions 850 may be transferred from any combination of the peripheral device 804 or the database 806 to the hardware resource 800. Thus, the memory of the processor 810, the memory / storage device 820, the peripheral device 804, and the database 806 are examples of computer-readable and machine-readable media.

[0106] Figure 9 9 is a block diagram of an example processor platform according to some embodiments of the present disclosure. Processor platform 900 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cell phone, a smartphone, a tablet computer such as an iPad™), a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a game console, a personal video recorder, a set-top box, a headset or other wearable device, or any other type of computing device.

[0107] The processor platform 900 of the illustrated example includes a processor 912. The processor 912 of the illustrated example is hardware. For example, the processor 912 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired series or manufacturer. The hardware processor can be a semiconductor-based (e.g., silicon-based) device. In some embodiments, the processor implements one or more of the methods or processes described above.

[0108] The processor 912 of the illustrated example includes a local memory 913 (e.g., a cache). The processor 912 of the illustrated example communicates with a main memory including a volatile memory 914 and a non-volatile memory 916 via a bus 918. The volatile memory 914 may be comprised of synchronous dynamic random access memory (SDRAM), dynamic random access memory (DRAM), Dynamic Random Access Memory The non-volatile memory 916 may be implemented by flash memory and / or any other desired type of memory device. Access to the main memories 914, 916 is controlled by a memory controller.

[0109] The processor platform 900 of the illustrated example also includes an interface circuit module 920. The interface circuit module 920 can be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), interface, a near field communication (NFC) interface, and / or a PCI Express interface.

[0110] In the illustrated example, one or more input devices 922 are connected to the interface circuit module 920. The input device(s) 922 permit a user to input data and / or commands to the processor 912. The input device(s) may be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touch screen, a track pad, a trackball, and / or a voice recognition system.

[0111] One or more output devices 924 are also connected to the interface circuit module 920 of the illustrated example. The output device 924 can be implemented, for example, by a display device (e.g., a light-emitting diode (LED), an organic light-emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-situ switching (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. Therefore, the interface circuit module 920 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0112] The interface circuit module 920 of the illustrated example also includes a communication device, such as a transmitter, a receiver, a transceiver, a modem, a residential gateway, a wireless access point, and / or a network interface, to facilitate exchanging data with an external machine (e.g., any type of computing device) via a network 926. This communication may occur via, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-site wireless system, a cellular telephone system, etc.

[0113] For example, the interface circuit module 920 may include a training data set input through input device(s) 922 or retrieved from a network 926 .

[0114] The processor platform 900 of the illustrated example also includes one or more mass storage devices 928 for storing software and / or data. Examples of such mass storage devices 928 include floppy disk drives, hard disk drives, optical disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

[0115] The machine-executable instructions 932 may be stored in the mass storage device 928, in the volatile memory 914, in the non-volatile memory 916, and / or on a removable non-transitory computer-readable storage medium such as a CD or DVD.

[0116] The following paragraphs describe examples of various embodiments.

[0117] Example 1 includes a method for optimizing a deep learning computation graph, comprising: obtaining a deep learning computation graph including compute-intensive operators and memory-intensive operators; fusing the memory-intensive operators into the compute-intensive operators to generate a new computation graph; dividing the new computation graph into sub-computation graphs; and fusing the compute-intensive operators in each sub-computation graph in the sub-computation graph to generate an optimized computation graph.

[0118] Example 2 includes the method of Example 1, wherein fusing the memory-intensive operator into the compute-intensive operator comprises fusing one or more sequential memory-intensive operators into a preceding or succeeding compute-intensive operator.

[0119] Example 3 includes the method of Example 1 or 2, wherein the new computation graph includes multiple layers, and each of the layers includes multiple compute-intensive operators, and wherein the new computation graph is divided into sub-computation graphs based on an output property of each layer of the new computation graph and a platform capacity of a platform on which the new computation graph is to be executed.

[0120] Example 4 includes the method of any one of Examples 1-3, wherein dividing the new computation graph into sub-computation graphs includes: obtaining a partition parameter by means of a heuristic rule based on the output characteristics of each layer; obtaining a buffer size of the layer allocated to the output batch and weight of the layer based on the output characteristics of each layer; and sequentially dividing the layers into sub-computation graphs according to the topological order of the new computation graph based on the partition parameter, the buffer size, and the platform capacity.

[0121] Example 5 includes the method of any one of Examples 1-4, wherein based on the partitioning parameters, the buffer size and the platform capacity, the layers are sequentially partitioned into sub-computation graphs in the topological order of the new computation graph, including: obtaining the reduced buffer size of the layer based on the buffer size and the partitioning parameters of each layer; and partitioning one or more consecutive layers into sub-computation graphs, for the one or more consecutive layers, the sum of the reduced buffer sizes of the one or more consecutive layers is less than or equal to the platform capacity, and the sum of the reduced buffer sizes of the one or more consecutive layers and the subsequent layer is greater than the platform capacity.

[0122] Example 6 includes the method of any of Examples 1-5, wherein each of the compute-intensive operators of the new computation graph outputs an output activation, and the output activations of the compute-intensive operators of each layer form an output batch, and wherein the output attributes include a size of the output batch and / or a size of the output activation of each compute-intensive operator of each layer of the new computation graph.

[0123] Example 7 includes the method of any of Examples 1-6, wherein the partitioning parameter includes a batch partitioning number and a spatial partitioning number, wherein the batch partitioning number corresponds to the number of sub-batches into which the output batch is to be partitioned, and the spatial partitioning number corresponds to the number of sub-activations into which the output activation is to be partitioned.

[0124] Example 8 includes the method of any of Examples 1-7, wherein the reduced buffer size is represented by where a R_i Indicates the reduced buffer size for layer i of the new computation graph in topological order, w i Indicates the weight of the i-th layer, a iindicates the buffer size of the i-th layer, x indicates the number of batch divisions, y indicates the number of spatial divisions, and a R_i 、i、w i 、a i Each of , x, and y is greater than 0.

[0125] Example 9 includes the method of any of Examples 1-8, wherein the platform capacity includes the size of a data cache unit (DCU) and the size of a middle layer cell (MLC) of a central processing unit (CPU) on which the new computation graph is to be executed.

[0126] Example 10 includes the method of any of Examples 1-9, wherein for each sub-computation graph, Wherein, N indicates the starting layer of the sub-computation graph, M indicates the last layer of the sub-computation graph, T indicates a threshold value set based on the CPU, L1 indicates the size of the DCU, L2 indicates the size of the MLC, N, M, L1 and L2 are greater than 0, and T is greater than 0 and less than or equal to 1.

[0127] Example 11 includes the method of any one of Examples 1-10, wherein in each sub-computation graph in the sub-computation graph, fusing the compute-intensive operators includes: for each sub-computation graph, dividing the output batch of each layer of the sub-computation graph into sub-batches by a batch partitioning number; dividing the output activation of each compute-intensive operator of the sub-computation graph into sub-activations by a spatial partitioning number; and fusing the compute-intensive operators in the sub-computation graph based on the sub-batches and the sub-activations.

[0128] Example 12 includes the method of any of Examples 1-11, wherein each output activation includes one or more output samples, and wherein dividing the output activation of each computationally intensive operator of the sub-computation graph into sub-activations by a spatial partitioning number includes dividing each output sample of the output activation into sub-samples along a height direction of the samples by a spatial partitioning number.

[0129] Example 13 includes a device for optimizing a deep learning computation graph, comprising: an interface circuit module; and a processor circuit module coupled to the interface circuit module and configured to: obtain a deep learning computation graph including compute-intensive operators and memory-intensive operators; fuse the memory-intensive operators into the compute-intensive operators to generate a new computation graph; divide the new computation graph into sub-computation graphs; and fuse the compute-intensive operators in each of the sub-computation graphs to generate an optimized computation graph.

[0130] Example 14 includes the device of Example 13, wherein the processor circuit module is configured to fuse the memory-intensive operator into the compute-intensive operator by fusing one or more sequential memory-intensive operators into a preceding or succeeding compute-intensive operator.

[0131] Example 15 includes the apparatus of Example 13 or 14, wherein the new computation graph comprises a plurality of layers, and each of the layers comprises a plurality of compute-intensive operators, and wherein the new computation graph is partitioned into sub-computation graphs based on an output property of each layer of the new computation graph and a platform capacity of a platform on which the new computation graph is to be executed.

[0132] Example 16 includes the apparatus of any one of Examples 13-15, wherein the processor circuit module is configured to partition the new computation graph into sub-computation graphs in the following manner: obtaining a partition parameter by means of a heuristic rule based on the output characteristics of each layer; obtaining a buffer size of the layer allocated to the output batch and weight of the layer based on the output characteristics of each layer; and sequentially partitioning the layers into sub-computation graphs according to the topological order of the new computation graph based on the partition parameter, the buffer size, and the platform capacity.

[0133] Example 17 includes the apparatus of any one of Examples 13-16, wherein the processor circuit module is configured to sequentially partition the layers into sub-computation graphs in the topological order of the new computation graph based on the partitioning parameter, the buffer size, and the platform capacity in the following manner: obtaining a reduced buffer size of the layer based on the buffer size of each layer and the partitioning parameter; and partitioning one or more consecutive layers into the sub-computation graphs, for the one or more consecutive layers, the sum of the reduced buffer sizes of the one or more consecutive layers is less than or equal to the platform capacity, and the sum of the reduced buffer sizes of the one or more consecutive layers and the subsequent layer is greater than the platform capacity.

[0134] Example 18 includes the apparatus of any of Examples 13-17, wherein each of the compute-intensive operators of the new computation graph outputs an output activation, and the output activations of the compute-intensive operators of each layer form an output batch, and wherein the output attributes include a size of the output batch and / or a size of the output activation of each compute-intensive operator of each layer of the new computation graph.

[0135] Example 19 includes the apparatus of any of Examples 13-18, wherein the partitioning parameter comprises a batch partitioning number and a spatial partitioning number, wherein the batch partitioning number corresponds to the number of sub-batches into which the output batch is to be partitioned, and the spatial partitioning number corresponds to the number of sub-activations into which the output activation is to be partitioned.

[0136] Example 20 includes the apparatus of any of Examples 13-19, wherein the reduced buffer size is represented by: where a R_i Indicates the reduced buffer size for layer i of the new computation graph in topological order, w i Indicates the weight of the i-th layer, a i indicates the buffer size of the i-th layer, x indicates the number of batch divisions, y indicates the number of spatial divisions, and aR_i 、i、w i 、a i Each of , x, and y is greater than 0.

[0137] Example 21 includes the apparatus of any of Examples 13-20, wherein the platform capacity includes the size of a data cache unit (DCU) and the size of a middle layer cell (MLC) of a central processing unit (CPU) on which the new computation graph is to be executed.

[0138] Example 22 includes the apparatus of any of Examples 13-21, wherein for each sub-computation graph: Wherein, N indicates the starting layer of the sub-computation graph, M indicates the last layer of the sub-computation graph, T indicates a threshold value set based on the CPU, L1 indicates the size of the DCU, L2 indicates the size of the MLC, N, M, L1 and L2 are greater than 0, and T is greater than 0 and less than or equal to 1.

[0139] Example 23 includes the apparatus of any of Examples 13-22, wherein the processor circuit module is configured to fuse the compute-intensive operators in each sub-computation graph in the sub-computation graph in the following manner: for each sub-computation graph, divide the output batch of each layer of the sub-computation graph into sub-batches by a batch division number; divide the output activation of each compute-intensive operator of the sub-computation graph into sub-activations by a spatial division number; and fuse the compute-intensive operators in the sub-computation graph based on the sub-batches and the sub-activations.

[0140] Example 24 includes the apparatus of any of Examples 13-23, wherein each output activation comprises one or more output samples, and wherein the processor circuit module is configured to divide the output activation of each computationally intensive operator of the sub-computation graph into sub-activations by a spatial division number in the following manner: each output sample of the output activation is divided into sub-samples along a height direction of the sample by the spatial division number.

[0141] Example 25 includes a device for optimizing a deep learning computation graph, comprising: a component for obtaining a deep learning computation graph including compute-intensive operators and memory-intensive operators; a component for fusing the memory-intensive operators into the compute-intensive operators to generate a new computation graph; a component for partitioning the new computation graph into sub-computation graphs; and a component for fusing the compute-intensive operators in each sub-computation graph to generate an optimized computation graph.

[0142] Example 26 includes the apparatus of Example 25, wherein the means for fusing a memory-intensive operator into a compute-intensive operator comprises means for fusing one or more sequential memory-intensive operators into a preceding or succeeding compute-intensive operator.

[0143] Example 27 includes the apparatus of Example 25 or 26, wherein the new computation graph comprises a plurality of layers, and each of the layers comprises a plurality of compute-intensive operators, and wherein the new computation graph is partitioned into sub-computation graphs based on an output property of each layer of the new computation graph and a platform capacity of a platform on which the new computation graph is to be executed.

[0144] Example 28 includes the apparatus of any one of Examples 25-27, wherein the component for partitioning the new computation graph into sub-computation graphs includes: a component for obtaining a partitioning parameter by means of a heuristic rule based on the output characteristics of each layer; a component for obtaining a buffer size of the layer allocated to the output batch and weight of the layer based on the output characteristics of each layer; and a component for sequentially partitioning the layers into sub-computation graphs in the topological order of the new computation graph based on the partitioning parameter, the buffer size, and the platform capacity.

[0145] Example 29 includes the apparatus of any one of Examples 25-28, wherein the component for sequentially partitioning the layers into sub-computation graphs in the topological order of the new computation graph based on the partitioning parameter, the buffer size, and the platform capacity includes: a component for obtaining a reduced buffer size of the layer based on the buffer size of each layer and the partitioning parameter; and a component for partitioning one or more consecutive layers into sub-computation graphs, for the one or more consecutive layers, the sum of the reduced buffer sizes of the one or more consecutive layers is less than or equal to the platform capacity, and the sum of the reduced buffer sizes of the one or more consecutive layers and the subsequent layer is greater than the platform capacity.

[0146] Example 30 includes the apparatus of any of Examples 25-29, wherein each of the compute-intensive operators of the new computation graph outputs an output activation, and the output activations of the compute-intensive operators of each layer form an output batch, and wherein the output attributes include a size of the output batch and / or a size of the output activation of each compute-intensive operator of each layer of the new computation graph.

[0147] Example 31 includes the apparatus of any of Examples 25-30, wherein the partitioning parameter comprises a batch partitioning number and a spatial partitioning number, wherein the batch partitioning number corresponds to the number of sub-batches into which the output batch is to be partitioned, and the spatial partitioning number corresponds to the number of sub-activations into which the output activation is to be partitioned.

[0148] Example 32 includes the apparatus of any of Examples 25-31, wherein the reduced buffer size is represented by where a R_i Indicates the reduced buffer size for layer i of the new computation graph in topological order, w i Indicates the weight of the i-th layer, a i indicates the buffer size of the i-th layer, x indicates the number of batch divisions, y indicates the number of spatial divisions, and a R_i 、i、wi 、a i Each of , x, and y is greater than 0.

[0149] Example 33 includes the apparatus of any of Examples 25-32, wherein the platform capacity includes the size of a data cache unit (DCU) and the size of a middle layer cell (MLC) of a central processing unit (CPU) on which the new computation graph is to be executed.

[0150] Example 34 includes the apparatus of any of Examples 25-33, wherein for each sub-computation graph: Wherein, N indicates the starting layer of the sub-computation graph, M indicates the last layer of the sub-computation graph, T indicates a threshold value set based on the CPU, L1 indicates the size of the DCU, L2 indicates the size of the MLC, N, M, L1 and L2 are greater than 0, and T is greater than 0 and less than or equal to 1.

[0151] Example 35 includes the apparatus of any of Examples 25-34, wherein the means for fusing compute-intensive operators in each sub-computation graph in the sub-computation graph comprises: for each sub-computation graph, means for dividing an output batch of each layer of the sub-computation graph into sub-batches by a batch partitioning number; means for dividing an output activation of each compute-intensive operator of the sub-computation graph into sub-activations by a spatial partitioning number; and means for fusing the compute-intensive operators in the sub-computation graph based on the sub-batches and the sub-activations.

[0152] Example 36 includes the apparatus of any of Examples 25-35, wherein each output activation comprises one or more output samples, and wherein the component for dividing the output activation of each computationally intensive operator of the sub-computation graph into sub-activations by a spatial division number comprises: a component for dividing each output sample of the output activation into sub-samples along a height direction of the sample by a spatial division number.

[0153] Although certain embodiments have been illustrated and described herein for purposes of illustration, a wide range of alternative and / or equivalent embodiments or implementations calculated to achieve the same purpose may be substituted for the embodiments shown and described without departing from the scope of the present disclosure. This application is intended to cover any modifications or variations of the embodiments discussed herein. Therefore, it is expressly intended that the embodiments described herein be limited only by the appended claims and their equivalents.

Claims

1. A method for optimizing a deep learning computation graph, comprising: Obtaining a deep learning computational graph that includes both compute-intensive and memory-intensive operators; fusing the memory-intensive operator into the computation-intensive operator to generate a new computation graph; Dividing the new computation graph into sub-computation graphs; as well as The computationally intensive operators are fused in each of the sub-computation graphs to generate an optimized computation graph.

2. The method according to claim 1, wherein Fusing the memory-intensive operator into the compute-intensive operator includes: Fuse one or more sequential memory-intensive operators into the preceding or following compute-intensive operator.

3. The method according to claim 1, wherein The new computation graph includes a plurality of layers, and each of the layers includes a plurality of computationally intensive operators, and The new computation graph is divided into the sub-computation graphs based on the output attributes of each layer of the new computation graph and the platform capacity of the platform on which the new computation graph is to be executed.

4. The method according to claim 3, wherein: Dividing the new computation graph into the sub-computation graphs includes: Based on the output characteristics of each layer, obtaining a partitioning parameter by means of a heuristic rule; Based on the output characteristics of each layer, obtaining a buffer size of the layer allocated for the output batch and weight of the layer; and Based on the partitioning parameters, the buffer size, and the platform capacity, the layers are sequentially partitioned into the sub-computation graphs according to the topological order of the new computation graph.

5. The method according to claim 4, wherein Sequentially dividing the layers into the sub-computation graphs according to the topological order of the new computation graph based on the partitioning parameters, the buffer size, and the platform capacity includes: obtaining a reduced buffer size for each layer based on the buffer size and the partitioning parameter of the layer; and One or more consecutive layers are divided into sub-computation graphs, wherein for the one or more consecutive layers, the sum of the reduced buffer sizes of the one or more consecutive layers is less than or equal to the platform capacity, and the sum of the reduced buffer sizes of the one or more consecutive layers and a subsequent layer is greater than the platform capacity.

6. The method according to claim 5, wherein: Each of the compute-intensive operators of the new computation graph outputs an output activation, and the output activations of the compute-intensive operators of each layer form an output batch, and The output attributes include the size of an output batch of each layer of the new computation graph and / or the size of an output activation of each compute-intensive operator.

7. The method according to claim 6, wherein: The division parameters include batch division number and space division number, wherein the batch division number corresponds to the number of sub-batches into which the output batch is to be divided, and The number of spatial divisions corresponds to the number of sub-activations into which the output activation is to be divided.

8. The method according to claim 7, wherein: The reduced buffer size is expressed as follows: where a R_i Indicates the reduced buffer size of the i-th layer of the new computation graph in the topological order, w i Indicates the weight of the i-th layer, a i indicates the buffer size of the i-th layer, x indicates the batch division number, y indicates the space division number, and the a R_i 、i、w i 、a i Each of , x, and y is greater than 0.

9. The method according to claim 8, wherein The platform capacity includes the size of a data cache unit (DCU) and the size of a middle layer cell (MLC) of a central processing unit (CPU) on which the new computation graph is to be executed.

10. The method according to claim 9, wherein: For each subgraph: Wherein, N indicates the starting layer of the sub-computation graph, M indicates the last layer of the sub-computation graph, T indicates a threshold value set based on the CPU, L1 indicates the size of the DCU, L2 indicates the size of the MLC, N, M, L1 and L2 are greater than 0, and T is greater than 0 and less than or equal to 1.

11. The method according to any one of claims 7 to 10, wherein: In each of the sub-computation graphs, fusing the computation-intensive operator comprises: for each sub-computation graph, Divide the output batch of each layer of the sub-computation graph into the sub-batches according to the batch division number; Partitioning the output activation of each computationally intensive operator of the sub-computation graph into the sub-activations by the spatial partition number; and The computationally intensive operators are fused in the sub-computation graph based on the sub-batch and the sub-activation.

12. The method according to claim 11, wherein Each output activation consists of one or more output samples, and wherein dividing the output activation of each computationally intensive operator of the subcomputation graph into the subactivations by the spatial partition number comprises: Each output sample of the output activation is divided into subsamples along the height direction of the sample by the spatial division number.

13. A device for optimizing a deep learning computation graph, comprising: Interface circuit module; as well as a processor circuit module coupled to the interface circuit module and configured to: Obtaining a deep learning computational graph that includes both compute-intensive and memory-intensive operators; fusing the memory-intensive operator into the computation-intensive operator to generate a new computation graph; Divide the new computation graph into sub-computation graphs; as well as The computationally intensive operators are fused in each of the sub-computation graphs to generate an optimized computation graph.

14. The apparatus according to claim 13, wherein The processor circuit module is configured to fuse the memory-intensive operator into the compute-intensive operator by: Fuse one or more sequential memory-intensive operators into the preceding or following compute-intensive operator.

15. The apparatus according to claim 13, wherein The new computation graph includes a plurality of layers, and each of the layers includes a plurality of computationally intensive operators, and The new computation graph is divided into the sub-computation graphs based on the output attributes of each layer of the new computation graph and the platform capacity of the platform on which the new computation graph is to be executed.

16. The apparatus according to claim 15, wherein The processor circuit module is configured to divide the new computation graph into the sub-computation graphs in the following manner: Based on the output characteristics of each layer, obtaining a partitioning parameter by means of a heuristic rule; Based on the output characteristics of each layer, obtaining a buffer size of the layer allocated for the output batch and weight of the layer; as well as Based on the partitioning parameters, the buffer size, and the platform capacity, the layers are sequentially partitioned into the sub-computation graphs according to the topological order of the new computation graph.

17. The apparatus according to claim 16, wherein The processor circuit module is configured to sequentially partition the layers into the sub-computation graphs according to the topological order of the new computation graph based on the partitioning parameters, the buffer size, and the platform capacity in the following manner: obtaining a reduced buffer size for each layer based on the buffer size and the partitioning parameter; as well as One or more consecutive layers are divided into sub-computation graphs, wherein for the one or more consecutive layers, the sum of the reduced buffer sizes of the one or more consecutive layers is less than or equal to the platform capacity, and the sum of the reduced buffer sizes of the one or more consecutive layers and a subsequent layer is greater than the platform capacity.

18. The apparatus according to claim 17, wherein Each of the compute-intensive operators of the new computation graph outputs an output activation, and the output activations of the compute-intensive operators of each layer form an output batch, and The output attributes include the size of an output batch of each layer of the new computation graph and / or the size of an output activation of each compute-intensive operator.

19. The apparatus according to claim 18, wherein The division parameters include batch division number and space division number, wherein the batch division number corresponds to the number of sub-batches into which the output batch is to be divided, and The number of spatial divisions corresponds to the number of sub-activations into which the output activation is to be divided.

20. The apparatus according to claim 19, wherein The reduced buffer size is expressed as follows: where a R_i Indicates the reduced buffer size of the i-th layer of the new computation graph in the topological order, w i Indicates the weight of the i-th layer, a i indicates the buffer size of the i-th layer, x indicates the batch division number, y indicates the space division number, and the a R_i 、i、w i 、a i Each of , x, and y is greater than 0.

21. The apparatus according to claim 20, wherein The platform capacity includes the size of a data cache unit (DCU) and the size of a middle layer cell (MLC) of a central processing unit (CPU) on which the new computation graph is to be executed.

22. The apparatus according to claim 21, wherein For each subgraph: Wherein, N indicates the starting layer of the sub-computation graph, M indicates the last layer of the sub-computation graph, T indicates a threshold value set based on the CPU, L1 indicates the size of the DCU, L2 indicates the size of the MLC, N, M, L1 and L2 are greater than 0, and T is greater than 0 and less than or equal to 1.

23. The apparatus according to any one of claims 19 to 22, wherein: The processor circuit module is configured to fuse the computationally intensive operators in each of the sub-computation graphs in the following manner: for each sub-computation graph, Divide the output batch of each layer of the sub-computation graph into the sub-batches according to the batch division number; Dividing the output activation of each computationally intensive operator of the sub-computation graph into the sub-activations by the spatial partition number; as well as The computationally intensive operators are fused in the sub-computation graph based on the sub-batch and the sub-activation.

24. A computer-readable medium having stored thereon instructions that, when executed by a processor, cause the processor to: Obtaining a deep learning computational graph that includes both compute-intensive and memory-intensive operators; fusing the memory-intensive operator into the computation-intensive operator to generate a new computation graph; Dividing the new computation graph into sub-computation graphs; as well as The computationally intensive operators are fused in each of the sub-computation graphs to generate an optimized computation graph.

25. The computer-readable medium of claim 24, wherein: The new computation graph includes a plurality of layers, and each of the layers includes a plurality of computationally intensive operators, and The new computation graph is divided into the sub-computation graphs based on the output attributes of each layer of the new computation graph and the platform capacity of the platform on which the new computation graph is to be executed.

Images