Hierarchical partitioning of operators

Abstract

Methods and apparatus are provided for hierarchical partitioning of operators of a neural network for execution on an acceleration engine. A neural network is constructed in a machine learning framework using neural network operators. The neural network operators are compiled into executable code for the acceleration engine. Development of new framework-level operators can outpace the ability to map newly developed framework-level operators to the acceleration engine. To enable the neural network to execute on the acceleration engine, the operators of the neural network can be partitioned using hierarchical partitioning. The hierarchical partitioning can identify operators that a compiler supports for execution on the acceleration engine, operators that are compiled for execution on a host processor, and operators that are to be executed on the machine learning framework.

Description

Background Technology

[0001] Neural networks are built within machine learning frameworks such as TensorFlow, PyTork, and MXNet by combining operators (e.g., convolutional neural networks (CNNs), nonmaximum suppression, ragged softmax, top-k, etc.). Thousands of framework-level operators are available, and the development of new framework-level operators may exceed the capacity to map them to deep learning accelerators. Neural networks built using newly developed framework-level operators may not be able to execute on deep learning accelerators. Attached Figure Description

[0002] Various embodiments according to this disclosure will be described with reference to the accompanying drawings, in which:

[0003] Figure 1 This is a diagram illustrating an example of hierarchical partitioning of a neural network operator according to some aspects of this disclosure;

[0004] Figure 2 This is a flowchart illustrating an example of a partitioning method according to some aspects of this disclosure;

[0005] Figure 3 A block diagram is included illustrating an instance of a host system on which a compiler and partitioner according to some aspects of this disclosure may be executed;

[0006] Figure 4 This is a block diagram illustrating an example of an integrated circuit device according to some aspects of this disclosure;

[0007] Figure 5 A block diagram illustrating an example of an acceleration engine according to some aspects of this disclosure; and

[0008] Figure 6 A block diagram illustrating an example of a host system according to some aspects of this disclosure. Detailed Implementation

[0009] Machine learning frameworks like TensorFlow and PyTorch contain a large number of operators that can be executed on accelerators. An accelerator, also known as a speedup engine, is an integrated circuit designed to efficiently perform processing operations on neural networks. To enable accelerators to perform these operations, neural network operators are compiled into executable code for the accelerator. Neural network operators will be simply referred to as "operators" in this paper. Neural networks execute different operators sequentially. For example, an operator could be convolution, where computation is performed on the input image. Another operator could be matrix multiplication (e.g., MatMul). There are many types of operators, such as nonmaximum suppression and ragged softmax.

[0010] Many operators are common across different machine learning frameworks, but each framework may also support some different operators. For example, new operators are created because neural network developers identify new operations and create macros to perform these new operations, as macros can perform operations efficiently.

[0011] Operators are added separately to the accelerator's compiler. The increasing number of new operators in machine learning frameworks may exceed the accelerator compiler's ability to support them. Therefore, neural networks containing operators that the accelerator compiler does not support may cause the accelerator compiler to fail to compile the neural network correctly.

[0012] In the following description, various examples will be described. Specific configurations and details are set forth for ease of explanation, in order to provide a thorough understanding of the examples. However, it will be apparent to those skilled in the art that the examples can be implemented without specific details. Furthermore, well-known features may be omitted or simplified so as not to obscure the described embodiments.

[0013] Accelerating neural networks on deep learning accelerators can be implemented by isolating operators that can run efficiently on deep learning accelerators. Deep learning accelerators may be simply referred to herein as "accelerators" or "acceleration engines." In some cases, hierarchical partitioning of operators can be implemented. The systems, methods, and apparatuses according to this disclosure can provide multi-level, parameter-aware, fault-tolerant, or any combination of these partitioning methods.

[0014] Multi-level partitioning allows operators to execute within a machine learning framework on the host processor, either natively on the host processor or on an accelerator. Furthermore, multi-level partitioning enables the building of additional machine learning inference or training services on top of it.

[0015] The parameter awareness of partitioning methods allows for the execution of a given instance of an operator on an accelerator. An operator can have many different parameters. For example, the max pooling operator can accept different input tensor shapes and data types, strides, kernel sizes, padding, data formats, etc. In some cases, not all combinations of operator parameters can be effectively mapped to deep learning accelerators. Therefore, the parameter awareness capability of partitioning methods can determine whether a given instance of an operator with a specific set of parameters can be executed on an accelerator.

[0016] Partitioning methods may fail due to downstream failures, such as unsupported combinations of operator parameters. Testing all combinations of parameters and operators is impractical. Failure handling capabilities of partitioning methods can provide a way to address downstream failures.

[0017] Neural networks can be developed on machine learning frameworks. Various operators or operator parts of a neural network can execute on a host processor running the machine learning framework, execute natively on the host processor, or execute on an accelerator. Compilers can compile operators for different kinds of target devices. In some cases, compilers can compile operators for both accelerators and one or more processors in the host computer system. For example, a compiler might support a portion of the operator vectors, but the accelerator hardware might only support a subset of the vectors. In these cases, the remaining vectors of the operator can be compiled to run on the host processor.

[0018] According to some aspects of this disclosure, a method for hierarchical partitioning of neural network operators is provided. Figure 1 This is a diagram illustrating an example of hierarchical partitioning of a neural network operator, based on some aspects of this disclosure. (See reference) Figure 1 The example of flow graph 110 shows a series of neural network operators, including: FOR loop operator 115, recurrent neural network operator (RNN) 120, addition operator (ADD) 125, activation operator (ACT) 130, and nonmaximum suppression operator (NMS) 135.

[0019] The initial partitioning of neural network operators can be performed by a partitioner 165 running on a host processor, such as CPU0 155, on a host computer system 150. The initial partitioning determines which operators are executed locally (e.g., via a machine learning framework), and the partitioner can be determined independently of the compiler. Partitioner 165 can be software executing on CPU0 155. In some embodiments, the partitioner can be a circuitry system, either internal or external to the processor, configured to perform partitioning operations on neural network operators. The initial partitioning performed by partitioner 165 allows compiler-supported neural network operators to be passed to the compiler, and compiler-independent neural network operators to be passed to a machine learning framework 160 executing on a host processor, such as CPU0 155. Within the compiler, additional partitioning can be performed based on whether a given neural network operator should be executed by, for example, the processing engine of accelerator 180, or whether a given neural network operator should be executed by, for example, the processor of host processor CPU0 170. This auxiliary partitioning separates operators that can run efficiently on the accelerator from those that cannot run at all.

[0020] refer to Figure 1 Partitioner 165 can determine NMS operator 135, a neural network operator not supported by the compiler. Partitioner 165 can determine to execute the NMS operator on machine learning framework 160, but the remaining neural network operators (FOR 115, RNN 120, ADD 125, ACT 130) can be sent to the compiler.

[0021] The compiler can determine that FOR operator 115 is supported on a processor such as CPU1 170, but not on accelerator 180. The compiler can compile FOR operator 115 to execute on a host processor such as CPU1 170. The compiler can compile the remaining neural network operators (RNN 120, ADD 125, ACT 130) to execute on accelerator 180. Therefore, a two-tier partitioning can be implemented: one tier is integrated into the machine learning framework itself and is framework-specific; the other tier is used to identify neural network operators that are not suitable for the accelerator.

[0022] Although the above example is explained using two host processors, fewer or more host processors may be used without departing from the scope of this disclosure. Furthermore, although the above example uses an accelerator as the processing engine, other processing engines, such as (but not limited to) activation engines, pooling engines, etc., may be used without departing from the scope of this disclosure.

[0023] In some implementations, the compiler can inform the partitioner that neural network operators can be compiled by providing it with a list of supported neural network operators (e.g., a "whitelist"). The whitelist may contain operators such as, but not limited to, addition, convolution, RNN, MatMul, sigmoid, max pooling, and other operators supported by compilers of various machine learning frameworks. The whitelist is independent of the network being compiled. The partitioner may receive the operator whitelist from the compiler and perform first-level partitioning based on the whitelist. As used herein, partitioning operators can refer to allocating said operators from a set of operators of a neural network for compilation or execution on one of a machine learning framework, a host processor, or an acceleration engine. Neural network operators not included in the whitelist cannot be compiled by the compiler but can be directly executed by a machine learning framework operating on a host processor.

[0024] In some implementations, interactive feedback from the compiler to the partitioner can be provided. The partitioner can query the compiler to determine whether the compiler can support a specific instance of an operator. For example, the partitioner can query the compiler for a given operator name and specified parameters of that operator. The compiler can then provide feedback to the partitioner, indicating whether the compiler can support a specific instance of the given operator. The partitioner can then determine whether to partition the operator instance by assigning it to the compiler or to execute the given operator instance on a machine learning framework.

[0025] The remaining neural network operators that are supported by the compiler can be sent to the compiler. Within the compiler, a second level of partitioning can be performed to identify neural network operators that are not suitable for the accelerator (or other processing engine). The compiler can then partition the operators that are not suitable for execution on the accelerator engine for compilation and execution on the processor. The compiler can generate a compilation model that includes a processor execution model for neural network operators that cannot be executed on the accelerator and an inference accelerator model for neural network operators that can be executed on the accelerator engine.

[0026] Each neural network operator can have a set of associated parameters for performing its operations. For example, a convolution operator can have "stride" and "padding" parameters. Different instances of the same neural network operator can use different parameters from this set of associated parameters. In some cases, even if included in the compiler-provided operator whitelist, different instances of a neural network operator using a particular set of associated parameters may not be supported by the processing engine (e.g., an acceleration engine) and may instead be partitioned for execution on the host processor.

[0027] When a compiler receives a neural network with specific operators, each instance of the operator specifies the associated parameters to be used. Based on the specified parameters used by a specific instance of the operator, the compiler can determine whether that specific instance of the operator can be supported by an accelerator. For example, the input feature map may be too large for the accelerator to accommodate. When the compiler decides that a specific instance of the operator is not supported due to its parameters, the compiler can partition the instance of the operator for execution on a processor (e.g., CPU1170).

[0028] Because accelerators may not support all combinations of operator parameters, partitioning can be parameter-aware, meaning the compiler considers operator instances with different associated parameters. The compiler can implement various procedural aspects of whitelisting to evaluate operator parameters. The compiler can examine all parameters of an operator instance to determine if the accelerator (or other processing engine) supports an operator instance with specific parameters. Based on whether the accelerator supports these parameters, the compiler can determine whether the operator is executed on a processor, such as the host processor, or on the accelerator. When the compiler determines that an operator will not execute on the accelerator, it can compile the operator for execution on the processor.

[0029] The compiler can pre-define an operator whitelist to identify supported operator names. Parameters, however, are not pre-declared in a whitelist because there are many of them. For each specific neural network, parameters are processed programmatically. The compiler can evaluate whether an operator with a given combination of parameters is suitable for accelerator execution. If an operator is not supported, or cannot be effectively supported, it can be executed by the processor instead of the accelerator.

[0030] The compiler can implement a programmatic whitelist containing code that processes each operator and its parameters in sequence. If the compiler determines a combination of operators whose parameters are not supported by the accelerator, it can repartition the operators by assigning them to be executed on a processor (e.g., the processor that manages the accelerator).

[0031] Parameter partitioning can be implemented via a callback mechanism, where operators are parsed to extract their names and various parameters passed to the serializer to generate accelerator code. The callback mechanism may return a value indicating that a particular combination of parameters or operators is unsupported when a specific combination of operators is not supported. In some cases, different instances of the same operator or different combinations of specific operators may be supported.

[0032] Compiler partitioning can also involve efficiency. Operators can be performance-analyzed for execution on processors and accelerators to determine more efficient operator execution platforms. For example, the FOR operator might functionally run on an accelerator but be slower, and could run much faster (e.g., more efficiently) on a processor. The compiler might take this efficiency into account and partition the FOR operator for execution on processors. In some cases, partitioning decisions can be related to execution efficiency or compiler efficiency. For example, certain operator sequences or operators with specific parameter combinations might compile faster on a processor compiler than on an accelerator compiler.

[0033] Therefore, for neural networks with unknown operators or operators that do not support execution on accelerators, compiler failures can be reduced by partitioning known neural network operators for execution on accelerators, while neural network operators that do not support execution on accelerators can be executed on the main processor.

[0034] Various aspects of this disclosure provide for handling compilation failures. In some cases, operators can be partitioned for execution on an acceleration engine. During the compilation of the acceleration engine, if an error occurs, the operators compiled for the acceleration engine may be repartitioned for execution on a machine learning framework. The compilation process of the acceleration engine may fail because of errors in the compiler code that prevent the compilation of specific operators, or because a specific sequence of operators being compiled may cause failure. For example, a convolution operator followed by a pooling operator may cause a kernel size mismatch, resulting in a compiler failure. When a failure occurs, the compiler may generate error messages or code. The compiler can repartition the affected operators for compilation and execution on a machine learning framework.

[0035] Figure 2 This is a flowchart illustrating an example of partitioning method 200 according to some aspects of this disclosure. (See also:) Figure 2At box 210, a neural network generated by a machine learning framework can be received. The neural network can be received by a partitioner configured to determine whether the operators of the neural network are supported by the compiler or whether the operators should be executed on the machine learning framework.

[0036] At box 220, a list of supported operators can be received from the compiler. The compiler can inform the partitioner that neural network operators can be compiled by providing the partitioner with a list of supported neural network operators (e.g., a "whitelist"). The whitelist may contain operators such as, but not limited to, addition, convolution, RNN, MatMul, sigmoid, max pooling, and other operators that are supported by compilers of various machine learning frameworks. The whitelist is independent of the network being compiled.

[0037] At box 230, it can be determined whether the operators from the neural network are supported by the compiler. The partitioner compares the neural network operators with a whitelist to determine whether each operator of the neural network is supported by the compiler or whether the operator should be executed on the machine learning framework.

[0038] In response to determining that an operator is not supported by the compiler (230-No), at box 235, the operator can be partitioned by assigning it to execution on a machine learning framework. The partitioner determines that a neural network operator is not in the whitelist and partitions the operator by assigning it to execution on a machine learning framework.

[0039] In response to determining that an operator is supported by the compiler (230-Yes), at box 240, it can be determined whether the acceleration engine supports the execution of the operator. Within the compiler, executable partitions are used to identify neural network operators that are not suitable for the acceleration engine. The compiler can partition operators that are not suitable for execution on the acceleration engine for compilation and execution on the processor natively.

[0040] In response to determining that an operator is not supported by the acceleration engine (240-No), at box 245, the operator can be partitioned for compilation and execution on the processor natively. For example, the compiler can partition operators that are not suitable for execution on the acceleration engine and separate those operators for execution on the processor natively.

[0041] In response to determining that the acceleration engine supports the execution of the operator (240-Yes), at box 250, it can be determined whether the acceleration engine supports the execution of a specific instance of the operator. Each neural network operator may have a set of associated parameters for performing its operations. Different instances of the same neural network operator may use different parameters from said set of associated parameters. In some cases, even if included in the operator whitelist provided by the compiler, different instances of a neural network operator using a particular set of associated parameters may not be supported by the acceleration engine.

[0042] The compiler can implement various procedural aspects of the whitelist to evaluate operator parameters. The compiler can examine all parameters of an operator instance to determine if the accelerator (or other processing engine) supports an operator instance with specific parameters. Based on whether the accelerator engine supports these parameters, the compiler can determine whether the operator will be executed on the accelerator engine.

[0043] Parameter partitioning can be implemented via a callback mechanism, where operators are parsed to extract their names and various parameters passed to the serializer to generate accelerator code. The callback mechanism may return a value indicating that a particular combination of parameters or operators is unsupported when a specific combination of operators is not supported. In some cases, different instances of the same operator or different combinations of specific operators may be supported.

[0044] In response to determining that the acceleration engine does not support a specific instance of the operator (250-No), at box 245, the operator can be partitioned for compilation and execution on the processor. In response to determining that the acceleration engine supports the execution of a specific instance of the operator (250-Yes), at box 260, the operator can be compiled for execution on the acceleration engine.

[0045] At box 270, it can be determined whether a failure occurred during the compilation of the operator for execution on the acceleration engine. For example, the compilation process of the acceleration engine may fail because of a bug in the compiler code that prevents the compilation of a specific operator, or because a specific sequence of operators being compiled may cause a failure. The compiler may generate error messages or codes indicating the failure.

[0046] In response to determining that no failure occurred (270-No), the partitioning process ends at box 275. In response to determining that a failure occurred during the compilation of the operator for execution on the acceleration engine (270-Yes), at box 245, the compiler can repartition the operator by assigning it for execution on the machine learning framework.

[0047] Figure 3 A block diagram is included illustrating an example of a host system 300 on which a compiler 330 and a partitioner 334 as described herein can be executed. The host system 300 shown is an example of a computing device and includes a set of processors 302, 303, a processor memory 304, at least one storage device 306, various input / output (I / O) devices 308, and at least one network interface 310. Figure 3In some instances, host system 300 also includes acceleration engine 312, which is an integrated circuit device that can accelerate certain operations or calculations performed by host system 300. In various instances, host system 300 may be implemented as a server in a data center, a desktop computer, a laptop computer, a tablet computer, or a smartphone, and other instances. In some instances, operations or components described below as performing or included in host system 300 may perform or be included in other computer devices. For example, compiler 330 and partitioner 343 may execute on host system 300, while acceleration engine 312 resides on another host system.

[0048] Processors 302 and 303 are integrated circuit devices capable of executing program code in the form of instructions. In some embodiments, processors 302 and 303 may be processing cores of the same integrated circuit, each configured to independently execute various software applications or tools. The program code can be used in various software applications or tools, such as operating system 320, machine learning framework 332, or compiler 330. While processors 302 and 303 are executing a program, program instructions may be stored in processor memory 304. Instructions may also be stored elsewhere, such as on storage device 306, and may be loaded into processor memory 304 when needed by processors 302 and 303. Processors 302 and 303 may also use processor memory 304 to temporarily store other data that processors 302 and 303 are operating on. In various instances, processor memory 304 is a volatile memory type, such as a certain type of random access memory, but processor memory 304 may alternatively or additionally use a non-volatile memory type.

[0049] Storage device 306 is an example of a device that may include non-volatile memory. For example, storage device 306 may be a disk drive, a solid-state drive, or an optical disk drive, and other examples. Storage device 306 may also be non-transitory, such that program code and other data stored on storage device 306 remain when storage device 306 is not powered on.

[0050] Storage device 306 is an example of a peripheral device, which is a component that can be coupled to host system 300 to add functionality to host system 300. Other examples of peripheral devices include input / output device 308 and network interface 310. Input / output device 308 may include user input and output devices such as keyboard, mouse, touchscreen, microphone, display, speaker, printer, and scanner, and other examples. Network interface 310 may be implemented using a network interface card, which provides access to one or more networks. For example, network interface 310 may include physical ports for connecting network cables and / or wireless antennas to communicate with Wi-Fi and / or cellular networks. Network interface 310 may also be described as an I / O device.

[0051] Acceleration engine 312 is also another type of peripheral device or I / O device. Acceleration engine 312 is a device dedicated to performing certain operations that can be performed by processors 302 and 303, but can be performed faster or more efficiently by acceleration engine 312. For example, acceleration engine 312 can be a neural network accelerator, and therefore can perform massively parallel computations of neural networks more efficiently than when computations are performed by processors 302 and 303. As another example, acceleration engine 312 can be a graphics processing unit (GPU) and can be optimized for performing the computations required for graphics rendering. Other examples of devices that can be implemented by acceleration engine 312 include encryption accelerators, compression and decompression accelerators, 3D accelerators, regular expression accelerators, security accelerators, etc.

[0052] In various instances, the acceleration engine 312 can execute program code to perform certain operations. For example, when the acceleration engine 312 is a neural network accelerator, it can be programmed to execute a specific neural network, such as a neural network for image recognition or a neural network for machine translation. As another example, to support the execution of a neural network, the acceleration engine 312 can be programmed to perform operations such as copying neural network data from processor memory 304 (e.g.) to the acceleration engine 312, copying neural network input data from processor memory 304 to the acceleration engine 312, and / or copying the results of the acceleration engine 312 to processor memory 304, and other instances.

[0053] In various instances, to generate the program code for the acceleration engine 312, the host system 300 can execute a compiler 330. Generally, a compiler is a software program that translates program code written in a human-readable language into a format (e.g., machine instructions) that can be read and processed by integrated circuit devices. Figure 3In this example, acceleration engine 312 is a neural network accelerator, and compiler 330 can compile the portion describing the neural network into instructions to be executed by acceleration engine 312 and processor 303. When acceleration engine 312 implements another type of accelerator, another compiler can be used.

[0054] The partitioner 334 of the machine learning framework 332 can perform a first-level partitioning of neural network operators based on the input code 342 to identify the neural network operators to be executed by the machine learning framework 332 and the neural network operators to be sent to the compiler. The output of the partitioner 334 can be neural network operators included in the neural network operator whitelist received from the compiler 330. For example, when the partitioner 334 identifies a neural network operator supported by the compiler 330 based on the input code 342, the compiler 330 can be activated. Once the compiler 330 is activated, for example, the processor of the processor 303 can load the instructions of the compiler 330 into the processor memory 304 and execute the instructions.

[0055] Within compiler 330, a second-level partition can be executed to identify neural network operators that are not suitable for acceleration engine 312. Compiler 330 can generate a compilation model that has a processor execution model for neural network operators that cannot be executed on acceleration engine 312 and an inference accelerator model for neural network operators that can be executed on acceleration engine 312.

[0056] exist Figure 3 In this example, compiler 330 includes a first level 336 and a second level 340, each performing different operations to produce compiled code. In other examples, compiler 330 may combine the operations of the first level 336 and the second level 340 into fewer levels, or may divide the operations of one level into multiple levels.

[0057] The first stage 336 may perform intermediate processing on the whitelisted neural network operators output from the partitioner 334. For example, the first stage 336 may determine operations that can be performed by the acceleration engine 312 and operations that can be performed natively by the processor 303. The processing of the first stage 336 may include other steps, such as arranging or determining the order in which the acceleration engine 312 and / or the processor 302 perform operations, and other instances thereof.

[0058] The first stage 336 can evaluate operators received from the partitioner 334 to determine whether the operators are supported for execution on the acceleration engine 312, and can also partition the operators for execution on processors such as processor 303. When the compiler 330 receives a neural network with a specific operator, each instance of the operator specifies associated parameters to be used. Based on the specified parameters used by a specific instance of the operator, the first stage 336 of the compiler 330 can determine whether the acceleration engine 312 can support a specific instance of the operator. When the first stage 336 of the compiler 330 determines that the acceleration engine 312 cannot support the operator, the first stage 336 can partition the operator by assigning it to be compiled by the second stage 340 for execution on processors such as processor 303.

[0059] Level 1 336 can generate a compilation model containing various operations performed by the components of acceleration engine 312 on the supported operators in the order they are to be executed. For example, the output can be represented as a data flow graph, where nodes in the graph represent memory operations, computations, and other operations, and edges or connections between nodes represent dependencies between nodes, such as data dependencies, memory dependencies, or operation dependencies, and other instances.

[0060] The second level 340 can compile the operators from the first level 336 partition to the second level 340 and execute various operations to generate instructions to be executed by the processor 303. For example, these operations may include removing redundant dependencies, resolving or handling dependencies between nodes by inserting synchronization instructions into the code, identifying possible optimizations for memory usage or memory bandwidth usage, and other operations.

[0061] The second level 340 can generate a compiled model that includes various operations performed by the components of processor 303 on the supported operators in the order in which the operations are to be executed. For example, the output can be represented as a data flow graph, where nodes in the graph represent memory operations, computations, and other operations, and edges or connections between nodes represent dependencies between nodes, such as data dependencies, memory dependencies, or operation dependencies, and other instances.

[0062] exist Figure 3 In this example, host system 300 may be executing driver 322 that manages acceleration engine 312; this driver may also be referred to as a device driver or runtime driver. Driver 322 provides an interface between the host system 300 (or another host system) and an application running on acceleration engine 312. For example, driver 322 may provide an application programming interface (API) that defines functions for feeding input data to acceleration engine 312 and defining operations performed on the input data.

[0063] In this and other instances, driver 322 can configure acceleration engine 312 to perform operations. For example, driver 322 can identify the neural network that acceleration engine 312 wants to execute, and the location of the compiled code of the neural network in processor memory 304 or on storage device 306. Driver 322 can also load into or cause acceleration engine 312 to load compiled code, load input data for the neural network to operate on, or cause acceleration engine 312 to load said input data, and / or cause acceleration engine 312 to execute on the input data. Once acceleration engine 312 finishes, acceleration engine 312 can notify driver 322, and driver 322 can deliver the results back to the application that requested the results.

[0064] Figure 4 This is a block diagram illustrating an example of an integrated circuit device capable of implementing hierarchical partitioning of neural network operators. Figure 4 The example illustrates accelerator 402. In various instances, for a set of input data (e.g., input data 450), accelerator 402 can perform computations using processing engine array 410, activation engine 416, and / or pooling engine 418. In some instances, instance accelerator 402 can be an integrated circuit component of a processor such as a neural network processor. The processor may have other integrated circuit components containing additional accelerator engines.

[0065] In various implementations, memory subsystem 404 may include multiple memory groups 414. In these implementations, each memory group 414 is accessible independently, meaning that a read of one memory group does not depend on a read of another memory group. Similarly, a write to a memory group does not affect or limit a write to another memory group. In some cases, each memory group can be read and written simultaneously. Various techniques can be used to enable independent access to memory groups 414. For example, each memory group may be a physically independent memory component with a separate address space independent of the address spaces of each other memory group. In this example, each memory group may have at least one read channel and at least one separate write channel that can be used simultaneously. In these examples, memory subsystem 404 may allow simultaneous access to the read or write channels of multiple memory groups. As another example, memory subsystem 404 may include arbitration logic such that arbitration, for example, between the outputs of multiple memory groups 414, allows the outputs of more than one memory group to be used. In these and other examples, although managed overall by memory subsystem 404, each memory group can operate independently of any other memory group.

[0066] Enabling independent access to memory bank 414 can improve the efficiency of accelerator 402. For example, values ​​can be read and served to each row of processing engine array 410 simultaneously, allowing the entire processing engine array 410 to be used within a single clock cycle. As another example, memory bank 414 can be read while the results computed by processing engine array 410 are written to memory subsystem 404. In contrast, a single memory may only be able to serve one read or write at a time. With a single memory, multiple clock cycles might be required, for example, to read input data from each row of processing engine array 410 before processing engine array 410 can begin.

[0067] In various implementations, memory subsystem 404 can be configured to serve multiple clients simultaneously, including processing engine array 410, activation engine 416, pooling engine 418, and any external clients accessing memory subsystem 404 via communication structure 420. In some implementations, the ability to serve multiple clients may mean that memory subsystem 404 has at least as many memory groups as the number of clients. In some cases, each row of processing engine array 410 can be considered a separate client. In some cases, each column of processing engine array 410 can output results, such that each column can be considered a separate write client. In some cases, the output of processing engine array 410 can be written to memory group 414, which can then provide input data to processing engine array 410. As another example, activation engine 416 and pooling engine 418 can include multiple execution channels, each of which can be a separate memory client. For example, memory group 414 can be implemented using static random access memory (SRAM).

[0068] In various implementations, the memory subsystem 404 may include control logic. For example, the control logic may record the address space of each of the memory groups 414, identify the memory group 414 to be read from or written to, and / or move data between memory groups 414. In some implementations, the memory groups 414 may be hardwired to specific clients. For example, a set of memory groups 414 may be hardwired to provide values ​​to rows of the processing engine array 410, with one memory group serving one row. As another example, a set of memory groups may be hardwired to receive values ​​from columns of the processing engine array 410, with one memory group receiving data for each column.

[0069] The processing engine array 410 is the computation matrix of the instance accelerator 402. For example, the processing engine array 410 can perform parallel integration, convolution, correlation, and / or matrix multiplication, etc. The processing engine array 410 includes multiple processing engines 411 arranged in rows and columns, such that the output of one processing engine 411 can be directly input into another processing engine 411. Therefore, processing engines 411 not located on the outer edge of the processing engine array 410 can receive data from other processing engines 411 instead of the memory subsystem 404 for operation.

[0070] In various instances, the processing engine array 410 uses pulsed execution, where data arrives at each processing engine 411 from different directions at regular intervals. In some instances, input data may flow into the processing engine array 410 from the left, and weight values ​​may be loaded at the top. In some instances, weights and input data may flow from the left, and partial sums may flow from the top to the bottom. In these and other instances, the multiply-accumulate operation moves through the processing engine array 410 as a diagonal wavefront, where data moves downward to the right across the entire array. Control signals may be input simultaneously with the weights at the left and may flow laterally and vertically along with the computation.

[0071] In various implementations, the number of columns in the processing engine array 410 determines the computing capacity of the processing engine array 410, and the number of rows determines the memory bandwidth required to achieve maximum utilization of the processing engine array 410. The processing engine array 410 may have, for example, 64 columns and 428 rows, or some other number of columns and rows.

[0072] Figure 4 An example of processing engine 411 is illustrated in the figure. As shown in this example, processing engine 411 may include multiplier-accumulator circuitry. The input from the left side may include, for example, input data i and weight values ​​w, where the input data is a set of values ​​taken from a set of input data or a set of intermediate results, and the weight values ​​are a set of weight values ​​that connect one layer of the neural network to the next layer. For example, a set of input data may be an image submitted for recognition or object identification, an audio clip provided for speech recognition, a text string for natural language processing or machine translation, or the current state of a game that needs to be analyzed to determine the next action, etc. In some instances, the input data and weight values ​​are output to the right side for input to the next processing engine 411.

[0073] In the example shown, the input from above may contain a partial sum p_in provided by a previous round of computation from another processing engine 411 or from the processing engine array 410. When computation begins for a new set of input data, the top row of the processing engine array 410 may receive a fixed value for p_in, such as zero. As shown in this example, i and w are multiplied, and the result is summed with p_in to produce a new partial sum p_out that can be input into another processing engine 411. Various other implementations of the processing engine 411 are possible.

[0074] The output from the last row in the processing engine array 410 can be temporarily stored in the result buffer 412. The result can be an intermediate result, which can be written to the memory bank 414 to be provided to the processing engine array 410 for further computation. Alternatively, the result can be a final result, which, once written to the memory bank 414, can be read from the memory subsystem 404 via the communication structure 420 for system output.

[0075] In some implementations, accelerator 402 includes activation engine 416. In these implementations, activation engine 416 can combine results from processing engine array 410 into one or more output activations. For example, for a convolutional neural network, convolutions from multiple channels can be summed to produce an output activation for a single channel. In other instances, it may be necessary to accumulate results from one or more columns in processing engine array 410 to produce an output activation for a single node in the neural network. In some instances, activation engine 416 can be bypassed.

[0076] In various instances, the activation engine 416 may contain multiple individual execution channels. In these instances, the execution channels may correspond to columns of the processing engine array 410 and can perform operations on the outputs of the columns, with the results stored in the memory subsystem 404. In these instances, the activation engine 416 can perform one to n parallel computations, where n equals the number of columns in the processing engine array 410. In some cases, one or more computations can be performed simultaneously. Examples of computations that each execution channel can perform include exponential, square, square root, identity, binary step, bipolar step, sigmoidal, and ramp computations, among others.

[0077] In some implementations, accelerator 402 may include a pooling engine 418. Pooling is a combination of the outputs of columns from processing engine array 410. Combinations may include, for example, calculating maximum, minimum, average, median, sum, multiplication, or another logical or mathematical combination. In various instances, pooling engine 418 may include multiple execution channels that operate on the values ​​of corresponding columns from processing engine array 410. In these instances, pooling engine 418 is capable of performing 1 to n parallel computations, where n equals the number of columns in processing engine array 410. In various instances, the execution channels of pooling engine 418 may operate in parallel and / or simultaneously. In some instances, pooling engine 418 may be bypassed.

[0078] Here, activation engine 416 and pooling engine 418 can be collectively referred to as execution engines. Processing engine array 410 is another instance of an execution engine. Another instance of an execution engine is a direct memory access (DMA) engine that can be located outside of accelerator 402.

[0079] Input data 450 can be reached via communication structure 420. Communication structure 420 connects accelerometer 402 to other components of the processor, such as a DMA engine, memory driver, or network interface that can obtain input data 450 from input / output (I / O) devices. For example, input data 450 can be one-dimensional data, such as a string or numerical sequence, or two-dimensional data, such as an array of pixel values ​​of an image or time-varying frequency and amplitude values ​​of an audio signal. In some instances, input data 450 can be three-dimensional, such as for contextual information used in autonomous vehicles or virtual reality data. In some embodiments, memory subsystem 404 may include a separate buffer for input data 450. In some embodiments, input data 450 may be stored in memory group 414 when accelerometer 402 receives input data 450.

[0080] In some instances, accelerator 402 may implement a neural network processing engine. In these instances, given a set of input data 450, accelerator 402 may execute a neural network to perform a task that the neural network has been trained on. Executing a neural network on a set of input data may be referred to as inference or performing inference.

[0081] The weights of the neural network, along with the input data 450 to which the neural network will operate, can be stored in the memory subsystem 404. The neural network may also include instructions that program the processing engine array 410 to perform various computations on the weights and input data. These instructions may also be stored in the memory subsystem 404, the memory bank 414, or a separate instruction buffer. The processing engine array 410 may output intermediate results representing the outputs of individual layers of the neural network. In some cases, the activation engine 416 and / or the pooling engine 418 may be enabled for computations invoked by certain layers of the neural network. The accelerator 402 may store intermediate results in the memory subsystem 404 to be input into the processing engine array 410 to compute the results of the next layer of the neural network. The processing engine array 410 may also output the final result from the last layer of the neural network. The final result may be stored in the memory subsystem 404 and then copied to the host processor memory or to another location.

[0082] Figure 5 The diagram includes an example of an acceleration engine 500. The acceleration engine 500 is an example of an integrated circuit that may include one or more accelerators 502a-502n, which may be similar to... Figure 4 The accelerator shown.

[0083] exist Figure 5 In the example, the acceleration engine 500 includes multiple accelerators 502a-502n, each of which can perform a set of operations. In various instances, accelerators 502a-502n are used for specific types of operations, making them significantly faster than a general-purpose processor when performing similar operations. In various instances, to perform a set of operations, the input data for those operations must first be moved into accelerators 502a-502n. Additionally, in some cases, program code that programs the operations performed on the data into accelerators 502a-502n is also moved into them. In the example shown, the acceleration engine 500 includes n accelerators 502a-502n. Examples of accelerators that can be included in the acceleration engine 500 include graphics accelerators, floating-point accelerators, neural network accelerators, etc. In various instances, accelerators 502a-502n can be the same accelerator (e.g., each of them is a graphics accelerator) or different accelerators (e.g., accelerators 502a-502n include graphics accelerators, floating-point accelerators and neural network accelerators).

[0084] The instance acceleration engine 500 also includes DRAM controllers 542a-542k for communicating with external memory. In this instance, the external memory is implemented using DRAM 530. In the illustrated instance, the acceleration engine 500 includes k DRAM controllers 542a-542k, each capable of communicating with an independent set of DRAM banks. In other instances, the external memory may use other types of RAM technology. The DRAM controllers 542a-542k may also be referred to as memory controllers.

[0085] In various instances, the input data and / or program code for accelerators 502a-502n may be stored in DRAM 530. Different programs may enable accelerators 502a-502n to perform different operations. For example, when one of the accelerators is a neural network accelerator, one program may configure the neural network accelerator to perform speech recognition, while another program may configure it to perform image recognition. In various instances, different accelerators 502a-502n may be programmed with different programs, causing each accelerator to perform a different set of operations. In various instances, processors 548a-548s may manage the movement of program code from DRAM 530 to accelerators 502a-502n.

[0086] The instance acceleration engine 500 also includes I / O controllers 544a-544p for communicating with I / O devices 532 in the system. The acceleration engine 500 may communicate with the I / O devices, for example, via a processor bus. In some instances, the processor bus may be implemented using Peripheral Component Interconnect (PCI) and / or a variant of the PCI bus protocol. The processor bus can connect the acceleration engine 500 to I / O devices, such as input and output devices, memory controllers, storage devices, and / or network interface cards, etc. In some instances, the I / O controllers 544-544p enable the acceleration engine 500 to act as an I / O device for the host processor. For example, the acceleration engine 500 may be a receiver of input data from the host processor and commands instructing the operation to be performed on the input data (e.g., a specific calculation or analysis). In the illustrated instance, the acceleration engine 500 includes p I / O controllers 544a-544p, where each I / O controller may contain a separate root complex and can communicate with a separate set of I / O devices 532. In other instances, the host bus may use other standardized bus protocols, such as Hyperpath Interconnect (UPI). In other instances, proprietary bus protocols may be used.

[0087] The movement of data within the acceleration engine 500 can be managed by one or more processors 548a-548s, which may also be referred to as data management processors. Figure 5In one example, the accelerator engine 500 includes several processors 548a-548s, which are incorporated into the device (e.g., on the same silicon die). In other examples, the processors 548a-548s may be external to the accelerator engine 500 (e.g., on another die and / or in another package). In some examples, the processors 548a-548s may manage the movement of data from I / O devices 532 to accelerators 502a-502n or DRAM 530. For example, input data may be located at I / O devices 532 or in processor memory, and the processors 548a-548s may move input from I / O devices 532 or processor memory to the accelerator or DRAM 530. As another example, the program code for accelerators 502a-502n may be located on I / O devices 532 or in processor memory.

[0088] The instance acceleration engine 500 also includes DMA engines 546a-546d, which can move data between accelerators 502a-502n, DRAM controllers 542a-542k, and I / O controllers 544a-544p. In the illustrated example, the acceleration engine 500 includes d DMA engines 546a-546d. In some implementations, the DMA engines 546a-546d can be assigned to specific tasks, such as moving data from DRAM controllers 542a-542d to accelerators 502a-502n, or moving data between I / O controllers 544a-544p and accelerators 502a-502n. For example, these tasks can be assigned by queuing descriptors using the DMA engines 546a-546d, where the descriptor identifies the address of a data block and the operation to be performed (e.g., read or write). For example, the descriptor can direct the DMA engine to instruct the DMA controller to read a data block from DRAM 530. As another example, a descriptor can instruct the DMA engine to write data read by the DMA controller to the accelerator. Other descriptors can be used to move data from the accelerator to the DRAM 530.

[0089] In various instances, each of the processors 548a-548s can be responsible for managing data movement to different accelerators. In some instances, a processor can manage data movement to more than one accelerator. Similarly, in various instances, each of the processors 548a-548s can be assigned to one or more DMA engines 546a-546d. In these and other instances, the association between the processors 548a-548s, accelerators 502a-502n, and DMA engines 546a-546d is determined by the program code being executed by each corresponding processor.

[0090] In the instance acceleration engine 500, various components can communicate via chip interconnect 520. Chip interconnect 520 primarily contains wiring for routing data between the components of the acceleration engine 500. In some cases, chip interconnect 520 may contain very little logic, such as multiplexers for controlling data direction, flip-flops for handling cross-clock domain issues, and timing logic.

[0091] Figure 6 Includes a block diagram showing an instance of a host system 600 in which the acceleration engine 660 can be used. Figure 6 The acceleration engine 660 can include, for example, Figure 5 Examples of one or more accelerator devices are shown in the figure. Figure 6 The instance host system 600 includes an acceleration engine 660, a host processor 672, DRAM 630 or processor memory, I / O devices 632, and a support system 674. In various embodiments, the host system 600 may include other hardware not shown herein.

[0092] The host processor 672 is a general-purpose integrated circuit capable of executing program instructions. In some instances, the host processor 672 may contain multiple processing cores. A multi-core processor may contain multiple processing units located within the same processor. In some instances, the host system 600 may contain more than one host processor 672. In some instances, the host processor 672 and the acceleration engine 660 may be a single chip, for example, one or more integrated circuits within the same package.

[0093] In various embodiments, host processor 672 may communicate with other components in host system 600 via one or more communication channels. For example, host system 600 may include a host processor bus, which host processor 672 may use to communicate with DRAM 630. As another example, host system 600 may include an I / O bus, such as a PCI-based bus, which host processor 672 may use to communicate with acceleration engine 660 and / or I / O device 632. In various embodiments, alternatively or additionally, host system 600 may include other communication channels or buses, such as serial buses, power management buses, memory device buses, etc.

[0094] In some instances, a software program executing on host processor 672 may receive or generate input for processing by acceleration engine 660. In some instances, the program may select an appropriate neural network to execute for a given input. For example, the program may be used for language translation and may select one or more neural networks capable of speech recognition and / or machine translation. In these and other instances, the program may configure acceleration engine 660 with the neural network to be executed and / or select a neural network processing engine on acceleration engine 660 that has been previously configured to execute the desired neural network. In some instances, once acceleration engine 660 begins inference of input data, host processor 672 may manage the movement of data (e.g., weights, instructions, intermediate results, results of conditional layers, and / or final results) into or out of acceleration engine 660.

[0095] In some instances, software programs using the acceleration engine 660 for inference can read the results of the conditional layer from the acceleration engine 660 and / or storage locations such as DRAM 630. In these instances, the program can determine what action the neural network should take next. For example, the program can determine to terminate inference. As another instance, the program can determine to change the direction of inference, which can be transferred from low-level code and / or the neural network processor to the next layer of execution. In these and other instances, the execution flow of the neural network can be coordinated by software.

[0096] DRAM 630 is a memory used by the host processor 672 to store program code being executed by the host processor 672 and values ​​being operated on. In some instances, neural network data (e.g., weight values, instructions, and other data) may be stored wholly or partially in DRAM 630. DRAM is a common term for processor memory, and although DRAM is volatile memory, processor memory can be volatile and / or non-volatile. Although not shown here, the host system 600 may include other volatile and non-volatile memories for other purposes. For example, the host system 600 may include read-only memory (ROM) storing boot code for powering on the host system 600, and / or Basic Input / Output System (BIOS) code.

[0097] Although not shown here, DRAM 630 may store instructions for various programs that can be loaded into and executed by the host processor 672. For example, DRAM 630 may store an operating system, one or more data storage areas, one or more application programs, one or more drivers, and / or instructions for implementing the features disclosed herein.

[0098] The operating system can manage and schedule the overall operation of the host system 600, such as scheduling tasks, executing applications and / or controlling peripherals, and other operations. In some instances, the host system 600 can control one or more virtual machines. In these instances, each virtual machine can be configured to run its own operating system. Examples of operating systems include Unix, Linux, Windows, Mac OS, iOS, Android, etc. Alternatively or additionally, the operating system can be a proprietary operating system.

[0099] Data storage areas may contain persistent or temporary data used and / or manipulated by operating systems, applications, or drivers. Examples of such data include web pages, video data, audio data, images, user data, etc. In some instances, information in the data storage area may be made available to user devices via a network. In some cases, the data storage area may additionally or alternatively contain stored applications and / or drivers. Alternatively or additionally, the data storage area may store standard and / or proprietary software libraries, and / or standard and / or proprietary application user interface (API) libraries. The information stored in the data storage area may be machine-readable object code, source code, interpreted code, or intermediate code.

[0100] A driver may contain a program that provides communication between components within the host system 600. For example, some drivers provide communication between the operating system and peripheral devices or I / O devices 632. Alternatively or additionally, some drivers provide communication between applications and the operating system and / or applications and peripheral devices accessible to the host system 600. In many cases, a driver may contain a driver that provides easily understood functionality (e.g., printer driver, monitor driver, hard disk driver, solid-state device driver, etc.). In other cases, a driver may provide proprietary or special-purpose functionality.

[0101] I / O device 632 may include hardware for connecting to user input and output devices, such as keyboards, mice, pens, tablets, voice input devices, touch input devices, displays or monitors, speakers, printers, and other devices. I / O device 632 may also include storage drives and / or network interfaces for connecting to network 680. For example, in various instances, host system 600 may use the network interface to communicate with storage devices, user terminals, other computing devices, servers, and / or other networks.

[0102] In various instances, one or more of the I / O devices 632 may be storage devices. In these instances, the storage device includes non-volatile memory and can store program instructions and / or data. Examples of storage devices include magnetic storage, optical disks, solid-state drives, flash memory, and / or magnetic tape memory, etc. The storage device may be located in the same chassis as the host system 600 or in an external rack. The storage device may be fixed (e.g., screw-attached) or movable (e.g., having a physical release mechanism, possibly with a hot-swappable mechanism).

[0103] Storage devices, DRAM 630, and any other memory components in host system 600 are examples of computer-readable storage media. A computer-readable storage medium is a physical medium capable of storing data in a format readable by a device such as host processor 672. Computer-readable storage media can be non-transitory. Non-transitory computer-readable media can retain data stored thereon when no power is supplied to the medium. Examples of non-transitory computer-readable media include ROM devices, magnetic disks, magnetic tapes, optical disks, flash drives, and solid-state drives. As used herein, computer-readable storage media does not include computer-readable communication media.

[0104] In various instances, data stored on a computer-readable storage medium may include program instructions, data structures, program modules, libraries, other software program components, and / or other data that can be transmitted within a data signal, such as a carrier wave or other transmission. Additionally or alternatively, a computer-readable storage medium may include documents, images, video, audio, and other data that can be operated on or manipulated using a software program.

[0105] In various instances, one or more of the I / O devices 632 may be PCI-based devices. In these instances, the PCI-based I / O devices include a PCI interface for communicating with the host system 600. The term “PCI” or “PCI-based” can be used to describe any protocol in the PCI family of bus protocols, including the original PCI standard, PCI-X, Accelerated Graphics Port (AGP), and Fast PCI (PCIe), or any other improved or derived protocols based on the PCI protocols discussed herein. PCI-based protocols are standard bus protocols used to connect devices such as local peripherals to host devices. A standard bus protocol is a data transfer protocol whose specifications are defined and adopted by individual manufacturers. Manufacturers ensure that compatible devices are compatible with computing systems that implement the bus protocol, and vice versa. As used herein, PCI-based devices also include devices that communicate using the Fast Non-Volatile Memory Transfer Specification (NVMe). NVMe is a device interface specification for accessing non-volatile storage media attached to a computing system using PCIe.

[0106] PCI-based devices may include one or more functions. A "function" describes the hardware and / or software that can provide operation by a PCI-based device. Examples of functions include mass storage controllers, network controllers, display controllers, memory controllers, serial bus controllers, wireless controllers, and encryption / decryption controllers. In some cases, a PCI-based device may include more than one function. For example, a PCI-based device may provide a mass storage controller and a network adapter. As another example, a PCI-based device may provide two storage controllers for controlling two different storage resources. In some implementations, a PCI-based device may have up to eight functions.

[0107] In some instances, PCI-based devices may include single root I / O virtualization (SR-IOV). SR-IOV is an extension capability that can be included in PCI-based devices. SR-IOV allows a physical resource (e.g., a single network interface controller) to appear as multiple virtual resources (e.g., sixty-four network interface controllers). Therefore, a PCI-based device (e.g., a network interface controller) providing a certain function can appear as multiple devices providing the same function to devices utilizing PCI-based devices. The functionality of a storage adapter device with SR-IOV capability can be classified as physical functions (PF) or virtual functions (VF). Physical functions are full-featured functions of the device that can be discovered, managed, and manipulated. Physical functions have configuration resources that can be used to configure or control the storage adapter device. Physical functions contain the same configuration address space and memory address space as non-virtualized devices. A physical function may have multiple virtual functions associated with it. Virtual functions are similar to physical functions but are lightweight functions; they may typically not have configuration resources and are usually controlled by the configuration of their underlying physical function. Each of the physical and / or virtual functions can be assigned to a corresponding execution thread (e.g., a virtual machine) running on the host device.

[0108] In various implementations, support system 674 may include hardware for coordinating the operation of acceleration engine 660. For example, support system 674 may include a microprocessor coordinating the activities of acceleration engine 660, including moving data around acceleration engine 660. In this example, the microprocessor may be an integrated circuit that executes microcode. Microcode is program code that gives an integrated circuit some flexibility in the operations it can perform, but because the program code uses a limited instruction set, the capabilities of the microprocessor may be more limited than those of the host processor 672. In some instances, the program executed by the microprocessor is stored on the microprocessor's hardware or on a non-volatile memory chip in the host system 600. In some instances, the microprocessor and acceleration engine 660 may be on a single chip, such as on the same die and in the same package.

[0109] In some instances, support system 674 may be responsible for fetching instructions from host processor 672 when a program executing on host processor 672 requests to execute a neural network. For example, host processor 672 may provide support system 674 with a set of input data and a task to be performed on said set of input data. In this instance, support system 674 may identify a neural network capable of performing the task and may program acceleration engine 660 to execute the neural network on said set of input data. In some instances, support system 674 may simply select an appropriate neural network processing engine among the neural network processors. In some instances, support system 674 may need to load data for the neural network onto acceleration engine 660 before acceleration engine 660 can begin executing the neural network. In these and other instances, support system 674 may also receive the output of executing the neural network and provide the output back to host processor 672.

[0110] In some instances, operations supporting system 674 can be handled by host processor 672. In these instances, support for system 674 may not be required and can be omitted in host system 600.

[0111] In various instances, host system 600 may include a combination of host system, processor node, storage subsystem and I / O chassis representing user device, service provider computer or third-party computer.

[0112] A user device may include a computing device for accessing applications (e.g., web browsers or mobile device applications). In some instances, the application may be controlled, managed, and / or provided by a computing resource service or service provider. For example, the application allows a user to interact with a service provider's computer to access web content (e.g., web pages, music, videos, etc.). The user device may be a computing device such as a mobile phone, smartphone, personal digital assistant (PDA), laptop, netbook, desktop computer, lightweight client device, tablet computer, e-book reader, game console, etc. In some instances, the user device may communicate with a service provider's computer over one or more networks. Additionally, the user device may be part of a distributed system managed, controlled, or integrated with a service provider's computer (e.g., a console device integrated with the service provider's computer).

[0113] The host system 600 may also represent one or more service provider computers. Service provider computers can provide native applications configured to run on user devices with which users can interact. In some instances, service provider computers can provide computing resources, such as, but not limited to, client entities, low-latency data storage, persistent data storage, data access, management, virtualization cloud-based software solutions, e-content performance management, and so on. Service provider computers can also be used to provide network hosting, databases, computer application development and / or implementation platforms, combinations of the foregoing, and so on. In some instances, service provider computers may be provided as one or more virtual machines implemented in a managed computing environment. A managed computing environment may contain one or more computing resources that can be rapidly provisioned and released. These computing resources may include computing, networking, and / or storage devices. A managed computing environment may also be referred to as a cloud computing environment. Service provider computers may contain one or more servers, which may be arranged in a cluster, a server cluster, or as separate, unrelated servers, and may control applications and / or cloud-based software services. These servers may be configured as part of an integrated, distributed computing environment. In some instances, alternatively or additionally, the service provider computer may include computing devices such as mobile phones, smartphones, personal digital assistants (PDAs), laptops, desktop computers, netbooks, server computers, lightweight client devices, tablets, game consoles, etc. In some instances, the service provider computer may communicate with one or more third-party computers.

[0114] As used below, any reference to a series of instances should be understood as a separate reference to each of those instances (e.g., "Instances 1-4" should be understood as "Instances 1, 2, 3, or 4").

[0115] Example 1 is a method comprising: identifying neural network operators in a neural network that are supported by a compiler of an acceleration engine; partitioning the neural network operators supported by the compiler into a first set of operators executable by the acceleration engine; and partitioning the neural network operators supported by the compiler into a second set of operators executable by a host processor.

[0116] Example 2 is the method according to Example 1, wherein the identification includes: receiving a list of neural network operators supported by the compiler; and comparing the neural network operators in the neural network with the list.

[0117] Example 3 is the method according to Example 1 or 2, further comprising partitioning the neural network operators not in the list for execution on a machine learning framework.

[0118] Example 4 is a method according to any one of Examples 1-3, wherein the partitioning includes: determining that one of the first set of operators is not supported by the acceleration engine; determining that the operator is executable faster on the host processor than on the acceleration engine; and partitioning the operator for execution on the host processor.

[0119] Example 5 is a method according to any one of Examples 1-4, wherein the partitioning includes: evaluating parameters of the first set of operators; determining that the parameters of one of the operators in the first set of operators are not supported by the acceleration engine; and partitioning the operator for execution on the host processor.

[0120] Example 6 is a method according to any one of Examples 1-5, further comprising: evaluating the parameters of an operator, and partitioning the operator for execution on the host processor when a parameter not supported by the acceleration engine is detected.

[0121] Example 7 is the method according to any one of Examples 1-6, wherein the partitioning includes: evaluating a series of operators in the set of first operators; determining that the series of operators are not supported by the acceleration engine; and partitioning the series of operators for execution on the host processor.

[0122] Example 8 is the method according to any one of Examples 1-7, further comprising: compiling the set of first operators to produce a compiled model for execution on the acceleration engine.

[0123] Example 9 is a method according to any one of Examples 1-8, further comprising: determining that the compilation of the first set of operators has failed; and repartitioning the first set of operators for execution on the machine learning framework.

[0124] Example 10 is the method according to any one of Examples 1-9, further comprising: compiling the set of second operators to produce a compiled model for execution on the host processor.

[0125] Example 11 is a system comprising: a set of processors; and an acceleration engine, wherein the set of processors is configured to: execute a machine learning framework to generate operators for a neural network; determine which operators are supported by a compiler of the acceleration engine; generate a first compiled model of a first subset of the operators supported by the compiler for execution on the acceleration engine; and generate a second compiled model of a second subset of the operators supported by the compiler for execution on the set of processors.

[0126] Example 12 is the system according to Example 11, and further includes a partitioner configured to: determine the operators supported by the compiler; and partition the operators not supported by the compiler for execution on the machine learning framework.

[0127] Example 13 is a system according to Example 11 or 12, wherein the partitioner is further configured to: receive from the compiler a list of operators supported by the compiler, compare the operators of the neural network with the list, and partition the neural network operators supported by the compiler and the operators not supported by the compiler based on the comparison.

[0128] Example 14 is a system according to Examples 11-13, wherein the set of processors is further configured to: evaluate parameters of the first subset of operators; determine that the parameters of a certain operator in the first subset of operators are not supported by the acceleration engine; and partition the operator for execution on the host processor.

[0129] Example 15 is a system according to Examples 11-14, wherein the set of processors is further configured to: evaluate the parameters of the operator; and when a parameter not supported by the acceleration engine is detected, partition the operator for execution on the set of processors.

[0130] Example 16 is a system according to Examples 11-15, wherein the set of processors is further configured to: evaluate a series of operators of the first subset of operators; determine that the series of operators are not supported by the acceleration engine; and partition the series of operators for execution on the host processor.

[0131] Example 17 is a system according to Examples 11-16, wherein the acceleration engine is configured to execute the first compilation model.

[0132] Example 18 is a system according to Examples 11-17, wherein a set of processors is configured to execute the second compilation model.

[0133] Example 19 is a non-transitory computer-readable medium storing instructions for causing one or more processors to perform a method of partitioning operators of a neural network, the processor-executable instructions including instructions for performing operations comprising: identifying neural network operators in the neural network supported by a compiler of an acceleration engine; partitioning the compiler-supported neural network operators into a first set of operators executable by the acceleration engine; and partitioning the compiler-supported neural network operators into a second set of operators executable by a host processor.

[0134] Example 20 is a method for hierarchically partitioning operators of a neural network for compilation on an acceleration engine, the method comprising: obtaining a set of neural network operators of a neural network generated by a machine learning framework; comparing a list of neural network operators supported by a compiler with the set of neural network operators of the neural network; based on the comparison, partitioning the set of neural network operators of the neural network into a first set of neural network operators not supported by the compiler and a second set of neural network operators supported by the compiler; providing the second set of neural network operators to the compiler; and partitioning the second set of neural network operators into a third set of neural network operators executable on the acceleration engine and a fourth set of neural network operators to be executed by a host processor.

[0135] Example 21 is the method according to Example 20, further comprising partitioning the set of third neural network operators into a set of fifth neural network operators to be executed on the host processor, the set of fifth neural network operators including instances of neural network operators having parameters not supported by the acceleration engine.

[0136] Example 22 is the method according to Example 20 or 21, further comprising compiling the third set of neural network operators for execution on the acceleration engine.

[0137] Example 23 is the method according to any one of Examples 20-22, further comprising compiling the fourth set of neural network operators for execution on the host processor.

[0138] Example 24 is a computer program product having instructions for causing one or more processors to perform a method of partitioning operators of a neural network, the processor-executable instructions including instructions for performing operations comprising: identifying neural network operators in the neural network supported by a compiler of an acceleration engine; partitioning the neural network operators supported by the compiler into a first set of operators executable by the acceleration engine; and partitioning the neural network operators supported by the compiler into a second set of operators executable by a host processor.

[0139] Example 25 is a data carrier signal that carries a computer program product according to Example 24.

[0140] The modules described herein can be software modules, hardware modules, or suitable combinations thereof. If a module is a software module, then the module may be embodied on a non-transitory computer-readable medium and processed by a processor in any computer system described herein. It should be noted that the described processes and architectures may be executed in real-time or asynchronously prior to any user interaction. Modules may be configured as recommended in the previous figures, and / or the functionality described herein may be provided by one or more modules existing as separate modules, and / or the module functionality described herein may be distributed across multiple modules. Any method described herein may be implemented as a computer-readable medium or computer program product including instructions that, when executed by one or more computers, cause the one or more computers to perform the method steps. Such computer program products may be transmitted over a data carrier signal carrying the computer program product via a wired or wireless network.

[0141] Therefore, the specification and drawings should be viewed in an illustrative rather than restrictive sense. However, it will be apparent that various modifications and changes can be made therein without departing from the broad spirit and scope of this disclosure as set forth in the claims.

[0142] Other variations are within the spirit of this disclosure. Therefore, while certain illustrated examples of the disclosed technology have been shown in the accompanying drawings and described in detail above, various modifications and alternative constructions are possible with the disclosed technology. However, it should be understood that this disclosure is not intended to be limited to the specific forms disclosed, but rather, it is intended to cover all modifications, alternative constructions, and equivalents that fall within the spirit and scope of this disclosure as defined in the appended claims.

[0143] Unless otherwise indicated herein or clearly contradicted by the context, the use of the words “a / an” and “the,” and similar indicators, in the context of describing the disclosed examples (especially in the context of the appended claims), shall be understood to cover both singular and plural. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” shall be understood as open-ended terms (i.e., meaning “including but not limited to”). The term “connected” shall be understood as being partially or wholly contained therein, attached to, or joined together, even in the presence of intermediates. Unless otherwise indicated herein, the description of ranges of values ​​herein is intended merely as a shorthand method of individually referring to each individual value falling within the range, and each individual value is incorporated into this specification as if it were described separately herein. Unless otherwise indicated herein or clearly contradicted by the context, all methods described herein may be performed in any suitable order. Unless otherwise asserted, the use of any and all instances or exemplary language (e.g., “for example”) provided herein is intended only to better illustrate instances of this disclosure and does not constitute a limitation on the scope of this disclosure. No language in this specification should be construed as indicating any unrequired element necessary for the practice of this disclosure.

[0144] Unless otherwise specifically stated, delimited language such as the phrase “at least one of X, Y or Z” is intended to be understood in the context in which it is used to generally represent that an item, term, etc., can be X, Y or Z, or any combination thereof (e.g., X, Y and / or Z). Therefore, such delimited language generally does not and should not imply that certain instances require the presence of at least one of X, at least one of Y, or at least one of Z.

[0145] Various examples of this disclosure are described herein, including the best mode known to the inventors for carrying out this disclosure. After reading the above description, those skilled in the art will understand variations of those examples. The inventors expect those skilled in the art to adopt such variations as needed, and the inventors intend to practice this disclosure in ways other than those specifically described herein. Therefore, this disclosure includes all modifications and equivalents of the subject matter set forth in the appended claims as permitted by applicable law. Furthermore, unless otherwise indicated herein or otherwise clearly contradicted by the context, this disclosure covers any combination of the elements described above in all their possible variations.

Claims

1. A method comprising: Identify neural network operators in the neural network that are supported by the compiler of the acceleration engine; The neural network operators supported by the compiler are partitioned into a first set of operators that can be executed by the acceleration engine. The neural network operators supported by the compiler are partitioned into a second set of operators that can be executed by the host processor. Evaluate the parameters of the first set of operators. It is determined that a specific instance of a neural network operator in the first group of operators, which was originally determined to be executable on the acceleration engine, contains parameters not supported by the acceleration engine. Specific instances of the neural network operator are partitioned for execution on the host processor.

2. The method according to claim 1, wherein the identification includes: Receive a list of neural network operators supported by the compiler; and Compare the neural network operators in the neural network with the list.

3. The method of claim 2, further comprising partitioning the neural network operators not in the list for execution on a machine learning framework.

4. The method according to claim 1 or 2, wherein the partition comprises: Determine that one of the operators in the first group is supported by the acceleration engine; It is determined that the operator can be executed faster on the host processor than on the acceleration engine; as well as The operator is partitioned for execution on the host processor.

5. The method according to claim 1 or 2, wherein the partition comprises: Evaluate a series of operators in the first group of operators; It was determined that the aforementioned series of operators were not supported by the acceleration engine; as well as The series of operators are partitioned for execution on the host processor.

6. The method according to claim 1 or 2, further comprising: It was determined that the compilation of the first set of operators had failed; as well as The first set of operators is repartitioned for execution on the machine learning framework.

7. A system comprising: A set of processors; as well as Acceleration engine, The set of processors is configured as follows: Operators that execute machine learning frameworks to generate neural networks; The first subset of the determined operators is supported by the compiler of the acceleration engine; Evaluate the parameters of the first subset of the operator; The first subset of the operators is determined to contain specific instances of neural network operators that were originally determined to be executable on the acceleration engine, which contain parameters that are not supported by the acceleration engine. Partition specific instances of the neural network operator to execute them on the set of processors; A first model of a first subset of the operators supported by the compiler is compiled for execution on the acceleration engine; as well as A second model of a second subset of the operators supported by the compiler is compiled for execution on the set of processors.

8. The system of claim 7, further comprising a partitioner configured to: Determine the operators supported by the compiler; and The operators not supported by the compiler are partitioned for execution on the machine learning framework.

9. The system according to claim 7 or 8, wherein the partitioner is further configured to: Receive a list of operators supported by the compiler from the compiler. Compare the operators of the neural network with the list, and Based on the comparison, the neural network operators supported by the compiler and those not supported by the compiler are partitioned.

10. A computer program product having instructions for causing one or more processors to perform a method of partitioning operators of a neural network, the processor-executable instructions including instructions for performing operations comprising: Identify neural network operators in the neural network that are supported by the compiler of the acceleration engine; The neural network operators supported by the compiler are partitioned into a first set of operators that can be executed by the acceleration engine. The neural network operators supported by the compiler are partitioned into a second set of operators that can be executed by the host processor. Evaluate the parameters of the first set of operators. It is determined that a specific instance of a neural network operator in the first group of operators, which was originally determined to be executable on the acceleration engine, contains parameters not supported by the acceleration engine. Specific instances of the neural network operator are partitioned for execution on the host processor.

11. A non-transient computer-readable medium having a computer program product according to claim 10 stored thereon.

Images