Use of hardware-accelerated instructions

Abstract

To provide a technology that pertains to a computer execution method (600) for solving a constraint satisfaction problem and thereby executing calculations using the hardware-accelerated instructions of a processor system.SOLUTION: A solution to a constraint satisfaction problem represents a possible call to hardware-accelerated instructions in a calculation. The constraint satisfaction problem assigns a calculation data flow graph node to an instruction dataflow graph node. The constraint satisfaction problem includes a constraint requiring that the assigned node of a calculation dataflow graph must have an equivalent dataflow to an instruction dataflow graph and a constraint limiting which node of the calculation dataflow graph can be assigned to the input of a hardware-accelerated instruction, the limiting being imposed by the hardware-accelerated instruction and / or its programming interface.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a computer execution method for performing calculations using hardware-accelerated instructions of a processor system, and to a corresponding system. The present invention further relates to a computer-readable medium including instructions for performing the above method and / or instructions for performing calculations using hardware-accelerated instructions. [Background technology]

[0002] Background of the Invention As the amount of data collected and processed increases, efficiently performing large-scale computations is becoming increasingly important. One area where the need to process large amounts of data is particularly prevalent is machine learning. For example, not only is the evaluation of neural networks such as deep neural networks (DNNs) necessary, but training in particular also involves evaluating numerous matrix multiplications, convolutions, dot products, and similar operations. There are various large-scale and continuously evolving DNN architectures that are adapted to specific applications.

[0003] To speed up such calculations, an increasing amount of specialized hardware is becoming available. Such hardware includes hardware accelerators optimized to efficiently perform specific instructions, such as matrix multiplication or convolution. Therefore, by using such hardware-accelerated instructions, it becomes possible to perform calculations more efficiently, especially in resource-constrained settings.

[0004] To utilize hardware-accelerated instructions, it is crucial to provide compiler technology that translates code in high-level programming languages ​​into machine-executable instructions that invoke hardware-accelerated instructions. This is especially important because both the computations to be performed and the hardware architectures that perform these computations are constantly evolving. However, automatically determining how computations, such as those related to evaluating neural network layers, should be performed so that hardware-accelerated instructions are utilized most effectively is a complex task. One reason this task is so complex is that hardware-accelerated instructions can have complex data flows, sometimes involving hundreds or thousands of parallel and sequential operations across multidimensional input and output arrays.

[0005] In the paper “ISA Mapper: A Compute and Hardware Agnostic Deep Learning Compiler” by M. Sotoudeh et al. (available at https: / / arxiv.org / abs / 1810.09958, and incorporated herein by reference), a technique for performing code generation for instruction sets including matrix multiplication instructions is presented. The kernel and hardware capabilities of the deep learning computation are described by intermediate representations that function at the loop-level representation of operators. Instruction mapping is applied to determine possible methods for performing the computation on the hardware system. In this instruction mapping, numerous “IR transformations” are applied to the computation, and then a “deterministic mapper” attempts to map this transformed computation to hardware instructions at the loop level. A scheduler selects a specific mapping. Heuristics, cost models, and, in some cases, machine learning are used to manage the large search space for mappings and schedules. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] “ISA Mapper: A Compute and Hardware Agnostic Deep Learning Compiler,” by M. Sotoudeh et al. (available at https: / / arxiv.org / abs / 1810.09958) [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] Summary of the Invention One drawback of existing techniques for performing computations using hardware-accelerated instructions is that the use of loop-level representations of operators introduces implicit implementation decisions, such as the ordering or access function notation of loops and tensors. For example, the tensor access function for a stencil in an operator may be x*stride+w, while the accelerator may provide an instruction with access function i+k. While consistency is possible when stride=1, it is generally difficult to automatically establish such consistency, for example, by pattern matching methods. Thus, possible implementations may be overlooked.

[0008] Furthermore, existing techniques rely on an explicitly specified set of IR transformations, which is a disadvantage. Directly detecting a specific sequence of transformations that leads to an embedding is difficult. Ultimately, this can lead to a non-deterministic search process that must generate many different candidate implementations. Also, the more transformations added, the larger the search space becomes, and the return in terms of the number of solutions decreases. Thus, there is a conflict between the search required to find the optimal implementation and the limit of the search space required to find a solution in a reasonable amount of time. Moreover, even when a large set of transformations is used, there remains the possibility that subclasses of implementations are hidden behind transformations that are not available.

[0009] Due to these shortcomings, existing technologies may fail to discover implementations of computations that most effectively utilize hardware-accelerated instructions. In some cases, existing technologies may even fail to discover at all how hardware-accelerated instructions can be used in computations. Therefore, it is desirable to have technologies that can discover more feasible implementations of computations using hardware-accelerated instructions. It is also desirable to have technologies that can discover such implementations more efficiently. [Means for solving the problem]

[0010] According to a first aspect of the present invention, a computer implementation method and a corresponding system for performing calculations using hardware-accelerated instructions of a processor system are provided, as defined by claims 1 and 14, respectively. According to another aspect of the present invention, a computer-readable medium is provided, as defined by claim 15.

[0011] Various means are directed toward performing computations using hardware-accelerated instructions of the processor system. Performing computations using hardware-accelerated instructions means discovering possible call calls to instructions in a computation, that is, discovering the parts of a computation that can be performed by calling instructions. It is possible, but not essential, to automatically generate machine-executable instructions that call hardware-accelerated instructions according to the determined possible call in order to perform the computation and, for example, compile the computation into executable code. For example, discovering possible call calls is already useful in itself, for example, to determine whether such an implementation is even possible when performing the computation manually, or for use by other systems that perform the actual compilation.

[0012] Computations and hardware-accelerated instructions can be represented by their respective dataflow graphs. Nodes in the dataflow graph can represent inputs and operations for a computation or instruction, and edges represent dataflow relationships between nodes; for example, an incoming edge of an operation can be used to represent the value on which the operation acts. Hardware-accelerated instructions typically perform a predetermined operation with a fixed dataflow; for example, an instruction dataflow graph can be fixed. An operation is typically a scalar operation, such as an operation applied to individual numeric or non-numeric elements.

[0013] Interestingly, the inventors have recognized that the problem of determining possible calls of hardware-accelerated instructions in a computation can be expressed as a constraint satisfaction problem that assigns nodes in the computation dataflow graph to nodes in the instruction dataflow graph. Thus, the solution to the constraint satisfaction problem can be to assign nodes in the computation dataflow graph, e.g., computation operations or inputs, to nodes in the instruction dataflow graph, e.g., hardware-accelerated instruction operations or inputs. Such an assignment is also referred to herein as a “mapping” of the instruction dataflow graph to the computation dataflow graph. Effectively, a bottom-up approach is proposed that allows for the incremental construction of instruction-to-computation mappings by discovering the solution to the constraint satisfaction problem, rather than discovering possible calls through a top-down approach that involves transforming computations and then mapping those computations to hardware instructions.

[0014] A constraint satisfaction problem can be defined such that its solution represents the possible calls of hardware-accelerated instructions in a computation. For this purpose, a constraint satisfaction problem may include at least two types of constraints, referred to as dataflow constraints and input constraints.

[0015] Dataflow constraints can enforce that nodes in a computation dataflow graph assigned to nodes in an instruction dataflow graph have a dataflow equivalent to that of the instruction dataflow graph. For example, hardware-accelerated instructions are mapped to the portion of a computation that corresponds to the functionality of the hardware-accelerated instructions. In particular, by performing operations on the instruction dataflow graph based on the values ​​of the mapped computation, according to the dataflow relationships shown by the instruction dataflow graph, these operations can produce computational results equivalent to performing operations on the mapped computation dataflow. One way to enforce this is to compel that the subgraph of the computation dataflow graph to which the instruction dataflow graph is mapped is isomorphic to the instruction dataflow graph itself, although this requirement can be relaxed.

[0016] However, as the inventors have recognized, simply using dataflow constraints is often insufficient to ensure that the mapping from the instruction dataflow graph to the compute dataflow graph represents possible calls to a hardware-accelerated instruction. This is because hardware-accelerated instructions and / or their programming interfaces typically impose additional restrictions on the values ​​of the compute to which the instruction can be applied. For example, a hardware-accelerated instruction may require that its inputs be organized according to a specific memory layout and / or accessed according to a specific access pattern. To represent such requirements, the constraint satisfaction problem may also include one or more input constraints that limit which nodes in the compute dataflow graph can be assigned to the inputs of the hardware-accelerated instruction. This allows possible calls to be made compliant with the restrictions imposed by the hardware-accelerated instruction and / or its programming interface.

[0017] Interestingly, by defining possible calls in relation to a constraint satisfaction problem, it becomes possible to determine possible calls by solving the constraint satisfaction problem. The technique itself is publicly known. Data flow and input constraints can guarantee that a call is possible. First, one does not need to apply transformations to the entire computation and then try to perform the transformed computation using hardware-accelerated instructions if it is feasible. This can be inefficient. Instead, a constraint solver can incrementally construct a mapping representing hardware-accelerated instruction calls, and the constraints guarantee that the solution is a possible call. This also eliminates the need to define an explicit set of code transformations that would be extremely difficult or impossible to complete.

[0018] Mapping is performed at the computation level, as opposed to the loop level, and any mapping that satisfies the constraints can be found by the constraint solver, thus avoiding implicit implementation decisions such as loop ordering, memory layout, and memory access functions. In particular, calls can be determined by a more flexible memory layout than those provided by known approaches based on a given set of transformations. Experiments in benchmarks have shown that this actual increase in flexibility allows for the discovery of calls that perform better than known techniques. Thus, it becomes possible to discover a wide range of possible calls for hardware-accelerated instructions, and to do so by more efficient, for example, computationally less labor-intensive methods. It becomes possible to discover even more effective calls to these hardware-accelerated instructions, or even discover calls for computations that were previously unexecutable.

[0019] A further advantage of using a constraint solver to discover possible calls is that it allows us to separate the requirements that possible calls must satisfy from preferences that are considered beneficial. In particular, by adapting the variable selection strategy and / or value selection strategy applied by the constraint solver, we can gain control over the solution space during constraint resolution without affecting what it means for a solution to be valid. By controlling the constraint solver, we can guide it to discover a particular solution first. Thus, we can improve the quality of the possible calls determined and further improve the efficiency of the constraint solver by effectively proposing to try a portion of potential solutions without discarding any part of the solution space. That is, even if a proposal does not lead to a solution, the constraint solver can continue to consider other possible solutions.

[0020] It is possible to optionally generate machine-executable instructions that perform calculations. These instructions can call hardware-accelerated instructions according to the determined possible calls. In this way, an efficient implementation of calculations utilizing hardware-accelerated instructions can be obtained. Thus, a method can be provided for compiling calculations into machine-executable instructions.

[0021] Optionally, computation dataflow graphs and instruction dataflow graphs may include nodes representing scalar operations. For example, a scalar operation can act on a fixed number of distinct values ​​(usually one or two) and output a fixed number of distinct values ​​(usually one). Thus, constraint satisfaction problems can map scalar operations to scalar operations, as opposed to operations at the loop level. According to some embodiments, each node in the graph can represent a scalar operation, with the exception of commutative contraction operations, which are described elsewhere. As described elsewhere, performing mapping at the scalar level minimizes implicit modeling assumptions that could otherwise limit which calls can be discovered, making it possible to identify calls with better performance.

[0022] Optionally, hardware-accelerated instructions can perform one or more of the following operations: matrix multiplication, convolution, dot product, matrix-vector product, cumulative sum, pooling, and Hadamard product. These are common operations that are repeatedly performed in large-scale computations, and since hardware acceleration can be utilized for this purpose, optimizing these operations is particularly important.

[0023] Generally, a computation is a computation that includes operations that are expected to be performed by hardware-accelerated instructions. For example, a computation may be a sub-computation of a relatively large computation, such as the application of a convolution operator in neural network evaluation, which is manually selected as potentially suitable for the application of hardware-accelerated instructions. With respect to dataflow, there may be many different methods by which instructions can be called in a computation, but only some of these methods can comply with the limitations imposed by hardware-accelerated instructions and / or their programming interfaces, and these possible calls may differ significantly in performance. Therefore, it is important that possible calls can be automatically discovered and / or enumerated.

[0024] Optionally, the instruction dataflow graph may contain nodes relating only to a subset of the hardware-accelerated instruction's outputs. Other outputs of the hardware-accelerated instruction, as well as the operations used to compute these other outputs, can be excluded, making the constraint satisfaction problem more efficient. Based on the determined possible calls, these determined possible calls can then be generalized to the complete computation performed by the instruction. For example, in the case of a matrix multiplication instruction, a dataflow graph of the computation of a single output element may suffice to extrapolate to the inputs and outputs of the complete matrix multiplication. A subset of the outputs that uses all the inputs of the hardware-accelerated instruction can be selected, so that the possible calls map all inputs, thereby facilitating extrapolation, or even allowing extrapolation of the mapping of instruction inputs to computations.

[0025] Optionally, the computation may be one applied to the evaluation of the neural network. For example, the computation may include convolution, matrix multiplication, and / or pooling operations applied to the inputs of the neural network layer. The computation can access each node of the inputs of the neural network layer. Neural network evaluation is an computation performed both during training and use of the neural network, and involves repeatedly applying hardware-accelerated operations such as matrix multiplication, and is therefore a particularly important optimization target. Furthermore, there are many techniques to which these operations can be applied with respect to data flow. In particular, for neural network training to be effective, it is typically necessary to train the neural network on a large dataset, while neural network evaluation is often performed in resource-constrained environments. In both cases, optimization is especially important. Furthermore, given the wide variety and continuous development of neural network layers and acceleration hardware, it is especially important to have implementations for new types of neural network layers and / or new hardware targets.

[0026] Optionally, at least a portion of the computation can be specified by a polyhedron representation. A polyhedron representation can symbolically define a set of nodes in the computation dataflow graph. When solving a constraint satisfaction problem, a polyhedron representation can be instantiated to obtain a specific node from this set of nodes, and then a node in the instruction dataflow graph can be mapped to this obtained node.

[0027] Polyhedral representations themselves are well known as a compact method for representing loop-based programs. Interestingly, the inventors realized that, in addition to modeling loops in loop-based programs, polyhedral representations can also be used to model dataflow. It is undesirable to determine calls by directly converting polyhedral representations to each other, because, for example, the representations represent modeling assumptions such as the ordering of loops or tensor dimensions, which may require defining explicit conversions of representations, leading to the associated drawbacks described elsewhere. Interestingly, these drawbacks are avoided by using polyhedral representations of the computation dataflow graph to which the instruction dataflow graph should be mapped.

[0028] In combination with this technology, polyhedral representation can be used to symbolically define a set of polyhedra in a computation dataflow graph. However, interestingly, the mapping between the instruction dataflow graph and the computation dataflow graph can still be performed on individual nodes, such as individual operations, rather than on a set of nodes, such as a set of operations. Polyhedral notation avoids the need to store the complete computation dataflow graph; that is, only the individual elements mapped from the instruction dataflow graph need to be instantiated. In many cases, this significantly reduces computational and memory complexity.

[0029] Polyhedral representations are typically not used to represent computational dataflow graphs, or at least, polyhedral representations of computational dataflow graphs are not used during constraint resolution. Polyhedral nodes of a computational dataflow graph are typically stored individually as variables in the constraint satisfaction problem.

[0030] Optionally, one or more input constraints can be defined to enforce that the inputs of a hardware-accelerated instruction have permitted memory layouts and / or memory access patterns. Generally, a hardware-accelerated instruction and / or its programming interface can enforce that the inputs to which the instruction is applied are stored in memory according to a specific arrangement, for example, that the inputs form a contiguous block of memory, or that memory addresses are aligned in a particular manner. These are examples of memory layout requirements that restrict which inputs can be to which a hardware-accelerated instruction can be applied. These requirements can be expressed as input constraints to ensure that they are complied with. Similarly, a hardware-accelerated instruction can support a given set of access patterns depending on which instructions it can be to, for example, whether the inputs are accessed according to a given stride, a given stencil pattern, etc. In this way, the inputs to which a hardware-accelerated instruction can be applied can be further restricted.

[0031] For example, to represent instructions imposed by an instruction and / or its programming interface, the set of constraints may also include restrictions on memory layout and / or memory access patterns related to the intermediate results and outputs of hardware-accelerated instructions.

[0032] Optionally, one or more constraints can be defined to enforce that pairs of hardware-accelerated instructions that are mutually parallelizable must be mapped to pairs of computations that are mutually parallelizable. For example, pairs can be marked as mutually parallelizable by including a special type of edge to indicate parallelizability to the dataflow graph. Explicitly marking pairs of operations as mutually parallelizable, and enforcing this in the mapping, is not mandatory, but doing so is beneficial for the performance of constraint resolution because it allows the resolution process to directly eliminate potential assignments where mutually parallelizable operations are not mapped to mutually parallelizable operations.

[0033] Interestingly, the inventors realized that for a set of operations in an instruction dataflow graph where all operations are mutually parallelizable, it is not necessary to include parallelizability constraints for each separate pair. Including constraints only on a subset of pairs improves solver performance because solver efficiency generally scales to the number of constraints, and subsets help the solver eliminate allocations. Furthermore, transitivity implies that enforcing parallelizability constraints on a set of pairs in an instruction dataflow graph also satisfies parallelizability constraints for each pair in the transitive closure. In a computational dataflow graph, parallelizability can be shown for each pair of operations in a set of mutually parallelizable operations, thereby avoiding unnecessary restrictions on which allocations can be made. In a computational dataflow graph, this can reduce performance degradation, especially if the graph is not fully instantiated during resolution, for example, due to the use of a polyhedral representation.

[0034] Optionally, nodes in an instruction dataflow graph can represent commutative reduction operations. For example, a node for a commutative reduction operation, such as sum or multiplication, can represent that this operation is applied to all incoming edges of the node. By using such nodes, the number of nodes and edges required to specify common hardware-accelerated operations, such as matrix multiplication or convolution, can be significantly reduced without affecting the accuracy of the dataflow graph as a representation of computations or instructions. This also allows for more flexible mapping, because the graph does not specify a particular order in which elements must be reduced. While commutative reduction operations can be implicitly defined by nodes with, for example, three or more incoming edges, it is preferable to explicitly mark nodes as representing commutative reduction operations. For example, this marking can be done by including self-edges in the graph. Such explicit marking makes it more efficient for the constraint solver to determine the mapping for commutative reduction operations.

[0035] Dataflow constraints can compel nodes representing commutative contraction operations in the instruction dataflow graph to map to corresponding nodes in the computation dataflow graph. These corresponding nodes may represent the commutative contraction operations of the computation, for example, nodes with incoming edges for each element on which the commutative contraction operation acts. However, this is not mandatory. Alternatively, individual contractions within the overall contraction operation can be represented separately in the instruction dataflow graph. In such cases, nodes in the instruction dataflow graph can be mapped to nodes representing individual contractions. This simplifies the computation modeling process by eliminating the need for separate modeling effort for commutative contraction operations.

[0036] When solving constraint satisfaction problems, a variable selection strategy that propagates backward through the instruction dataflow graph can be used, albeit arbitrarily. In practice, it has been observed that this variable selection strategy has the greatest potential for propagation and is therefore generally the most efficient in finding solutions for the types of constraint satisfaction problems defined by dataflow and input constraints. This has been found to be particularly true for the computation of neural network layer evaluations.

[0037] A constraint satisfaction problem can be defined to optionally allow mapping the operations of hardware-accelerated instructions to dummy operations that do not affect the computation, such as dummy operations that produce equivalent computation results. Solving the constraint satisfaction problem may involve mapping nodes to dummy operations, for example, mapping a node to an operation not included in the computation dataflow graph, or mapping multiple nodes to the same node in the computation dataflow graph. Allowing dummy operations increases flexibility in discovering possible calls. In particular, it becomes possible to apply hardware-accelerated instructions to workloads that could not otherwise be performed by these instructions. For example, the relevant part of the computation can be smaller than the hardware-accelerated instruction, for example, only a subset of nodes in the instruction dataflow graph can be mapped to nodes in the computation dataflow graph. For example, an instruction can calculate the sum of four elements, while a computation only requires the sum of three elements. By including dummy operations that do not affect the calculation, for example, by adding a neutral element for the operation performed by the instruction—for example, 0 in the case of addition, or 1 in the case of multiplication—the relevant part of the calculation can still be executed by hardware-accelerated instructions.

[0038] Optionally, one or more constraints can be included in the constraint satisfaction problem that compel the inputs of hardware-accelerated instructions to form a hypercube. This constraint allows the constraint solver to quickly eliminate a large portion of the search space after, for example, only a few assignments to instruction dataflow nodes. This significantly improves search efficiency and ensures that a regular solution is provided that can be more easily generalized to obtain calls for the remaining computation. In many practical applications, typical calls of hardware-accelerated instructions have hypercube-shaped inputs, so these restrictions in such cases do not result in good-performing calls being missed.

[0039] Multiple solutions to a constraint satisfaction problem can be determined at will. The performance of these multiple solutions can then be evaluated, and one solution can be selected from among them based on this performance evaluation. A constraint solver can find possible calls that satisfy the constraints, but typically does not optimize the discovered solutions in terms of performance (although a constraint solver can be configured to favor a particular solution, as described herein). However, a constraint solver may be configured to output multiple solutions to a constraint satisfaction problem or to enumerate all possible solutions. By evaluating the performance of these multiple solutions, a solution with desired performance can be selected. Various strategies can be employed; for example, it may be possible to determine a fixed number of solutions, or to use a fixed amount of time to find a solution and then evaluate the discovered solution, or to repeatedly determine and evaluate solutions until a solution with desirable performance is found or until no further improvements are found within a certain period.

[0040] Those skilled in the art will understand that two or more of the above-described embodiments, implementations, and / or optional configurations of the present invention may be combined in any way that seems useful.

[0041] Any modified and modified forms of any system and / or any computer-readable medium corresponding to the described modified and modified forms of the corresponding computer implementation can be implemented in accordance with this specification by those skilled in the art.

[0042] These and other aspects of the present invention will be apparent from and will be illustrated with reference to the embodiments described as examples in the following description and the accompanying drawings. [Brief explanation of the drawing]

[0043] [Figure 1] This is a diagram showing the system for performing calculations. [Figure 2] This figure shows a system for performing calculations using hardware-accelerated instructions. [Figure 3] This figure shows a detailed example of how to perform the calculation. [Figure 4a] This figure shows a detailed example of a data flow graph. [Figure 4b] This figure shows a detailed example of a polyhedron representation that defines a data flow graph. [Figure 5a] This figure shows a detailed example of applying the super-rectangular system convention. [Figure 5b] This figure shows a detailed example of applying the super-rectangular system convention. [Figure 5c] This figure shows a detailed example of applying the super-rectangular system convention. [Figure 5d] This figure shows a detailed example of applying the super-rectangular system convention. [Figure 6] This diagram shows the computer method used to perform the calculations. [Figure 7] This is a diagram showing a computer-readable medium containing data.

[0044] It should be noted that each drawing is purely schematic and is not illustrated to scale. Elements corresponding to elements already described in the drawings may share the same reference numeral. [Modes for carrying out the invention]

[0045] Detailed description of the embodiment Figure 1 shows a system 100 for performing calculations using hardware-accelerated instructions of a processor system. For example, calculations can be performed using the processor system 240 of system 200 in Figure 2, for example, system 100 can generate instructions that can be executed by the processor system 240, or at least determine calls to hardware-accelerated instructions supported by the processor system 240 that are available for performing calculations on the processor system 240. System 100 and system 200 can be combined, for example, the processor system 140 (described later) of system 100 can support hardware-accelerated instructions.

[0046] System 100 may include data interfaces for accessing computation data 030 and instruction data 040. Computation data 030 can define a computation data flow graph representing computations. Nodes in the computation data flow graph can represent inputs or operations for computations. Instruction data 040 can define an instruction data flow graph representing hardware-accelerated instructions. Nodes in the instruction data flow graph can represent inputs or operations for hardware-accelerated instructions.

[0047] The instruction dataflow graph may have at least 16, at least 32, or at least 128 nodes. The number of nodes may be up to 2048 or up to 4096. The computation dataflow graph is typically more complex than the instruction dataflow graph, for example, 2 6 = 64 times, 2 10 = 1024 times, or 2 12 = 4096 times, having more nodes. The number of nodes in the instruction data flow graph is, for example, at least 2 16 =65536 pieces, 2 32 = 4,294,967,296 pieces, or 2 48 It can be considered an individual entity.

[0048] For example, as illustrated in Figure 1, the data interface can be configured by a data storage interface 120 that can access data 030,040 from data storage 021. For example, the data storage interface 120 may be a memory interface or a persistent storage interface, such as a hard disk or SSD interface, or it may be a personal, local or wide-area network interface such as Bluetooth, Zigbee or Wi-Fi interface, or Ethernet or fiber optic interface. The data storage 021 may be an internal data storage of the system 100, such as a hard drive or SSD, or it may be an external data storage, such as a network-accessible data storage. In some embodiments, data 030,040 can be accessed from various data storages, for example, through various subsystems of the data storage interface 120. Each subsystem may be of the type described above with respect to the data storage interface 120.

[0049] A data interface, such as a data storage interface 120, may be for accessing a determined list of possible calls to hardware-accelerated instructions, or, as shown in the drawings, for accessing machine-executable instructions 050, which perform calculations by calling hardware-accelerated instructions according to the determined list of possible calls. These instructions 050 may be for use by a processor system that performs calculations, such as the processor system 240 in Figure 2.

[0050] The system 100 may further include a processor subsystem 140, which can be configured to acquire computation data 030 and instruction data 040 during the operation of the system 100.

[0051] The processor subsystem 140 can be further configured to define constraint satisfaction problems based on computation data and instruction data. A constraint satisfaction problem can be defined by a set of variables, a set of domains, and a set of constraints. To “define” a constraint satisfaction problem means to provide information about the constraint satisfaction problem in a form that enables the constraint satisfaction problem to be solved by a constraint solver. For example, it is not necessary to store the set of variables, the set of domains, and / or the set of constraints explicitly and separately; for example, these sets can be defined implicitly and / or evaluated lazily as is well known. Furthermore, part or all of the constraint satisfaction problem can be provided in the form of computation data 030 and / or instruction data 040, for example, instruction data 040 may include, or otherwise define, a set of nodes in an instruction data flow graph and thereby a set of variables for the constraint satisfaction problem.

[0052] A solution to a constraint satisfaction problem can represent the possible calls of hardware-accelerated instructions in a computation. The constraint satisfaction problem allows assigning nodes in the computation dataflow graph to nodes in the instruction dataflow graph. The constraint satisfaction problem may include one or more dataflow constraints that compel assigned nodes in the computation dataflow graph to have a dataflow equivalent to that of the instruction dataflow graph. The constraint satisfaction problem may further include one or more input constraints that restrict which nodes in the computation dataflow graph can be assigned to the inputs of hardware-accelerated instructions. The latter constraints may be imposed by the hardware-accelerated instruction and / or its programming interface. The definition of constraints may be done, for example, by automatically defining constraints based on computation data 030 and / or instruction data 040, by receiving manual definitions of constraints, or by a combination of the above.

[0053] The processor subsystem 140 is further configurable to solve a constraint satisfaction problem to determine the possible calls of hardware-accelerated instructions in a computation. The processor subsystem is further configurable to output data that defines the possible calls.

[0054] The system 100 may further include an output interface for outputting data that defines possible calls, such as solutions to constraint satisfaction problems, or data that defines machine-executable instructions 050 derived from solutions. For example, as illustrated in Figure 1, the output interface can be configured by a data interface 120, which in these embodiments is an input / output ("IO") interface through which data can be stored in the data storage 021. According to some embodiments, the output interface may be separate from the data storage interface 120, but in general it may be of the same type as described above with respect to the data storage interface 120. The output interface may be any other type of output interface, such as a network interface.

[0055] Figure 2 shows a system 200 for performing calculations using hardware-accelerated instructions.

[0056] System 200 may include a data interface 220 for accessing instructions 050 for performing calculations. For example, instructions 050 can be generated by system 100 in Figure 1, or can be generated based on possible calls determined by system 100. For example, as also illustrated in Figure 2, the data interface may be configured by a data storage interface 220 that can access instructions 050 from data storage 022. Generally, the data interface 220 and data storage 022 may be of the same type as those described with reference to Figure 1 with respect to data interface 120 and data storage 201.

[0057] System 200 may further include a processor subsystem 240. The processor subsystem 240 can assist with hardware-accelerated instructions. Generally, hardware-accelerated instructions are those assisted by the processor subsystem 240 that performs dedicated tasks. Dedicated tasks are not basic CPU instructions such as scalar addition, scalar multiplication, bit operations, etc. Instead, a single call of a hardware-accelerated instruction can correspond to executing some of these basic instructions, but the execution of hardware-accelerated instructions is typically faster. By executing hardware-accelerated instructions, the performance of the computation can be improved, for example, resulting in reduced latency and / or increased throughput.

[0058] Hardware-accelerated instructions typically have multiple, for example, at least 4, at least 16, or at least 64 scalar inputs (such as numerical values like integers or floating-point numbers). A hardware-accelerated instruction can have a single output (such as a numerical value like an integer or a floating-point number), but can also have multiple, for example, at least 4, at least 16, or at least 64 such outputs. A hardware-accelerated instruction can correspond to a predetermined number N of basic CPU instructions. For example, a basic CPU instruction can correspond to a node of an instruction data flow graph. Thus, N can be made equal to the number of nodes of the instruction data flow graph, or N can be made larger if the instruction data flow graph models only part of the instruction. For example, N can be at least 2 12 = 4096, at least 2 16 = 65536, or at least 2 24 = 16777216.

[0059] Generally, the processor subsystem 240 can be implemented by a single central processing unit (CPU), but it can also be implemented by a combination or system of such CPU and / or other types of processing units. Hardware-accelerated instructions can be supported by the processor system 240 in various ways. In some embodiments, the hardware-accelerated instructions are instructions of the CPU of the processor subsystem 240, and for example, the hardware-accelerated instructions can be implemented on-chip. In some embodiments, as shown in the drawings, the hardware-accelerated instructions can be executed by a hardware accelerator 241 separate from the CPU, for example, a dedicated processing unit. The hardware accelerator 241 may be, for example, a coprocessor, an application-specific instruction set processor (ASIP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a composite programmable logic device (CPLD), or any other type of programmable logic. For example, hardware accelerator 241 may be a VTA accelerator as discussed in "A Hardware-Software Blueprint for Flexible Deep Learning Specialization" by T. Moreau et al. (available at https: / / arxiv.org / abs / 1807.04188, and incorporated herein by reference).

[0060] Instruction 050 can invoke a hardware-accelerated instruction. The processor subsystem 240 can be configured to perform a computation that includes invoking a hardware-accelerated instruction according to instruction 050, for example by invoking a hardware accelerator 241. Since the determined possible calls are used, the efficiency of the computation can be improved.

[0061] Optionally, the computation may be part of the application of a trained machine learning model, such as a deep neural network. The processor subsystem 240 can be configured to acquire input to the machine learning model via a sensor interface from sensors such as image sensors, LiDAR sensors, radar sensors, pressure sensors, and containment temperature sensors. The processor subsystem 240 can be configured to apply the machine learning model to the input and acquire an output. Based on the output, control data can be derived and supplied via an actuator interface to actuators such as electrical, hydraulic, pneumatic, thermal, magnetic, and / or mechanical actuators. For example, the system 200 can be used to control robots, autonomous vehicles, etc.

[0062] Generally, but not limited to, each system described herein, including system 100 in Figure 1 and system 200 in Figure 2, can be embodied as a single device or apparatus, such as a workstation or server, or in the form of a single device or apparatus. The device may be an embedded device. The device or apparatus may include one or more microprocessors that run appropriate software. The example given with respect to processor system 240 in Figure 2 also applies to processor system 140 in Figure 1. The software run by the processor system can be downloaded and / or stored in corresponding memory, for example, volatile memory such as RAM, or non-volatile memory such as flash. Generally, each functional unit of each system 100, 200 can be implemented in the form of a circuit. Unless otherwise stated, it will be understood that, generally, the same considerations and optional implementations as those for system 100 in Figure 1 may be applied to system 200.

[0063] Figure 4a shows a detailed but non-exclusive example of a dataflow graph. A dataflow represents a computation in terms of operations obtained by subdividing the computation and the dataflow relationships between these operations. In particular, the graph may include nodes and edges. Nodes can represent inputs and operations of a computation. Edges can represent dataflow relationships between nodes. These edges include input / output relationships (referred to herein as sequential edges), but may optionally include other types of relationships. The graph may also have other nodes and edges separate from these. A dataflow graph is typically directed and, for example, includes at least one directed edge. A dataflow graph may be a connected graph, but this is not strictly necessary.

[0064] For illustrative purposes, the drawings are based on the following calculations.

number

[0065] One or more nodes in a dataflow graph can represent the inputs to a computation. These nodes are also referred to herein as data nodes. Typically, data nodes have only sequential outgoing edges and, for example, no incoming edges of any kind, nor any other types of outgoing edges. The diagram shows data nodes 411, 412, 413, and 414 corresponding to the respective inputs x1, x2, x3, and x4 of the computation represented by the graph, and data nodes 421, 422, 423, and 424 corresponding to the respective inputs y1, y2, y3, and y4. The diagram also shows a further input node 461 corresponding to the constant value 0.

[0066] One or more nodes in a dataflow graph can represent computational operations. These nodes are also referred to herein as operation nodes. Operation nodes can be labeled according to the operation they perform. While a dataflow graph can be used to represent any type of operation, operations as used herein are typically scalar operations, such as operations performed on a fixed number of numerical values ​​(e.g., scalar addition, scalar multiplication, scalar division, negation) or a fixed number of non-numerical values ​​(e.g., bitwise AND, bitwise XOR, bitwise NOT). For example, a scalar value may be a value that has a fixed-size representation in memory, such as up to 256 bits, up to 128 bits, up to 64 bits, or up to 32 bits.

[0067] Typically, an operation node has one or more sequential incoming edges corresponding to the data consumed by the operation. For example, the diagram shows operation node 431 representing the scalar multiplication of input x1,411 and input y1,421, operation node 432 representing the scalar multiplication of input x2,412 and input y2,422, operation node 433 representing the scalar multiplication of input x3,413 and input y3,423, and operation node 434 representing the scalar multiplication of input x4,414 and input y4,424. As in this example, the operation may be a commutative operation, in which case the order of the inputs to the operation is typically not explicitly represented. However, in the case of a non-commutative operation, for example, the order of inputs can be represented by labeling the sequential incoming edges.

[0068] Interestingly, operation nodes can be used to represent commutative contraction operations such as addition or multiplication. For example, node 441 represents the addition of the incoming node of node 441, operation nodes 431-434, and node 461, which represents the initialization value of the operation. Adding a node for the initialization value is optional, and an initialization value can be assumed unless, for example, the neutral element of the operation is explicitly specified as equal to zero or one. The initialization value of an operation can be mapped to the initialization value of an equal operation. Using a single node 441 to represent this operation has several advantages. For example, it reduces the need to include separate nodes to apply operations such as adding values ​​431 and 432, adding value 433 to the result, and adding value 434 to the result separately, pair by pair. This improves efficiency and also prevents the dataflow graph from forcing additions to be performed in a specific order, for example, by prohibiting determining the total by adding values ​​433 and 434 first, then 432, and then 431. Nodes representing commutative reduction operations can be used in the instruction dataflow graph, the computation dataflow graph, or both.

[0069] As shown in this example, nodes in a graph can be explicitly marked to represent commutative contraction operations. In this example, this marking is done by adding a self-edge 451 to node 441. Other marking methods are also possible, for example, by labeling nodes as representing commutative contraction operations. Such marking has the advantage that, for example, when selecting which nodes should be mapped to commutative contraction operations, the constraint solver can more efficiently determine the mapping for commutative contraction operations, and the constraint solver can more easily abandon nodes that do not have self-edges than, for example, when attempting to map the inputs of the operations.

[0070] When commutative reduction nodes are used in the instruction dataflow graph but not in the computation dataflow graph, it is particularly useful to use self-edges to mark commutative reduction operations. For example, when mapping a commutative reduction node A in the instruction dataflow graph to a corresponding node B in the computation dataflow graph, a self-edge can be mapped to the corresponding edge in the computation dataflow graph between subsequent applications of the reduction operation. By forcing the existence of such corresponding edges, it becomes easier to detect mappings between the two graphs.

[0071] An operation node can generate one or more sequential outgoing edges to other operations that use the results of its operation. A node that does not have sequential outgoing edges to other operations (but can have self-edges that outgo itself) can represent the output of a computation represented by the graph. For example, node 441 can represent an output. In this way, an output in the dataflow graph can be made equivalent to the resulting operation. Outputs can also be explicitly marked, for example, by labeling. Output nodes can be labeled with additional information, for example, representing the shape and / or data type of the output, and this additional information can be used for consistency between computations and instructions.

[0072] Optionally, pairs of operations in a dataflow graph can be marked as mutually parallelizable. This marking can be done, for example, by adding an edge between pairs of nodes (not shown in this diagram). Such edges are referred to herein as spatial edges. In particular, spatial edges can be added between nodes that perform similar operations for different parts of a computation, for example, for different output elements. Spatial edges can be used again to assist a constraint solver in mapping mutually parallelizable operations to mutually parallelizable operations, as described herein.

[0073] By including only a subset of the set of edges for a set of mutually parallelizable operations, the number of spatial edges can be reduced, and consequently, efficiency can be improved. For example, instead of having a set of k nodes that are fully connected by spatial edges, the number of edges can be reduced by pruning the connections while maintaining similarly connected components. For instance, a star-shaped configuration can be chosen where one internal node is connected to k-1 leaves by spatial edges. Transitivity allows for the preservation of parallelizability information.

[0074] Mathematically, a dataflow graph as used herein can be formulated, for example, as a labeled directed graph defined as G=(N,E,l), where N is the set of nodes, E⊆N×N is the set of directed edges, and l() is the set L. N ∪L E This is a function that assigns labels from L. N ={{Operation},{Data}} is a set of node label classes that hold, for example, tensor shape, data type and / or arithmetic operations. E ={Spatial,Sequential} is the set of edge labels.

[0075] In various embodiments, a dataflow graph referred to as a computation dataflow graph can be used to represent computations. A dataflow graph referred to as an instruction dataflow graph can be used to represent instructions. These dataflow graphs can represent the complete input / output behavior of a computation or instruction, for example, including all inputs and outputs of the computation or instruction, and nodes that model how the outputs follow the inputs. In some cases, it is also possible to model only a subset of computations and / or instructions. For example, an instruction dataflow graph can represent only the operations and / or inputs used for computations of a subset of the outputs of this instruction dataflow graph, and the remaining inputs, operations, and / or outputs are determined by extrapolation, as described elsewhere.

[0076] Graph operations do not necessarily need to be explicitly performed by computation or hardware-accelerated instructions; for example, addition and multiplication can be modeled separately in the instruction dataflow graph, while they are performed by hardware as combinations of multiplication and addition. Therefore, nodes representing intermediate results of instructions or computations do not necessarily correspond to values ​​that are explicitly computed or stored at a given point in time during the instruction or computation. However, inputs and outputs are typically explicitly computed in the implementation.

[0077] In various embodiments, data is used to define a dataflow graph, such as a computation dataflow graph or an instruction dataflow graph. “Defining” a dataflow graph means that a set of nodes and edges can be derived from the data, or at least that, given a node or edge, it is possible to determine whether that node or edge belongs to the graph. In particular, the data itself does not need to contain explicit data items corresponding to individual nodes and edges. For example, as will be discussed with respect to Figure 4b, a set of nodes and edges can be defined symbolically, or edges can be defined implicitly; for example, the presence of a particular type of edge can be defined by the data by providing a pair of nodes that do not have that edge. Various options are possible.

[0078] Figure 3 shows a detailed but non-limiting example of how computations should be performed using hardware-accelerated instructions.

[0079] The diagram shows an instruction data flow graph (DFG) IDG,330, which represents a hardware-accelerated instruction, or in other words, the function or calculation performed by a hardware-accelerated instruction. For example, the instruction DFG IDG may be the one described in relation to Figure 4a. In particular, the nodes of the graph IDG can represent the input and calculation of a hardware-accelerated instruction.

[0080] The diagram also shows a computation DFG CDG,340, which represents a computation to be performed using hardware-accelerated instructions, such as in Figure 4a. Nodes in the graph CDG can represent computation inputs and operations. The instruction dataflow graph IDG and the computation dataflow graph CDG are distinct and, for example, do not share common nodes or edges.

[0081] The instruction DFG and computation DFG are two separate graphs that can be represented by instruction data and computation data (not shown), respectively. As described above, this data can implicitly or explicitly define the nodes and edges of the graph, making it possible, for example, to enumerate nodes and edges, or at least to test the belonging relationships of sets of nodes and edges. Graph IDG,CDG are generally defined over similar sets of operations, for example, having equivalent granularity. This makes it easier and more efficient to map operations in the two graphs to each other.

[0082] In the definition operation Def,355, the constraint satisfaction problem CSP,360 can be defined using the instruction DFG IDG and the computation DFG CDG. The solution to this problem CSP can represent the possible calls of hardware-accelerated instructions in the computation.

[0083] As is well known, a constraint satisfaction problem CSP is typically defined by a set of variables VARS 361, a set of domains DOMS 362 for each variable, and a set of constraints CONS 363. Mathematically, a constraint satisfaction problem CSP can be represented by the triple (X, D, C), where, ·X={x j |0≦j≦n} is a set of variables, and the problem is to find values ​​within this set of variables. ·D={d j |0≦j≦n} is a set of value domains, and values ​​can be assigned to each variable from this value domain. Assignment Asn(d j ,x j ):x j =v is x j The value v∈d should take j You can select this option. This will allow you to select domain d j To obtain a value only from the variable x j It is possible to restrict this. ·C={c i |0≦i≦m} is a set of constraints. Constraint c i g is a subset of variables x It can be formed over ⊂X, and g d All assignments Asn(g) that have ⊂D d ,g x We evaluate whether the following is valid. The solution to the constraint satisfaction problem CSP may be an assignment that assigns values ​​to all variables, in which case there are no assignments that violate the conjunction of constraint C.

[0084] The constraint satisfaction problem CSP can represent the problem of discovering possible calls of hardware-accelerated instructions in a computation by mapping nodes in the computation dataflow graph CDG to nodes in the instruction dataflow graph IDG. The set of variables VARS may include variables for each node in the instruction dataflow graph. Each node in the computation dataflow graph may be included in the domain DOM with respect to at least one of the variables. The set of constraints CONS may include dataflow constraints and input constraints. The number of constraints may be, for example, at least 10 times, at least 100 times, or at least 1 times, 2 times, or 5 times the number of nodes in the instruction dataflow graph.

[0085] In this way, the embedding of instructions into computations can be effectively described at the scalar level. Interestingly, all potential assignments between nodes in instructions and nodes in computations can be represented as mappings between nodes in a graph, and thus, this formulation can be used to capture all possible solutions to the embedding problem. The consistency problem itself is not constrained by decisions of implicit implementations such as tensor loop ordering or memory layout, or access functions. This eliminates the need to perform transformations in the search to explain such decisions. Instead, such transformations can be derived from the results of the embedding.

[0086] In particular, the set of variables VARS for each node of the instruction DFG IDG can include variables, for example, X = {x | ∀x ∈ N}. i Therefore, all scalar operations and data elements in hardware-accelerated instructions can be represented by variables. The set of domains (DOMS) is given by D={d|d⊆S}. d It can be defined as}. Here, S dThe domain may include all nodes of the computation DFG CDG, or it may include a subset of the set of all nodes, for example, a subset of all nodes having the same operation as node d. However, this latter restriction can also be imposed by constraints. The domain may include, for example, additional values ​​corresponding to dummy operations or dummy inputs.

[0087] Given variables VARS and domains DOMS, a set of constraints CONS can be used to represent the conditions that an assignment can satisfy in order to represent a possible call to a hardware-accelerated instruction.

[0088] An important aspect that can be enforced by the constraint CONS is that the data flow of nodes in the instruction data flow graph IDG is equivalent to the data flow of nodes in the computation data flow graph CDG to which they are assigned. For example, by taking the portion of the computation data flow graph given by the assigned nodes and replacing this portion with the nodes in the instruction data flow graph, it is possible to obtain a computation data flow graph with equivalent output. This constraint that enforces equivalence can be called a data flow constraint.

[0089] In particular, dataflow constraints can compel that the instruction dataflow graph IDG is isomorphic to a subgraph of the computation dataflow graph CDG induced by the solution to the constraint satisfaction problem CSP. In this case, by replacing the mapped subgraph of the computation dataflow graph with the instruction dataflow graph, a graph identical in terms of node connections and node labeling can be obtained. However, strict isomorphism can be relaxed by several techniques described herein.

[0090] To enforce equivalence, one or more constraints representing the data flow can be included in the data flow constraints for an edge of the instruction data flow graph. In mathematical notation, this is the instruction DFG G i =(N i ,E i ,l i ) and calculate DFG G o =(N o ,E o ,l o Given that ) and , the solution to the constraint satisfaction problem CSP is, for example, ∀(s,t)∈E i ⇒(f(s),f(t))∈E o Like G i It is consistent with, and for example, ∀s∈N i :l i (s)≡l o G o An injective function f:G that describes a distinct subset of nodes and edges in [the system]. i →G o The representation can be required by dataflow constraints. Injectivity can be enforced by requiring that each node in the computational dataflow graph exists only once in the solution. A global "AllDiff" constraint that enforces this can be included in the dataflow constraints.

[0091] For example, the following pseudocode can be used to model the data flow: [Table 1]

[0092] Dataflow constraints can enforce that pairs of mutually parallelizable operations in hardware-accelerated instructions map to pairs of mutually parallelizable operations in computations. In this example, this is done by including spatial edges in each dataflow graph IDG,CDG and enforcing that spatial edges in instruction dataflows map to spatial edges in computation dataflow graphs. As mentioned above, including constraints for each pair in the set of mutually parallelizable operations can result in a large number of constraints. To avoid this, the transitivity property of pairwise constraints can be leveraged. Instead of enforcing mutual parallelism on each pair in the set, it can be enforcing it only on a subset of pairs. In particular, if the subset still has the whole set as a transitive closure, parallelizability can be implied with respect to that whole set. For example, any first node in instruction DFG can be selected, and constraints can be added to all parallel nodes of this first node. Once values ​​are assigned to all nodes parallel to the first node, the domain of the first node can be pruned to include only nodes parallel to those assigned. This can be propagated to all other nodes parallel to the first node. Therefore, the same number of values ​​can be efficiently pruned.

[0093] When solving the constraint satisfaction problem CSP, values ​​can be assigned to the variable VARS from their respective domains DOMS. During the solution, nodes in the computation dataflow graph CDG can be selected as possible values ​​for nodes in the instruction dataflow graph IDG. The constraint solver can apply a so-called propagation algorithm, which attempts to assign values ​​to the variables and checks whether this assignment is permitted by the constraint CONS, and / or whether this assignment compels assignments to other variables VARS according to the constraint. Below is an example of a propagation algorithm for enforcing dataflow constraints. Given a node s that has a value assigned and has an edge to another node t, the algorithm in this example checks whether the assignment to s is correct and what correct values ​​can be assigned to the other node t: [Table 2]

[0094] In this example, the propagator directly filters values ​​based on the relationships between nodes in T. The propagator evaluates the relationships ("evaluate_relation") and removes values ​​from the domain of intercepted nodes where no connection exists ("t.intersect"). If the relationships between pairs are functional, the propagator can directly assign a solution ("if t.size==1"). Even if this is not the case, the proposed propagation is strong enough to encompass the domain, meaning that only valid solutions for this constraint remain in the domain, and no further propagation is needed. The remaining domain values ​​are evaluated against other constraints across their respective variables. If the propagation results in an empty domain, the assignment fails.

[0095] Once values ​​are assigned to both nodes s and t in the instruction dataflow graph IDG, the constraint can be checked for correctness by verifying that there are edges connecting the pair in the computation dataflow graph CDG, for example, Asn((s,t),(d_s,d_t))∈E o In particular, this may involve forcing the self-edges of the instruction dataflow graph IDG representing commutative reduction operations to be mapped to the self-edges of the computation dataflow graph CDG.

[0096] Dataflow constraints do not need to enforce strict isomorphism. Various relaxations can be employed. For example, instead of requiring that source operations in the instruction dataflow graph IDG map to the same target operations in the computation dataflow graph CDG, it may suffice to require that the source operations be generalizations of the target operations, e.g., operations with at least the precision required by the target operations, or generalizations of operations that can be instantiated by the source operations. As described elsewhere, relaxations can be employed to allow the mapping of commutative contraction operations in the instruction dataflow graph to nodes in the computation dataflow graph that represent a single contraction of this commutative contraction operation. Another example of not enforcing strict isomorphism, as described elsewhere, is the use of dummy operations.

[0097] As the inventors have recognized, defining dataflow equivalence by dataflow constraints alone is often insufficient to define what constitutes a possible call of a hardware-accelerated instruction in a computation. That is, as shown in the drawings, a hardware-accelerated instruction and / or its programming interface can impose a set of hardware constraints HWR 320 on possible calls. Defining a constraint satisfaction problem CSP Def may include defining one or more constraints CONS representing these hardware constraints. Hardware constraints HWR can be represented in various ways, for example, as constraints, as templates from which constraints are instantiated when defining a constraint satisfaction problem, or as code that implicitly defines constraints based on an instruction dataflow graph and / or computation dataflow graph.

[0098] In particular, hardware constraints can impose one or more input constraints that limit which nodes in the computation dataflow graph CDG can be assigned to the inputs of hardware-accelerated instructions, for example, to the input nodes of the instruction dataflow graph IDG. Such constraints can, for example, enforce that the inputs have permitted memory layouts and / or permitted access patterns. Similar constraints can also be imposed on the outputs of hardware-accelerated instructions, for example, if the outputs are to be used in other computations or in other parts of the current computation that require a particular memory layout. In addition to the constraints imposed by the instruction, it is also possible to include constraints on obtaining calls with other desired characteristics, such as calls that use the first element of the input array. This can make it easier to generalize the discovered calls to generate code for the complete computation. In general, an appropriate set of constraints CONS can depend on at least the hardware target and the available code generation methods.

[0099] For example, the constraint CONS may include one or more of the following: • Restricting input to permitted memory access patterns. For example, input access patterns can be restricted to stencil calculations, or to access patterns with regular strides and / or offsets. Propagation can constrain values ​​in such dimensions to zero, thereby removing a large portion of the search space in calculations where these access patterns reside. • Restricting the mapping instruction input to a specific axis in the calculation. • Restricting the instructions to which they should be applied to specific values ​​in the calculation. For example, by restricting the first output of an instruction to the first output element in the calculation, the search can be fixed to values ​​around those points.

[0100] Interestingly, the solution space can be made more specific by including additional constraints, while relaxing constraints can serve as a tool for exploring implementation strategies. Constraint resolution is beneficial because it provides expressiveness in the program formulation and offers customizable propagation and search algorithms.

[0101] A constraint satisfaction problem CSP can be defined in such a way that it is possible to map the operations of hardware-accelerated instructions to dummy operations that do not affect the computation. Effectively, it is possible to obtain "incomplete" (e.g., non-isomorphic) embeddings of instructions into computations. This makes it possible to discover calls that would otherwise be impossible. For example, there may be cases where the computation performed by an instruction cannot be evenly divided into smaller subproblems. By allowing dummy operations, it becomes possible to use hardware-accelerated instructions even in such situations. For example, an 8x8 matrix multiplication can be performed using a hardware-accelerated instruction for a 16x16 matrix multiplication. This instruction can be used by padding the input and output tensors; that is, the shape can be made to match the instruction, and the instruction can be used. In a constraint satisfaction problem, dummy operations can be allowed by any combination of allowing or restricting duplicate / replicated computations, allowing or restricting duplicate / replicated memory accesses, and allowing or restricting the insertion of dummy values ​​into computations. Typically, dummy values ​​are included, which are neutral elements of the operation, such as 0 (for example, in addition) or 1 (for example, in multiplication), so as not to affect the result of the calculation. These values ​​can be added to the domain DOM of each variable.

[0102] To make constraint satisfaction problems (CSPs) easier to solve and thus more efficiently, it is possible to include constraints in the set of constraints CONS. For example, by applying pruning constraints to groups of variables in the instruction data flow graph IDG that share the same domain, such as all the variables describing the output operations of an instruction, the domain size of the variable group can be reduced. For example, domain d m The range is the vector (e0,···,e m ) can be described by, for example, d m=(4,4,4) can represent a cube with side length 4. Therefore, the coordinates in the cube can be described as (i,j,k) having 0≦i<4, 0≦j<4, and 0≦k<4. Thus, the upper bound e * By setting e i By setting =2, the domain scope can be controlled. For example, using the following unary pruning constraint, for all variables in g, the m-dimensional domain d m =(e0,···,e m The size of all dimensions within )⊂S is

number

[0103] Therefore, the constraint satisfaction problem CSP can assign nodes in the computation dataflow graph to nodes in the instruction dataflow graph. In some embodiments, nodes in the computation DFG are assigned to each node in the instruction DFG. In some embodiments, each node in the instruction dataflow graph is assigned either a node in the computation dataflow graph or a node representing a dummy input or dummy operation. In some embodiments, each node in the instruction DFG is assigned a different value, and for example, each node in the computation DFG can be assigned at most once.

[0104] The diagram also shows the operation Solve,365, which obtains the constraint satisfaction problem CSP, solves this constraint satisfaction problem CSP, and determines the possible call INV,370 of hardware-accelerated instructions in the computation, if any. The call INV can map nodes in the computation dataflow graph CDG to nodes in the computation dataflow graph CDG, which is shown in the diagram by dashed arrows pointing from nodes b, c, and d in the instruction dataflow graph IDG to nodes B, C, and D in the computation dataflow graph CDG. Several desirable characteristics of the call INV are shown in this example. As shown in the diagram, each node in the instruction dataflow graph IDG can be mapped to a node in the computation dataflow graph CDG. However, the computation dataflow graph typically has additional nodes, nodes A and E in this example. The existence of edges between each node in the instruction dataflow graph IDG can imply the existence of edges between each node in the compute dataflow graph CDG to which those nodes are mapped, but additional edges may exist, for example, between node A and node C, and between node D and node E. However, a non-input node in the instruction dataflow graph, for example node d, is typically mapped to a node in the compute dataflow graph that has the same number and / or the same type of incoming edges, for example node D.

[0105] Various constraint programming techniques are publicly known in the literature for performing the Solve operation and can be applied herein. Interestingly, the algorithms used by constraint solvers to find solutions to problems are often customizable. This allows for control over the solution space, for example, the prioritization of specific solutions, without changing what it means for a solution to be valid. In particular, many existing constraint solvers allow for customization of the propagation algorithm, variable selection strategy, and / or value selection strategy.

[0106] A propagation function can be used to remove values ​​from domains (DOMS) that cannot be part of a valid solution. A propagation function can be considered a monotonic filtering algorithm in the sense that it removes values ​​from a domain but does not add them. Typically, a propagation function is specific to each constraint (CONS). Based on the domains (DOMS) and the assignments of other variables under similar constraints, a propagation function can infer which values ​​should be removed from a domain. To find a solution, the solver (Solve) can use a search algorithm to systematically perform assignments and propagate those assignments through the domains (DOMS). Examples of propagation algorithms for enforcing dataflow constraints are described elsewhere in this specification.

[0107] The solver Solve often uses a backtracking-based search algorithm to find all possible solutions to a given problem. This may involve a variable selection strategy that determines which variable x ∈ X from the set of variables VARS should be assigned a value to next. The value selection strategy can determine the order in which the domain of variables should be searched, and can be considered, for example, as a specific implementation of Asn(d,x). Variable selection and value selection affect the time to find the solution and also affect the order in which the solution is output. Therefore, optimizing the value selection strategy and variable selection strategy can affect both the efficiency of the solution and the quality of the determined solution.

[0108] In variable selection strategies, it is preferable to use a strategy that propagates backward through the instruction dataflow graph. The output can be selected first, and then the solution can proceed by traversing the graph in reverse. This strategy has been found to have the greatest potential for propagation for computations related to neural network layer evaluation, for example.

[0109] One example of a value strategy is the use of lexicographical search. Often, value selection strategies can be optimized based on additional information about the structure of the input data for hardware-accelerated instructions. For example, hardware-accelerated instructions can often have multidimensional inputs, and the values ​​to which the instructions can be applied can also be multidimensional. In such cases, a portfolio search strategy can be used as a value selection strategy. In such a search, multiple so-called assets are searched in parallel, each providing an order in which values ​​are assigned to variables. Each different asset can correspond to a different mapping between the dimensions of the hardware-accelerated instruction and the dimensions of the possible input values. Assets can prioritize assignments according to their respective dimensional mappings. These multiple different assets can be searched in parallel, resulting in a stable search duration over a wide range of computations.

[0110] Interestingly, when using a portfolio search strategy, the number of possible assets can be limited by restricting which dimensions of the instruction can be mapped to which dimensions of the computation. This depends on the available instructions and computations and can be done, for example, by manual annotation. For example, the dimensions of an instruction and an operation can be divided into reduced and unreduced dimensions, for example, in the case of matrix multiplication. In a portfolio search strategy, in this case, the reduced dimension of the operation can be mapped only to the reduced dimension of the computation, and similarly for the unreduced dimension. In this way, the number of assets can be significantly limited, thereby improving performance. For example, matrix multiplication over (I,J,K) has two spatial dimensions (I,J) and one reduced dimension (K). A value selection strategy can attempt to directly embed between the dimensions in the computation and the dimensions of the matrix multiplication. In the assets, the search can prioritize the three selected dimensions. s n is the number of spatial dimensions. k If is the number of contractions in the calculation, then the number of assets is {(n s!) / (n s -2)!}·{(n k !) / (n k It could be (-1)!}. This means that otherwise it could have occurred (n s +n k This is a remarkable abbreviation compared to the possible domain permutations of +1)!. Within the asset, lexicographical search can be applied, for example, as a value selection strategy.

[0111] If the solver Solve fails to find a solution in these initial portfolios, the constraints on the search space can be relaxed, allowing the search to be repeated across a wider solution space.

[0112] Generally (and, as mentioned above, when using portfolio search, either instead of or in addition to repeating the search), if the solver Solve fails to find a solution to a constraint satisfaction problem CSP, the solver Solve can be repeated by relaxing the constraint satisfaction problem CSP, for example, until a solution is found or until no further appropriate relaxations can be utilized. For example, the constraint satisfaction problem can be relaxed to allow dummy operations, or constraints that enforce the use of specific inputs can be relaxed. Such relaxations may result in less-than-optimal solutions, for example, fewer operations in the operation may be optimized or more preprocessing may be required to obtain data of the correct shape that can be processed by hardware-accelerated instructions. However, by implementing these relaxations, it becomes possible to apply hardware-accelerated instructions to such less-than-ideal situations.

[0113] The diagram also shows an optional compile operation Comp,345, which can generate a machine-executable instruction INSTR,350 that performs the calculation using a determined call INV. Instruction INSTR can call hardware-accelerated instructions according to the determined possible call INV. For example, a compileable code can be generated using a call INV, which is then compiled to obtain instruction INSTR. The rules used to generate code from a call compile INV for a particular hardware-accelerated instruction typically depend on the provided code generation interface. Generally, the conversion of a call INV to instruction INSTR can be carried out relatively straightforwardly by including sufficient input constraints in the constraint satisfaction problem CSP, depending on how the programming interface allows hardware-accelerated instructions to be called.

[0114] According to some embodiments, the instruction dataflow graph may include nodes relating only to a subset of the outputs of hardware-accelerated instructions. To generate a machine-executable instruction (INSTR), a call INV can be used to infer a mapping of some or all of the remaining outputs of a hardware-accelerated instruction to an operation of the computation. Thus, a hardware-accelerated instruction can be applied to a portion of the computation larger than what is indicated by the possible call INV mapping. For example, in matrix multiplication, the computation of the output elements shares a set of input nodes with its neighbors, but does not share intermediate results. Therefore, a call INV may provide a mapping relating only to one output element, which can then be generalized by using it to perform the full matrix multiplication of the computation.

[0115] Interestingly, generating an instruction INSTR can also involve generalizing the determined call INV to obtain further calls to the hardware-accelerated instruction. In many real-world computations, including neural network workloads such as convolutions, it is possible to extrapolate the execution of a computation by multiple calls from a single determined call INV. To this end, one or more features can be extracted from the determined possible call INV, and based on these features, the determined call INV can be generalized. For example, in this way, the computation Comp can be fully executed by a call to a hardware-accelerated instruction.

[0116] For example, in the case of convolution, the variables in the computation dataflow graph CDG related to the input and output values ​​of the instruction dataflow graph IDG can be evaluated to calculate which dimensions of the instruction align with which dimensions in the workload, and what the tiling coefficients are. From this, an instruction INSTR can be executed to perform a complete convolution. For example, the aligned dimensions can be tiled by the discovered coefficients and moved to become the innermost dimension. These tilings and rearrangements can be fixed for embedding. Other loop and tensor dimensions may still be transformable for further performance optimization, for example, loop tiling, rearrangement, fusion, etc. Such further performance optimizations can be determined automatically using AutoTVM or a similar tool. An instruction INSTR with embedded hardware-accelerated instruction calls can be generated, for example, by the VTA programming tool flow of TVM.

[0117] An optional evaluation operation, Eval,355, is also shown. Often, a constraint solver, Solve, can determine multiple call invs that solve a constraint satisfaction problem CSP, for example, by enumerating all possible solutions to the constraint satisfaction problem CSP. In the evaluation operation Eval, the performance of multiple such calls can be evaluated, and based on this performance evaluation, one solution can be selected from the multiple solutions. The evaluation can be performed, for example, by executing or simulating an instruction and measuring its performance based on a determined machine-executable instruction INSTR, or by directly calculating a cost function (typically hardware-specific) for the call inv without first determining the instruction INSTR.

[0118] Generally, the techniques described herein are applicable to various types of computations and various types of hardware-accelerated instructions. Hardware-accelerated instructions can perform one or more of the following: matrix multiplication, convolution, dot product, matrix-vector product, cumulative sum, pooling, and Hadamard product. The computation may include, for example, the evaluation of convolution, multiplication, or pooling operations in a neural network layer.

[0119] Typically, at least the hardware limit HWR depends on the target hardware, while the computation dataflow graph CDG depends on the computation to be performed. Therefore, for example, it is possible to define multiple implementations for multiple computations by applying the presented approach to multiple different computation dataflow graph CDGs using the same instruction dataflow graph IDG and the same hardware limit HWR, and it is possible to define multiple implementations for equivalent computations by applying the presented approach to the same computation dataflow graph CDG and multiple different hardware limit HWRs and / or instruction dataflow graph IDGs.

[0120] Figure 4b shows a detailed but non-limiting example of a polyhedron representation defining a dataflow graph. In various embodiments, a computational dataflow graph or a portion thereof can be represented by a polyhedron.

[0121] For example, as discussed in relation to Figure 3, the possible calls of hardware-accelerated instructions in a computation can be determined by solving a constraint satisfaction problem in which nodes in the computation dataflow graph are assigned to nodes in the instruction dataflow graph. In many practical cases, the number of nodes in the computation dataflow graph is often very large, for example, at least 1024, at least 4096, or at least 104876 nodes. Interestingly, however, the inventors have found that by using a polyhedral representation to store the computation dataflow graph or at least a portion thereof, it is possible to avoid explicitly determining and storing a list of all nodes and / or edges in the computation dataflow graph. This improves the performance of the constraint solver.

[0122] Generally, in polyhedral representations, a set of nodes in a dataflow graph is represented symbolically, for example, by using a representation that describes which nodes are in the set without explicitly listing them. A set of edges, such as incoming and / or outgoing edges of a set of nodes, can also be represented symbolically. When solving constraint satisfaction problems, a specific node can be obtained from the set of nodes by instantiating the polyhedral representation, for example by selecting nodes according to the value selection strategies described herein. The nodes of the instruction dataflow graph can then be mapped to these instantiated nodes.

[0123] As an example, the diagram graphically represents the dataflow graph using a polyhedron representation. In the case of Figure 4a, 4x4 matrix multiplication is represented, for example, by the tensor expression T:A i,j =Σ k X i,k ·Y k,jThis can be expressed as follows. This calculation can be expressed as a loop nest, that is, if we substitute i for I, j for J, or k for K, then A[i][j]+=X[i][k]*Y[k][j].

[0124] An input tensor can be described by a set of shapes of these input tensors. For example, the figure shows a polyhedral representation x,410 of the set of input nodes in a dataflow graph. This set corresponds to, for example, the set containing nodes 411-414 in Figure 4a, and is the following set X={(i,k)|0≦i <I,0≦k<K} This can be described by: A polyhedral representation y,420 of a further set of input nodes in the dataflow graph, including nodes 421-424 in Figure 4a, is also shown. This set is: Y={(k,j)|0≦k <K,0≦j<J} It can be written as follows.

[0125] Scalar multiplication operations in matrix multiplication calculations can be represented, for example, by a polyhedron representation *,430 including multiplications 431-434 in Figure 4a. Scalar addition operations in matrix multiplication calculations can be represented, for example, by a polyhedron representation +,440 including addition 441 in Figure 4a. Both scalar multiplication and scalar addition are in a single set. S={(i,j,k,n)│0≦i <I,0≦j<J,0≦k<K,0≦n<#T} It can be represented by I, J, and K being domain boundaries. S also includes an additional dimension that describes which elements of the original tensor representation T are selected, for example, whether a node represents addition or multiplication in this case.

[0126] The set of nodes in a dataflow graph is the union of sets defined by polyhedral representations, for example, N o It can be formed as ≡S∪X∪Y.

[0127] The set of edges between nodes in a dataflow graph can also be defined using a polyhedron representation. This polyhedron representation can correspond to a binary relationship between two domains.

number

[0128] For example, in Figure 4b, the edges between the sets of nodes 410, 420, 430, and 440 are related by the following relationship: R1:*→+={i,j,k,n→i',j',k',n'│i'=i,j'=j',k'=k,n=n * ,n'=n +}; R2:+→+'={i,j,k,n→i',j',k',n'│i'=i',j'=j,k'=k+1,n'=n=n +}; A X :X→*={i,k→i',j',k',n'│i'=i,k'=k,n'=n *}; A Y :Y→*={j,k→k',j',i',n'|j'=j,k'=k,n'=n *} It can be defined by:

[0129] In the example above, relation R1 specifies that multiplication and addition occur within the same loop iteration, but are ordered by the positions of multiplication and addition in the tensor expression T. R2 can be interpreted as two sequential addition operations performed in iteration dimension k. X and A Y This encodes the access function itself and, in this case, the multiplication at T, which indicates which node performs the access. The union of all relations is the dataflow graph G o It is possible to describe the edges of, specifically, for example, E o ≡R1∪R2∪A X ∪AY That is the case.

[0130] In this example, a commutative contraction operation, in this example addition, is represented using a set of nodes 440 and a corresponding relation R2. As will be discussed with respect to Figure 4a, in the instruction dataflow graph, such a commutative contraction operation can be represented by a single node, for example node 441 in Figure 4a. In such a case, node 441 in the instruction dataflow graph can effectively correspond to a set of nodes 440 in the computation dataflow graph. Effectively, relation R1 can correspond to incoming non-self edges of node 441, while relation R2 can correspond to self edges. In this case, a constraint satisfaction problem can be constructed to allow node 441 to be mapped to any node in the set of nodes 440. To address the mapping of commutative contraction operations to nodes in the set of nodes 440, terms controlling the order of contractions can be relaxed, for example, the terms with respect to k' in R1 and R2 can be relaxed so as not to determine any ordering of contractions. This relaxation effectively allows each node in the set of 440 to be treated as a commutative reduction node. Interestingly, although this relaxation significantly increases the number of edges in the graph, it eliminates the need to fully instantiate the computational dataflow graph, making it possible to efficiently implement the graph using a polyhedral representation. Alternatively, even when using a polyhedral representation, the commutative reduction operation can be represented as a single node.

[0131] Labels for nodes and / or edges in a dataflow graph can be provided, for example, as a common label for a set, or as separate, symbolically defined labels. Labels can indicate the type of input or output, the type of operation, etc. For example, node 410 can be labeled as “OutType:int8”, node 420 can be labeled as “OutType:int8”, node 430 can be labeled as “InType:int8;OutType:f32;Op:mul”, and node 440 can be labeled as “InType:f32;OutType:f32;Op:add”.

[0132] Pairs of mutually parallelizable operations can also be represented using polyhedral notation. For example, as discussed in Figure 4a, pairs of mutually parallelizable operations can be represented by spatial edges. Such sets of edges can also be represented using polyhedral representation.

[0133] Regarding relations that describe a set of edges, several cases can arise. Some or all of the relations may be asymmetric. One or more relations, such as the relation *→X in this example, may be surjective and functional, for example, multiplication may map to exactly one tensor element that it consumes in X. One or more inverse relations, such as the relation X→*, may be nonfunctional, for example, input elements in X may be available through multiple multiplications, but not all of them. This is A X and A Y This is observable by the relationship, because A X and A Y The relationship does not contain any terms that have properties related to i' or j', respectively. For a given input value in X, this relationship can describe a subset of all multiplications that use this value.

[0134] Generally, a set S of operations of a data flow graph defined by a polyhedron representation can be referred to as an instance set, and its elements are called dynamic execution instances. The set S can be described by a set of integer tuples, and the integer tuples describe dynamic execution instances, such as nodes. For example, to describe a set of integer tuples, the following notation: {(e0,···,e n )|τ0,···,τ n} can be used.

[0135] For example, each term τ j can provide a fixed lower and / or upper bound for each tuple member e j . Thus, a complete set can be defined by the conjunction of all τ terms. The domain of the constraint satisfaction problem can be defined as a subset of a set of integer tuples, for example, defined by the type of node. In the case of input nodes, the domain can be described by a set of tuples in the shape of each tensor. In the case of operation nodes, the domain may be a set of tuples representing the operation or a subset thereof.

[0136] The union of binary relations between pairs of instances, such as sequential edges of a graph, may also be referred to as a data dependence relation D. The relation between elements from a source set to a target set can be s source →s target ={(e0,···,e n )→(e’^0,···,e’ n )|Φ0,···,Φ m} denoted as, where the term Φ k describes the conditions in the relation. The elements in the target tuple can be denoted as e’. The relation condition Φ k can be used to describe the source element e that is mapped to the element e’ in the target set. The entire relation domain can be described by the conjunction of all Φ terms.

[0137] The use of polyhedral representations is particularly beneficial for computations that are part of neural network layer evaluation, such as convolution, matrix multiplication, or pooling operations. Such computations typically operate over tensors with bounds known at compile time, such as n-dimensional arrays. Typically, various output elements are computed using similar methods, but with different slices of input values. The computations are highly structured, typically involving deep loop nesting without conditional statements. This makes representing the computations using polyhedral representations particularly efficient.

[0138] Interestingly, using polyhedral representations in constraint satisfaction problems as described herein does not result in a loss of generality. The mapping of instructions to computations still occurs at the level of individual nodes, for example, at the scalar level. The constraint solver can assign any computation dataflow graph node from the described set of nodes to an instruction dataflow graph node, regardless of how the set is described, for example, regardless of the order of dimensions. This differs from other approaches, where explicit transformations are applied to the polyhedral representation and the way the set is described involves difficult-to-overcome modeling assumptions.

[0139] Figures 5a to 5d provide a detailed but non-restrictive example of applying the hypercuboid construct. In various settings, it is beneficial to restrict the input of hardware-accelerated instructions to form a regular shape. This can result in implementations with more efficient memory access and / or, for that purpose, enable the use of efficient memory layout transformations to allow hardware-accelerated instructions to be invoked on the input. In particular, this example shows how the hypercuboid construct can be used to force the input to form a hypercuboid.

[0140] In this specification, the term hyper-rectangle is generally used to refer to a rectangle having any number of dimensions n>0, for example, n=1, n=2, n=3, or n≧4. For example, in the case of 0≦δi≦Δi, the set of points (x1+···+x n Points can be placed on a hypercube as a square lattice forming ) + δ1·(s1,0,···,0) + δ2·(0,s2,0,···,0) +···, where s i These are stride values ​​that can be individually fixed to 1, for example, set or dynamically determined.

[0141] For example, the use of a hypercube is suitable for computations that occur in neural network evaluation because such computations typically involve (often large) tensors, such as n-dimensional arrays. The shape of a hypercube is particularly beneficial for efficient memory access and for enabling efficient calling of hardware-accelerated instructions by using memory layout transformations, such as transposing or tiling the tensor dimensions. Constraints can be used to ensure that the variables assigned to the solution's input values ​​form a hypercube, for example, a hypercube aligned to a vector basis in tensor space. Another advantage is that by constraining the inputs to form a hypercube, potential assignments to input nodes in the instruction dataflow graph can be relatively easily removed by the constraint solver, thereby significantly improving the efficiency of constraint resolution.

[0142] The ultra-rectangular system is, for example, the point V = (v0,···,v) corresponding to the input of a hardware-accelerated instruction. nIt is possible to construct a consistent propagation of the shape of a hypercube with many dimensions n>0 from an ordered tuple of ). The constraint can be configured to infer a bounding box from the set of input points to which computational values ​​have been assigned so far and the total number of points in V. Values ​​outside the bounding box can be removed as possible values ​​for the remaining input points of the hardware-accelerated instruction. Thus, typically, it is possible to eliminate most of the domain after only a few assignments. For example, after selecting only two points along one axis of the tensor, this dimension can be constrained to the size of bound=#V·|v0-v1|.

[0143] A rectangle can be computed by performing a linear iteration of a set of points V, which involves iteratively attempting to infer whether the points form a rectangle with any number of dimensions. In particular, if the points describe a lattice in lexicographical order, the iteration may involve performing one of the following steps: The basic step can represent a movement from one element to the next in the innermost dimension. A dimension jump can represent a movement to the next line in the innermost dimension, for example, a movement diagonally through space. To enforce a lattice structure, it can be required that the basic step is identical and repeated a fixed number of times before a dimension jump occurs. A dimension jump can add a new dimension to the rectangle. Dimensions [n0,···,n m For ]∈Rec, the dimensional jump n is 0≦k≦m. k but,

number

[0144] Figures 5a to 5d show examples of forcing a hypercube constraint, which forces a lattice structure onto a set of points. The drawings illustrate how constraints can be forced and propagated. In the drawings, the black dots 512, 522, and 532 represent domain values, the × marks 511, 521, 531, and 541 represent selections for one of 16 variables, and the white dots 533 and 543 represent values ​​removed by propagation.

[0145] In this example, propagation may still be impossible in the first four steps (Figures 5a and 5b) because the size of the dimension along the x-axis, 8, is smaller than the total number of variables, 16. However, once a fifth value is selected, a value can be removed from the domain. In this example, the value (y,x) with x>3 can be removed because this is the size of the innermost dimension and must be equivalent for all outer dimensions. From the selected value (1,0) for the fifth variable, the propagation function extends to the y dimension,

number

[0146] Interestingly, by performing a hypercube constraint through iteration across a set of points V, the resulting constraints can be made transparent to the relative dimensional order between instructions and workloads. As a result, any dimension in the workload can be mapped to any dimension in the instruction without having to perform operations such as transposition. Interestingly, this mapping can be used to derive what memory transformations, such as transposition, tiling, or fusion, should be used when generating code to invoke hardware-accelerated instructions.

[0147] Next, we provide a detailed example of the implementation of the supercuboid algorithm. This algorithm also works for partially assigned rectangles. The domain of residual values ​​can be propagated by directly using the information discovered during inference to compute the bounding box, which is the largest possible rectangle that can be formed by the unsolved assignments.

[0148] The algorithm begins by checking the first step, returning failure if the points are not aligned parallel to the coordinate axes. Next, the algorithm removes the completed steps from the vector basis so that each tensor dimension is used only once. Then, the remaining points are iterated over, and all steps are tabled. `live_tbl` records the steps being executed. k When a dimensional jump to d occurs, k-1 The number of steps performed is recorded in dim_table. Every step that occurs is checked to see if it has already been recorded. If it has already occurred, counters_valid checks whether the step occurred at the correct time and resets all counters in the inner dimension. In the example in Figure 5, there must be four steps in x dimension for each step in y dimension. When y step occurs, the counter for x is reset. [Table 3]

[0149] Here, we provide details of an experiment in which the techniques considered for generating code for a VTA hardware accelerator that provides hardware-accelerated matrix multiplication (GEMM) instructions are applied. The hardware is instantiated on a ZynqUltrascale+ FPGA having a 256kb weight buffer, a 128kb data buffer, and a GEMM core that consumes an int8 input and produces a float32 result. The GEMM unit is for the case (x,y,z)=(1,16,16)

number

[0150] The code generated according to the presented technique is compared to the TVM's conv2d reference implementation, which statically maps the three axes x,y,z of the GEMM unit in convolution to the dimensions of batch n=x, the dimensions of output channel oc=y, and the dimensions of input channel ic=z. The TVM convolution implementation expects an NCHW (N=batch size, C=channels, H=height, W=width) memory layout.

[0151] In the experiment, machine-executable instructions were generated using TVM's machine learning compiler framework. When TMV encounters an operator that can be accelerated by hardware, the implementation is handled by a specified deployment strategy. The strategy implements the operator in TVM's IR, optimized for the hardware target. TVM's code generation toolflow for VTA is available by incorporating TVM's approach into VTA's strategy. Instructions DFG G i This is generated based on the hardware configuration. Based on this, the constraint satisfaction problem is defined as described above. G i The nodes are variables, and the domain is a collection of dynamic instances of operators.

[0152] In one experiment, a constraint solver was used to determine possible calls similar to those in the reference implementation. For this purpose, the following constraints were applied: • Data flow constraints to ensure consistent data flow for GEMM instructions. • To enable simpler memory conversion based on transposition and reshaping operations, and to ensure that input and output elements are mapped to shapes aligned parallel to the coordinates, - The allDiff constraint prevents the same dynamic execution instance from appearing multiple times in the same instruction call. • A fixed origin constraint that enforces that the initial alignment of all input and output tensors is fixed to the origin of each domain. • Density constraints that do not allow strides in any dimension for input and output tensors. • Memory access constraints that allow only alignment in the workload dimension, which has an access pattern consistent with the instruction and does not have, for example, a stride or stencil pattern. It was used.

[0153] Given the computation, the constraint solver is used to perform instruction G i Each operation and data element is calculated G oPossible calls can be obtained that describe how they map to the nodes. Due to the regularity of DNN workloads such as convolution, it is possible to extrapolate the discovered calls to implementations that repeatedly call hardware-accelerated instructions. In the solution, variables related to input and output values ​​are evaluated to calculate which dimensions of the instructions align with which dimensions in the workload, and what the tiling coefficients are. This information makes it possible to generate code. The aligned dimensions are tiled by the discovered coefficients and moved to become the innermost dimension. These tilings and rearrangements are fixed for embedding. Other loop and tensor dimensions can be freely transformed for further performance optimization. These optimizations include tiling, rearranging, or merging loops. Using AutoTVM, the best optimization parameters for each conv2d layer were automatically determined. Code with embedded instructions was generated by the TVM VTA programming tool flow.

[0154] Experiments demonstrated that implementations with similar structure and performance to expert-created reference implementations can be automatically generated. Interestingly, in some benchmark calculations, the code generated based on determined possible calls achieved significant speed improvements compared to the reference implementation. Specifically, in some benchmark calculations, the reference implementation achieved very low hardware utilization due to its static embedding strategy. In these calculations, speed improvements of up to 2.49 times were achieved, and improvements of as much as 238 times were observed for individual operators. Therefore, in these scenarios, the automatically generated code according to determined calls resulted in significant performance improvements.

[0155] In other experiments, the computational tensor layout of the DNN was dynamically changed using the presented technique. The solution to the constraint satisfaction problem indicates which dimensions are required and, consequently, which dimensions are free, allowing the memory layout of the free dimensions to be changed during code generation. In this experiment, a constraint solver was used to generate possible calls according to an NHWC (N=batch size, H=height, W=width, C=channels) layout. In many cases, this proved to improve performance compared to the NCHW layout used by the reference implementation.

[0156] In further experiments, memory access constraints were relaxed compared to previous experiments. This increased the number of solutions to the constraint satisfaction problem. As mentioned above, the above experiments included constraints to obtain solutions similar to those of the reference implementation. In some benchmark calculations, these constraints resulted in the need to perform zero-padding in the ic dimension because ic / z < 1, resulting in reduced utilization and larger tensors. Interestingly, it was found that relaxing these constraints made it possible to determine possible calls that require less padding or no padding at all, thereby providing performance improvements. In particular, relaxations were applied that allowed for mapping of multiple dimensions of convolutions included in the filter stencil. Relaxing constraints means that it is still feasible, although it is necessary to handle more different cases when generating the code to perform the calculations. Specifically, the code generation was adapted to help merge multiple tensor dimensions into one and to linearize data access in stencil calculations. The latter involves explicitly replicating in memory the access patterns created by the stencil, such as W[h+kh]. The new footprint of the expanded dimension is h' = h / stride*kh, while the total number of operations to compute the result remains the same. To minimize the overhead created, only the stencil dimension required for the embedding was expanded. To generate the code for the VTA, the tensor decomposition coefficient, which is an even divisor of the original dimension, was used. Where an implementation without any padding was impossible, the contracted dimension was automatically added as needed to the next even divisor of the instruction size.

[0157] In particular, memory layout transformations were performed using relay functions. The operator for expanding the stencil is “relay.take()”, which is a gather function that obtains an index list to determine which values ​​should be duplicated. This is because the direct im2col operator cannot be used in relays.

[0158] The performance of the implementation generated according to this technology was evaluated with respect to the performance of inference and memory conversion and the data footprint. Compared with the reference implementation of TVM with padding, it is often possible to improve the memory footprint, the performance of operators, and / or the overall performance. In some cases, this involves a trade-off, and various optimal implementations can be discovered according to which performance parameter is optimized. In particular, in many cases, the effective hardware utilization controlled by padding was an important factor for improving performance. That is, when ic < z, only ic / z·(h·w) elements in the input image significantly contribute to the result, suppressing the effective hardware utilization rate and making this technology particularly advantageous. Another situation where the generated call was particularly well implemented is when padding generates a weight tensor that exceeds the capacity of the accelerator's weight buffer. The implementation generated by the presented technology can avoid this, which can have the effect that the accelerator can hold the complete weight tensor in the on-chip buffer. In such cases, a particularly significant performance acceleration was observed.

[0159] Figure 6 shows a block diagram of a computer-implemented method 600 for performing calculations using hardware-accelerated instructions of a processor system. Method 600 can correspond to the operations of system 100 of FIG. 1. However, it is not limited thereto, and thus, method 600 may be implemented using other systems, other devices, or other equipment.

[0160] Method 600 may include obtaining calculation data 610 in an operation titled "obtain calculation data". The calculation data can define a calculation data flow graph representing the calculation. The nodes of the calculation data flow graph represent the inputs or operations of the calculation.

[0161] Method 600 may include obtaining instruction data 620 in an operation titled “Obtaining Instruction Data”. Instruction data can define an instruction data flow graph representing hardware-accelerated instructions. Nodes in the instruction data flow graph can represent inputs or operations of hardware-accelerated instructions.

[0162] Method 600 may include defining a constraint satisfaction problem 630 based on computation data and instruction data in an operation titled “Define an instruction -> compute a CSP”. The solution to the constraint satisfaction problem may represent possible calls of hardware-accelerated instructions in the computation. The constraint satisfaction problem may include assigning nodes in the computation dataflow graph to nodes in the instruction dataflow graph. The constraint satisfaction problem may include one or more dataflow constraints that compel the assigned nodes in the computation dataflow graph to have a dataflow equivalent to that of the instruction dataflow graph. The constraint satisfaction problem may further include one or more input constraints that restrict which nodes in the computation dataflow graph can be assigned to the inputs of hardware-accelerated instructions. These restrictions may be imposed by the hardware-accelerated instructions and / or their programming interface. The constraint satisfaction problem may include additional constraints.

[0163] Method 600 may include solving a constraint satisfaction problem 640 to determine the possible calls of hardware-accelerated instructions in a computation, in an operation titled “Solving CSP to determine calls”. Method 600 may include outputting data defining the possible calls 650 in an operation titled “Outputting calls”.

[0164] Generally, it will be understood that the operations of method 600 in Figure 6 may be performed in any suitable order, for example, sequentially, simultaneously, or in combination thereof, and, where applicable, subject to a specific order required by the input / output relationship.

[0165] This method may be implemented on a computer as a computer implementation, as dedicated hardware, or as a combination of both. As shown in Figure 7, instructions for the computer, such as executable code, may be stored on a computer-readable medium 700, for example, in the form of a series of machine-readable physical marks 710 and / or as a series of elements having various electrical, such as magnetic or optical properties or values. The executable code may be stored temporarily or permanently. Examples of computer-readable media include memory devices, optical memory devices, integrated circuits, servers, online software, etc. Figure 7 shows an optical disk 700.

[0166] Alternatively, the computer-readable medium 700 may include temporary or non-temporary data 710 which represents instructions for causing the processor system to perform calculations when executed by a processor system supporting hardware-accelerated instructions, for example by system 200 in Figure 2, and which instructions are for calling hardware-accelerated instructions according to possible calls determined as described herein.

[0167] Furthermore, a computer implementation is envisioned in which a processor system supporting hardware-accelerated instructions executes instructions to invoke hardware-accelerated instructions according to a determined possible call.

[0168] Examples, embodiments, or optional features, whether stated to be non-limiting or not, should not be understood as limiting the claimed invention.

[0169] It should be noted that the embodiments described above are illustrative and not limiting to the present invention, and that a number of alternative embodiments can be designed by those skilled in the art without departing from the scope of the appended claims. In the claims, no reference numerals in parentheses should be construed as limiting the scope of the claims. The use of the verb "includes" and its conjugations does not preclude the existence of elements or stages other than those described in the claims. The articles "a" or "an" preceding an element do not preclude the existence of multiple such elements. Expressions such as "at least one" preceding a list or group of elements represent a selection of all or any subset of elements from that list or group. For example, the expression "at least one of A, B, and C" should be understood as including only A, only B, only C, both A and B, both A and C, both B and C, or all of A, B, and C. The present invention may be implemented by hardware comprising several distinct elements and by a appropriately programmed computer. In an apparatus claim listing several means, some of these means may be embodied by the same hardware item. The mere fact that several specific means are listed in different dependent claims does not mean that combinations of those means cannot be used advantageously.

Claims

1. A computer implementation method (600) that performs calculations using hardware-accelerated instructions of a processor system, The method (600) includes acquiring calculation data (610), wherein the calculation data defines a calculation data flow graph representing the calculation, and the nodes of the calculation data flow graph represent the input or operation of the calculation. The method (600) includes acquiring instruction data (620), the instruction data defining an instruction data flow graph representing hardware-accelerated instructions, and the nodes of the instruction data flow graph representing inputs or operations of the hardware-accelerated instructions. The method (600) includes defining a constraint satisfaction problem (630) based on the computation data and the instruction data, wherein the solution to the constraint satisfaction problem represents the possible calls of the hardware-accelerated instruction in the computation, the constraint satisfaction problem assigns nodes in the computation data flow graph to nodes in the instruction data flow graph, and the constraint satisfaction problem is at least - One or more constraints that compel the assigned node in the computation data flow graph to have a data flow equivalent to that of the instruction data flow graph, - Includes one or more constraints that limit which nodes of the computation dataflow graph can be assigned to the input of the hardware-accelerated instruction, the constraints being imposed by the hardware-accelerated instruction and / or its programming interface. The method (600) includes solving the constraint satisfaction problem (640) to determine the possible calls of the hardware-accelerated instruction in the calculation, and outputting data (650) that defines the possible calls. Method (600).

2. The method (600) further includes generating a machine-executable instruction to perform the calculation, The instruction invokes the hardware-accelerated instruction according to the determined possible call. The method according to claim 1 (600).

3. The calculations of the aforementioned calculations, and / or the calculations of the hardware-accelerated instructions, are scalar operations. The method according to claim 1 or 2 (600).

4. The hardware-accelerated instructions perform one or more of the following: matrix multiplication, convolution, dot product, matrix-vector product, cumulative sum, pooling, and Hadamard product. The method according to claim 3 (600).

5. The calculation includes applying one or more of the following operations to the neural network layer: convolution, matrix multiplication, and pooling. The method according to claim 4 (600).

6. The instruction data flow graph includes nodes relating only to a subset of the outputs of the hardware-accelerated instructions. The method further includes inferring a mapping from the possible calls to the computation of the further outputs of the hardware-accelerated instructions. The method according to any one of claims 1 to 5 (600).

7. The computational data includes a polyhedron representation that symbolically defines a set of nodes in the computational data flow graph. Solving the aforementioned constraint satisfaction problem means, Instantiating the polyhedron representation in order to obtain one node from the set of nodes, The acquired nodes are mapped to the nodes of the instruction data flow graph. The method according to any one of claims 1 to 6 (600), including the method according to any one of claims 1 to 6.

8. The method includes defining one or more constraints to enforce that the input of the hardware-accelerated instruction has an permitted memory layout and / or permitted memory access pattern. The method according to any one of claims 1 to 7 (600).

9. The method includes defining one or more constraints to enforce that pairs of mutually parallelizable operations of hardware-accelerated instructions are mapped to pairs of mutually parallelizable operations of computations. The method according to any one of claims 1 to 8 (600).

10. The nodes in the instruction data flow graph represent commutative reduction operations, The method includes defining one or more constraints to enforce that the node is mapped to the corresponding node in the computed dataflow graph. The method according to any one of claims 1 to 9 (600).

11. The method includes defining the constraint satisfaction problem in order to allow mapping the operations of the hardware-accelerated instructions to dummy operations that do not affect the computation, The method according to any one of claims 1 to 10 (600).

12. The method includes including one or more constraints in the constraint satisfaction problem that compel the input of the hardware-accelerated instruction to form a hypercube, The method according to any one of claims 1 to 11 (600).

13. This method is Determining multiple solutions to the aforementioned constraint satisfaction problem, Evaluating the performance of the aforementioned multiple solutions, Selecting one solution from the multiple solutions based on the performance evaluation. The method according to any one of claims 1 to 12, including (600).

14. A system (100) for performing calculations using hardware-accelerated instructions of a processor system, The system includes a data interface (120) for accessing calculation data (030) and instruction data (040). The aforementioned calculation data defines a calculation data flow graph representing the calculation, and the nodes of the calculation data flow graph represent the input or operation of the calculation. The instruction data defines an instruction data flow graph representing hardware-accelerated instructions, and the nodes of the instruction data flow graph represent the input or operation of the hardware-accelerated instructions. The system includes a processor system (140), The processor system (140) is configured to acquire the calculation data and the instruction data, The processor system (140) is configured to define a constraint satisfaction problem based on the computation data and the instruction data, the solution to the constraint satisfaction problem represents the possible calls of the hardware-accelerated instruction in the computation, the constraint satisfaction problem assigns nodes in the computation data flow graph to nodes in the instruction data flow graph, and the constraint satisfaction problem is at least, - One or more constraints that compel the assigned node in the computation data flow graph to have a data flow equivalent to that of the instruction data flow graph, - Includes one or more constraints that limit which nodes of the computation dataflow graph can be assigned to the input of the hardware-accelerated instruction, the constraints being imposed by the hardware-accelerated instruction and / or its programming interface. The processor system (140) is configured to solve the constraint satisfaction problem to determine the possible calls of the hardware-accelerated instructions in the computation, and to output data defining the possible calls. System (100).

15. A computer-readable medium (700) containing temporary or non-temporary data (710), - The temporary or non-temporary data (710), when executed by the processor system, represents an instruction causing the processor system to perform the computer implementation method described in any one of claims 1 to 13, and / or - The temporary or non-temporary data (710) represents an instruction to cause the processor system to perform a calculation when executed by the processor system supporting the hardware-accelerated instruction, the instruction being for calling the hardware-accelerated instruction in accordance with possible calls determined using the computer implementation method described in any one of claims 1 to 13. Computer-readable media (700).

Images