Efficient Data Processing

Abstract

A processor and method for manipulating data by retrieving operations from storage, analyzing each of the operations to determine an associated operation space, and generating at least one operation set, the operations of the operation set having substantially similar operation spaces. Receive input data in the form of tensors and assign the input data as input to a given operation of the operation set. The input data has predetermined input characteristics associated with the given operation. Perform the given operation using the input to produce an output having known output characteristics. Store the input data and the output associated with one operation of the operation set in a segment associated with that operation of the operation set.

Description

[Technical Field]

[0001] The present invention relates to a method, a processor, and a non-transitory computer-readable storage medium for handling data for processing by a set of operations, such as neural network processing operations and graphics processing operations. [Background technology]

[0002] Description of the Prior Art Certain data processing techniques, such as neural network processing, graphics processing, etc., involve the processing and generation of large amounts of data using operations, and it is desirable to efficiently handle the data as it is processed by a set of operations. Summary of the Invention

[0003] According to a first aspect of the present invention, there is provided a processor for handling data, the processor comprising a handling unit, a plurality of storage elements and a plurality of execution units, the processor being configured to: retrieve from a storage device task data describing a task to be performed in the form of a graph of operations, each of the operations mapping to a corresponding execution unit of the processor and each connection between the operations in the graph mapping to a corresponding storage element of the processor, the task data further defining an operation space representing a dimension of a multidimensional array of operations to be performed; transforming, for each of a plurality of portions of the operation space, said portions to generate, for each of a plurality of operations of the acyclic graph, a respective operation-specific local space; and dispatching, to each of a plurality of execution units associated with the operations for which the transformed local space has been generated, call data describing the operation-specific local space and at least one of a source storage element and a destination storage element corresponding to a connection between the particular operation to be performed by the execution unit and a further adjacent operation in the graph to which the particular operation is connected.

[0004] According to a second aspect of the present invention, there is provided a method for handling data in a processor comprising a handling unit, a plurality of storage elements and a plurality of execution units, the method comprising: retrieving from a storage device task data describing a task to be performed in the form of a graph of operations, wherein each of the operations maps to a corresponding execution unit of the processor and each connection between the operations in the graph maps to a corresponding storage element of the processor, the task data further defining an operation space representing a dimension of a multidimensional array of the connected operations to be performed; transforming, for each of a plurality of portions of the operation space, said portions to generate, for each of a plurality of operations of the graph, a respective operation-specific local space; and dispatching, to each of a plurality of execution units associated with the operations for which the transformed local space was generated, call data describing the operation-specific local space and at least one of a source storage element and a destination storage element corresponding to a connection between a particular operation to be performed by the execution unit and a further adjacent operation in the graph to which the particular operation is connected. [Brief explanation of the drawings]

[0005] Further features and advantages will become apparent from the following description of preferred embodiments, given by way of example only, made with reference to the accompanying drawings, in which like reference symbols are used to denote like features and in which: [Figure 1a] 1 illustrates an exemplary directed acyclic graph in which sections are interconnected by a series of pipes, according to the present disclosure. [Figure 1b] 1 illustrates a schematic diagram of an example data processing system according to the present disclosure. [Figure 2] 1 shows a schematic diagram of a neural engine according to the present disclosure. [Figure 3] 1 illustrates a schematic diagram of an exemplary system for allocating handling data according to the present disclosure. [Figure 4] 1 shows a table showing data relating to several sections according to the present disclosure. [Figure 5]1 shows a table showing data relating to several sections according to the present disclosure. [Figure 6] 1 shows an exemplary chain of operations that may be performed. [Figure 7] 1 shows an exemplary corresponding coordinate space. [Figure 8] An example of scheduling of the blocks shown in FIG. 7 is shown below. [Figure 9] 1 shows a flowchart of efficient data processing according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0006] Examples herein relate to a processor for manipulating data, the processor comprising a manipulating unit, a plurality of storage elements, and a plurality of execution units. The processor is configured to retrieve task data from a storage device that describes tasks to be performed in the form of a graph of operations, such as a directed acyclic graph. Each of the operations maps to a corresponding execution unit of the processor, each connection between operations in the graph maps to a corresponding storage element of the processor, and the task data further defines an operation space representing dimensions of a multidimensional array of connected operations to be performed. The examples described below refer to a directed acyclic graph of operations, but it will be understood that any type of graph of operations may be used.

[0007] For each of a plurality of portions of the operation space, the processor is configured to transform the portion of the operation space to generate, for each of a plurality of operations of the graph, a respective operation-specific local space.

[0008] The processor is further configured to perform clipping, if necessary, on lower and upper bounds of the task and operation space before performing the transform. Clipping may be functionally necessary on the edges of the tensor, allowing for a smaller operation space than the full tensor. A smaller operation space than the full tensor is advantageous because it allows a larger sequence of operations to be split across multiple independent tasks, optionally performed on separate cores.

[0009] The processor is further configured to dispatch, to each of a plurality of execution units associated with the operation for which the transformed local space was generated, invocation data describing the operation-specific local space and at least one of a source storage element and a destination storage element corresponding to a connection between the particular operation to be performed by the execution unit and a further adjacent operation in the acyclic graph to which the particular operation is connected.

[0010] This disclosure relates to executing a graph of operations (called sections) connected by various connections (called pipes). By providing the ability to operate on a sequence of connected operations (sections), which can be defined within a common operation space for the sequence of operations, it is possible to ensure that all coordinates required by the operations in the operation space are reachable when executing the sequence of operations. For each execution of an operation (or portion of an operation), the operation space is transformed into a local section space for that operation. More generally, a directed acyclic graph of operations includes vertices (see operations) connected by edges, with each edge directed from one vertex to another such that the direction of the edge does not form a closed loop. As noted above, a task can be performed in the form of a graph of operations representing a given sequence of operations. This can be represented as any type of graph, not just a directed acyclic graph. In such examples, the graph of operations includes vertices (see operations) connected by directed or undirected edges. In some examples, the directed edges may form a closed loop.

[0011] Each operation (section) is linked by a corresponding pipe to form a directed acyclic graph of operations. For each operation, a source pipe and a destination pipe can be defined, and under the control of the handling unit, execution of a section can be issued by issuing call data that defines the source and destination pipes for the operation. This execution of the graph of operations by each execution unit is therefore implicitly ordered by dependencies on the specific inputs to the operation. The result of this implicit ordering is a simplified orchestration of operations among the execution units of the processor. In other words, the sections and their directed acyclic relationships to each other can be determined by their pipe usage (e.g., their producers / consumers).

[0012] In this disclosure, by transforming from the operation space, it is guaranteed that for each possible operation, there exists a specific coordinate space, called a section space (or section-specific local space). For all operations, there may be a fixed function transformation from their individual section space to each of their input and output data (pipes), which may be different for multiple inputs / outputs. For element-wise operations, the transformation from the section space to the input and output pipes is an identity mapping, and no transformation is required. For convolution, the output is also the identity of the section space, and transformation is only required for the input. An exception to this is that for some operations (e.g., convolution), the output space is only the outer four dimensions. Furthermore, inputs to some operations may have non-identity transformations from the section space and may be different from each other. However, in this disclosure, every operation is defined in its own independent section space, specific to that section (or operation), without requiring a mapping to the outputs of other operations.

[0013] Different operations with different types are chained together by defining a common operation space for the entire graph (or chain of operations) and then defining transformations from the operation space to each operation's individual section space. Here, each hardware unit only needs to understand the fixed-function transformation from the section space to the input / output space, not the chain of operations before and after it. For example, as long as valid operation space conditions exist, it is possible to chain additional operations before or after a convolution operation to stitch together more diverse operations. Because all sections iterate through the same operation space during execution, blocks of data are aligned. For example, the first block from a memory read operation becomes the first block to a data processing operation, which in turn moves incrementally to the first block in a memory write operation. This is a simplification, but generally holds true in principle, considering some operations (reduction and broadcast operations) where blocks can be grouped with data from other blocks to form a new merged block. The operation space is typically mapped to the space of unique operations in the graph, and programmatic transformations are provided for all other operations.

[0014] An operation that accesses a pipe may have additional transformations to access the data stored in the pipe. For example, this may be a different transformation for different pipes, a different transformation for multiple inputs, a different transformation for the output. This transformation is defined in the nature of the operation and is a fixed function.

[0015] In summary, the section space of an operation can be mapped to the input and / or output (which may be the same), or the section space of an operation can be mapped separately if a fixed function transformation is required. In this way, the proposed approach enables more compartmentalized functionality in separate execution units. Therefore, the execution units of a processor can be implemented with a simpler structure, since it is not necessary to provide each execution unit with the ability to perform complex transformations on the front end or output of the execution unit. Instead, the transformation from the operation space to the section space (and thus the management of data compatibility and correct structuring between successive operations) is managed and issued centrally by a single handling unit based on the dimensionality of a given operation space, e.g., by descriptors that define the operation space and the sections and pipes that form the graph.

[0016] Because a single transformation unit can perform the transformation from operation to section space, the processor can add support for additional operations in the future, allowing them to be chained before or anywhere within the chain, without requiring significant hardware modifications to the execution unit. This allows new functionality to be easily added. As an example, for convolution operations, dynamic weights can be easily added by adding a data reordering unit or transformation that can convert tensors in an activation layout to a weight layout, which can be handled by the convolution engine. Attributes of operations, such as padding around the edges of the input, can also be implemented through the transformation mechanism.

[0017] Furthermore, many less common operations can be decomposed into smaller units of execution (e.g., by simpler base operations from which more complex (or less common) operations can be constructed). Repetition of more common operations can enable support for larger operations that could not otherwise be accommodated within the constraints of a processor, rather than implementing native support within the execution unit. For example, for an operation, a convolution operation with a stride value > 1 can be implemented by dividing the kernel into single-element increments and repeatedly calling a convolution engine with a 1-element kernel, thus supporting larger strides. A similar example exists for operations requiring a dilation value > 1. 3D convolution operations can similarly be implemented as repeated 2D convolution operations.

[0018] In some examples, the processor is optionally configured such that two or more operations in the acyclic graph of operations are mapped to the same execution unit of the processor, and two or more connections in the acyclic graph of operations are each mapped to different portions of the same storage element.

[0019] In some examples, the processor is optionally configured such that each execution unit of a plurality of execution units of the processor is configured to perform a unique operation type, and the mapping between the operations in the acyclic graph and the execution units is defined based on execution compatibility between the operations in the acyclic graph and the unique operation type of the execution unit.

[0020] In some examples, the processor is optionally configured such that the task data includes an element count value indicating a count of the number of elements that map to each execution unit having a unique operation type, each element corresponding to an instance of use of an execution unit to execute a respective operation in the acyclic graph, and a pipe count value indicating a count of the number of pipes required to execute the task.

[0021] There is an element that describes each type of section and each type of pipe, and thus an element can be defined as a structured definition of a pipe or section. As described herein, a section has various parameters that describe the details of its execution.

[0022] In some examples, the processor is optionally configured such that the task data further includes element configuration data that defines, for each element in the acyclic graph, data used to configure a particular execution unit when performing the operation.

[0023] In some examples, the processor is optionally configured such that the element configuration data includes an offset value pointing to a location in memory of transformation data indicating, for each of a plurality of operations of the acyclic graph, a transformation to the portion of the operation space that is performed to generate a respective operation-specific local space.

[0024] In some examples, the processor is optionally configured such that the task data includes transformation program data defining a plurality of programs, each program including a sequence of instructions selected from a transformation instruction set. The processor is optionally configured such that the transformation program data is stored for each of a set of predetermined transformations, and such that, for each of a plurality of operations of the acyclic graph, a particular transformation is selected from the set of transformations to transform the portion of the operation space to generate a respective operation-specific local space.

[0025] In some examples, the processor is optionally configured such that the transformation program data is configured to perform a particular transformation on a plurality of values ​​stored in the bounds registers defining the arithmetic space to generate new values ​​in the bounds registers.

[0026] In some examples, the processor is optionally configured to iterate over the computation space in blocks, the blocks being created according to a predetermined block size.

[0027] In some examples, the processor is optionally configured such that dispatching of the call data is controlled based on a value identifying a dimension of the operation space, where a change in coordinate in the dimension while performing the task causes the operation to perform the operation, and a further value identifying a dimension of the operation space, where a change in coordinate in that dimension while performing the task causes the operation to store data in a storage device, where the stored data is ready to be consumed by the operation.

[0028] Directed Acyclic Graph (DAG) execution The examples described below refer to the execution of directed acyclic graphs, but it will be understood that the methods described may be utilized for the execution of any type of graph, not just directed acyclic graphs.

[0029] Many data structures executed in a processor can be represented as graphs, such as directed acyclic graphs. Examples of such data structures include neural networks, which can be represented as directed acyclic graphs of operations that completely constitute the operations required to execute the network (i.e., perform the operations performed across the layers of the neural network). A directed acyclic graph is a data structure of operations (also referred to herein as "sections") with directed connections between them that indicate the flow of operations such that the directed connections do not form closed loops. The connections between operations (or sections) that exist in a graph of operations are also referred to herein as "pipes." An acyclic graph can include any number of diverging and converging branches.

[0030] FIG. 1a illustrates an exemplary directed acyclic graph 100, whose sections are interconnected by a series of pipes. Specifically, an initial section, Section 1 (1110), represents a point in the acyclic graph where an operation, Operation A, is performed when executing the graph. The output of Operation A in Section 1 (1100) is connected to two further sections, Section 2 (1120) and Section 3 (1130), where Operations B and C, respectively, are performed. The connection between Section 1 (1110) and Section 2 (1120) can be identified as a pipe with a unique identifier, Pipe 1 (1210). The connection between Section 1 (1110) and Section 3 (1130) can be identified as a pipe with a different unique identifier, Pipe 2 (1220). The output of Section 1 is the result of performing Operation A on the input to Section 1 and can be provided to multiple subsequent sections in a branching manner.

[0031] More generally, a section in the acyclic graph may each receive multiple inputs from different sections in the acyclic graph via different pipes. For example, section 1150 in Figure 1a receives a first input data set from section 1120 via pipe 1240 and a second input data set via pipe 1250. Any number of input pipes and output pipes may be connected to a particular section in the acyclic graph, depending on the nature of the operations performed in the particular section and the dependencies of subsequent operations on the outputs of the operations.

[0032] An acyclic graph can be represented by several subgraphs, each containing a subset of the sections in the graph. Figure 1a shows an arrangement in which graph 110 is decomposed into three subgraphs 1310, 1320, and 1330, which can be connected together to form a complete graph. For example, subgraph 1310 contains sections 1110 and 1130 (and corresponding pipes 1220 and 1260), subgraph 1320 contains sections 1120, 1140, and 1150 (and corresponding pipes 1210, 1230, 1240, and 1250), and subgraph 1330 contains sections 1160 and 1170 (and corresponding pipes 1270, 1280, and 1290).

[0033] Decomposing graph 100 into subgraphs is particularly useful when attempting to execute the graph because the subgraphs can be executed separately, thereby enabling parallelization of execution when there are no dependencies between the subgraphs. This can be particularly useful in a multiprocessor environment where subgraphs can be allocated for execution by different processors in the multiprocessor environment. However, as shown in FIG. 1a, subgraph 1320 has a dependency on the execution of operation A and section 1110, and subgraph 1330 has a dependency on subgraph 1310. Therefore, execution of subgraph 1330 may need to be stalled until subgraph 1310 is completed. Therefore, it will be appreciated that careful selection of an appropriate subgraph arrangement is required to maximize or improve the execution efficiency of a graph.

[0034] The operations performed when executing a neural network can be decomposed into a sequence of operations that form an acyclic graph in the manner described with respect to Figure 1a. The detailed description herein describes an arrangement for executing the acyclic graph of operations in an improved manner.

[0035] computational space For example, when performing a chain of operations structured in a directed acyclic graph, each section can represent a different operation. Each operation need not be of the same type or nature. This is particularly true when a graph of operations is used to represent the processing of a neural network. The machine learning software ecosystem allows for diverse structures of neural networks applicable to many different problem spaces, and therefore there is a very large set of possible operators that can comprise a neural network. The inventors have recognized that the set of possible operations that can form a section can be difficult to manage when attempting to design hardware to enable the execution (also referred to as "acceleration") of these operations, especially when chained together. For example, allowing for fixed-function operations of each possible type of operation can result in inefficient hardware by requiring support for obscure or complex operations (sections).

[0036] As a result, significant challenges exist in designing and building hardware capable of running all types of neural networks created by current machine learning toolsets. Consequently, the inventors have recognized the desirability of defining a set of predetermined low-level operations from which a wide range of possible higher-level operations corresponding to various machine learning toolsets can be constructed. One example of such a set of low-level operations is the Tensor Operator Set Architecture (TOSA). The Tensor Operator Set Architecture (TOSA) provides a complete set of tensor operations commonly used by deep neural networks. The intent is to enable various implementations running on a diverse range of processors with consistent TOSA-level results across those implementations. Thus, applications or frameworks targeting TOSA can be deployed on a wide range of different processors with defined precision and compatibility constraints, including single-instruction multiple-data (SIMD) CPUs, graphics processing units (GPUs), and custom hardware, such as neural processing units (NPUs) / tensor processing units (TPUs). Most operators from popular ML frameworks (TensorFlow, PyTorch, etc.) should be expressible in TOSA.

[0037] However, even if such an operator set exists, the inventors recognize the need to implement the operator set in a manner that can be executed efficiently, both in terms of complexity and while minimizing the need to perform external memory transactions. To enable this, the inventors recognize that it is useful to consider that many of the operations in the defined operation set (such as TOSA) can be expressed as loops of scalar operations.

[0038] For example, consider a 2D convolution operation, which can be expressed as a multidimensional loop of scalar operations. These may need to be performed on input 2D input data having dimensional input X (input X, IX) and dimensional input Y (input Y, IY): (Input) Input channel (IC) - a dimension representing the input channel on which the operation is performed (in the image example, this could be three channels, each representing one of the red, green, and blue input channels) - (Input) Kernel Dimension X (KX) - the first dimension X of the 2D kernel, - (Input) Kernel dimension Y(KY) - the second dimension Y of the 2D kernel, - (Output) Output X (Output X, OX) - the first dimension of the output feature map for the convolution operation; - (Output) Output Y (Output Y, OY) - the second dimension of the output feature map for the convolution operation, - (Output) Batch(N) - The batch dimension of operations over which operations are batched, - (Output) Output channel (OC) - A dimension representing the output channel generated for a 2D convolution operation.

[0039] In one proposed ordering, KY / KX can be considered the innermost dimension and OC the outermost dimension.

[0040] In the example 2D convolution operation above, the operation being performed can be represented as a "nested for-loop" of scalar operations, as shown in the pseudocode below. In practice, when performing this operation, the processor needs to perform the operation across each of these dimensions by performing a multiply-accumulate operation (MAC), and the results are then written to an accumulator (e.g., an accumulator buffer in hardware). Having iterated through all of these dimensions, the 2D convolution is completed; therefore, the contents of the accumulator represent the result of the 2D convolution operation across all dimensions of the operation. for(output channel) for(batch N) for(output Y) for(output X) for(input channel) for(kernel Y) for(kernel X) MAC write accumulator

[0041] The inventors have recognized that the seven dimensions of a convolution operation can be used collectively to define the "operation space" in which the 2D convolution operation is performed. More specifically, the size of each dimension can be used to define an effective "bounding box" that defines the size of the operation space in which the operation is performed, the number of elements in each dimension. To explain this in more detail, consider an example in which a 3x3 (i.e., KX=3, KY=3) convolution operation with padding is performed on input data having dimensions IX=15, IY=15, N=1, and IC=32. This operation results in the following minimum and maximum index values, as shown in Table 1, which represent the upper and lower bounds (i.e., size) of the dimensionality of the convolution operation:

[0042] [Table 1]

[0043] The output of a 2D convolution operation may have dimensions N=1, OY=15, OX=15, and OC=64. While these values ​​represent the size of the output of the 2D convolution operation, they alone do not fully represent the size of the operations required to produce that output. To fully represent the operation space of an operation, all of the dimensions of the operation are required, as shown in the table above. A shorthand representation of the dimensions of a 2D convolution operation is [OC N OY OX IC KY KX], which in this specific example can be presented as minimum and maximum index values, as shown in the example above, i.e., [64 1 15 15 32 3 3].

[0044] An operation, such as the convolution operation described above, can be divided into blocks, with each block representing a subset of the operation, with each dimension of the block covering a subset of the full range of the corresponding dimension in the operation. In the following example, the 2D convolution in Table 1 is divided into multiple blocks by decomposing the operation in the OY, OX, and IC dimensions. Dividing an operation into blocks involves dividing the operation space of the operation into multiple blocks, each of which individually represents a portion of the operation but collectively represents the operation space. This block generation involves dividing the operation space into sub-blocks that represent non-overlapping subsets of dimensions in the operation space that completely cover the operation space dimension (e.g., the set of nested for loops shown above). In examples where an operation is divided into several blocks, the operation space is decomposed into sub-blocks based on a predetermined block size that defines a fixed size for each dimension of the operation. This fixed-size block is referred to herein as a block quantum. In the following example, the block sizes are as follows:

[0045] [Table 2]

[0046] For the block sizes above, the operation space is divided by dividing four of the seven dimensions of the operation by two. In the example below, OY, OX, and IC are divided by two, and OC is divided by four. The block below shows a portion of a block that completely represents the operation space (only the first quarter of the OC dimension is represented):

[0047] [Table 3]

[0048] For a given block of operation space, e.g., [OC N OY OX IC KY KX], it is possible to determine which input feature map coordinates are needed to perform the operation on that block. In the example of a 2D convolution operation, the input feature map coordinates (and other input parameters) on which the output feature map coordinates depend can be defined as follows (stride X,Y=1 (i.e., no stride), dilation X,Y=1 (i.e., no dilation), and top,left pad=1 (i.e., the input is padded): - N = N, where N is the number of batches, -IY=(OY * (Stride Y))+((Expansion Y) * KY)-Upper pad, -IX=(OX * (Stride X))+((Expansion X) * KX)-Left pad, and -IC=IC

[0049] where strideX and strideY, dilationX and dilationY represent the stride and dilation values ​​in the X and Y dimensions, respectively, when performing the convolution operation, and toppad and leftpad represent the top and left padding values, respectively, when performing the operation. If the above relationship is simplified for a stride and dilation value of 1 with zero padding, this can be more simply expressed as [N, OY+K Y-1, O X+K X-1, IC]. These equations for calculating the input feature map for processing a block can be expressed as an affine transformation, as shown in Table 4 below.

[0050] [Table 4]

[0051] Thus, for a given block in the operation space, it is possible to represent a transformation (either affine or semi-affine) to transform the block to determine the input feature map coordinate range needed to perform the operation as defined by the block. In the example of the affine transformation above being applied to block #2, the resulting input range of input feature map indices can be shown as in Table 5 below.

[0052] [Table 5]

[0053] The affine transformation defined above can be used to separately represent the transformations required to define each of the input feature map, output feature map, and weights (as shown above). Typical examples of each of the input feature map, output feature map, and weight transformations are shown in Tables 6-8 below.

[0054] Input transformation for 2D convolution

[0055] [Table 6]

[0056] Weight Transformation for 2D Convolution

[0057] [Table 7]

[0058] Output transformation for 2D convolution

[0059] [Table 8]

[0060] It will be appreciated that the operation space therefore defines the dimensionality of the operations performed when performing a particular operation. While the above example is provided with respect to 2D convolution, the concepts are applicable to all types of operations performed. For example, a similar transformation for the input and output of a transpose operation (e.g., transposing dimensions {0, 1, 3, 2}) can be derived as shown below:

[0061] Input transformation for {0,1,3,2} transposition

[0062] [Table 9]

[0063] Output transformation for {0,1,3,2} transposition

[0064] [Table 10]

[0065] By utilizing an input transformation on the input, swapping dimensions 2 and 3 in the input transformation matrix can perform a transpose operation. More generally, the input and output matrices may then be applied to a block in the operation space to determine the ranges of input and output values ​​for the operation. These determined value ranges represent a local section space for the operation, which forms the local coordinate system in which the operation can be performed on that block of the operation space.

[0066] Lower and upper bound clipping of the task and operation space may be performed before performing the transformation. Clipping may be functionally necessary for the edges of the tensor, allowing for a smaller operation space than the full tensor. A smaller operation space than the full tensor is advantageous because it allows a larger sequence of operations to be split across multiple independent tasks, optionally performed on separate cores.

[0067] In such a clipping model, code may be used to initialize the upper / lower bounds before performing the transformation, where low=op_space, high=op_space, and the initial coordinates default to op_space+block_size-1. Coordinates are clipped to the actual operation space and task bounds before the transformation is performed.

[0068] Given the acyclic graph data structure described above with respect to Figure 1a, the operations performed on each section of the graph can be defined by a set of input and output transformation matrices for that operation. Thus, it is possible to represent at least a portion of an acyclic graph by a chain of operations corresponding to a chain of sections each connected by a pipe. Furthermore, an operation space for the chain of operations can be established.

[0069] Hardware Implementation As noted above, a data structure in the form of a directed acyclic graph may include multiple sequenced operations connected together for execution in a chain. Other data structures in the form of different graphs may also be represented. Described below is an exemplary hardware arrangement for performing chained operations on at least a portion of a directed acyclic graph such as that shown in FIG. 1a, although it will be understood that the exemplary hardware configuration may be used to perform chained operations on any type of graph.

[0070] 1b schematically illustrates an example of a data processing system 600 including a processor 630 that can function as a co-processor or hardware accelerator unit for a host processing unit 610. It will be understood that the type of hardware accelerator for which the processor 630 can provide dedicated circuitry is not limited to neural processing units (NPUs) or graphics processing units (GPUs), but can be dedicated circuitry for any type of hardware accelerator. GPUs may be well suited to performing certain types of arithmetic operations, such as neural processing operations, because these operations are generally similar to arithmetic operations that may be required when performing graphics processing tasks (but on different data formats or structures). Furthermore, GPUs typically support high levels of parallelism (e.g., support multiple execution threads) and are optimized for data plane (rather than control plane) processing, all of which means that GPUs may be well suited to performing other types of operations.

[0071] That is, rather than using an entirely separate hardware accelerator, such as a machine learning processing unit separate from a graphics processor such as an NPU, or even using the GPU's hardware entirely to perform machine learning processing operations, dedicated circuitry may be built into the GPU itself.

[0072] This means that the hardware accelerator circuitry incorporated into the GPU can be operated to utilize some of the existing resources of the GPU (e.g., so that at least the functional units and resources of the GPU can be effectively shared, e.g., between different hardware accelerator circuitry), while still allowing improved (more optimized) performance compared to performing all processing in a general-purpose implementation.

[0073] Thus, the processor 630 may be a GPU adapted with several dedicated hardware resources, such as those described below.

[0074] In some examples, this may be particularly beneficial when performing machine learning tasks that are themselves related to graphics processing work, in which case all of the associated processing can (and preferably is) performed local to the graphics processor, thus improving data locality and reducing the need for external communication along the interconnect with (for example) other hardware units (e.g., NPUs). At least a portion of the machine learning processing work can then be offloaded to the machine learning processing circuitry, thereby freeing up the execution units to perform the actual graphics processing operations desired.

[0075] In other words, in some examples, if the machine learning processing circuitry is provided within a graphics processor, this means that the machine learning processing circuitry is preferably operable to perform at least some machine learning processing operations while other functional units of the graphics processor are simultaneously performing graphics processing operations. In situations where the machine learning processing relates to a portion of the overall graphics processing task, this can therefore improve the overall efficiency of the overall graphics processing task (in terms of energy efficiency, throughput, etc.).

[0076] 1b, processor 630 is configured to receive a command stream 620 from a host processor 610, such as a central processing unit (CPU). Command stream 620 includes at least one command in a given sequence, each command being executed, and each command may be broken down into several tasks, such as those discussed in this document. These tasks may be self-contained operations, such as a given machine learning operation or a graphics processing operation. It will be understood that other types of tasks may be present depending on the command.

[0077] Command stream 620 is sent by host processor 610 and received by command processing unit 640. Command processing unit 640 is configured to schedule the commands in command stream 620 according to their sequence. Command processing unit 640 is configured to schedule the commands and to decompose each command in command stream 620 into at least one task. Once command processing unit 640 has scheduled the commands in command stream 620 and generated multiple tasks for the commands, command processing unit 640 issues each of the multiple tasks to at least one computing unit 650 a, 650 b, each of which is configured to process at least one of the multiple tasks.

[0078] The processor 630 includes multiple computational units 650a, 650b. Each computational unit 650a, 650b may be a shader core of a GPU specifically configured to perform several different types of operations, although it will be understood that other types of specifically configured processors may be used, such as general-purpose processors configured with individual computational units, such as computational units 650a, 650b. Each computational unit 650a, 650b includes several components, including at least a first processing module 652a, 652b for performing tasks of a first task type and a second processing module 654a, 654b for performing tasks of a second task type different from the first task type. In some examples, the first processing module 652a, 652b may be a processing module for processing neural processing operations, such as those typically performed by a separate NPU. In these cases, the first processing module 652a, 652b may be, for example, a neural engine. Similarly, the second processing modules 654a, 654b may be processing modules for processing graphics processing operations that form a set of predetermined graphics processing operations that enable the implementation of a graphics processing pipeline that may be referred to as a graphics processor. For example, such graphics processing operations include graphics compute shader tasks, vertex shader tasks, fragment shader tasks, tessellation shader tasks, and geometry shader tasks. All of these graphics processing operations may form part of a set of predetermined operations as defined by an application programming interface (API). Examples of such APIs include Vulkan, Direct3D, and Metal. Such tasks are typically performed by a separate / external GPU. It will be appreciated that any number of other graphics processing operations may be capable of being processed by the second processing module.

[0079] Thus, the command processing unit 640 issues tasks of a first task type to the first processing module 652a, 652b of a given computing unit 650a, 650b and tasks of a second task type to the second processing module 654a, 654b of the given computing unit 650a, 650b. The command processing unit 640 issues machine learning / neural processing tasks to the first processing module 652a, 652b of a given computing unit 650a, 650b, where the first processing module 652a, 652b is optimized to handle neural network processing tasks, for example, by having efficient means for handling a large number of multiply-accumulate operations. Similarly, the command processing unit 640 issues graphics processing tasks to the second processing module 654a, 654b of a given computing unit 650a, 650b, where the second processing module 652a, 654a is optimized to handle such graphics processing tasks. In some examples, both the first and second neural processing tasks may be issued to a first processing module 652 a, 652 b that is a neural engine. Such neural processing tasks may involve processing tensors, for example, representing feature maps with weights associated with layers of a neural network.

[0080] In addition to comprising a first processing module 652a, 652b and a second processing module 654a, 654b, each computing unit 650a, 650b also comprises memory in the form of a local cache 656a, 656b used by the respective processing module 652a, 652b, 654a, 654b during task processing. An example of such a local cache 656a, 656b is an L1 cache. The local cache 656a, 656b may comprise, for example, synchronous dynamic random-access memory (SDRAM). For example, the local cache 656a, 656b may include double data rate synchronous dynamic random-access memory (DDR-SDRAM). It will be understood that the local cache 656a, 656b may comprise other types of memory.

[0081] The local cache 656a, 656b is used to store data related to a task being processed on a given computing unit 650a, 650b by the first processing module 652a, 652b and the second processing module 654a, 654b. It may also be accessed by other processing modules (not shown) that form part of the computing unit 650a, 650b with which the local cache 656a, 656b is associated. However, in some examples, it may be necessary to provide access data associated with a given task executing on a processing module of a given computing unit 650a, 650b to a task executing on a processing module of another computing unit (not shown) of the processor 630. In such examples, the processor 630 may also include a memory device 660, e.g., a cache such as an L2 cache, for providing access to usage data for processing of tasks executing on different computing units 650a, 650b.

[0082] By providing local caches 656a, 656b, tasks issued to the same compute unit 650a, 650b may access data stored in the local cache 656a, 656b, regardless of whether they form part of the same command in command stream 620. The command processing unit 640 is responsible for allocating tasks of a command to a given compute unit 650a, 650b so that the compute unit can most efficiently use available resources, such as the local cache 656a, 656b, thus reducing the number of required read / write transactions to memory external to the compute unit 650a, 650b, such as storage device 660 (L2 cache) or higher level memory. One such example is that a task of one command issued to a first processing module 652a of a given computing unit 650a may store its output in a local cache 656a so that it is accessible by a second task of a different (or the same) command issued to a given processing module 652a, 654a of the same computing unit 650a.

[0083] One or more of the command processing unit 640, the computing units 650a, 650b, and the storage device 660 may be interconnected using a bus, which allows data to be transferred between the various components. The bus may be or include any suitable interface or bus. For example, an ARM® Advanced Microcontroller Bus Architecture (AMBA®) interface, such as the Advanced eXtensible Interface (AXI), may be used.

[0084] FIG. 2 is a schematic diagram of a neural engine 700, used in this example as a first processing module 652a, 652b in a data processing system 600 according to FIG. 1b. The neural engine 700 includes a command and control module 710. The command and control module 710 receives tasks from the command processing unit 640 (shown in FIG. 1b) and also serves as an interface to storage external to the neural engine 700 (such as local caches 656a, 656b and / or L2 cache 660) configured to store data to be processed by the neural engine 700, such as data representing tensors or stripes of tensors. In the context of this disclosure, a stripe is a subset of a tensor, where each dimension of the stripe covers a subset of the full range of the corresponding dimension in the tensor. The external storage may also store other data for configuring the neural engine 700 to perform a particular operation and / or data used by the neural engine 700 to implement the operation, such as neural network weights.

[0085] The command and control module 710 interfaces to a handling unit 720, e.g., a traversal synchronization unit (TSU). In this example, each task corresponds to a stripe of tensors that are operated on according to a sequence of operations through at least a portion (e.g., a subgraph) of the acyclic graph representation of the neural network. The tensors represent, e.g., feature maps for processing using the neural network. A neural network typically includes a sequence of layers of processing, with the output from each layer being used as input to the next layer. Each layer processes an input feature map, e.g., by performing an operation on the input feature map to generate an output feature map that is used as the input feature map for the next layer. The term "feature map" is used herein generically to refer to either an input feature map or an output feature map. The processing performed by a given layer may be considered to correspond to an operation.

[0086] In this example, handling unit 720 divides data representing stripes of a feature map into multiple blocks of data, each representing a respective portion of the feature map. Handling unit 720 also retrieves task data from storage external to neural engine 700, such as L2 cache 660, that defines an operation selected from an operation set containing multiple operations. In this example, the operations are structured as a chain of operations representing a sequence of layers of a neural network. The blocks of data are assigned as input by handling unit 720 to one of the operations.

[0087] The handling unit 720 coordinates the interaction of the neural engine 700's internal components, including a weight fetch unit 722, an input reader 724, an output writer 726, a direct memory access (DMA) unit 728, a dot product unit (DPU) array 730, a vector engine 732, a transform unit 734, an accumulator buffer 736, and a storage device 738, to process blocks of data. Data dependencies across functional units are tracked by the handling unit 720. When all input blocks are available and space is available within the neural engine 700's storage device 738, processing is initiated by the handling unit 720 within a functional unit. The storage device 738 may be considered a shared buffer in that the various functional units of the neural engine 700 share access to the storage device 738.

[0088] In the context of a directed acyclic graph representing the operations to be performed, each of the internal components that operate on data can be considered to be one of two types of components. The first type of component is an execution unit (and is identified as such within neural engine 700) that maps to a section that performs a unique instance of an operation in the acyclic graph. For example, weight fetch unit 722, input reader 724, output writer 726, dot product unit array 730, vector engine 732, and transform unit 734 are each configured to perform one or more predetermined fixed operations on the data they receive. Each of these sections can be uniquely identified with an identifier, and each execution unit can also be uniquely identified.

[0089] Similarly, all physical storage elements within the neural engine (and in some instances, portions of those physical storage elements) can be considered to be uniquely identified within the neural engine. The connections between sections within the acyclic graph representing a neural network are also referred to as pipes within the context of the acyclic graph. These pipes can be mapped to uniquely identified physical storage elements within the neural engine. For example, accumulator buffer 736 and storage device 738 (and portions thereof) can each be considered storage elements that can function to store data for pipes within the acyclic graph. Pipes serve as connections between sections (as executed by execution units) to allow sequences of operations defined in the acyclic graph to be chained together within neural engine 700. In other words, the logical data flow of the acyclic graph can be mapped to a physical arrangement of execution units and storage elements within neural engine 700. Under the control of the handling unit 720, execution can be scheduled on the execution units and data can be passed between execution units via storage elements according to the mapping, so that chained operations of the graph can be performed without having to write data to memory external to the neural engine 700 between executions. The handling unit 720 is configured to control and dispatch work representing performing the operations of the graph on at least a portion of the data provided by the pipes.

[0090] A weight fetch unit 722 fetches weights associated with a neural network from external storage and stores the weights in storage 738. An input reader 724 reads data from external storage to be processed by neural engine 700, such as blocks of data representing portions of tensors. An output writer 726 writes data obtained after processing by neural engine 700 to external storage. The weight fetch unit 722, input reader 724, and output writer 726 interface with external storage (e.g., local caches 656a, 656b, which may be L1 caches such as load / store caches) via a DMA unit 728.

[0091] Data is processed by the DPU array 730, the vector engine 732, and the transform unit 734 to generate output data corresponding to the operations in the acyclic graph. The results of each operation are stored in a unique pipe within the neural engine 700. The DPU array 730 is configured to perform one or more operations associated with a dot product operation between two operands, such as between an array of weights and a corresponding block of data (e.g., representing a portion of a tensor). The vector engine 732 is configured to scale the output of the dot product calculated by the DPU array 730, e.g., by applying a scale parameter, to perform the element-wise operation. Data generated during the course of processing performed by the DPU array 730 and the vector engine 732 may be transmitted for temporary staging in an accumulator buffer 736, which serves as a conduit between previous and subsequent operations, from which it may be retrieved by either the DPU array 730 or the vector engine 732 (or another different execution unit) for further processing as desired.

[0092] Transform unit 734 is configured to perform intra-block transforms such as dimension broadcast or axis swap. Transform unit 734 obtains data from a pipe such as storage 738 (e.g., after processing by DPU array 730 and / or vector engine 732) and writes the transformed data back to storage 738.

[0093] To efficiently use the available storage 738 within neural engine 700, handling unit 720 determines an available portion of storage 738 that is available during execution of a portion of a first task (e.g., during processing of blocks of data associated with the first task by DPU array 730, vector engine 732, and / or transform unit 734). Handling unit 720 determines a mapping between at least one logical address associated with data generated during execution of a second task (e.g., by processing blocks of data associated with the second task by DPU array 730, vector engine 732, and / or transform unit 734) and at least one physical address in storage 738 that corresponds to the available portion. The logical address is, for example, a global address in a global coordinate system. Thus, by changing the physical address corresponding to a given logical address, handling unit 720 can effectively control the use of storage 738 without requiring changes in the software defining the operations being performed, because the same logical address can still be used to reference a given element of a tensor being processed. The handling unit 720 identifies at least one physical address corresponding to the at least one logical address based on the mapping, so that data associated with the logical address is stored within the available portion. The handling unit 720 can perform the mapping process according to any of the examples described herein.

[0094] It will be appreciated that there need not be only a single instance of a particular type of operation in a graph of operations. For example, multiple instances of a convolution operation may exist in a graph of operations. In the exemplary hardware arrangement above, there may be only a single convolution engine. It will be appreciated, therefore, that there need not be a direct 1:1 mapping between operations (sections) in the graph and execution units, and similarly, there need not be a direct 1:1 mapping between pipes and storage elements. In particular, a single execution unit may be configured at different instances to perform different instances (e.g., first and second sections) of a convolution operation. Similarly, an input reader may be required to read data as part of different sections in the graph. The same is true for storage elements and pipes.

[0095] All storage devices within the Neural Engine 700 may be mapped to corresponding pipes, including lookup tables, accumulators, etc. Some storage devices may be relatively fixed in purpose; for example, if the hardware is limited to one convolution operation per graph, the accumulator buffer may also be limited to being mapped to one pipe, and the scale / bias / shift buffers may also be limited to being mapped to one pipe. However, both are likely to be double-buffered. If the Neural Engine supports two lookup tables (LUTs), up to two pipes can be used to target the LUT to avoid having to thrash the LUT storage. The LUT pipe may then be single-buffered. All other pipes may be mapped to a common shared buffer (or a portion thereof) with fewer restrictions. The width and height of the pipes are also programmable, resulting in a highly configurable mapping between pipes and storage elements within the Neural Engine 700.

[0096] The ordering of execution of sections is implied by dependencies on inputs. Memory load operations have no data dependencies (unless they are collect operations) and are therefore implicitly early in the graph. Consumers of the pipe that a memory read generates are implicitly after the memory read. Memory store operations are near the end of the graph because they do not generate pipes that other operations consume. The sequence of execution of a chain of operations is therefore handled by the handling unit 720, as described in more detail below.

[0097] FIG. 3 illustrates generally a system 800 for allocating transaction data and, in some examples, for generating multiple blocks of input data for processing.

[0098] System 800 includes a host processor 810, such as a central processing unit or any other type of general processing device, that issues a command stream containing multiple commands, each command having multiple tasks associated with it.

[0099] System 800 may also include a processor 830, which may be similar to or the same as processor 630 of FIG. 1b and may include at least some of the components for implementing the above-described methods and / or may be configured to implement the above-described methods. Processor 830 includes at least a plurality of computing units 650a, 650b and a command processing unit 640. Each computing unit may include a plurality of processing modules, each configured to perform at least one type of operation. System 800 may also include at least one additional processor (not shown), which may be the same as processor 830. Processor 830 and host processor 810 may be combined as a system on a chip (SoC) or in multiple SoCs to form one or more application processors.

[0100] System 800 also includes memory 820 for storing data generated by tasks external to processor 830, so that the data may be easily accessed by other tasks running on other processors. However, it will be appreciated that due to the allocation of tasks as described above, the use of external memory is used sparingly, and tasks that require use of data generated by other tasks, or that require the same data as other tasks, are allocated to the same computational units 650a, 650b of processor 830 in a manner that maximizes the use of local caches 656a, 656b.

[0101] In some examples, system 800 may include a memory controller (not shown), which may be a dynamic memory controller (DMC). The memory controller is coupled to memory 820. The memory controller is configured to manage the flow of data into and out of the memory. The memory may comprise main memory, also referred to as “primary memory.” The memory may be external memory in that the memory is external to system 800. For example, memory 820 may include “off-chip” memory. The memory may have a larger storage capacity than the local cache of processor 830 and / or host processor 810. In some examples, memory 820 is included in system 800. For example, memory 820 may include “on-chip” memory. Memory 820 may include, for example, a magnetic or optical disk and disk drive or a solid-state drive (SSD). In some examples, memory 820 includes synchronous dynamic random access memory (SDRAM). For example, memory 820 may include double data rate synchronous dynamic random access memory (DDR-SDRAM).

[0102] One or more of the host processor 810, the processor 830, and the memory 820 may be interconnected using a system bus 840, which allows data to be transferred between the various components. The system bus 840 may be or include any suitable interface or bus. For example, an ARM® Advanced Microcontroller Bus Architecture (AMBA®) interface, such as the Advanced eXtensible Interface (AXI), may be used.

[0103] Neural engine program descriptor (NED)

[0104] Neural engine 700 receives tasks from command processing unit 640 to execute operations from a graph, such as the directed acyclic graph described above with reference to FIG. 1a. Neural engine 700 is configured to execute operations selected from a base set of operations that define an operator set. One example of such an operator set is the Tensor Operator Set Architecture (TOSA)-based inference profile, which defines a set of operations that can be collectively used to define operations for a wide range of neural network operations. One exception to the TOSA operator set is a control flow operation that can be implemented by a command stream processed by command processing unit 640. It will be understood that there may be multiple neural engines with processors 630, and thus multiple tasks may be issued to different neural engines simultaneously.

[0105] In an exemplary implementation, tasks issued by the command processing unit 640 for execution by the neural engine 700 are described by task data, which in this example is embodied by a neural engine program descriptor (NED), a data structure stored in memory and retrieved by the neural engine when executing a task issuance by the command processing unit. The NED describes at least a portion of a complete graph of operations (sections) to be performed when executing a graph of operations (e.g., representing a neural network). As described above, sections are mapped to various hardware execution units within the neural engine 700 and essentially represent instantiations of specific operators at locations within the graph. In one example, these sections are described by unique "elements" that collectively define the operations that form part of the NED. Furthermore, the NED has an unordered list of pipes (graph vertices) and an unordered list of sections / operations (graph nodes). Each operation specifies its input and output pipes, which create the adjacency of the operation in the acyclic graph to which the particular operation is connected.

[0106] An exemplary NED includes a NED structure that includes a header, with elements each corresponding to a section in the graph. The NED describes various requirements for the ordering, number, and relationships of these sections and pipes. In one implementation, each of the execution units and each storage element (or portion of a storage element) of neural engine 700 has a sub-descriptor definition that defines how that execution unit / storage element can be configured for use in implementing a specific section or pipe in the graph. Examples of hardware units and their corresponding elements are provided below. -Weight Fetch (WF): NEDWeightFetchElement -Input Reader (IR): NEDInputReaderElement -Output Writer (OW): NEDOutputWriterElement -Convolution Engine (CE): NEDConvolutionEngineElement -Transform Unit (TU): NEDTransformUnitElement -Vector Engine (VE): NEDVectorEngineElement

[0107] Thus, the NED may specify an execution unit, or in other words, a compatible execution unit for each operation. In embodiments, there may be two or more execution units of a given type, such as an InputReader, and they may have two command queues that can operate simultaneously. The NED may specify which of the queues are assigned so that there remains a 1:1 relationship between what the NED specifies and the physical hardware it points to.

[0108] The data flow and dependencies of a task graph are described by pipes, which are described in a separate element, the NEDpipeElement, as part of the NED. Pipes represent data storage elements within the neural engine 700 and are used to describe the relationships between sections (operations) in a producer-consumer relationship, with the output destination pipe (e.g., pipe number) for each section and each input source pipe (e.g., pipe number) defined in the NED element of the NED. A pipe has only a single producer but may have multiple consumers. A pipe may be mapped to one of several different locations (e.g., storage elements within the neural engine 700), although not all locations are suitable for different section operations. It will be appreciated that in some arrangements, a pipe may map to only a portion of a storage element, e.g., several physical buffers, thereby allowing the pipe to describe (for example) double-buffering behavior between its producer and consumer. The output data produced by a section and stored in a pipe is equivalently referred to as a block (of data) or a (virtual) buffer, with a block of data occupying one physical buffer location. Regardless of location, the pipe may be non-coherent with the broader memory system associated with the neural engine 700 and processor 630, with data being stored using the output writer element of the neural engine 700.

[0109] In some arrangements, the NED may be configured such that the same pipe is used for multiple inputs, and any relevant usage constraints (such as format or position) are met. For example, an element-wise multiplication may have the same pipe for two input operands to square the input.

[0110] In some embodiments, sections such as InputReader and WeightFetcher do not have pipes; instead, their data comes from external memory such as an external cache or DRAM. In contrast, some sections, such as OutputWriter, do not have output pipes; in this case, their data is written to external memory.

[0111] For a section to execute, it must have all appropriate buffers available for its input source pipe. A section may generate a new buffer in its output destination pipe, and therefore, space must be available in the pipe for this new buffer. In the case of a reduction operation (e.g., a convolution), a section may repeatedly read back and update previous buffers it generated. As a result, for a reduction operation, due to this update process, there is a distinction between when the reduction operation initially generated an output buffer and when the reduction is complete and the output buffer is fully available. In other words, there is a point in time when an output buffer exists in the input pipe of a subsequent operation but is not yet ready to be consumed by that subsequent operation. The neural engine 700 is responsible for tracking all of these dependencies; buffers are tracked like FIFO entries, but only have buffers available to consumers when a producer has completed any sequence of any given reduction, and only have buffers that are released when all consumers have completed the operations that depend on them.

[0112] In one example, the graph of tasks has a directed acyclic data flow. As such, in this example, it is not legal to use an input pipe as a destination pipe in the same section, or to have any form of loop in the graph. Note that while reduction operations both read from and write to the buffers of their output destination pipes, this is still acyclic behavior; for example, a convolution engine may repeatedly accumulate into the same accumulator buffer.

[0113] In this exemplary implementation, the neural engine is stateless between tasks; all control state is encapsulated in the task's NED, and all data is encapsulated in pipes defined by the NED. There are no shared pipes between tasks, and therefore no architectural sharing of data between tasks within the neural engine 700. Data reuse and sharing is achieved solely through memory, using output writers in earlier tasks and input readers in later tasks. The neural engine caches memory descriptors, including NEDs, between tasks. This cache is invalidated whenever a complete neural workload (e.g., the entire neural network, not just the subgraph associated with a unique task) is completed. However, it will be understood that this is merely an exemplary implementation.

[0114] The NED is divided into multiple data structures that may appear contiguously in memory as read by neural engine 700. In this exemplary implementation, the NED header defines the dimensions of the operation space of the operations to be performed. Specifically, the NED header defines the total size of the NED (e.g., the number of bytes used to represent the NED) and a count of the number of sections and pipes present in the graph.

[0115] For each section and pipe in the graph, a count of the corresponding mapped subdescriptor element type is represented in the NED header. For example, if a graph (or subgraph) contains several sections, each of those sections will execute on a specific compatible execution unit in the neural engine 700. Therefore, for each section, elements of the appropriate type are counted in the NED header to represent the hardware requirements needed to invoke the execution of the graph. For example, for a section defining a convolution operation, a corresponding configuration and invocation of the convolution engine execution unit is required. Similar counts of instantiations of weight fetch and input read execution units are counted based on the presence of sections that use those operations. This is reflected in the counts in the NED header for the weight fetch unit and input reader elements associated with the weight fetch and input reader units in the neural engine 700.

[0116] The NED also contains information describing any divergent or convergent branches between sections and pipes. For example, for each pipe in the graph, the NED identifies the number of producers and consumers associated with that pipe.

[0117] Thus, the NED header essentially identifies the operation space and counts of all instances of sections and pipes defined by the NED (for each type of hardware element allocated to instantiate the section or pipe needed to execute the graph (or subgraph)). An illustrative example of at least a portion of the fields stored in the NED header is shown below: In addition to the NED header, the NED further includes sub-descriptor elements for each instance of a section and / or pipe (which define the configuration of execution units or storage elements for operation as a section or pipe). Each sub-descriptor element defines the configuration of the associated hardware elements (execution units or storage elements) needed to execute the section and / or pipe.

[0118] Below are examples of at least some of the fields in the NED header:

[0119] [Table 11]

[0120] The theoretical minimum and maximum operation space dimension sizes may be defined at compile time based on the configuration of the neural engine, specifically so that the operations of a task (e.g., a subgraph) can be performed without the need to store intermediate data in memory elements external to the neural engine. A practical approach for defining a task and its corresponding operation space is presented in more detail below.

[0121] The NED header may also comprise pointers to each of the sub-descriptor elements to allow the unique configuration of each element to be read by the handling unit 720.

[0122] As noted above, each instance of a sub-descriptor element defines the configuration of the hardware element (e.g., execution unit or memory element) with which it is associated. The following description provides an example sub-descriptor for a convolution engine.

[0123] In one example, a convolution engine is an execution unit configured, when invoked, to perform a convolution or pooling operation selected from one or more convolution operations for which the convolution engine is configured. One such example is a 2D convolution operation, such as that described above. In the example 2D convolution operation described above, the operation space is 7D, i.e., [oc,n,oy,ox,ic,ky,kx].

[0124] [Table 12]

[0125] In this example, the operation type may take the form of, for example, one of pooling (average or max pooling), 2D convolution, or 2D depthwise convolution. The source 0 pipe field may identify which pipe the convolution engine should read input feature map data from, which may be, for example, a specific portion of the shared buffer. Similarly, the source 1 pipe field may indicate which (different) portion of the shared buffer the weight data will be taken from. Finally, the destination pipe may indicate which accumulation buffer serves as a pipe for the output of the operation performed by the convolution engine. By identifying a unique source pipe and / or destination pipe with a unique identifier for a section within the task definition (NED), any preceding or subsequent sections are implicitly connected and sequenced. Another subdescriptor element referencing a destination pipe of a different section as a source pipe essentially reads that data, and the buffer allocation for that destination pipe may only be released when all dependencies are resolved (e.g., all of the sections dependent on that portion of the accumulation buffer have completed reading their data).

[0126] Similar sub-descriptor elements exist for all sections based on configuring the execution units to perform the operation. For example, sub-descriptor elements may define the destination and source pipes, pointers to the translation from operation space to section space, and the operation mode of the section.

[0127] In this exemplary implementation, pipes represent all storage within the Neural Engine, and all allocation and memory management is handled through the NED pipe definitions of a task and traversal through the sections that produce and consume these pipes. There is no sharing of pipes between tasks, and therefore no architectural sharing of data between tasks within the Neural Engine. Subdescriptor elements are defined in the NED for each pipe in the graph. An example pipe subdescriptor is shown below:

[0128] [Table 13]

[0129] As will be explained in more detail below, these descriptors are used to configure the hardware elements when a call is triggered by the handling unit 720 .

[0130] Neural Engine Dimensions and Repetitions A neural engine task describes a 4D bounding box (dimensions 0-3) to be operated on by a section operation of the graph defined by the NED to which the task provides a pointer. In addition to describing the graph, the NED also defines four additional dimensions (dimensions 4-7), creating a total eight-dimensional operation space. The bounding boxes of the first four dimensions are full-sized subregions of these dimensions, and different tasks and / or jobs cover other subregions of these dimensions. As shown in Figures 4 and 5, the command processing unit 640 can issue different tasks to different neural engines. Thus, when the NED is created or a task is defined, the dimensions are 0-3. The latter four dimensions are described in their entirety in the NED and are therefore completely covered in each task. The NED further defines the increment size for each of these eight dimensions to be stepped through, known as the block size. Graph execution on this 8D operation space can be viewed as a series of nested loops.

[0131] This divides the execution of a task's operation space into a series of blocks, and a section is called for each block, operating on the true value of the block's data in all source and destination pipes. Thus, defining a general operation space in a coordinate system with, for example, eight dimensions, may provide a low-complexity pattern for the execution of any task involving operations on data, instead of relying on fixed functions per task type, which may involve a significant risk of missing a required pattern combination. By defining a common operation space in a coordinate space, it may be less complex to chain together multiple operations performed on data and coordinate the execution of these functions. The operation space dimensions have no inherent interpretation until they are projected into a space for a specific task.

[0132] The number of dimensions in use depends on the graph and its operations. Not all sections are executed for increments in each dimension. For example, a convolution operation has a 7D operation space, but only a 4D output space where the convolution operation increments and accumulates its output. Therefore, the VE scaling operations following the convolution are executed for increments in only the first four dimensions. This relationship is described by two variables: dims_inc_run (the "dimension increment run" value), which is the number of operation space dimensions that triggers each section increment, and dims_inc_buf (the "dimension increment buffer" value), which is the number of operation space dimensions that generate a new block for each pipe; both of these are encoded in each NED element. Both fields are specified by counting dimensions from the outermost dimension #0 to the innermost dimension #7.

[0133] dims_inc_run specifies how many operation space dimensions trigger a call to the section when these dimensions increment in the operation space. An example use of dims_inc_run is shown below: ○0: The section is independent of the computation space and is therefore invoked only once for a task; ○1: The section may depend on the computation space dimension #0 and is called for each computation space step through dimension #0; and ○8: Section may depend on all computation space dimensions and is called for each computation space step. dims_inc_buf specifies how many operation space dimensions produce new blocks to the pipe when the operation space dimension increments in the producer section, effectively defining how many blocks the pipe produces throughout the duration of the task. If dims_inc_buf has a value k (where k>0), then pipe.blocks = dim[0].blocks * dim[1].blocks * ... * dim[k-1].blocks, If dims_inc_buf has a value of k (where k==0), the pipe always has only a single block.

[0134] For simple operations, dims_inc_run will be equal to dims_inc_buf for all source input and output destination pipes, but for more complex operations, dims_inc_run can be larger.

[0135] For the source pipe, if dims_inc_run>dims_inc_buf, This relationship between the fields indicates buffer reuse across one or more operation spatial dimensions, with the difference between the two values ​​specifying the number of reused dimensions. In this context, reuse means that data is broadcast across extra dimensions, and buffers within the neural engine's internal memory are consumed multiple times. For example, feature maps input to a convolution operation are typically reused across the x and y dimensions of the convolution engine's weight kernel.

[0136] On the other hand, for a destination pipe, this relationship indicates a reduction of a set of buffers in one or more operation space dimensions, and the difference between the two values ​​specifies the number of reduction dimensions. In this context, reduction means that data from extra internal operation space dimensions is accumulated into a smaller number of external operation space dimensions (a section reads back and updates its output buffers over multiple calls). For example, a vector block reduction operation results in a smaller number of buffer increments.

[0137] If a pipe has multiple consumers, there is no relationship between those consumers and there are no restrictions or requirements on the value of a consumer's dims_inc_run relative to other consumers.

[0138] In the example described herein, the neural engine's handling unit is responsible for iterating through this 8D operation space for each section described in the NED graph. The handling unit uses two values, dims_inc_run and dims_inc_buf, to determine which increments are relevant and to correctly manage dependencies between sections and their pipes. Each section operates in its own local coordinate space, known as the section space, and the handling is responsible for transforming each relevant operation space block (related through increments in the run dimension) into this section space. In the example described herein, this transformation can be programmatic, written in a small program in a dedicated (or general-purpose) ISA that is executed for each block before the section is called.

[0139] The handling unit may synchronize the execution of different parts of these nested for loops in parallel, and therefore needs to keep track of where in the loop a component's function should be called, and where in the loop (based on a partially ordered set of data structures) data that may be needed by a subsequent component is produced. To achieve this in a flexible way, that still allows for simple hardware implementation, two types of dimensions are specified in each data structure.

[0140] In some embodiments, each data structure contains, for each of the N dimensions of the coordinate space, a vector of N binary values ​​that indicates whether a change in coordinate in that dimension while executing the task causes the associated component's function to execute and whether the associated component's function should store data in storage (DIMS_INC_RUN). Effectively, this allows the behavior of each component for each dimension to be encoded as a multi-hot vector of behaviors. Behaviors may include, for example, reuse, recalculate, reduce, output, and unmap / once.

[0141] In some types of tasks that involve operations on data, the data is frequently "reused" multiple times across several dimensions. For example, in operations on neural networks, the same weight may be applied to multiple elements in the X and Y dimensions of a batch of feature maps, but the weights are unique across the input and output channel dimensions. To inform the handling unit about the details of each function (based on the task at hand), each data structure may indicate a dimension of a coordinate space, and changes in the coordinates in that dimension while performing the task will cause the function of the associated component to be performed.

[0142] To save bits and reduce complexity, each data structure may instead include a first number 402 (as well as a second number, described further below in connection with FIG. 5) indicating a dimension of the coordinate space, such as a number between 0 and N (the number of dimensions in the computation space, 8 in the example of FIG. 4), where a change in coordinate in that dimension while performing the task causes the associated component's function to be executed. If the number is equal to 0, the section is called once per task (e.g., when an iteration over an N=>1 dimensional coordinate space begins or ends). This may correspond, for example, to a function that loads a table to be used in a subsequent subtask, regardless of coordinate or dimension. At the other extreme, the value may be equal to N, meaning that the component's function is executed at every iteration in all dimensions.

[0143] In Figure 4, the shaded elements correspond to the dimensions (of each section) whose coordinate changes cause a function to be executed (e.g., DIMS_INC_RUN). As can be seen from Figure 4, for the data structures described as "IFM Load", "Weight Load", and "conv", the functions associated with each component are executed when any dimension increments. "Bias" and "Scale Load" are only called (executed) when the batch or OFM channel increments. The "Scale" and "OFM Write" sections are called when the batch, OFM C, OFM Y, or OFM X increments.

[0144] In some types of tasks involving operations on data, functions performed on the data may result in a reduced number of dimensions being output. For example, as can be seen in Figure 4, a 2D convolution operation (conv) iterates over batch (N), output feature map height (OFM Y), output feature map width (OFM X), input channel (IFM C), output channel (OFM C), kernel X (KX), and kernel Y (K Y). However, this reduces these seven dimensions to four in its output (N, OFM X, OFM Y, OFM C). Similarly, so-called "reduction operators," such as ReduceSum, iterate over tensors, summing data across one or more dimensions to produce an output tensor with fewer dimensions than the input tensor. To inform the handling unit about the details of each function (based on the task at hand), each data structure may indicate a dimension of the coordinate space, and changes in coordinates in that dimension while performing the task will cause the associated component function to store data in storage, ready to be consumed by a component function associated with a subsequent data structure in the partially ordered set of data structures, or ready to store the task's final output data. In other words, when such a dimension increments (i.e., a coordinate changes), a new buffer is available in the pipe to be used by a component function associated with a subsequent data structure in the partially ordered set of data structures, or the final data for the task (i.e., for the portion of the bounding box currently being processed) has been stored in an output buffer.

[0145] In some embodiments, each section includes N dimension specifications, for each of the N dimensions of the coordinate space, indicating the storage implications for each dimension as coordinates in that dimension change during execution. To save bits and reduce complexity, each data structure may instead include a second number indicating the dimension of the coordinate space, such that changes in coordinates in that dimension while performing the task cause the associated component function to store data in storage, ready to be consumed by the associated component function in a subsequent data structure in the partially ordered set of data structures or ready to store the task's final output data. The second number (reference number 502 in FIG. 5) may be a number between 0 and N (the number of dimensions in the computation space, 8 in the example of FIG. 4). Because data storage may occur only as the associated component function executes, the second number may be less than or equal to the first number.

[0146] A second number of 0 indicates that the section (data structure) produces exactly one block of output ready to be consumed by the component function associated with the subsequent data structure / section. A second number of 1 indicates that the section produces output (ready to be consumed) only when the operation space dimension 0 increments (coordinates change). A second number of 2 indicates that the section produces output (ready to be consumed) when either the operation space dimension 0 or 1 increments, etc. If the second number is smaller than the first number, this indicates a reduction operation.

[0147] In Figure 5, shaded elements correspond to dimensions (for each data structure) whose coordinate changes cause the associated component's function (as opposed to Figure 4, e.g., DIMS_INC_BUF, which are associated with the function) to store data in storage, ready for consumption by the component's function associated with a subsequent data structure in the partially ordered set of data structures or for storing the task's final output data. As can be seen from Figure 5, for the data structures described as "IFM Load" and "Weight Load," the function associated with each component stores data ready for consumption by the component's function associated with a subsequent data structure in the partially ordered set of data structures when any dimension increments. "Bias" and "Scale Load" only store data ready for consumption by the subsequent function when the batch or OFM channel increments. "Scale" stores data ready for consumption by the subsequent function when the batch, OFM C, OFM Y, or OFM X increments. "OFM Write" stores the final output data of a task when Batch, OFM C, OFM Y, or OFM X increments. For "Conv," IFM C, Kernel X, and Kernel Y are marked as dimensions along which the associated function executes (see Figure 4), but are not marked as dimensions along which the associated function stores data ready to be consumed. This means that these three dimensions are so-called reduction dimensions, and seven dimensions are reduced to four at the output of Conv.

[0148] In the example, if a computation space dimension is marked as a dimension in which a change in coordinates in that dimension causes the associated component's function to execute (Figure 4), but a change in coordinates in that dimension is not marked as a dimension in which the associated component's function that generates the component's input buffer stores data in storage (Figure 5), this indicates a reuse of an input buffer by an execution section. For example, if we have a section A->B, and A's storage dimension is smaller than B's run dimension, then there is a reuse by B of an input buffer written by A. On the other hand, if B's ​​storage dimension is smaller than B's execution dimension, then this is a reduction by B to an output buffer.

[0149] The described data structures may be generated, for example, by a compiler connected to a processor, the compiler configured to generate code for execution by the processor. Execution of a neural engine task may be defined by two separate iterative processes implemented in the handling unit. In one process, the handling unit iteratively steps through the task's operation space in block units as defined by the block size of the NED. In the other process, the handling unit iteratively steps through the data flow graph defined by the NED, and, if permitted by the dimension rules described above, transforms each block into the associated section space before invoking the execution unit for the section with the transformed block by issuing call data.

[0150] In general, in most cases, these two processes are defined as architecturally independent in the examples described herein. This means that the execution of any given block is deterministic and completely defined in itself, independent of the state of any other block or operation space iteration of the handling unit. Execution of blocks that do not follow this operation space iteration and transformation will run to completion, but the results will not provide meaningful results with respect to the complete operation definition of the Tensor Operator Set Architecture (TOSA).

[0151] In all cases, execution of a block must not extend beyond the block's section space boundary. Data loads and stores (whether mapping the section space to the coordinates of a tensor in memory, to a pipe, or to any other memory or pipe storage) may extend beyond the section space as required by the implementation's access granularity, but must not extend beyond the size of the pipe's buffer or the total size of the tensor. If the section space is smaller than the pipe buffer, VE BlockReduce operations have the additional requirement not to modify data in the buffer beyond the section space; other operations or execution units do not have this requirement.

[0152] TSU operation space iteration may produce blocks with one or more execution dimensions (execution_dimension_empty) equal to zero, meaning that no functional operations are required. This may occur, for example, due to padding before the start of the operation space or clipping at the end of the operation space. As noted in TSU task iteration and block invocation, blocks must still be dispatched to execution units for accurate tracking of dependencies and execution order.

[0153] Thus, for a conversion from operation space to section space to be compatible when connected by a pipe, the following must hold for the conversion to be valid:

[0154] Assume the following scenario: ○Section S0 writes to pipe P. ○Section S1 reads from the same pipe P. ○T0() is the transformation of section S0. ○T1() is the transformation of section S1. ○B is a block in the computation space. ○B0 is the absolute tensor coordinate of the block written to pipe P by S0. ■ This becomes DST(T0(B)), where DST() is the fixed transformation of S0's execution unit into its destination output space. ○B1 is the absolute tensor coordinate of the block read from pipe P by S1. ■ This becomes SRC(T1(B)), where SRC() is a fixed transformation from S1's execution unit to its source input space.

[0155] In this case, the following must hold: - Compatible origins: Blocks B0 and B1 must have the same lower bound coordinates for each dimension. o This coordinate forms the origin of the block stored in the pipe buffer. - Sufficient size: The size of block B0 must be greater than or equal to the size of block B1 in each dimension.

[0156] An operation space iteration may produce blocks with one or more execution dimensions that are zero, meaning that no functional operations are required (this may occur, for example, due to padding before the start of the operation space or clipping at the end of the operation space). The blocks must still be dispatched to execution units for correct tracking of dependencies and execution order.

[0157] To implement a reduction operation, an operation space iteration issues a sequence of block calls to an execution unit (e.g., a convolution engine or vector engine) all targeting the same output block. The handling unit signals when it is time to execute the first block in this sequence, and the execution unit must begin by initializing a destination buffer (the entire buffer, limited by the size of the block as described above). For all subsequent blocks in the sequence, the unit reads back existing values ​​from the buffer. In this way, the destination buffer serves as an additional input to the operation from the perspective of individual block execution. In the case of a convolution engine, one or more reduction dimensions can be zero, meaning that no functional operation is required; however, the convolution engine must still initialize the destination buffer if it is the first block in the sequence and the execution dimension of the block is not empty.

[0158] When a handling unit calls an execution unit to execute a block, the handling unit is configured to issue call data to perform the operation on the block. Block iteration is defined based on the block size specified in the NED, and the issuance of call data is controlled by the DIMS_INC_RUN value, as described above. Additionally, any dependencies that need to be satisfied for the execution unit to operate on the block are required. These include that the necessary data is stored in the source pipe(s) for the operation, that sufficient storage is available in the destination pipe, and that a transformation of the operation space to the section space for that section is performed and the output of the transformation operation (i.e., the transformed coordinate data) is available to be issued to the execution unit. More specifically, sufficient availability for a new block or buffer in the pipe is guaranteed. However, if this is not the first step in a reduction block, this is not required, since in this instance the operation may involve a simple read-modify-write of the previous destination block / buffer. Determining the availability of source storage elements may involve determining that a suitable block / buffer exists in the source pipe.

[0159] In one example, the call data includes the output of the transformation program in the form of transformed coordinates, along with the relevant portions of the NED that describe that section (e.g., configuration data from subdescriptor elements of the NED for that section). This additional configuration data may also include the type of operation being performed (if the execution unit is capable of performing more than one type of operation), and any other attributes of the operation, such as stride and dilation values ​​in the example of a convolution operation.

[0160] The iterative process involves first reading the block size from the NED and iterating through the operation space one block at a time. For each block, a conversion program is run to convert the operation space coordinates into section space coordinates for that section. Further details about the conversion program are provided below. Once the section space coordinates are determined, the section operation is performed for that block. This process is repeated across all blocks until the operation is completed for all blocks.

[0161] 6 shows an exemplary chain of operations 200 that is performed. The chain includes a left-hand-side (LHS) input read operation 220 and a right-hand-side (RHS) input read operation 210. The output of the RHS input read operation 210 is input to a reverse operation 230, which, along with the output of the LHS input read operation 220, outputs to a matrix multiplication (MatMul) operation 240. The output of the MatMul 240 operation is input to a rescale operation 250, the output of which is provided to an output write operation 260, which writes the output to memory.

[0162] 7 shows the corresponding coordinate spaces (i.e., the section spaces for each of the operations). For example, RHS input read section space 215 is shown for RHS input read 210 operation. LHS input read section space 225 is shown for LHS input read operation 220. Reverse section space 235 is shown for reverse operation 230. MatMul section space 245 is shown for MatMul operation 240. Rescale section space 255 is shown for rescale operation 250. In this example, the section space for the output write operation is shown using section space 255 because it is unchanged from the section space for the rescale operation.

[0163] Each section space includes multiple dimensions, i.e., two dimensions (e.g., K, N, K, M). The section space is divided into blocks having a predetermined block size, and each of blocks A to H represents a different block to be operated on according to the example shown in this specification.

[0164] As can be seen, the reverse section space 230 has effectively reversed dimensionality relative to the RHS input read section space 215. The section space 225 for the LHS input read contains repeated blocks A / E, B / F, C / G, and D / H. The section space 255 for the rescale and output write operation contains two blocks, A-D and E-H. This is because the MatMul operation is a reduction operation. In the MatMul example of FIG. 7, MatMul is performed on two matrices 225 and 235. Matrix 225 has dimensions K×N, and matrix 235 has dimensions K×M. Output 255 has dimensions N×M, so K dimensions have been reduced. MatMul can be described in a 3D operation space of N, M, and K.

[0165] As will be appreciated, the operations shown in FIG. 7 are sections that can each be executed by a different execution unit. The handling unit can be configured to control the execution of various blocks so that a particular block can flow through a chain of operations defined by a graph or subgraph. The "A / E" notation in these figures indicates that the block is repeated. For example, blocks A and E have the same coordinates in some dimensions (K, N), but there is another dimension (M) that changes but does not map to the 220 coordinate space. The "AD" notation indicates that the block has been reduced and merged into a single block. For example, blocks A, B, C, and D have been reduced to a single block. These blocks change in dimension K, but dimension K has been reduced. An example of scheduling the blocks shown in FIG. 7 is shown in FIG. 8.

[0166] FIG. 8 illustrates an exemplary iteration through the blocks of the chain of operations of FIGS. 6 and 7 for a series of latency time instances 0 through 11. At latency time instance 0, block A is processed simultaneously by the execution unit executing LHS read and RHS read operations. These operations have no dependencies and can be handled in a single latency time instance in this example, and are therefore issued simultaneously. Because the LHS read and RHS read operations are independent of each other, for all subsequent latency time instances, the next block (e.g., block B at time instance 1) is called for execution until all blocks A through H have executed at time instance 7. This operation may still stall if there is no space in the destination pipe for that section.

[0167] Because the reverse operation is a subsequent operation that depends on the output of the RHS read operation, the processing of block B by the reverse operation can only be invoked at time instance 1. Therefore, the processing of the block by the reverse operation is delayed by one invocation time instance relative to the RHS read operation. Similarly, the MatMul operation depends on the output of the reverse operation, so the MatMul processing of block B is delayed by one further invocation time relative to the reverse operation.

[0168] The rescale operation operates on blocks of data derived from a set of four reduced blocks of data, e.g., A-D or E-H, in a single invocation. Thus, the rescale operation is not invoked until all input dependencies are satisfied, i.e., until a MatMul operation has been performed for each of blocks A-D at time instance 6. Similarly, blocks E-H are not invoked for execution until time instance 10. The output write operation is dependent on the completion of the rescale operation and therefore is not invoked until time instance 7 for blocks derived from processing blocks A-D, and similarly is not invoked until time instance 11 for blocks derived from processing blocks E-H.

[0169] In this manner, the process iterates through all blocks until the complete operation space has been executed.

[0170] The process for generating an operation space capable of representing each of these respective section spaces will be described in more detail later, but in this example, the operation space for this chain of operations is considered to be the section space 245 for the MatMul operation 240, since all other section spaces can be represented from the MatMul section space 245.

[0171] FIG. 9 shows a flowchart of an efficient data processing method 900 according to the present disclosure. The data processing method 900 is executed on a processor configured to handle task data, the processor comprising a handling unit, a plurality of storage elements, and a plurality of execution units. The task data includes a program including transformation program data describing a transformation from an operation space to a section space (local space) for a corresponding section. In step 902, the processor retrieves the task data from storage in the form of a directed acyclic graph of operations, as described above; the method and processor may be configured to operate on any type of graph, not just directed acyclic graphs. Each operation is mapped to a corresponding execution unit of the processor, and each connection between operations in the acyclic graph is mapped to a corresponding storage element of the processor. In step 904, for each corresponding portion of the operation space, the method 900 includes transforming the portion of the operation space to generate a respective operation-specific local space for each of the operations of the acyclic graph. In step 906, the method 900 includes dispatching, to each of a plurality of execution units associated with the operation for which the transformed local space was generated, invocation data describing the operation-specific local space and at least one of a source storage element and a destination storage element corresponding to a connection between the particular operation to be performed by the execution unit and a further adjacent operation in the acyclic graph to which the particular operation is connected. The processor is further configured to perform clipping 908, if necessary, on lower and upper bounds of the task and operation spaces before performing the transformation.

[0172] Programmability of the transformation from arithmetic space to section space As noted above, the operation space for a task (subgraph) may include a predetermined number of dimensions (e.g., 8), but the local section space for operations performed on a specific section within that graph may include fewer than 8 dimensions. Also, as noted above, the handling unit may iterate through the operation space in units known as blocks, converting each block from the common operation space to a section-specific space described by various fields in the NED.

[0173] In an exemplary implementation, the NED may further include, for each element in the NED (e.g., each section / pipe), a program containing translation program data describing the translation from operation space to section space (local space) for the corresponding section. In one such implementation, each element in the NED may include an offset value that points to a unique program in the NED for performing the translation. This offset value may be considered a pointer to a "program space," which is the space where all programs defining the various enabled translations are located. Alternatively, the offset value may be a pointer to a virtual address space in main memory. For example, this program space may be defined in the NED as a field tsu_space_size, e.g., 256 bytes in size. The offset may point to a memory location where the start of that section-space translation is located (e.g., the first instruction in a sequence of instructions that collectively define the program for performing the translation).

[0174] Each transformation program may end with an explicit END instruction and may be followed without any spacing or alignment by the next program, which defines a sequence of instructions to perform a different transformation associated with a different element. Alternatively, a start pointer may be used along with the total number of instructions to execute.

[0175] In an exemplary implementation, the sequence of instructions used for each transformation may be selected from a predetermined set of instructions that effectively form an instruction set. The instructions may be considered a transformation instruction set, which may be a specific set of instructions optimally selected to perform the transformation from the operation space to the section space. Alternatively, the transformation may be a general-purpose instruction set such as found in a central processing unit (CPU).

[0176] In an exemplary implementation, a transformation instruction may operate on a set of state values ​​for the transformation. The state values ​​include bounds registers (eight bounds registers b[0]-b[7] in one example), each containing a low and high component. Each block in the computational space is defined by the values ​​written to the low and high components of the eight bounds registers. These values ​​indicate the upper and lower bounds (inclusive) of the coordinates within the block relative to that axis of the "bounding box" computational space.

[0177] In this example, no other state is available for the instruction to operate to convert the operation space to the local section space for the specific operation being performed, and therefore all operations performed by the instruction operate on the bounding registers, including intermediate calculations.

[0178] Some instruction sequences transform one dimension at a time, starting with dimension 0 (e.g., b[0]), and work iteratively inward through the dimensions. In other, more complex instruction sequences, the more complex transformations may require jumping around by explicitly modifying the destination register identifier, for example, by using a SETD instruction within a set of instructions.

[0179] An example of a transformation program used to transform the output dimension of a convolution operation is shown below using a register swap instruction with destination modifier D and dimension d. program,4 instructions,4 bytes (d=0)Register swap b[d],b[1] / / swap OC and N (d=1)SWP.D b[d],b[2] / / swap OC and OY (d=2)SWP.D b[d],b[3] / / swap OC and OX END

[0180] This instruction sequence represents the following affine transformation on the output dimension of the convolution operation:

[0181] [Table 14]

[0182] The result of executing the translation program for a particular block defines a block in the section space that is ready to be used to invoke a particular hardware execution unit to execute the section. For many types of operations performed by the hardware execution units to execute a section, the execution units do not use the full 8-dimensional section space. Therefore, the handling unit defines a calling structure for each unit that defines the associated requirements for that operation.

[0183] At least some aspects of the examples described herein involve computer processes implemented on a processing system or processor. However, in some examples, the disclosure also extends to computer programs, particularly computer programs on or within an apparatus adapted to implement the disclosure. The program may be in the form of non-transitory source code, object code, code intermediate source and object code, such as partially compiled form, or any other non-transitory form suitable for use in implementing a process according to the disclosure. The apparatus may be any entity or device capable of carrying a program. For example, the apparatus may comprise a storage medium such as a solid-state drive (SSD) or other semiconductor-based RAM; a ROM, e.g., a CD-ROM or semiconductor ROM; a magnetic recording medium, e.g., a floppy disk or hard disk; a general optical memory device; or the like.

[0184] In the foregoing description, for purposes of explanation, numerous specific details of certain examples are set forth. Reference herein to "one example" or similar language means that a particular feature, structure, or characteristic described in connection with that example is included in at least one example, but not necessarily in other examples.

[0185] The above examples should be understood as illustrative examples of the present disclosure. Further examples of the present disclosure are contemplated. It should be understood that any feature described in connection with any one example may be used alone or in combination with other features described, and may be used in combination with one or more features of any other example, or in any combination of any other example. Furthermore, equivalents and modifications not described above may also be used without departing from the scope of the present disclosure, as defined in the appended claims.

Claims

1. 1. A processor for manipulating data, said processor comprising: a manipulation unit; a plurality of memory elements; and a plurality of execution units, said processor comprising: obtaining task data from a storage device that describes tasks to be performed in the form of a graph of operations, each of the operations mapping to a corresponding execution unit of the processor, each connection between operations in the graph mapping to a corresponding memory element of the processor, the task data further defining an operation space representing dimensions of a multi-dimensional array of the connected operations to be performed; For each of the plurality of portions of the computational space, transforming the portion of the operation space to generate, for each of a plurality of the operations of the graph, a respective operation-specific local space; dispatching to each of a plurality of execution units associated with an operation for which a transformed local space was generated, invocation data describing the operation-specific local space and at least one of a source storage element and a destination storage element corresponding to a connection between the particular operation to be performed by the execution unit and a further adjacent operation in the graph to which the particular operation is connected; A processor for manipulating data, the processor being configured to:

2. two or more operations in the graph of operations are mapped to the same execution unit of the processor; and two or more connections in the graph of operations each map to different portions of the same storage element; The processor of claim 1 , wherein the processor is one or more of:

3. 3. The processor of claim 1, wherein each execution unit of the plurality of execution units of the processor is configured to perform a unique operation type, and the mapping between operations in the graph and the execution units is defined based on execution compatibility between the operations in the graph and the unique operation type of the execution unit.

4. The task data is an element count value indicating a count of the number of elements that map to each execution unit having a unique operation type, each element corresponding to an instance of use of an execution unit to execute a respective operation in the graph; and a pipe count value indicating a count of the number of pipes required to execute the task; The processor of any one of claims 1 to 3, comprising:

5. 5. The processor of claim 4, wherein the task data further comprises element configuration data that defines, for each element in the graph, data used to configure the particular execution unit when performing the operation.

6. 6. The processor of claim 5, wherein the element configuration data includes, for each of a plurality of the operations of the graph, an offset value pointing to a location in memory of transformation data indicating the transformation to the portion of the operation space to be performed to generate a respective operation-specific local space.

7. The task data is conversion program data defining a plurality of programs, each program including a sequence of instructions selected from a conversion instruction set; 7. A processor according to claim 1, wherein the transformation program data is stored for each of a set of predetermined transformations, and wherein, for each of a plurality of the operations of the graph, a particular transformation from the set of predetermined transformations is selected to transform the portion of the operation space to generate a respective operation-specific local space.

8. 8. The processor of claim 1, wherein the task data includes transformation program data configured to perform a particular transformation on a plurality of values ​​stored in boundary registers defining the operation space to generate new values ​​in the boundary registers.

9. The processor of claim 8 , wherein clipping is performed on the values ​​stored in bounds registers defining the operational space before conversion.

10. A processor according to any preceding claim, comprising iterating over the computation space in blocks, the blocks being created according to a predetermined block size.

11. said dispatching of block invocation data a value identifying the dimension of the computation space along which a change in coordinates in the dimension while performing the task causes the computation to be performed; a further value identifying the dimension of the computation space, where a change in coordinates in the dimension while performing the task causes the computation to store data in the storage device, the stored data being ready to be consumed by the computation; and The processor of claim 10 , wherein the processor is controlled based on:

12. A processor according to any preceding claim, wherein dispatching of call data for the particular operation depends on the availability of the source storage data and the destination storage elements.

13. the handling unit, the plurality of memory elements, and the plurality of execution units form part of a first neural engine within the processor; and the processor: a plurality of further neural engines, each comprising a respective plurality of further storage elements, a plurality of further execution units and a further handling unit; a command processing unit configured to issue respective tasks to one or more neural engines for execution; The processor of any one of claims 1 to 12, comprising:

14. A processor according to any preceding claim, wherein the graph of operations is a directed acyclic graph of operations.

15. 1. A method for manipulating data in a processor comprising a manipulation unit, a plurality of memory elements, and a plurality of execution units, said method comprising: retrieving from a storage device task data describing a task to be performed in the form of a graph of operations, each of the operations mapping to a corresponding execution unit of the processor, each connection between operations in the graph mapping to a corresponding storage element of the processor, the task data further defining an operation space representing dimensions of a multi-dimensional array of the connected operations to be performed; For each of the plurality of portions of the computational space, transforming the portion of the operation space to generate, for each of a plurality of the operations of the graph, a respective operation-specific local space; dispatching, to each of a plurality of execution units associated with an operation for which a transformed local space was generated, invocation data describing the operation-specific local space and at least one of a source storage element and a destination storage element corresponding to a connection between the particular operation to be performed by the execution unit and a further adjacent operation in the graph to which the particular operation is connected; A method comprising:

16. two or more operations in the graph of operations are mapped to the same execution unit of the processor; and two or more connections in the graph of operations each map to different portions of the same storage element; 16. The method of claim 15, wherein the method is one or more of:

17. 17. The method of claim 15 or 16, wherein each execution unit of the plurality of execution units of the processor is configured to perform a unique operation type, and the mapping between operations in the graph and the execution units is defined based on execution compatibility between the operations in the graph and the unique operation type of the execution unit.

18. The task data is an element count value indicating a count of the number of elements that map to each execution unit having a unique operation type, each element corresponding to an instance of use of an execution unit to execute a respective operation in the graph; and a pipe count value indicating a count of the number of pipes required to execute the task; 18. The method of claim 17, comprising:

19. The method of any one of claims 15 to 18, wherein the graph of operations is a directed acyclic graph of operations.

20. A non-transitory computer-readable storage medium comprising a set of computer-readable instructions stored on the storage medium, the set of computer-readable instructions, when executed by at least one processor, causing the at least one processor to: retrieving from a storage device task data describing tasks to be performed in the form of a graph of operations, each of the operations mapping to a corresponding execution unit of the processor, each connection between operations in the graph mapping to a corresponding memory element of the processor, the task data further defining an operation space representing dimensions of a multi-dimensional array of the connected operations to be performed; For each of the plurality of portions of the computational space, transforming the portion of the operation space to generate, for each of a plurality of the operations of the graph, a respective operation-specific local space; dispatching to each of a plurality of execution units associated with an operation for which a transformed local space was generated invocation data describing the operation-specific local space and at least one of a source storage element and a destination storage element corresponding to a connection between the particular operation to be performed by the execution unit and a further adjacent operation in the graph to which the particular operation is connected; A non-transitory computer-readable storage medium configured to:

Images