Efficient data processing

Abstract

A processor and method for handling data by obtaining operations from a storage device, analyzing each of the operations to determine an associated operation space, and generating at least one operation set, where the operations of the operation set have substantially similar operation spaces. Receiving input data in a tensor form; and assigning the input data as input to a given operation in the set of operations. The input data has a predetermined input characteristic associated with the given operation. The given operation is performed using the input to produce an output having known output characteristics. Storing the input data in a segment associated with an operation in the operation set; and an output associated with the operation in the set of operations.

Description

Background Art Technical Field

[0001] The present invention relates to methods, processors, and non-transitory computer-readable storage media for handling data for processing by a set of operations, such as neural network processing operations and graphics processing operations.

[0002] Related technical notes

[0003] Certain data processing techniques, such as neural network processing and graphics processing, involve using operations to process and generate relatively large amounts of data. It is desirable to handle the data efficiently when processed by a set of operations. Summary of the Invention

[0004] According to a first aspect of the present invention, a processor for handling data is provided, the processor comprising a handling unit, a plurality of storage elements and a plurality of execution units, the processor being configured to: obtain task data describing a task to be performed in the form of an operation graph from a storage device, wherein each of these operations is mapped to a corresponding execution unit of the processor, and wherein each connection between the operations in the graph is mapped to a corresponding storage element of the processor, the task data also defining an operation space, which represents the dimensions of a multidimensional arrangement of the operations to be performed; and for each of a plurality of parts of the operation space: transform the part of the operation space to generate a corresponding operation-specific local space for each of a plurality of operations in an acyclic graph; and dispatch call data describing the operation-specific local space, and at least one of a source storage element and a destination storage element to each of a plurality of execution units associated with the operation for which the transformed local space has been generated, the source storage element and the destination storage element corresponding to a connection between a specific operation to be performed by the execution unit and another adjacent operation to which the specific operation in the graph is connected.

[0005] According to a second aspect of the present invention, a method for handling data in a processor is provided, the processor comprising a handling unit, a plurality of storage elements and a plurality of execution units, the method comprising: obtaining task data describing a task to be performed in the form of an operation graph from a storage device, wherein each of these operations is mapped to a corresponding execution unit of the processor, and wherein each connection between the operations in the graph is mapped to a corresponding storage element of the processor, the task data also defining an operation space, which represents the dimensions of a multidimensional arrangement of connected operations to be performed; and for each of a plurality of parts of the operation space: transforming the part of the operation space to generate a corresponding operation-specific local space for each of a plurality of operations of the graph; and dispatching call data describing the operation-specific local space and at least one of a source storage element and a destination storage element to each of a plurality of execution units associated with the operation for which the transformed local space has been generated, the source storage element and the destination storage element corresponding to a connection between a specific operation to be performed by the execution unit and another adjacent operation to which the specific operation in the graph is connected. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Further features and advantages will become apparent from the following description of preferred embodiments, given by way of example only, with reference to the accompanying drawings in which like reference numerals are used to indicate like features.

[0007] Figure 1a illustrates an example directed acyclic graph in which segments are interconnected by a series of pipes according to the present disclosure;

[0008] Figure 1b Schematically illustrates an example of a data processing system according to the present disclosure;

[0009] Figure 2 A schematic diagram illustrating a neural engine according to the present disclosure is illustrated;

[0010] Figure 3 schematically illustrates an example system for distributing treatment data according to the present disclosure;

[0011] Figure 4 illustrates a table showing data associated with a plurality of segments according to the present disclosure;

[0012] Figure 5 illustrates a table showing data associated with a plurality of segments according to the present disclosure;

[0013] Figure 6 illustrates a sample chain of operations to be performed;

[0014] Figure 7 An example corresponding coordinate space is illustrated;

[0015] Figure 8 Illustrated Figure 7 An example of the scheduling of blocks set forth in ; and

[0016] Figure 9 A flow chart illustrating efficient data processing according to the present disclosure is illustrated. DETAILED DESCRIPTION

[0017] Examples herein relate to a processor for processing data, the processor comprising a processing unit, a plurality of storage elements, and a plurality of execution units. The processor is configured to obtain task data from a storage device that describes tasks to be performed in the form of an operation graph, such as a directed acyclic graph. Each of these operations is mapped to a corresponding execution unit of the processor, and each connection between the operations in the graph is mapped to a corresponding storage element of the processor. The task data also defines an operation space that represents the dimensions of a multidimensional arrangement of connected operations to be performed. Although the examples described below relate to directed acyclic operation graphs, it should be understood that any type of operation graph can be used.

[0018] For each of the plurality of portions of the operation space, the processor is configured to transform the portion of the operation space to generate a corresponding operation-specific local space for each of the plurality of operations of the graph.

[0019] Where necessary, the processor is also configured to perform clipping on the lower and upper bounds of the task and operation spaces before running the transformation. Clipping may be functionally necessary for the edges of the tensor and allows for an operation space that is smaller than the full tensor. An operation space that is smaller than the full tensor is advantageous because it allows larger sequences of operations to be split into multiple independent tasks and optionally executed on separate cores.

[0020] The processor is also configured to dispatch, to each of a plurality of execution units associated with an operation for which a transformed local space has been generated, call data describing a specific local space for the operation, and at least one of a source storage element and a destination storage element corresponding to a connection between a specific operation to be performed by the execution unit and another adjacent operation to which the specific operation is connected in the acyclic graph.

[0021] The present disclosure relates to a graph that performs operations (called segments) connected by various connections (called pipelines). By providing the ability to operate a sequence of connection operations (segments) defined in an operation space shared by an operation sequence, it is possible to ensure that when the operation sequence is executed, all coordinates required for the operation in the operation space are reachable. For each execution of an operation (or part of an operation), the operation space is transformed into the local segment space of the operation. More generally, a directed acyclic operation graph includes vertices (see operations) connected by edges so that each edge points to another vertex from one vertex in a manner that does not form a closed loop in the direction of the edge. As described above, a task can be performed in the form of an operation graph representing a given operation sequence. This can be represented as any type of graph, not just a directed acyclic graph. In such an example, the operation graph includes vertices (see operations) connected by directed or undirected edges. In some examples, directed edges can form a closed loop.

[0022] Each operation (segment) is connected by a corresponding pipeline, forming a directed acyclic operation graph. For each operation, a source pipeline and a destination pipeline can be defined, and under the control of the processing unit, the execution of the segment can be issued by issuing call data defining the source pipeline and destination pipeline of the operation. Therefore, this execution of the operation graph by the corresponding execution unit is implicitly ordered according to the dependency on the specific input of the operation. The result of this implicit ordering is to simplify the orchestration of operations between the execution units of the processor. In other words, the directed acyclic relationship between the segments and each other can be determined by their pipeline usage (for example, their producers / consumers).

[0023] In the present disclosure, by transforming from the operation space, it is ensured that for each possible operation, there is a specific coordinate space called the segment space (or segment-specific local space). For each operation, there can be a fixed-function transformation from their individual segment space to each of their input and output data (pipelines); this may be different for multiple inputs / outputs. For element-by-element operations, the transformation from the segment space to the input and output pipelines will be an identity mapping: no transformation is required. For convolution, the output is similarly an identity of the segment space, where only the input needs to be transformed. An exception is that for some operations (e.g., convolution), the output space is only the outer four dimensions. In addition, the inputs of some operations may have non-identity transformations from the segment space and may be different from each other. However, in the present disclosure, each operation is defined with its own independent segment space, i.e., specific to that segment (or operation) without needing to map to the outputs of other operations.

[0024] By defining a common operation space for the entire graph (or operation chain), and then defining a transformation from the operation space to a separate segment space for each operation, different operations of different types are chained together. Now, each hardware unit only needs to understand its fixed-function transformation from the segment space to the input / output space, without having to understand the operation chain before or after it. For example, if the conditions for a valid operation space exist, additional operations can be chained before or after the convolution operation, and more operations can be stitched together. Since all segments iterate through the same operation space during execution, the data blocks are aligned. For example, the first block from a memory read operation will be the first block to enter the data processing operation, and this will gradually permeate into the first block in the memory write operation. This is a simplification considering that for some operations (reduction and broadcast operations), blocks can be combined with data from other blocks to form new merged blocks, but it generally holds true as a principle. The operation space is typically mapped to the space of a specific operation in the graph, where programming transformations are provided for all other operations.

[0025] Operations that access pipelines may have additional transformations added to access data stored in the pipeline. For example, this may be a different transformation for different pipelines: different for multiple inputs and different for outputs. The transformation is defined in the properties of the operation and is a fixed function.

[0026] In summary, the segment space of an operation can be mapped to the input and / or output (which can be the same), or the segment spaces of the operation can be mapped separately, in which case fixed-function transformations may be required. In this way, the proposed method allows for more compartmentalized functionality to be implemented in separate execution units. Consequently, the execution units of a processor can be implemented with a more simplified structure, since the ability to perform complex transformations on the front end or output of the execution unit does not need to be provided in each execution unit. Instead, the transformation from the operation space to the segment space (and therefore the management of compatibility and correct structuring of data between consecutive operations) is centrally managed and issued by a single handling unit based on the dimensions of a predefined operation space (e.g., by descriptors that define the operation space and the segments and pipelines that form the graph).

[0027] Since a single transform unit can perform the transformation from an operation to segment space, the processor is able to add support for other operations in the future without requiring major hardware modifications to the execution units, allowing additional operations to be chained to the front of the chain or anywhere in the chain. This allows new functionality to be added easily. As an example: for the convolution operation, dynamic weights can be easily added by adding a data reordering unit or transform that can transform tensors in activation layout to weight layout, which can be handled by the convolution engine. Properties of operations such as padding around the edges of the input can also be implemented through the transform mechanism.

[0028] Additionally, many less common operations can be broken down into smaller execution units (e.g., through simpler primitive operations from which more complex (or less common) operations can be constructed). Iteration of more common operations can enable support for larger operations that would otherwise not fit within the constraints of the processor, rather than implementing native support within the execution units. For example, for operations, convolution operations with stride values > 1 can be implemented by breaking the kernel into single element increments and iteratively calling the convolution engine with a 1-element kernel, thereby supporting larger strides. Similar examples exist for operations requiring dilation values > 1. 3D convolution operations can be similarly implemented as iterative 2D convolution operations.

[0029] In some examples, the processor is optionally configured such that more than one operation in the acyclic operation graph is mapped to the same execution unit of the processor; and more than one connection in the acyclic operation graph is respectively mapped to different portions of the same storage element.

[0030] In some examples, the processor is optionally configured such that each of a plurality of execution units of the processor is configured to perform a particular operation type, and wherein the mapping between the operations in the acyclic graph and the execution units is defined based on the compatibility of execution between the operations in the acyclic graph and the particular operation type of the execution units.

[0031] In some examples, the processor is optionally configured such that the task data includes an element count value indicating a count of the number of elements mapped to each execution unit having a particular operation type, where each element corresponds to an instance of use of an execution unit to perform each operation in the acyclic graph; and a pipeline count value indicating a count of the number of pipelines required to perform the task.

[0032] There are elements for describing each type of segment and each type of pipeline, so an element can be defined as a structured definition of a pipeline or segment.As described herein, a segment has various parameters that describe the execution details.

[0033] In some examples, the processor is optionally configured such that the task data also includes, for each element in the acyclic graph, element configuration data defining data for configuring a particular execution unit when performing the operation.

[0034] In some examples, the processor is optionally configured such that the element configuration data includes an offset value pointing to a location in memory of transformation data indicating a transformation to be performed on a portion of the operation space to generate a corresponding operation-specific local space for each of a plurality of operations of the acyclic graph.

[0035] In some examples, the processor is optionally configured such that the task data includes transformation program data defining a plurality of programs, each program including a sequence of instructions selected from a transformation instruction set. The processor is optionally configured such that the transformation program data is stored for each transformation in a set of predetermined transformations, a particular transformation from the set of predetermined transformations being selected to transform the portion of the operation space to generate a corresponding operation-specific local space for each of the plurality of operations of the acyclic graph.

[0036] In some examples, the processor is optionally configured such that the transformation program data is configured to perform a particular transformation on a plurality of values stored in boundary registers defining the operating space to generate new values in the boundary registers.

[0037] In some examples, the processor is optionally configured to iterate the operation space in units of blocks, where the blocks are created according to a predetermined block size.

[0038] In some examples, the processor is optionally configured such that the dispatch of call data is controlled based on the following two values: a value identifying a dimension of the operation space, for which a change in the coordinates of the dimension when performing a task causes the operation to be performed; and another value identifying a dimension of the operation space, for which a change in the coordinates of the dimension when performing the task causes the operation to store data in a storage device, where the stored data is ready for use by the operation.

[0039] Directed Acyclic Graph (DAG) Execution

[0040] Although the examples described below relate to the execution of directed acyclic graphs, it should be understood that the described methods can be used for the execution of any type of graph, not just directed acyclic graphs.

[0041] Many data structures to be executed in a processor can be represented as graphs, such as directed acyclic graphs. Examples of such data structures include neural networks, which can be represented as directed acyclic graphs of operations that fully constitute the operations required to execute the network (i.e., the operations performed across the layers of the neural network). A directed acyclic graph is an operation (also referred to herein as a "segment") data structure with directed connections between these operations that indicate the flow of operations, such that those directed connections do not form closed loops. The connections between the operations (or segments) present in the operation graph are also referred to herein as "pipelines." An acyclic graph can contain any number of divergent branches and convergent branches.

[0042] Figure 1aAn example directed acyclic graph 100 is illustrated in which segments are interconnected by a series of pipes. Specifically, an initial segment, segment 1 (1110), represents a point in the acyclic graph at which an operation, operation A, is performed when the graph is executed. The output of operation A at segment 1, 1100, is connected to two additional segments, segment 2 (1120) and segment 3 (1130), at which respective operations B and C are performed. The connection between segment 1 (1110) and segment 2 (1120) can be identified as a pipe with a unique identifier, pipe 1 (1210). The connection between segment 1 (1110) and segment 3 (1130) can be identified as a pipe with a different identifier, pipe 2 (1220). The output of segment 1, as a result of performing operation A on the input of segment 1, can be provided to multiple subsequent segments in a branching manner.

[0043] More generally, a segment in an acyclic graph can receive multiple inputs, each input coming from a corresponding different segment in the acyclic graph via a corresponding different pipe. For example, Figure 1a Segment 1150 in receives a first set of input data from segment 1120 via pipe 1240 and receives a second set of input data via pipe 1250. Any number of input and output pipes may be connected to a particular segment in the acyclic graph, depending on the nature of the operation performed in the particular segment and the dependency of subsequent operations on the output of that operation.

[0044] An acyclic graph can be represented by multiple subgraphs, each containing a subset of the segments in the graph. Figure 1a 13. The diagram 110 is illustrated as being decomposed into three sub-graphs 1310, 1320, and 1330, which can be connected together to form a complete diagram. For example, sub-graph 1310 includes segments 1110 and 1130 (and corresponding pipes 1220 and 1260), sub-graph 1320 includes segments 1120, 1140, and 1150 (and corresponding pipes 1210, 1230, 1240, and 1250), and sub-graph 1330 includes segments 1160 and 1170 (and corresponding pipes 1270, 1280, and 1290).

[0045] Deconstructing the graph 100 into subgraphs is particularly useful when attempting to execute the graph, as the subgraphs can be executed independently, which allows for parallel execution without dependencies between the subgraphs. This can be particularly useful in a multi-processor environment, where the subgraphs can be distributed for execution by different processors in the multi-processor environment. However, as Figure 1aAs shown, subgraph 1320 depends on the execution of operation A and section 1110, and subgraph 1330 depends on subgraph 1310. Therefore, the execution of subgraph 1330 may need to be stopped until subgraph 1310 has completed. Therefore, it should be understood that it is necessary to carefully select an appropriate subgraph arrangement to maximize or improve the execution efficiency of the graph.

[0046] The operations performed when executing a neural network can be broken down into Figure 1a

[0014] The detailed description herein will describe an arrangement for executing acyclic graphs of operations in an improved manner.

[0047] Operating space

[0048] When executing a chain of operations constructed, for example, in a directed acyclic graph, each segment can represent a different operation. Each operation does not have to be of the same type or nature. This is particularly true when the operation graph is used to represent the processing of a neural network. The machine learning software ecosystem allows for diverse structures of neural networks applicable to many different problem spaces, and therefore there is a very large set of possible operators from which a neural network can be composed. The inventors have recognized that when seeking to design hardware to enable execution of a set of possible operations (also known as "acceleration"), these sets of operations from which segments can be formed can be difficult to manage, particularly when chained together. For example, enabling fixed-function operations for every possible type of operation can lead to hardware inefficiencies by requiring support for obscure or complex operations (segments).

[0049] Therefore, there are significant challenges in designing and building hardware capable of executing all types of neural networks created by current machine learning tool sets. Therefore, the inventors have recognized that it is desirable to define a set of predetermined low-level operations from which a variety of possible high-level operations corresponding to various machine learning tool sets can be constructed. An example of such a low-level operation set is the Tensor Operator Set Architecture (TOSA). The Tensor Operator Set Architecture (TOSA) provides a set of full tensor operations commonly used by deep neural networks. Its purpose is to enable various implementations to run on various processors, making TOSA-level results consistent across these implementations. Therefore, applications or frameworks targeting TOSA can be deployed on a variety of different processors, including single instruction multiple data (SIMD) CPUs, graphics processing units (GPUs), and custom hardware such as neural processing units / tensor processing units (NPUs / TPUs), with defined accuracy and compatibility constraints. Most operators from common ML frameworks (TensorFlow, PyTorch, etc.) should be expressible using TOSA.

[0050] However, even if such an operator set exists, the inventors have recognized that it is necessary to implement the operator set in a manner that can be executed efficiently, both in terms of complexity and while minimizing the need to perform external memory transactions. To achieve this, the inventors have recognized that it is useful to consider that many operations in a defined operation set (such as TOSA) can be expressed as cycles of scalar operations.

[0051] For example, consider a 2D convolution operation, which can be expressed as a multidimensional loop of scalar operations. These may need to be performed on input 2D input data with dimensions inputX(IX) and inputY(IY):

[0052] - (Input) Input Channels (IC) – represents the dimension of the input channels on which the operation is to be performed (in the example of an image, this could be three channels, one for each of the red, green, and blue input channels)

[0053] -(Input) Kernel Dimension X (KX) – the first dimension X of the 2D kernel;

[0054] -(Input) Kernel dimension Y(KY) - the second dimension Y of the 2D kernel;

[0055] -(Output)Output X(OX)-the first dimension of the output feature map of the convolution operation;

[0056] -(Output) Output Y (OY) – the second dimension of the output feature map of the convolution operation;

[0057] - (output) batch (N) – the batch dimension of operations, where operations will be processed in batches;

[0058] - (Output) Output Channels (OC) - Indicates the dimension of output channels to be produced for the 2D convolution operation.

[0059] In one proposed ordering, KY / KX can be considered the innermost dimension, and OC the outermost dimension.

[0060] For the 2D convolution operation example above, the operation to be performed can be represented as a "nested for loop" of scalar operations, as illustrated in the pseudo code listed below. In practice, when performing this operation, the processor needs to perform the operation across each of these dimensions by performing a multiply-accumulate operation (MAC) and then write its result to the accumulator (e.g., an accumulator buffer in hardware). After iterating through all of these dimensions, the 2D convolution is completed, and the contents of the accumulator therefore represent the result of the 2D convolution operation across the entire operating dimension.

[0061] for(output channel)

[0062] for(batch N)

[0063] for(output Y)

[0064] for(output X)

[0065] for(input channel)

[0066] for(kernel Y)

[0067] for(kernel X)

[0068] MAC

[0069] Write to accumulator

[0070] The inventors have recognized that the seven dimensions of the convolution operation can be used together to define an "operation space" in which a 2D convolution operation is to be performed. More specifically, the size of each dimension can be used to define a valid "bounding box" that defines the size of the operation space on which the operation is to be performed, the number of elements in each dimension. To illustrate this in more detail, consider an example in which a 3x3 (i.e., KX=3; KY=3) convolution operation with padding is to be performed on input data of dimensions IX=15; IY=15; N=1; and IC=32. This operation produces the following minimum and maximum index values, which represent the upper and lower bounds (i.e., sizes) (inclusive) of the dimensions of the convolution operation, as shown in Table 1:

[0071]

[0072]

[0073] Table 1

[0074] The output of a 2D convolution operation will have dimensions N = 1; OY = 15; OX = 15; OC = 64. These values represent the size of the output of the 2D convolution operation, but they do not alone fully represent the size of the operation required to generate that output. In order to fully represent the operation space of the operation, all dimensions of the operation are required, as shown in the table above. The shorthand representation of the dimensions of the 2D convolution operation is [OC N OY OX IC KY KX], and in this particular example, can be presented as the minimum and maximum index values as illustrated in the example above, i.e., [64 1 15 15 32 3 3].

[0075] Operations such as the convolution operations described above can be divided into blocks, each block representing a subset of the operation, where each dimension of the block covers a subset of the entire range of the corresponding dimension in the operation. In the example below, the 2D convolution of Table 1 is divided into multiple blocks by decomposing the operation into OY, OX, and IC dimensions. Dividing the operation into blocks involves dividing the operation space of the operation into multiple blocks, each block individually representing a part of the operation, but collectively representing the operation space. This block generation involves dividing the operation space into sub-blocks representing non-overlapping subsets of the dimensions in the operation space, which completely cover the operation space dimensions (e.g., the set of nested for loops shown above). In examples where the operation is to be divided into multiple blocks, the operation space is decomposed into sub-blocks based on a predetermined block size that defines a fixed size for each dimension of the operation. This fixed size block is referred to as a block quantum in this article. In the example below, the block sizes are as follows:

[0076] OC N OY OX IC KY KX Block Quantum 16 1 8 8 16 3 3

[0077] Table 2

[0078] In the block sizes above, the operation space is decomposed by splitting four of the seven dimensions of operation into two dimensions. In the example below, OY, OX, and IC are split into two, while OC is split into four. The following block illustrates a portion of a block that fully represents the operation space (only the first quarter of the OC dimension is represented):

[0079]

[0080]

[0081] Table 3

[0082] For a given block of the operation space, such as [OC N OY OX IC KY KX], it is possible to determine which input feature map coordinates are required to perform the operation of the block. In the example of a 2D convolution operation, the input feature map coordinates (and other input parameters) that the output feature map coordinates depend on can be defined as follows (stride X, Y = 1 (i.e., no stride); expansion X, Y = 1 (i.e., no expansion) and top, left padding = 1 (i.e., the input is padded):

[0083] - N = N; (where N is the batch number);

[0084] - IY = (OY * (stride Y)) + ((expansion Y) * KY) - top filling;

[0085] - IX = (OX * (stride X)) + ((dilation X) * KX) - left padding; and

[0086] -IC=IC;

[0087] Wherein strideX and strideY, dilationX and dilationY are the corresponding stride and dilation values in the X and Y dimensions when performing the convolution operation, and wherein top padding and left padding represent the corresponding top padding values and left padding values when performing the operation. When the above relationship is simplified for a stride and dilation value of 1 with zero padding, this can be more simply expressed as [N, OY + KY-1, OX + KX-1, IC]. These expressions for calculating the input feature map for the processing block can be expressed as an affine transformation as set forth in Table 4 below:

[0088] OC N OY OX IC KY KX Offset N 1 IY 1 1 -1 IX 1 1 -1 IC 1 1

[0089] Table 4

[0090] Therefore, for a given block in the operation space, a transformation (affine or semi-affine transformation) can be expressed to transform the block to determine the input feature map coordinate range required to perform the operation defined by the block. In the example of applying the above affine transformation to block number 2, the resulting input range of the input feature map index can be as shown in Table 5 below:

[0091] Minimum Maximum N 0 0 IY -1 8 IX 7 15 IC 0 15

[0092] Table 5

[0093] The affine transformation defined above can be used to individually represent the transformation required to define each of the input feature map (as described above), the output feature map, and the weights. General examples of each of the input feature map, the output feature map, and the weight transformation are listed in Tables 6 to 8 below:

[0094] Input transformation for 2D convolution

[0095] IFM OC N OY OX IC KY KX Offset N IY Stride Y Expansion Y - Top padding IX Stride X Expansion X -Left padding IC 1 1

[0096] Table 6 Weight transformation of 2D convolution

[0097] Weight OC N OY OX IC KY KX Offset OC 1 KY 1 KX 1 IC 1 1

[0098] Table 7 Output transformation for 2D convolution

[0099] OFM OC N OY OX IC KY KX Offset N 1 OY 1 OX 1 OC 1 1

[0100] Table 8

[0101] Therefore, it should be understood that the operation space defines the dimensions of the operations to be performed when performing a specific operation. The above examples are provided with respect to 2D convolutions, but the concept applies to all types of operations to be performed. For example, a similar transformation of the input and output of a transpose operation (e.g., transposing dimensions {0, 1, 3, 2}) can be derived as follows:

[0102] {0,1,3,2} transposed input transformation

[0103]

[0104]

[0105] Table 9 {0,1,3,2} transposed output transformation

[0106] Output Dimension 0 Dimension 1 Dimension 2 Dimension 3 Offset Dimension 0 1 Dimension 1 1 Dimension 2 1 Dimension 3 1 1

[0107] Table 10

[0108] Using the input transform on the input allows swapping dimensions 2 and 3 in the input transform matrix to perform a transpose operation. More generally, the input and output matrices can then be applied to a block in the operation space to determine the value ranges for the input and output of the operation. These determined value ranges represent the local segment space for the operation, which forms a local coordinate system on which the operation can be performed for that block of the operation space.

[0109] It is possible to implement lower and upper bounds on the task and operation spaces before running the transformation. Clipping may be functionally necessary for the edges of a tensor and allows for an operation space that is smaller than the full tensor. An operation space that is smaller than the full tensor is advantageous because it allows larger sequences of operations to be split into multiple independent tasks and optionally executed on separate cores.

[0110] In such a clipping model, code can be used to initialize the upper / lower bounds before performing the transformation, where low = op_space; high = op_space, and the initial coordinates default to op_space + block_size - 1. Before the transformation occurs, the coordinates are clipped to the actual operation space and task boundaries.

[0111] When considering the above Figure 1a When describing an acyclic graph data structure, the operation performed on each segment of the graph can be defined by a set of input and output transformation matrices for that operation. Thus, at least a portion of the acyclic graph can be represented by a chain of operations corresponding to each chain of segments connected by a pipeline. Furthermore, an operation space can be established for the operation chain.

[0112] Hardware implementation

[0113] As described above, a data structure in the form of a directed acyclic graph may include multiple sequential operations connected to each other to be performed in a chain. Other data structures in different graph forms may also be represented. Figure 1a The example hardware arrangement is illustrated to perform chain operations on at least a portion of a directed acyclic graph, but it should be understood that the example hardware arrangement can be used to perform chain operations on any type of graph.

[0114] Figure 1b Schematically, an example of a data processing system 600 is shown that includes a processor 630 that can act as a coprocessor or hardware accelerator unit for a host processing unit 610. It should be understood that the type of hardware accelerator for which the processor 630 can provide dedicated circuitry is not limited to the type of neural processing unit (NPU) or graphics processing unit (GPU), but can be dedicated circuitry for any type of hardware accelerator. GPUs may be well suited to performing certain types of arithmetic operations, such as neural processing operations, because these operations are generally similar to arithmetic operations that may be required when performing graphics processing work (but on different data formats or structures). In addition, GPUs generally support high levels of concurrent processing (e.g., support a large number of execution threads) and are optimized for data plane (rather than control plane) processing, all of which means that GPUs may be well suited to performing other types of operations.

[0115] That is, instead of using a completely separate hardware accelerator, such as a machine learning processing unit (such as an NPU) independent of the graphics processor, or being able to perform machine learning processing operations entirely using the GPU's hardware, dedicated circuitry can be incorporated into the GPU itself.

[0116] This means that hardware accelerator circuits incorporated into a GPU may be operable to take advantage of some of the GPU's existing resources (e.g., such that at least some of the GPU's functional units and resources may be efficiently shared, for example, between different hardware accelerator circuits), while still allowing improved (more optimized) performance compared to performing all processing in a general-purpose implementation.

[0117] As such, processor 630 may be a GPU adapted to include a number of specialized hardware resources, such as those described below.

[0118] In some examples, this can be particularly beneficial when performing machine learning tasks that themselves involve graphics processing work, because in this case, all associated processing can (and preferably is) performed locally on the graphics processor, thereby improving data locality and (for example) reducing the need for external communication along interconnects with other hardware units (e.g., NPUs). In this case, at least some of the machine learning processing work can be offloaded to the machine learning processing circuitry, thereby freeing up execution units to perform actual graphics processing operations as needed.

[0119] In other words, in some examples, providing machine learning processing circuitry within a graphics processor means that the machine learning processing circuitry is then preferably operable to perform at least some machine learning processing operations while other functional units of the graphics processor are concurrently performing graphics processing operations. In cases where machine learning processing is involved in a portion of an overall graphics processing task, this can therefore improve the overall efficiency (in terms of energy efficiency, throughput, etc.) of the overall graphics processing task.

[0120] exist Figure 1b In the embodiment of the present invention, processor 630 is arranged to receive a command stream 620 from a host processor 610, such as a central processing unit (CPU). Command stream 620 includes at least one command in a given sequence, each command to be executed, and each command can be broken down into multiple tasks, such as the tasks discussed in this document. These tasks can be self-contained operations, such as a given machine learning operation or graphics processing operation. It should be understood that other types of tasks may exist depending on the command.

[0121] The command stream 620 is sent by the host processor 610 and received by a command processing unit 640, which is arranged to schedule commands according to the sequence of commands in the command stream 620. The command processing unit 640 is arranged to schedule the commands and decompose each command in the command stream 620 into at least one task. Once the command processing unit 640 has scheduled the commands in the command stream 620 and generated a plurality of tasks for the commands, the command processing unit 640 issues each of the plurality of tasks to at least one compute unit 650a, 650b, each of which is configured to process at least one of the plurality of tasks.

[0122] The processor 630 includes a plurality of compute units 650a, 650b. Each compute unit 650a, 650b can be a shader core of a GPU that is specifically configured to undertake a plurality of different types of operations, however it should be understood that other types of specially configured processors may be used, such as a general-purpose processor configured with separate compute units (such as compute units 650a, 650b). Each compute unit 650a, 650b includes a plurality of components, and at least a first processing module 652a, 652b for performing tasks of a first task type and a second processing module 654a, 654b for performing tasks of a second task type that is different from the first task type. In some examples, the first processing module 652a, 652b can be a processing module for processing neural processing operations (such as those typically undertaken by a separate NPU). In these cases, the first processing module 652a, 652b is, for example, a neural engine. Similarly, the second processing modules 654a, 654b can be processing modules for processing graphics processing operations that form a predefined graphics processing operation set, which enables the implementation of a graphics processing pipeline, which can be referred to as a graphics processor. For example, such graphics processing operations include graphics compute shader tasks, vertex shader tasks, fragment shader tasks, pattern mosaic shader tasks, and geometry shader tasks. These graphics processing operations can all form part of a predefined operation set as defined by an application programming interface (API). Examples of such APIs include Vulkan, Direct3D, and Metal. Such tasks will typically be undertaken by a separate / external GPU. It should be understood that any number of other graphics processing operations may be capable of being processed by the second processing module.

[0123] Thus, the command processing unit 640 issues tasks of a first task type to the first processing module 652a, 652b of a given compute unit 650a, 650b, and issues tasks of a second task type to the second processing module 654a, 654b of the given compute unit 650a, 650b. The command processing unit 640 will issue machine learning / neural processing tasks to the first processing module 652a, 652b of a given compute unit 650a, 650b, where the first processing module 652a, 652b is optimized to handle neural network processing tasks, for example, by efficiently handling a large number of multiply-accumulate operations. Similarly, the command processing unit 640 will issue graphics processing tasks to the second processing module 654a, 654b of a given compute unit 650a, 650b, where the second processing module 652a, 654a is optimized to handle such graphics processing tasks. In some examples, both the first and second processing modules 652a, 652b may be neural processing tasks issued to the first processing module 652a, 652b as a neural engine. Such neural processing tasks may involve processing tensors representing, for example, feature maps using weights associated with layers of a neural network.

[0124] In addition to including first processing modules 652a, 652b and second processing modules 654a, 654b, each computing unit 650a, 650b also includes a memory in the form of a local cache 656a, 656b for use by the corresponding processing module 652a, 652b, 654a, 654b during task processing. An example of such a local cache 656a, 656b is an L1 cache. The local cache 656a, 656b can be, for example, synchronous dynamic random access memory (SDRAM). For example, the local cache 656a, 656b can include double data rate synchronous dynamic random access memory (DDR-SDRAM). It should be understood that the local cache 656a, 656b can include other types of memory.

[0125] The local cache 656a, 656b is used to store data related to the task being processed by the first processing module 652a, 652b and the second processing module 654a, 654b on a given computing unit 650a, 650b. It can also be accessed by other processing modules (not shown) that form part of the computing unit 650a, 650b associated with the local cache 656a, 656b. However, in some examples, it may be necessary to provide access data associated with a given task being executed on the processing module of another computing unit (not shown) of the processor 630. In such an example, the processor 630 may also include a storage device 660, such as a cache (such as an L2 cache), for providing access to data used to process tasks executed on different computing units 650a, 650b.

[0126] By providing a local cache 656a, 656b, tasks issued to the same compute unit 650a, 650b can access data stored in the local cache 656a, 656b, regardless of whether they form part of the same command in the command stream 620. The command processing unit 640 is responsible for assigning command tasks to a given compute unit 650a, 650b so that they can make the most efficient use of available resources (such as the local cache 656a, 656b), thereby reducing the number of read / write transactions required to memory external to the compute unit 650a, 650b (such as the storage device 660 (L2 cache) or higher-level memory). One such example is that a task issuing one command to a first processing module 652a of a given compute unit 650a can store its output in the local cache 656a so that it can be accessed by a second task issuing a different (or the same) command to a given processing module 652a, 654a of the same compute unit 650a.

[0127] One or more of the command processing unit 640, the computing units 650a, 650b, and the storage device 660 may be interconnected using a bus. This allows data to be transferred between the various components. The bus may be or include any suitable interface or bus. For example, Advanced Microcontroller Bus Architecture Interfaces, such as Advanced Extensible Interface (AXI).

[0128] Figure 2 is a schematic diagram of a neural engine 700, which in this example is used as a neural engine according to Figure 1b The neural engine 700 includes a command and control module 710. The command and control module 710 receives data from the command processing unit 640 ( Figure 1b ) receives tasks and also acts as an interface to storage devices external to the neural engine 700 (such as local caches 656a, 656b and / or L2 cache 660), which are arranged to store data to be processed by the neural engine 700, such as data representing tensors or data representing stripes of tensors. In the context of the present disclosure, a stripe is a subset of a tensor where each dimension of the stripe covers a subset of the full range of the corresponding dimension in the tensor. The external storage device can additionally store other data for configuring the neural engine 700 to perform specific processing and / or data to be used by the neural engine 700 to implement the processing (such as neural network weights).

[0129] The command and control module 710 interfaces with a handling unit 720, which is, for example, a traversal synchronization unit (TSU). In this example, each task corresponds to a strip of tensors that will be operated on in a sequence of operations according to at least a portion (e.g., a subgraph) of the acyclic graph representation of the neural network. A tensor represents, for example, a feature map for processing using a neural network. A neural network typically includes a series of processing layers, with the output from each layer being used as the input to the next layer. Each layer processes an input feature map, for example, by operating on the input feature map to generate an output feature map, which is used as the input feature map for the next layer. The term "feature map" is generally used herein to refer to an input feature map or an output feature map. The processing performed by a given layer can be considered to correspond to an operation.

[0130] In this example, the processing unit 720 divides the data representing the strip of the feature map into a plurality of data blocks, each data block representing a corresponding portion of the feature map. The processing unit 720 also obtains task data defining an operation selected from an operation set including a plurality of operations from a memory external to the neural engine 700, such as the L2 cache 660. In this example, the operations are structured as a chain of operations representing a sequence of layers of a neural network. The processing unit 720 assigns the data block as input to one of the operations.

[0131] The handling unit 720 coordinates the interaction of the internal components of the neural engine 700, including the weight fetch unit 722, input reader 724, output writer 726, direct memory access (DMA) unit 728, dot product unit (DPU) array 730, vector engine 732, transform unit 734, accumulator buffer 736, and storage 738 for processing of data blocks. The handling unit 720 tracks data dependencies across functional units. If all input blocks are available and space is available in the storage 738 of the neural engine 700, the handling unit 720 in the functional unit initiates processing. The storage 738 can be considered a shared buffer because the various functional units of the neural engine 700 share access to the storage 738.

[0132] In the context of a directed acyclic graph representing operations to be performed, each internal component that operates on data can be considered to be one of two types of components. The first type of component is an execution unit that is mapped to a segment that performs a specific instance of an operation within the acyclic graph (and is identified as such within the neural engine 700). For example, the weight acquisition unit 722, input reader 724, output writer 726, dot product unit array 730, vector engine 732, and transform unit 734 are each configured to perform one or more predetermined and fixed operations on the data they receive. Each of these segments can be uniquely identified with an identifier, and each execution unit can also be uniquely identified.

[0133] Similarly, all physical storage elements within the neural engine (and in some cases portions of those physical storage elements) can be considered to be uniquely identified within the neural engine. In the context of acyclic graphs, the connections between segments in an acyclic graph representing a neural network are also referred to as pipelines. These pipelines can also be mapped to uniquely identified physical storage elements in the neural engine. For example, the accumulator buffer 736 and the storage device 738 (and portions thereof) can each be considered to be storage elements that can be used to store data for pipelines within the acyclic graph. The pipelines act as connections between segments (as performed by execution units) to enable the sequence of operations defined in the acyclic graph to be chained together within the neural engine 700. In other words, the logical data flow of the acyclic graph can be mapped to the physical arrangement of execution units and storage elements within the neural engine 700. Under the control of the handling unit 720, execution can be scheduled on the execution units, and data can be passed between the execution units via storage elements according to the mapping, so that chained operations of the graph can be performed without the need to write data memory to the outside of the neural engine 700 between executions. The handling unit 720 is configured to control and dispatch work representing execution of operations of a graph on at least a portion of the data provided by the pipeline.

[0134] The weight fetch unit 722 fetches weights associated with the neural network from the external storage device and stores the weights in the storage device 738. The input reader 724 reads data to be processed by the neural engine 700, such as a block of data representing a portion of a tensor, from the external storage device. The output writer 726 writes the data obtained after processing by the neural engine 700 to the external storage device. The weight fetch unit 722, the input reader 724, and the output writer 726 interact with the external storage device (e.g., the local cache 656a, 656b, which may be an L1 cache, such as a load / store cache) via the DMA unit 728.

[0135] Data is processed by the DPU array 730, the vector engine 732, and the transform unit 734 to generate output data corresponding to the operations in the acyclic graph. The results of each operation are stored in a specific pipeline within the neural engine 700. The DPU array 730 is arranged to perform one or more operations associated with a dot product operation between two operands, such as between a weight array and a corresponding data block (e.g., representing a portion of a tensor). The vector engine 732 is arranged to perform element-wise operations, such as applying a scaling parameter to scale the output of the dot product calculated by the DPU array 730. Data generated during the processing performed by the DPU array 730 and the vector engine 732 can be transferred for use in a temporary stage in the accumulator buffer 736, which acts as a pipeline between the previous operation and the subsequent operation, and the DPU array 730 or the vector engine 732 (or another different execution unit) can retrieve the data from the temporary stage for further processing as needed.

[0136] The transform unit 734 is arranged to perform intra-block transforms, such as dimension broadcasting or axis swapping. The transform unit 734 obtains data from a pipeline such as the storage device 738 (e.g., after being processed by the DPU array 730 and / or the vector engine 732) and writes the transformed data back to the storage device 738.

[0137] To efficiently utilize the available storage 738 within the neural engine 700, the handling unit 720 determines an available portion of the storage 738 that is available during execution of a portion of a first task (e.g., during processing of a data block associated with the first task by the DPU array 730, the vector engine 732, and / or the transform unit 734). The handling unit 720 determines a mapping between at least one logical address associated with data generated during execution of a second task (e.g., by processing a data block associated with the second task by the DPU array 730, the vector engine 732, and / or the transform unit 734) and at least one physical address of the storage 738 corresponding to the available portion. The logical address is, for example, a global address in a global coordinate system. Thus, by changing the physical address corresponding to a given logical address, the handling unit 720 can efficiently control the use of the storage 738 without requiring changes to the software defining the operations to be performed, as the same logical address can still be used to refer to a given element of a tensor to be processed. Based on the mapping, the handling unit 720 identifies at least one physical address corresponding to the at least one logical address, such that the data associated with the logical address is stored in the available portion. The handling unit 720 can perform a mapping process according to any example herein.

[0138] It should be understood that in an operation graph, there need not be only a single instance of a particular type of operation. For example, multiple instances of a convolution operation may be present in an operation graph. In the example hardware arrangement described above, there may be only a single convolution engine. Therefore, it should be understood that there does not need to be a direct 1:1 mapping between the operations in the graph (segments) and the execution units, and similarly, there does not need to be a direct 1:1 mapping between the pipeline and the storage element. Specifically, a single execution unit can be configured to perform different instances of the convolution operation (e.g., a first segment and a second segment) at different time instances. Similarly, an input reader may need to read data that is part of different segments in the graph. The same is true for storage elements and pipelines.

[0139] All storage devices in the neural engine 700 can be mapped to corresponding pipelines, including lookup tables, accumulators, etc. Some storage devices can be relatively fixed-purpose, for example, if the hardware is limited to one convolution operation per graph, the accumulator buffer may also be limited to mapping to one pipeline, and the scale / bias / shift buffer may be limited to mapping to one pipeline; however, both may be double-buffered. If the neural engine supports 2 lookup tables (LUTs), up to 2 pipelines can be used to target the LUTs to avoid the need to thrash the LUT storage devices; the LUT pipelines can then be single-buffered. All other pipelines can be mapped to a common shared buffer (or portion thereof) with fewer restrictions. The width and height of the pipelines can also be programmable, resulting in a highly configurable mapping between pipelines and storage elements within the neural engine 700.

[0140] Dependencies on inputs imply an execution ordering for each segment. Memory load operations have no data dependencies (unless they are aggregate operations) and are therefore implicitly placed early in the graph. Consumers of the pipeline created by a memory read are implicitly placed after the memory read. Memory store operations are placed near the end of the graph because they do not create a pipeline for consumption by other operations. Therefore, the execution order of the operation chain is handled by the handling unit 720, as will be explained in more detail later.

[0141] Figure 3 A system 800 for distributing treatment data and, in some examples, generating multiple input data blocks for processing is schematically illustrated.

[0142] System 800 includes a host processor 810, such as a central processing unit or any other type of general purpose processing unit. Host processor 810 issues a command stream comprising a plurality of commands, each command having a plurality of tasks associated therewith.

[0143] System 800 also includes a processor 830 that can communicate with Figure 1bThe processor 830 is similar to or identical to the processor 630 of the host computer 800 and may include at least some of the components of the above-described method and / or be configured to perform the above-described method. The processor 830 includes at least a plurality of computing units 650a, 650b and a command processing unit 640. Each computing unit may include a plurality of processing modules, each processing module being configured to perform at least one type of operation. The system 800 may also include at least one additional processor (not shown), which may be the same as the processor 830. The processor 830 and the host processor 810 may be combined into a system on a chip (SoC) or combined onto multiple SoCs to form one or more application processors.

[0144] The system 800 also includes a memory 820 for storing data generated by tasks external to the processor 830 so that other tasks running on other processors can easily access the data. However, it should be understood that due to the distribution of tasks as described above, external memory usage will be less used, so that tasks that need to use data generated by other tasks or need the same data as other tasks will be assigned to the same computing unit 650a, 650b of the processor 830 to maximize the use of the local cache 656a, 656b.

[0145] In some examples, system 800 may include a memory controller (not shown), which may be a dynamic memory controller (DMC). The memory controller is coupled to memory 820. The memory controller is configured to manage the flow of data into and out of the memory. The memory may include main memory, or "primary memory." The memory may be external memory, in that the memory is external to system 800. For example, memory 820 may include "off-chip" memory. The memory may have a larger storage capacity than the local cache of processor 830 and / or host processor 810. In some examples, memory 820 is included in system 800. For example, memory 820 may include "on-chip" memory. Memory 820 may, for example, include a magnetic disk or optical disk and a disk drive or solid-state drive (SSD). In some examples, memory 820 includes synchronous dynamic random access memory (SDRAM). For example, memory 820 may include double data rate synchronous dynamic random access memory (DDR-SDRAM).

[0146] One or more of the host processor 810, processor 830, and memory 820 may be interconnected using a system bus 840. This allows data to be transferred between the various components. The system bus 840 may be or include any suitable interface or bus. For example, Advanced Microcontroller Bus Architecture Interfaces, such as Advanced Extensible Interface (AXI).

[0147] Neural Engine Program Descriptor (NED)

[0148] The neural engine 700 receives tasks from the command processing unit 640 to perform operations from the graph, such as those described above with reference to Figure 1a The neural engine 700 is configured to execute operations selected from a basic set of operations that define an operator set. An example of such an operator set is the Tensor Operator Set Architecture (TOSA) basic inference profile, which defines a set of operations that can be used together to define operations for various neural network operations. An exception to the TOSA operator set is the control flow operations that can be implemented through the command stream processed by the command processing unit 640. It should be understood that the processor 630 may have multiple neural engines and therefore multiple tasks can be issued concurrently to different neural engines.

[0149] In an example implementation, tasks issued by the command processing unit 640 for execution by the neural engine 700 are described by task data, which is embodied in this example by a neural engine program descriptor (NED), which is a data structure stored in memory and retrieved by the neural engine when executing the tasks issued by the command processing unit. The NED describes at least a portion of a complete graph of operations (segments) to be executed when executing an operation graph (e.g., representing a neural network). As described above, segments are mapped to various hardware execution units within the neural engine 700 and essentially represent instantiations of specific operators at locations within the graph. In one example, these segments are described by specific "elements" that collectively define operations that form part of the NED. In addition, the NED has an unordered list of pipelines (graph vertices) and an unordered list of segments / operations (graph nodes). Each operation specifies its input and output pipelines, thereby generating adjacencies of the operations in an acyclic graph to which the specific operations are connected.

[0150] The example NED includes a NED structure that includes a header with each element corresponding to a segment in the diagram. The NED describes various requirements for the ordering, number, and relationship of these segments and pipelines. In one implementation, each execution unit and each storage element (or portion of a storage element) of the neural engine 700 has a subdescriptor definition that defines how the execution unit / storage element can be configured to implement a specific segment or pipeline in the diagram. Examples of hardware units and their corresponding elements are shown below:

[0151] -Weight Fetch (WF): NEDWeightFetchElement

[0152] - Input Reader (IR): NEDInputReaderElement

[0153] -Output Writer (OW): NEDOutputWriterElement

[0154] -Convolution Engine (CE): NEDConvolutionEngineElement

[0155] -Transform Unit (TU): NEDTransformUnitElement

[0156] -Vector Engine (VE): NEDVectorEngineElement

[0157] Thus, the NED can specify the execution unit, or in other words, specify a compatible execution unit for each operation. In an embodiment, there can be more than one execution unit of a given type, such as an InputReader can have two command queues that can operate concurrently. The NED can specify which queue to assign so that there is a 1:1 relationship between what the NED specifies and the physical hardware it targets.

[0158] The data flow and dependencies of the task graph are described by pipelines, which are described in another element that is part of NED:NEDPipeElement. Pipelines are used to represent data storage elements within the neural engine 700 and describe the relationship between segments (operations) in a producer-consumer relationship: the output destination pipeline (e.g., pipeline number) and each input source pipeline (e.g., pipeline number) of each segment are defined in the NED element of the NED. A pipeline has only a single producer, but may have multiple consumers. A pipeline can be mapped to one of several different locations (e.g., storage elements in the neural engine 700), but not all locations may be suitable for different segment operations. It should be understood that in some arrangements, a pipeline may be mapped to only a portion of a storage element - for example, multiple physical buffers, allowing it to describe double buffering (for example) behavior between its producers and consumers. The output data generated by a segment and stored in a pipeline is equivalently referred to as a (data) block and a (virtual) buffer, where a data block occupies one physical buffer location. Regardless of the location, the pipeline can be independent of the wider memory system associated with the neural engine 700 and the processor 630, and store the data using the output writer elements of the neural engine 700.

[0159] In some arrangements, the NED can be configured so that the same pipeline is used for multiple inputs, where any relevant usage constraints (such as format or position) are met. For example, an element-wise multiplication might have the same pipeline for both input operands in order to square the input.

[0160] In some embodiments, sections such as InputReader and WeightFetcher do not have pipelines, but instead their data comes from external memory, such as external cache or DRAM. In contrast, some sections, such as OutputWriter, do not have output pipelines. In this case, their data is written to external memory.

[0161] For a segment to run, it must have all the appropriate buffers available for its input source pipeline. A segment may produce a new buffer in its output destination pipeline, so there must be room in the pipeline for that new buffer. In the case of a reduce operation (e.g., a convolution), a segment may repeatedly read back and update the previous buffer it generated. Therefore, for a reduce operation, due to this update process, there is a difference between when the reduce operation first generates an output buffer and when the reduction has completed and the output buffer is fully available. In other words, there is a point in time when the output buffer exists in the input pipeline of a subsequent operation but is not yet ready to be consumed by the subsequent operation. The neural engine 700 is responsible for tracking all of these dependencies, where buffers are tracked like FIFO entries, but buffers are only available to consumers when the producer completes any sequence of reductions, and buffers are only released when all consumers have completed the operations that depend on them.

[0162] In one example, the task graph has a directed acyclic data flow. In this way, it is illegal to use an input pipeline as a destination pipeline in the same section, or to have any form of cycle within the graph. Note that a reduce operation will read from and write to the buffer of its output destination pipeline, but this is still acyclic behavior; for example, a convolution engine can repeatedly accumulate into the same accumulator buffer.

[0163] In this example implementation, the neural engine is stateless between tasks: all control state is encapsulated in the task's NED, and all data is encapsulated in the pipeline defined by the NED. There is no pipeline sharing between tasks, and therefore no architectural data sharing between tasks within the neural engine 700. Data reuse and sharing is achieved only through memory using output writers in the previous task and input readers in the next task. The neural engine will cache memory descriptors, including NEDs, between tasks; this cache is invalidated each time a complete neural workload is completed (e.g., the entire neural network, not just the subgraph associated with a specific task). However, it should be understood that this is merely an example implementation.

[0164] The NED is divided into multiple data structures that can appear contiguously in memory to be read by the neural engine 700. In this example implementation, the NED header defines the dimensions of the operation space of the operations to be performed. Specifically, the NED header defines the total size of the NED (e.g., the number of bytes used to represent the NED) and a count of the number of segments and pipelines present in the graph.

[0165] For each segment and pipeline in the graph, a count of the corresponding mapped sub-descriptor element type is represented in the NED header. For example, where a graph (or sub-graph) contains multiple segments, each of those segments will be executed on a specific compatible execution unit of the neural engine 700. Therefore, for each segment, elements of the appropriate type are counted in the NED header in order to represent the hardware requirements needed for the execution of the call graph. For example, for a segment that defines a convolution operation, a corresponding configuration and call of a convolution engine execution unit will be required. Similar counts of instantiations of weight acquisition and input read execution units are counted based on the presence of segments that use those operations. This is reflected in the counts of weight acquisition and input reader elements in the NED header relative to the weight acquisition and input reader units associated with the neural engine 700.

[0166] The NED also contains information describing any diverging or converging branches between segments and pipelines. For example, for each pipeline in the diagram, the NED identifies the number of producers and consumers associated with that pipeline.

[0167] Thus, the NED header essentially identifies the operation space and a count of all instances of the segments and pipelines defined by the NED (for each type of hardware element that will be allocated to instantiate the segments or pipelines required to execute the graph (or subgraph)). Illustrative examples of at least a portion of the fields stored in the NED header are set forth below. In addition to the NED header, the NED also includes a subdescriptor element for each instance of a segment and / or pipeline (defining the configuration of the execution unit or storage element to operate as a segment or pipeline). Each subdescriptor element defines the configuration of the associated hardware element (execution unit or storage element) required to execute the segment and / or pipeline.

[0168] Examples of at least some of the fields in the NED header are as follows:

[0169] Field Minimum Maximum Size of the operation space of dimension 1 - - The size of the operation space in dimension 2 - - Size of the operation space in dimension 3 - - The size of the operation space of dimension 4 - - The size of the operation space in dimension 5 - - The size of the operation space of dimension 6 - - The size of the operating space of dimension 7 - - Weight acquisition and number of decoding sections 0 1 Enter the number of reader segments 1 7 Output write segment number 1 7 Number of convolution engine sections 0 1 Number of transform unit segments 0 7 Number of vector engine sections 0 7 Number of pipes 1 15

[0170] Theoretical minimum and maximum operation space dimension sizes can be defined at compile time based on the configuration of the neural engine, specifically so that operations of a task (e.g., a subgraph) can be performed without storing intermediate data in memory elements external to the neural engine. A practical method for defining tasks and their corresponding operation spaces is described in more detail later.

[0171] The NED header may also include a pointer to each sub-descriptor element to enable the specific configuration of each element to be read by the handling unit 720 .

[0172] As described above, each instance of a subdescriptor element defines the configuration of the hardware element (eg, execution unit or storage element) to which it is associated. The following description will provide an example subdescriptor for a convolution engine.

[0173] In one example, a convolution engine is an execution unit that is configured to, when called, perform a convolution or pooling operation selected from one or more convolution operations for which the convolution engine is configured. One such example is a 2D convolution operation as described above. In the example of the 2D convolution operation above, the operation space is 7D, i.e., [oc, n, oy, ox, ic, ky, kx].

[0174] Field Stride X and Stride Y Dilate X and Dilate Y The type of operation (e.g., which type of convolution operation to perform) Enter the width and height Left padding Top padding Source 0 pipeline (input feature map pipeline) Source 1 pipeline (weight pipeline) Destination Pipeline

[0175] In this example, the operation type may take the form of one of pooling (average or max pooling), 2D convolution, or 2D depthwise convolution, for example. The source 0 pipeline field may identify which pipeline the convolution engine should read the input feature map data from, which may be, for example, a specific portion of a shared buffer. Similarly, the source 1 pipeline field may indicate which (different) portion of the shared buffer the weight data is to be retrieved from. Finally, the destination pipeline may indicate that the accumulation buffer will act as a pipeline for the output of the operation performed by the convolution engine. By identifying a segment-specific source and / or destination pipeline with a unique identifier in the task definition (NED), any previous or subsequent segments are implicitly connected and ordered. Referencing a destination pipeline for a different segment as another subdescriptor element of the source pipeline will inherently read that data, and the buffer allocation for that destination pipeline may only be released when all dependencies have been resolved (e.g., the segments that depend on that portion of the accumulation buffer have all finished reading that data).

[0176] Similar subdescriptor elements exist for all sections based on the execution unit being configured to perform an operation. For example, a subdescriptor element may define the destination and source pipelines, a pointer to the transformation from the operation to the section space, and the operation mode of the section.

[0177] In this example implementation, pipelines represent all storage within the Neural Engine: all allocation and memory management is handled by the NED pipeline definitions for tasks and the traversal of sections that create and use these pipelines. There is no sharing of pipelines between tasks, and therefore no architectural data sharing between tasks within the Neural Engine. A subdescriptor element is defined in the NED for each pipeline in the diagram. An example pipeline subdescriptor is shown below:

[0178]

[0179]

[0180] As will be described in more detail later, these descriptors are used to configure the hardware elements when a call is triggered by the handling unit 720 .

[0181] Neural Engine Dimensions and Iterations

[0182] A Neural Engine task describes a 4D bounding box (dimensions 0-3) that should be operated on by the segment operations of the NED-defined graph to which the task provides a pointer. In addition to describing this graph, NED also defines four more dimensions (dimensions 4-7), making a total of 8 dimensions of operation space. The bounding boxes of the first four dimensions are sub-regions of the full size of those dimensions, with different tasks and / or work covering other sub-regions of those dimensions. Figure 4 and Figure 5 As illustrated, the command processing unit 640 can issue different tasks to different neural engines. Therefore, when generating an NED or defining a task, the dimensions are 0-3. The last four dimensions are fully described in the NED and are therefore fully covered in each task. The NED also defines the increment size for each of these eight dimensions to be stepped through, called the block size. Based on this 8D operation space, the execution graph can be considered a series of nested loops.

[0183] This divides the execution of a task's operation space into a series of blocks, where sections are called on a block-by-block basis, operating on a block's worth of data in each source and destination pipeline. Thus, defining a common operation space in a coordinate system with, for example, eight dimensions can provide a low-complexity pattern for executing any task that involves data operations, rather than relying on fixed functions per task type, which can carry a significant risk of missing necessary pattern combinations. By defining a common operation space in a coordinate space, the complexity of chaining together multiple operations to be performed on the data and coordinating the execution of these functions can be reduced. The operation space dimensions have no specific interpretation until they are projected into the space of a specific task.

[0184] The number of dimensions in use depends on the graph and its operations; not every segment will run increments on every dimension. For example, a convolution operation has a 7D operation space, but only a 4D output space through which the convolution operation increments and accumulates the output; therefore, the VE scaling operation after the convolution operation can only run increments on the first four dimensions. This relationship is described by two variables, the number of operation space dimensions that trigger the increment for each segment, dims_inc_run (the "dimension increment run" value), and the number of operation space dimensions that generate a new block for each pipeline, dims_inc_buf (the "dimension increment buffer" value), both of which are encoded in their corresponding NED elements. Both fields are specified as counting dimensions from the outermost dimension number 0 to the innermost dimension number 7.

[0185] dims_inc_run specifies how many operation space dimensions trigger the call of this section when the operation space dimension increases. The following illustrates an example use of dims_inc_run:

[0186] ○0: This section is independent of the operation space and is therefore called only once for this task;

[0187] ○1: This section can depend on operation space dimension number 0 and is called for each operation space step through dimension number 0; and

[0188] ○8: This section can depend on all action space dimensions and is called for each action space step.

[0189] dims_inc_buf specifies how many operation space dimensions generate new blocks in the pipeline when these dimensions are incremented in the producer segment, effectively defining how many blocks the pipeline generates over the entire task duration;

[0190] If the value of dims_inc_buf is k (where k>0), then pipe.blocks=dim[0].blocks*dim[1].blocks*……*dim[k-1].blocks;

[0191] If the value of dims_inc_buf is k (where k=0), then the pipeline has only a single block.

[0192] For simple operations, dims_inc_run will be equal to dims_inc_buf for all source input pipes and output destination pipes, but for more complex operations, dims_inc_run may be larger.

[0193] In the case where dims_inc_run> dims_inc_buf:

[0194] For source pipelines: This relationship between the fields indicates the reuse of buffers across one or more operational spatial dimensions, with the difference between the two values specifying the number of reused dimensions. In this context, reuse means broadcasting data across additional dimensions: the buffer in the neural engine's internal memory is consumed multiple times. For example, feature maps input to a convolution operation are typically reused based on the convolution engine's weight kernel x and y dimensions.

[0195] Meanwhile, for the destination pipeline, this relationship indicates a reduction in the buffer set for one or more operation space dimensions, with the difference between the two values specifying the number of dimensions reduced. In this context, reduction means that data from additional inner operation space dimensions is accumulated in a smaller number of outer operation space dimensions (where the segment reads back and updates its output buffers over multiple calls). For example, a vector block reduction operation will result in a smaller number of buffer increments.

[0196] In the case of a pipeline with multiple consumers, there is no relationship between those consumers, and there are no restrictions or requirements on the value of dims_inc_run for one consumer relative to the other consumers.

[0197] In the examples described herein, the neural engine's handling unit is responsible for iterating through this 8D operation space for each segment described in the NED graph. The handling unit uses two values, dims_inc_run and dims_inc_buf, to determine which deltas are relevant and to properly manage the dependencies between the segment and its pipeline. Each segment operates in its own local coordinate space (called the segment space), and the handling unit is responsible for transforming each relevant operation space block (related by deltas in the run dimension) into this segment space. In the examples described herein, this transformation can be programmatic and can be described by a small routine in a dedicated (or general) ISA that is executed for each block before calling the segment.

[0198] The handling unit may synchronize the execution of multiple different parts of these nested for loops in parallel, and therefore needs to keep track of where in the loop a component's functionality should be called, and where in the loop to generate data that a subsequent component may need (based on a partially ordered set of data structures). In order to achieve this in a flexible way that still allows for simple hardware implementation, two types of dimensions are specified in each data structure.

[0199] In some embodiments, each data structure includes N vectors of binary values that indicate, for each of the N dimensions of the coordinate space, whether a change in the coordinates of that dimension causes the associated component's function to execute and store data in storage (DIMS_INC_RUN) when executing a task. Effectively, this allows the behavior of each component in each dimension to be encoded as a multi-hot behavior vector. Behaviors can include, for example, reuse, recalculate, reduce, output, and unmap / once.

[0200] In some types of tasks involving data manipulation, data is frequently "reused" multiple times across some number of dimensions. For example, in an operation in a neural network, the same weight may be applied to multiple elements in the batch, X, and Y dimensions of a feature map, but the weights are unique across the input and output channel dimensions. To inform the processing unit of the details of each function (based on the task at hand), each data structure can indicate the dimensions of the coordinate space whose coordinate changes when performing the task cause the associated component's function to execute.

[0201] To save bits and reduce complexity, each data structure may instead include a first number 402 (and the following in conjunction with Figure 5 A second number further described) indicates the dimension of the coordinate space, the coordinate changes of which in performing the task result in the execution of the function of the associated component, such as a number between 0 and N (the number of dimensions in the operating space, in Figure 4 In the example, 8). If this number is equal to 0, the section is called once per task (e.g., at the beginning or end of an iteration over an N=>1-dimensional coordinate space). This could, for example, correspond to a function that loads a table to be used in subsequent subtasks, regardless of the coordinates or dimensions. At the opposite extreme, the value could be equal to N, meaning that the section's functionality is executed on each iteration over each dimension.

[0202] exist Figure 4 In , the shaded elements correspond to the dimensions (for each part) where a change in coordinates causes a function to be executed (e.g., DIMS_INC_RUN). Figure 4 As can be seen in the data structures described as "IFM load", "weight load", and "conv", the functions associated with the corresponding components are executed when any dimension is incremented. "Bias" and "Scale load" are only called (executed) when the batch or OFM channel is incremented. "Scale" and "OFM write" sections are called when the batch, OFM C, OFM Y, or OFM X are incremented.

[0203] In some types of tasks that involve data manipulation, the function performed on the data may result in a smaller number of dimensions being output. For example, Figure 4 As can be seen in the figure, the 2D convolution operation (conv) iterates over the batch size (N), output feature map height (OFM Y), output feature map width (OFM X), input channels (IFM C), output channels (OFM C), kernel X (KX), and kernel Y (KY). However, it reduces these seven dimensions to four dimensions (N, OFM X, OFM Y, OFM C) in its output. Similarly, so-called "reduction operators" such as ReduceSum iterate over tensors and sum the data over one or more dimensions, thereby producing an output tensor with fewer dimensions than the input tensor. In order to inform the processing unit of the details of each function (based on the task at hand), each data structure can indicate the dimensions of the coordinate space for which the coordinate changes of said dimensions when performing the task cause the function of the associated component to store data in a storage device, where the stored data is ready to be consumed by the function of the component associated with the subsequent data structure in the partially ordered set of data structures, or to store the final output data of the task. In other words, when such a dimension increases (i.e., the coordinates change), new buffers are made available in the pipeline for use by functions of components associated with subsequent data structures in the partially ordered set of data structures, or the final data of the task (i.e., the currently processed portion of the bounding box) is stored in the output buffer.

[0204] In some embodiments, each section includes N dimension specifications that indicate, for each of the N dimensions of the coordinate space, the effect on storage of each dimension when a change in coordinates in that dimension occurs during execution. To conserve bits and reduce complexity, each data structure may instead include a second number indicating the dimension of the coordinate space for which a change in coordinates in that dimension during execution of the task causes the function of the associated component to store data in storage, the stored data being ready for consumption by the function of the component associated with a subsequent data structure in the partially ordered set of data structures, or to store final output data for the task. The second number ( Figure 5 502 in the figure) can be a number between 0 and N (the number of dimensions in the operating space, in Figure 4 Since storage of data may only occur when the function of the associated component is performed, the second number may be equal to or smaller than the first number.

[0205] A second number of 0 indicates that the segment (data structure) produces exactly one output block, which is ready for consumption by the functionality of the component associated with the subsequent data structure / segment. A second number of 1 indicates that the segment produces output (ready to be consumed) only when the operation space dimension 0 is incremented (coordinates changed). A second number of 2 indicates that the segment produces output (ready to be consumed) when either the operation space dimension 0 or 1 is incremented, etc. In the case where the second number is less than the first number, this indicates a reduction operation.

[0206] exist Figure 5 In the example, the shaded elements correspond to the dimensions (for each data structure) for which a change in coordinates causes the functionality of the associated component to store data in the storage device (with Figure 4 Instead, the diagram involves causing a function to execute, such as DIMS_INC_BUF), where the stored data is ready for consumption by a function associated with a subsequent data structure in a partially ordered set of data structures, or to store the final output data of a task. Figure 5 As can be seen in , for the data structures described as "IFM Load" and "Weight Load", when any dimension is incremented, the function associated with the corresponding component stores data ready to be consumed by the function of the component associated with the subsequent data structure in the partially ordered set of data structures. "Bias" and "Scale Load" only store data ready to be consumed by subsequent functions when batch or OFM channel is incremented. "Scale" stores data ready to be consumed by subsequent functions when batch, OFMC, OFM Y, or OFM X is incremented. "OFM Write" stores the final output data of the task when batch, OFM C, OFM Y, or OFM X is incremented. For "Conv", IFM C, Kernel X, and Kernel Y are marked as the dimensions that the associated function will execute (see Figure 4 ), but not the dimensions in which the associated function stores the data ready to be consumed. This means that these three dimensions are the so-called reduced dimensions, and in the output of Conv the seven dimensions are reduced to four.

[0207] In the example, if the action space dimensions are labeled ( Figure 4 ) is a dimension whose coordinate change results in the execution of the function of the associated component, but is not marked ( Figure 5 ) is the dimension in which the function of the component that generated the associated component's input buffer stores data in storage, which indicates that the execution segment is reusing the input buffer. For example, if we have segment A->B, and A's storage dimension is smaller than B's run dimension, then B reuses the input buffer written by A. On the other hand, if B's storage dimension is smaller than B's run dimension, then B will perform a reduction on the output buffer.

[0208] The described data structures can be generated, for example, by a compiler connected to a processor, where the compiler is configured to generate code for execution by the processor. The execution of a Neural Engine task is defined by two separate iterative processes implemented in a processing unit. In one process, the processing unit iteratively steps through the task's operation space in units of blocks defined by the NED's block size. In the other process, the processing unit iteratively steps through the dataflow graph defined by the NED and, where permitted by the aforementioned dimensionality rules, transforms each block into the associated segment space before calling the segment's execution unit with the transformed block by issuing call data.

[0209] In general, for most cases, the two processes are defined as architecturally independent in the examples described in this article. This means that the execution of any given block is well-defined and completely isolated from the state of any other block or processing unit operation space iteration. The execution of blocks that do not conform to this operation space iteration and transformation will run to completion, but the results will not provide meaningful results with respect to the full operational definition of the Tensor Operator Set Architecture (TOSA).

[0210] In all cases, the execution of a block must not extend beyond the block's extent space boundaries. Loads and stores of data (whether mapping the extent space to coordinates of tensors in memory, mapping to a pipeline, or any other memory or pipeline storage device) may extend beyond the extent space as required by the implementation's access granularity, but must not extend beyond the size of the pipeline's buffers or the total size of tensors. When the extent space is smaller than the pipeline buffers, VE BlockReduce operations have the additional requirement not to modify data in buffers outside the extent space; no other operations or execution units have this requirement.

[0211] TSU action space iteration can generate blocks with one or more execution dimensions (execution_dimension_empty) that are zero, meaning that no functional operations are required; for example, this can happen due to padding before the start of the action space or pruning at the end of the action space. As noted in TSU task iteration and block invocation, blocks still must be dispatched to execution units to correctly track dependencies and perform ordering.

[0212] In this way, when connected via pipes, the following conditions must hold for the transformation to be valid, making the transformation from action space to segment space compatible.

[0213] Assume the following scenario:

[0214] ○ Segment S0 is written to pipe P;

[0215] ○ Segment S1 is read from the same pipe P;

[0216] ○ T0() is the transformation of segment S0;

[0217] ○ T1() is the transformation of segment S1;

[0218] ○B is a block in the action space;

[0219] B0 is the absolute tensor coordinate of the block written to pipeline P by S0;

[0220] ■ This will be DST(T0(B)), where DST() is a fixed transformation of the execution units of S0 into their target output space;

[0221] B1 is the absolute tensor coordinate of the block read from pipeline P by S1;

[0222] ■ This will be SRC(T1(B)), where SRC() is a fixed transformation from the execution units of S1 to its source input space;

[0223] Then the following must hold:

[0224] - Origin compatibility: The lower bound coordinates of each dimension of blocks B0 and B1 must be the same;

[0225] o This coordinate forms the origin of the block stored in the pipeline buffer;

[0226] - Sufficient size: for each dimension, the size of block B0 must be greater than or equal to the size of block B1;

[0227] Operation space iteration can generate blocks with one or more execution dimensions equal to zero, meaning no functional operations are required; for example, this can occur due to padding before the start of the operation space or pruning at the end of the operation space. The block must still be dispatched to an execution unit to correctly track dependencies and perform ordering.

[0228] To implement a reduction operation, the operation space iteration will issue a sequence of block calls to an execution unit (e.g., a convolution engine or a vector engine), all targeting the same output block. When the first block in the sequence is executed, the handling unit will be signaled, and the execution unit must start by initializing the destination buffer (the entire buffer bounded by the size of the block, as described above), and for all subsequent blocks in the sequence, the unit will read back the existing values from the buffer. In this way, from the perspective of a single block execution, the destination buffer acts as an additional input to the operation. In the case of the convolution engine, one or more of the reduction dimensions may be zero, meaning that no functional operation is required, but the convolution engine still has to initialize the destination buffer if it is the first block in the sequence and the execution dimensions of that block are not empty.

[0229] When the handling unit calls an execution unit to execute a block, the handling unit is configured to issue call data to perform an operation on the block. Block iterations are defined based on the block size specified in the NED, and the issuance of call data is controlled by the DIMS_INC_RUN value as described above. In addition, any dependencies that the execution unit needs to satisfy in order to operate on the block are necessary. These include: the required data is stored in the source pipeline of the operation, sufficient storage is available in the destination pipeline, and the transformation of the operation space to the segment space of the segment has been performed, and the output of the transformation operation (i.e., the transformed coordinate data) is available for issuance to the execution unit. More specifically, it is ensured that there is sufficient availability in the pipeline for the new block or buffer. However, this is not necessary if this is not the first step in reducing the block, as in this case the operation may involve a simple read-modify-write of the previous destination block / buffer. Determining the availability of the source storage element may involve determining the presence of the appropriate block / buffer in the source pipeline.

[0230] In one example, the call data includes the output of the transform procedure in the form of transformed coordinates and the relevant portion of the NED that describes the section (e.g., configuration data from the sub-descriptor element of the NED for the section). This additional configuration data may also include the type of operation being performed (where the execution unit is capable of performing more than one type of operation) and any other properties of the operation, such as the stride and dilation values in the example of a convolution operation.

[0231] The iterative process first involves reading the block size from the NED and iterating through the operation space one block at a time. For each block, a transformation procedure is performed to transform the operation space coordinates into the segment space coordinates for that segment. More details on the transformation procedure are provided below. Once the segment space coordinates are determined, the segment operation is performed on the block. This process is repeated for all blocks until the operation is completed for all blocks.

[0232] Figure 6 An example chain 200 of operations to be performed is illustrated. The chain includes a left-hand side (LHS) input read operation 220 and a right-hand side (RHS) input read operation 210. The output of the RHS input read operation 210 is input to an inverse operation 230, which in turn is output to a matrix multiplication (MatMul) operation 240 along with the output of the LHS input read operation 220. The output of the MatMul 240 operation is input to a rescale operation 250, which (if provided) is provided to an output write operation 260, which writes the output to memory.

[0233] Figure 7The corresponding coordinate spaces (i.e., the segment spaces for each operation) are instantiated. For example, the RHS input read segment space 215 is instantiated for the RHS input read 210 operation. The LHS input read segment space 225 is instantiated for the LHS input read operation 220. The reverse segment space 235 is instantiated for the reverse operation 230. The MatMul segment space 245 is instantiated for the MatMul operation 240. The rescale segment space 255 is instantiated for the rescale operation 250. In this example, the segment space 255 is used to instantiate the segment space for the output write operation because it is unchanged compared to the segment space used for the rescale operation.

[0234] Each segment space includes multiple dimensions, namely two dimensions (eg, K, N; K, M). The segment space is divided into blocks of predefined block sizes, where each of blocks A to H represents a different block to be operated according to the examples set forth herein.

[0235] As can be seen, the reverse segment space 230 has dimensions that are effectively reversed relative to the RHS input read segment space 215. The segment space 225 for the LHS input read contains duplicate blocks A / E, B / F, C / G, D / H. The segment space 255 for the rescale and output write operations contains two blocks, AD and EH. This is because the MatMul operation is a reduction operation. Figure 7 In the MatMul example in

[15] , a MatMul operation is performed on two matrices 225 and 235. Matrix 225 has dimensions KxN, and matrix 235 has dimensions KxM. Output 255 has dimensions NxM, so the K dimension has been reduced. MatMul can be described using a 3D operation space of N, M, and K.

[0236] As will be understood, Figure 7 The operations described in are sections that can be executed by different execution units respectively. The handling unit can be configured to control the execution of each block so that a specific block can flow through the operation chain defined by the graph or subgraph. The "A / E" symbols in these figures illustrate that the blocks are repeated. For example, blocks A and E have the same coordinates in some dimensions (K, N), but there is another dimension (M) in the coordinate space that has been changed but not mapped to 220. The "AD" symbol indicates that the blocks have been reduced and merged into a single block. For example, blocks A, B, C, D have been reduced to a single block. The dimension K of these blocks is different, but the dimension K has been reduced. Figure 8 exemplified in Figure 7 An example schedule of the blocks shown in .

[0237] Figure 8 Illustrated Figure 6 and Figure 7Figure 1. An example of block iteration over a chain of operations for a series of call time instances 0 to 11. At call time instance 0, block A is processed concurrently by execution units that perform LHS and RHS read operations. These operations have no dependencies and can be handled at a single call time instance in this example and are therefore issued concurrently. Since the LHS and RHS read operations are independent of each other, for all subsequent call time instances, the next block is called for execution (e.g., block B is called at time instance 1) until all blocks A to H have been executed at time instance 7. The operation may still stall if there is no room in the destination pipeline for the segment.

[0238] Since the reverse operation is a subsequent operation that depends on the output of the RHS read operation, the processing of block B by the reverse operation can only be called at time instance 1. Therefore, the processing of the block by the reverse operation is delayed by one invocation time instance relative to the RHS read operation. Similarly, the MatMul operation depends on the output of the reverse operation, and therefore the MatMul processing of the block is further delayed by one invocation time instance relative to the reverse operation.

[0239] The rescaling operation operates on data blocks derived from a set of four reduced data blocks, for example, from A to D or from E to H, in a single call. Thus, the rescaling operation is not called until all input dependencies are satisfied, i.e., the MatMul operation has been performed on each of blocks A to D at time instance 6. Similarly, blocks E to H are not called for execution until time instance 10. The output write operation depends on the completion of the rescaling operation, and thus is not called until time instance 7 for blocks derived from the processing of blocks A to D, and again at time instance 11 for blocks derived from the processing of blocks E to H.

[0240] In this manner, processing iterates through all blocks until the complete operation space has been executed.

[0241] The process for generating an operation space from which each of these respective segment spaces can be represented will be described in more detail later, but in this example, since all other segment spaces can be represented from the MatMul segment space 245, the operation space of the operation chain is considered to be the segment space 245 of the MatMul operation 240.

[0242] Figure 9The flowchart of the efficient data processing method 900 according to the present disclosure is illustrated. The data processing method 900 is executed on a processor, which is configured to handle task data and includes a handling unit, multiple storage elements and multiple execution units. The task data includes a program, which includes transformation program data describing the transformation from the operation space to the segment space (local space) of the corresponding segment. In step 902, the processor obtains the task data in the form of a directed acyclic operation graph from the storage device. As mentioned above, the method and the processor can be configured to operate on any type of graph, not just a directed acyclic graph. Each operation in the operation is mapped to the corresponding execution unit of the processor, and each connection between the operations in the acyclic graph is mapped to the corresponding storage element of the processor. In step 904, for each corresponding part of the operation space, the method 900 includes transforming the part of the operation space to generate a corresponding operation-specific local space for each operation in the multiple operations of the acyclic graph. At step 906, method 900 includes dispatching, to each of a plurality of execution units associated with an operation for which a transformed local space has been generated, call data describing a specific local space for the operation, and at least one of a source storage element and a destination storage element corresponding to a connection between the specific operation to be executed by the execution unit and another adjacent operation to which the specific operation is connected in the acyclic graph. If necessary, the processor is further configured to perform clipping 908 on the lower and upper bounds of the task and operation spaces before running the transformation.

[0243] Programmability of transformation from operation space to segment space

[0244] As described above, the operation space of a task (subgraph) may contain a predetermined number of dimensions (e.g., eight), but the local segment space of the operations to be performed on a particular segment of the graph may contain fewer than 8 dimensions. Furthermore, as described above, the handling unit may iterate through the operation space in units called blocks, transforming each block from the common operation space into the segment-specific space described by various fields in the NED.

[0245] In an example implementation, the NED may also include a program for each element in the NED (e.g., each segment / pipe) that includes transformation program data that describes the transformation of the corresponding segment from the operating space to the segment space (local space). In one such implementation, each element in the NED may include an offset value that points to a specific program within the NED for performing the transformation. The offset value may be viewed as a pointer to a "program space," which is the space where all programs for the various enabled transformations are defined. Alternatively, the offset value may be a pointer to a virtual address space in main memory. For example, the program space may be defined in the NED as a field tsu_space_size, which may be, for example, 256 bytes in size. The offset may point to a memory location where the start of its segment space transformation is placed (e.g., the first instruction in a sequence of instructions that collectively define a program for performing the transformation).

[0246] Each transformation program may end with an explicit END instruction and may be followed, without any spacing or alignment, by the next program defining a sequence of instructions for performing different transformations associated with different elements. Alternatively, a start pointer may be used in conjunction with the total number of instructions to be executed.

[0247] In an example implementation, the instruction sequence for each transformation can be selected from a set of predetermined instructions that effectively form an instruction set. The instructions can be considered a transformation instruction set, which can be a specific instruction set that is optimally selected to perform the transformation from the operation space to the segment space. Alternatively, the transformation can be a general instruction set as found in a central processing unit (CPU).

[0248] In an example implementation, a transform instruction may operate on a set of state values for the transform. The state values include boundary registers (eight boundary registers b[0] to b[7] in one example), each of which includes a low component and a high component. Each block in the operating space is defined by the values described in the low and high components of the eight boundary registers. These values indicate the upper and lower limits (inclusive) of the coordinates in the block for that axis of the "bounding box" operating space.

[0249] In this example, no additional state is available to the instruction for transforming the operation space into the local segment space of the specific operation to be performed. Therefore, all operations performed by the instruction operate on boundary registers, including intermediate calculations.

[0250] Some instruction sequences will transform one dimension at a time, starting at dimension 0 (e.g., b[0]) and working iteratively through the dimensions inward. In other, more complex instruction sequences, more complex transformations may require jumping by explicitly modifying the destination register identifier (e.g., by using the SETD instruction in the instruction set).

[0251] An example of a transform procedure for transforming the output dimension of a convolution operation is illustrated below using a register exchange instruction with a destination modifier D and dimension d:

[0252] Program, 4 instructions, 4 bytes

[0253] (d=0)Register swap b[d],b[1] / / swap OC and N

[0254] (d=1)SWP.Db[d],b[2] / / swap OC and OY

[0255] (d=2)SWP.Db[d],b[3] / / swap OC and OX

[0256] Finish

[0257] This instruction sequence represents the following affine transformation of the output dimension of the convolution operation:

[0258] OFM OC N OY OX IC KY KX Offset N 1 OY 1 OX 1 OC 1 1

[0259] The result of executing the transformation program on a particular block defines a block in the segment space, which is ready to be used to call a specific hardware execution unit to execute the segment. For many types of operations to be performed by a hardware execution unit to execute a segment, the execution unit does not use the full 8-dimensional segment space. Therefore, the handling unit defines a calling structure for each unit that defines the relevant requirements for the operation.

[0260] At least some aspects of the examples described herein include computer processes executed in a processing system or processor. However, in some examples, the present disclosure also extends to computer programs suitable for putting the present disclosure into practice, in particular computer programs on or in an apparatus. The program may be in the form of non-transient source code, object code, code intermediate source and object code, such as partially compiled form or any other non-transient form suitable for use in the implementation of a process according to the present disclosure. The apparatus may be any entity or device capable of carrying the program. For example, the apparatus may include a storage medium, such as a solid state drive (SSD) or other semiconductor-based RAM; a ROM, for example, a CD ROM or a semiconductor ROM; a magnetic recording medium, for example, a floppy disk or a hard disk; typically, an optical storage device; and the like.

[0261] In the previous description, for the purpose of explanation, numerous specific details of certain examples are set forth. References in this specification to "example" or similar language mean that a particular feature, structure, or characteristic described in conjunction with the example is included in at least one example, but not necessarily in other examples.

[0262] The above examples are to be understood as illustrative examples of the present disclosure. Additional examples of the present disclosure are contemplated. It should be understood that any feature described for any one example may be used alone or in combination with the other features described, and may also be used in combination with one or more features of any other example or with any combination of any other examples. In addition, equivalents and modifications not described above may also be employed without departing from the scope of the present disclosure as defined in the appended claims.

Claims

1. A processor for processing data, the processor comprising a processing unit, a plurality of storage elements, and a plurality of execution units, the processor being configured to: obtaining, from a storage device, task data describing a task to be performed in the form of an operation graph, wherein each of the operations maps to a corresponding execution unit of the processor, and wherein each connection between the operations in the graph maps to a corresponding storage element of the processor, the task data further defining an operation space representing the dimensions of a multi-dimensional arrangement of connected operations to be performed; and For each of the plurality of sections of the operating space: transforming the portion of the operation space to generate a corresponding operation-specific local space for each of a plurality of operations of the graph; and To each of the plurality of execution units associated with an operation for which a transformed local space has been generated, call data describing a specific local space for the operation is dispatched, along with at least one of a source storage element and a destination storage element corresponding to a connection between a specific operation to be performed by the execution unit and another adjacent operation in the graph to which the specific operation is connected.

2. The processor of claim 1 , wherein one or more of the following occurs: More than one operation in the operation graph is mapped to the same execution unit of the processor; and More than one connection in the operational graph is respectively mapped to different portions of the same storage element.

3. The processor of claim 1 , wherein each of the plurality of execution units of the processor is configured to perform a specific operation type, and wherein the mapping between the operations in the graph and the execution units is defined based on compatibility of execution between the operations in the graph and the specific operation type of the execution units.

4. A processor according to any preceding claim, wherein the task data comprises: an element count value indicating a count of the number of elements mapped to each execution unit of a particular operation type, wherein each element corresponds to an instance of use of an execution unit to perform each operation in the graph; and A pipeline count value indicating a count of the number of pipelines required to perform the task. 5 . The processor of claim 4 , wherein the task data further comprises, for each element in the graph, element configuration data, the element configuration data defining data for configuring a particular execution unit when performing the operation.

6. The processor of claim 5 , wherein the element configuration data comprises an offset value pointing to a location of transformation data in memory, the transformation data indicating the transformation to be performed on the portion of the operation space to generate a corresponding operation-specific local space for each of the plurality of operations of the graph.

7. A processor according to any preceding claim, wherein the task data comprises: transformation program data defining a plurality of programs, each program comprising a sequence of instructions selected from a transformation instruction set, wherein the transformation program data is stored for each transformation in a set of predetermined transformations, and a specific transformation is selected from the set of predetermined transformations to transform the portion of the operation space to generate a corresponding operation-specific local space for each operation in the plurality of operations of the graph.

8. A processor according to any preceding claim, wherein the task data comprises transformation program data configured to perform a specific transformation on a plurality of values stored in boundary registers defining the operating space to generate new values in the boundary registers.

9. The processor of claim 8, wherein clipping is performed on the plurality of values stored in boundary registers defining the operating space prior to transformation.

10. A processor according to any preceding claim, comprising iterating over the operation space in blocks, wherein the blocks are created according to a predetermined block size.

11. The processor of claim 10 , wherein the dispatch of call data of a block is controlled based on: a value identifying a dimension of the operation space, for which a change in coordinates causes the operation to be performed when performing the task, and Another value identifying a dimension of the operation space, for which a change in coordinates of the dimension when performing the task causes the operation to store data in the storage device, wherein the stored data is ready for use by the operation.

12. A processor according to any preceding claim, wherein dispatch of call data for the particular operation is dependent on availability of source storage data and the destination storage element.

13. A processor according to any preceding claim, wherein the handling unit, the plurality of storage elements and the plurality of execution units form part of a first neural engine within the processor; and The processor comprises: a plurality of additional neural engines, each additional neural engine comprising a corresponding plurality of additional storage elements, a plurality of additional execution units, and an additional processing unit; and A command processing unit is configured to issue corresponding tasks to one or more neural engines for execution.

14. A processor according to any preceding claim, wherein the operation graph is a directed acyclic operation graph.

15. A method for processing data in a processor, the processor comprising a processing unit, a plurality of storage elements, and a plurality of execution units, the method comprising: obtaining, from a storage device, task data describing a task to be performed in the form of an operation graph, wherein each operation in the operation graph maps to a corresponding execution unit of the processor, and wherein each connection between the operations in the graph maps to a corresponding storage element of the processor, the task data further defining an operation space representing the dimensions of a multi-dimensional arrangement of connected operations to be performed; as well as For each of the plurality of sections of the operating space: transforming the portion of the operation space to generate a corresponding operation-specific local space for each of a plurality of operations of the graph; and To each of the plurality of execution units associated with an operation for which a transformed local space has been generated, call data describing a specific local space for the operation is dispatched, along with at least one of a source storage element and a destination storage element corresponding to a connection between a specific operation to be performed by the execution unit and another adjacent operation in the graph to which the specific operation is connected.

16. The method of claim 15, wherein one or more of the following occurs: More than one operation in the operation graph is mapped to the same execution unit of the processor; and More than one connection in the operational graph is respectively mapped to different portions of the same storage element.

17. A method according to claim 15 or 16, wherein each of the plurality of execution units of the processor is configured to perform a specific operation type, and wherein the mapping between the operations in the graph and the execution units is defined based on the compatibility of execution between the operations in the graph and the specific operation type of the execution units.

18. The method according to claim 17, wherein the task data comprises: an element count value indicating a count of the number of elements mapped to each execution unit of a particular operation type, wherein each element corresponds to an instance of use of an execution unit to perform each operation in the graph; and A pipeline count value indicating a count of the number of pipelines required to perform the task.

19. The method according to any one of claims 15 to 18, wherein the operation graph is a directed acyclic operation graph.

20. A non-transitory computer-readable storage medium comprising a set of computer-readable instructions stored thereon, the set of computer-readable instructions being arranged, when executed by at least one processor, to cause the at least one processor to: obtaining, from a storage device, task data describing a task to be performed in the form of an operation graph, wherein each of the operations maps to a corresponding execution unit of the processor, and wherein each connection between the operations in the graph maps to a corresponding storage element of the processor, the task data further defining an operation space representing the dimensions of a multi-dimensional arrangement of connected operations to be performed; and For each of the plurality of sections of the operating space: transforming the portion of the operation space to generate a corresponding operation-specific local space for each of a plurality of operations of the graph; and To each of the plurality of execution units associated with an operation for which a transformed local space has been generated, call data describing a specific local space for the operation is dispatched, along with at least one of a source storage element and a destination storage element corresponding to a connection between a specific operation to be performed by the execution unit and another adjacent operation in the graph to which the specific operation is connected.

Images