Programmable accelerator for use in data-dependent irregular calculation

Abstract

To provide an accelerator which can accelerate data-dependent calculation, irregular calculation, and / or memory-bound calculation.SOLUTION: A hardware circuit 101 includes: a sparse accelerator 103; a coprocessor 104 for accelerating predictable calculation for calculation load and behavior on the coprocessor during design and manufacturing; a high-band width memory 107; and an on-chip interconnection 108. The sparse accelerator 103 includes one or more tiles 102A to 102F, and each of the tiles is equipped with its vector processing unit (VPU) and has cross lane processing units (XPU) 101A to 101F therefor. The sparse accelerator has a tile sequencer 106 for coordinating input data and output data for the tiles 102A to 120F.SELECTED DRAWING: Figure 1A

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. Provisional Patent Application No. 17 / 981,617, filed November 7, 2022, which claims the benefit of the filing dates of U.S. Provisional Patent Application Nos. 63 / 357,281, filed June 30, 2022, 63 / 322,285, filed March 22, 2022, 63 / 281,960, filed November 22, 2021, and 63 / 279,262, filed November 15, 2021, the disclosures of which are incorporated herein by reference. This application is related to U.S. patent application Ser. No. 17 / 972,681, filed October 25, 2022, Ser. No. 17 / 972,663, filed October 25, 2022, and Ser. No. 17 / 722,782, filed April 18, 2022, the disclosures of which are incorporated herein by reference. [Background technology]

[0002] background Hardware acceleration is the use of computer hardware to perform certain types of operations more efficiently. Exemplary types of operations that can be accelerated include linear algebra operations, such as matrix-matrix multiplication or matrix-vector multiplication. A device or processor built to perform hardware-accelerated operations is sometimes referred to as an accelerator.

[0003] Accelerators are designed and built to accelerate a small subset of desired operations. During the accelerator design and build process, assumptions are made about the nature of the operations desired to be accelerated, such as the size and type of inputs the accelerator will receive, the regularity with which the accelerator will receive the inputs, or the computational requirements for performing the operations. As a result, accelerators often end up being highly specialized, capable of only accelerating a small class of predetermined operations, and unable to efficiently perform other operations, if at all.

[0004] Operations outside this class include data-dependent operations where the computational load on the accelerator cannot be determined prior to execution of the operation. Multiple instances of this type of accelerated operation may vary depending on various factors, and a given accelerator design may be inefficient in accelerating at least some of these instances. Other types of operations that are difficult to accelerate include memory-bound operations that have low computational intensity and limited data reuse. Yet another type of operation that is difficult to accelerate is irregular operations that may be characterized by random memory accesses, complex code patterns, and diverse use of parallel execution of multiple simultaneous sub-operations.

[0005] In reality, a processing pipeline, such as one for training or deploying a machine learning model, requires the execution of a wide variety of different types of operations. Incorporating an accelerator into the pipeline to accelerate only some types of operations and relying on devices to execute other types of operations without hardware acceleration imposes unacceptable latency and memory bandwidth stress on the links and interconnects between the accelerator and non-accelerator, limiting overall performance. In most cases, it is impossible or infeasible to design and build accelerators to cover all types of operations. Data-dependent operations do not contribute to acceleration, and the logistical effort required to accelerate other types of operations may not be worth the investment in designing, building, and deploying the corresponding accelerators. Summary of the Invention

[0006] overview Aspects of the present disclosure provide accelerators capable of accelerating data-dependent, irregular, and / or memory-bound operations. The accelerators described herein include a programmable engine for efficiently performing on-chip computations that are dynamic, irregular, and / or memory-bound, along with a coprocessor configured during design and fabrication to accelerate operations that are predictable in terms of computational load and behavior on the coprocessor.

[0007] Dynamic operations are operations in which the computations performed are data-dependent or input-dependent, meaning that the inputs are not known prior to performing the operation. The irregularity of the operations can result from random memory accesses, complex code patterns, and varying amounts of computational resources and parallelism required to perform different instances of the operation on different input data. Memory-bound operations are often operations with low computational intensity, e.g., a small number of operations are performed per unit of data transferred during acceleration of the operation, and data reuse is limited.

[0008] The accelerators described herein can coordinate and distribute cross-chip data scatter and gather operations for data of various sizes to scale acceleration on host devices or data centers that implement multiple accelerators and other processors. As described herein, the accelerators utilize configurable architectural primitives to adapt to the acceleration of various types of data-dependent, irregular, and / or memory-bound operations without requiring physical redesign or modifications to the hardware circuitry that implements the accelerator itself.

[0009] Aspects of the present disclosure provide accelerators that can accelerate computations of neural network layers that exhibit sparsity, for example, in an embedded form. Sparse computation refers to computations in which the fractional part of the computed data values ​​(e.g., input values, output values, or intermediate values) is zero. The fractional part can vary, for example, between 0.1% and 50%. Aspects of the present disclosure provide for the acceleration of training and processing of embeddings as part of a machine learning processing pipeline.

[0010] An aspect of the present disclosure provides a processor including a plurality of tiles, each of the plurality of tiles including a vector core and a slice of a shared software-controlled scratch-pad memory, the processor further including a scalar core configured to dispatch tasks to the plurality of tiles, and a memory coupled to the plurality of tiles and the scalar core.

[0011] In one example, each tile is configured to perform an independent computation. In another example, the vector core in each of the tiles includes multiple single instruction, multiple data (SIMD) processing lanes. In yet another example, multiple tiles of the plurality of tiles issue memory requests to main memory in parallel.

[0012] In yet another example, the vector core in each of the plurality of tiles is configured to generate a data-dependent address stream to any level of a memory hierarchy. In yet another example, each data-dependent address stream corresponds to a sequence of addresses, the length and specific values ​​of the addresses in the sequence being data-dependent and known only at run time. In yet another example, the vector core in each of the plurality of tiles is configured to represent the data-dependent address stream while decoupling the high performance services of the data-dependent address stream to the microarchitecture. In one example, the microarchitecture includes a scatter-gather engine for the high performance servicing of the data-dependent address stream. In yet another example, the data-dependent address stream includes multiple addressing modes, runtime configurable transfer sizes, and indirect memory accesses with atomic arithmetic updates.

[0013] In yet another example, the vector core in each of the plurality of tiles includes circular buffer instructions that enable transfer and access of dynamically sized data streams on a statically sized region of memory. In yet another example, the processor further includes a microarchitecture configured to track runtime buffer sizes of the dynamically sized data streams. In yet another example, the vector core in each of the plurality of tiles is configured to provide runtime configuration and access regions of the tile-local scratch pad memory as in-order circular first-in-first-out (FIFO) accesses without precluding out-of-order accesses to the same region of tile-local scratch pad memory. In yet another example, the in-order circular FIFO accesses, in conjunction with the microarchitecture, enable dynamically sized data-dependent address streams on the statically sized region of the tile-local scratch pad memory.

[0014] In yet another example, each tile includes a scatter-gather engine configured to manage issuing, fetching, tracking, and ordering of data streams. In yet another example, each scatter-gather engine is further configured to maintain at least 256 outstanding read requests in-board per tile. In yet another example, each scatter-gather engine is further configured to track and update buffer occupancy to manage flow control.

[0015] In yet another example, a subset of the plurality of tiles each further includes a prefetch unit configured to cooperatively prefetch data stream instructions. In yet another example, the processor further includes a cross-lane processing unit configured to accelerate at least one of irregular control flow sequences or intra-vector dependent operations. In yet another example, each tile is configured to support scatter from an off-chip memory to its scratch-pad memory and gather from its scratch-pad memory to the off-chip memory.

[0016] In yet another example, the subset of tiles is grouped based on a logically configurable vector width. In yet another example, the logically configurable vector width comprises a logical SIMD width.

[0017] In yet another example, the processor is part of a machine learning accelerator configured to execute a neural network layer that exhibits semantic sparsity. In yet another example, the neural network layer includes an embedded neural network or a graph neural network. In yet another example, the processor is connected to several other processors via a network configured to perform distributed scatter-gather and computations required by the neural network layer computations, which are dynamic, irregular, and memory-bound. [Brief explanation of the drawings]

[0018] [Figure 1A] FIG. 2 is a block diagram illustrating a hardware circuit for accelerating data-dependent operations in accordance with an aspect of the present disclosure. [Figure 1B] FIG. 2 is a block diagram illustrating an example data path implemented as part of a hardware circuit, according to an aspect of the present disclosure. [Figure 2] FIG. 1 is a block diagram illustrating an exemplary environment for implementing hardware circuitry according to aspects of the present disclosure. [Figure 3A]FIG. 1 is a block diagram illustrating an example tile according to an aspect of the present disclosure. [Figure 3B] FIG. 10 is a block diagram illustrating another example tile implementing an XPU for stream forwarding, according to aspects of the present disclosure. [Figure 4] FIG. 2 is a block diagram illustrating a tile sequencer according to an aspect of the present disclosure. [Figure 5] FIG. 2 is a block diagram illustrating an exemplary scratch pad memory along with memory across multiple tiles of a sparse accelerator, in accordance with aspects of the present disclosure. [Figure 6] FIG. 10 is an exemplary block diagram illustrating a scalar core complex of a tile sequencer according to an aspect of the present disclosure. [Figure 7] FIG. 2 is a block diagram illustrating an example XPU according to an aspect of the present disclosure. [Figure 8] FIG. 1 is an exemplary functional diagram illustrating a scatter-gather engine according to an aspect of the present disclosure. [Figure 9] 10 is a flow diagram illustrating an example process for expanding a stream descriptor into a configuration off-tile or tile-local stream request according to an aspect of the present disclosure. [Figure 10] FIG. 1 is a flow diagram illustrating an example process for ordering stream transfers according to an aspect of the present disclosure. [Figure 11] FIG. 2 is an exemplary diagram illustrating stream ordering according to an aspect of the present disclosure. [Figure 12] FIG. 10 is a logical diagram illustrating connectivity between tiles and an exemplary sparse accelerator, according to aspects of the present disclosure. [Figure 13] FIG. 10 illustrates additional exemplary aspects of an instruction router according to aspects of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0019] Detailed Description Overview Aspects of the present disclosure provide an accelerator that can accelerate data-dependent, irregular, and / or memory-bound operations. An accelerator configured to accelerate data-dependent operations on sparse inputs may be referred to herein as a sparse accelerator. The sparse accelerator may be configured to efficiently and highly performantly execute neural network layers that exhibit semantic sparsity, such as embedded neural networks or graph neural networks (GNNs). A graph neural network may correspond to a neural network for processing data that can be represented as a graph.

[0020] A sparse accelerator can be organized as a tiled processor. The sparse accelerator may include a tile sequencer used to dispatch tasks to the tiles. Each tile of the sparse accelerator has a vector core augmented with a cross-lane processing unit (XPU) and a slice of shared software-controlled scratchpad memory. Both the tile and the sequencer can be connected to high-bandwidth main memory.

[0021] The organization and structure of the sparse accelerator improves data-dependent, irregular, and / or memory-bound operations through various combinations of the separate features described herein. The accelerator tiles may include one or more of the following features to more efficiently utilize available computations in the presence of data-dependent, irregular, and / or memory-bound operations: A cross-lane processing unit (XPU) within each tile accelerates common irregular control flow sequences and / or intra-vector dependent operations. Each XPU allows for the provision of a custom high-speed data path for common intra-vector dependent operations.

[0022] Other features that may be implemented on the accelerators described herein include cooperative prefetching and stream instruction and ordering. Cooperative prefetching enables improved performance and energy efficiency by reducing instruction fetch bandwidth requirements between tiles. The stream instructions described herein enable high-bandwidth data-dependent scatter-gather. Each vector processing unit of the accelerator may generate data-dependent addresses to any level of the memory hierarchy at high bandwidth.

[0023] Tile-local scatter and gather may enable uninterrupted computation in the presence of data irregularities. Each accelerator tile can inherently support high-performance scatter and gather into its local memory exposed by the instruction set architecture (ISA), with available instructions for indirect vector loads, stores, and store appends.

[0024] The implemented software-controlled tile grouping of the XPU can provide flexible amortization of control overhead. Software can flexibly group tiles to exhibit logically configurable vector widths, e.g., logical SIMD widths. This can help amortize control overhead, such as eliminating redundant instruction fetches between tiles in the same group or enabling coalesced memory accesses between tiles. This can result in better bandwidth efficiency, e.g., high-bandwidth memory (HBM) requires high-speed memory with access granularity. Different inputs may have different bandwidth efficiencies. A data-dependent operation (also referred to as an "input-dependent operation") is an operation for which the amount of computational effort to perform the operation is not known in advance and depends on the properties of the data. A data-dependent operation can be represented as a data-dependent address stream, which corresponds to a sequence of addresses where the length and specific values ​​of the addresses in the sequence are known at run time. The computational effort can be measured, for example, in the number of operations or processing cycles required to perform the data-dependent operation. Exemplary data-dependent operations include an operation for vector sorting, an operation for counting duplicate values ​​in a vector, and an operation for processing the shapes or sizes of vectors of various lengths. Data-dependent operations are irregular at least because there are differences in the random memory access patterns for performing the same type of operation on different inputs. As a result, data-dependent operations are difficult to optimize for performance, in contrast to other types of operations for which the computational effort does not change based on the properties of the input data, such as its shape, degree, or sparsity.

[0025] Data-dependent operations include operations performed on sparse data. The sparsity of a data structure is a measure of the ratio of non-empty elements to empty elements. Depending on the data structure, an empty element may be zero, a reserved word indicating the absence of a value for the element, or may have such a small value that it is considered to contribute only marginally to operations performed on the data structure as input. A data structure is sparse if it has more empty elements than non-empty elements. Some data structures may be more or less sparse than others.

[0026] An exemplary data-dependent operation includes generating an embedding for an input training example. The embedding may be a vector or some other data structure mapped from the input having a higher dimensionality than the embedding. The embedding generation may be performed as part of a workload processed according to a pipeline.

[0027] The XPUs in each tile of the sparse accelerator may also perform other data-dependent operations such as vector scatter or gather operations, segment sums, and / or The XPUs described herein may partition sparse data structures such as kernels. The XPUs described herein may be complementary processing units to other or connected components of a processor, such as vector processing units built according to the SIMD parallel processing paradigm. One or more XPUs may be connected in each processor core of a larger processor, which itself may include other components for accelerating the performance of specific workloads, such as training neural networks.

[0028] Furthermore, the XPU is not limited to performing a specific type of data-dependent operation; therefore, a processor can be designed to include an XPU to complement other types of processing units for multiple different pipelines. Because the XPU can be configured for each workload, the physical footprint of the XPU is reduced compared to other approaches in which specialized circuits are physically fabricated on the processor as complementary units for sparse data computation. The functionality of the XPU can also be extended through instruction sets or extensions to the host processor's existing instruction set, further improving the adaptability of various data-dependent operations in response to changes in pipeline data reception. Instructions can be provided as signals to the XPU's components, which translate the instructions to configure the XPU's individual processing cells and crossbar. The XPU can be configured using a program compiled by a corresponding compiler for hardware circuits that implement the XPU.

[0029] Aspects of the present disclosure provide at least the following technical advantages: The accelerators described herein provide scalable distributed training of large machine learning models that require performing data-dependent operations whose operands are generally not known until runtime.

[0030] The accelerator allows flexible computational configuration of tiles implementing XPUs within the accelerator to achieve various forms of parallelism, such as task, data, pipeline, and model parallelism, alone and in combination with other processors. In task-level parallelism, each tile may perform independent computations (tasks) in parallel. For memory-level parallelism, tiles may issue high-bandwidth memory requests in parallel to absorb available memory bandwidth. This allows a sparse accelerator to handle varying amounts of work in separate tasks.

[0031] The accelerators described herein may be dedicated co-processors alongside another accelerator, or may be general-purpose processors that are not configured to accelerate data-dependent operations. The accelerators described herein may accelerate sparse or other data-dependent computations to achieve performance gains, e.g., in increasing processing speed, over approaches that rely on host memory to retrieve data for performing data-dependent operations.

[0032] The accelerators described herein can function as architectural analogs to coprocessors configured to perform dense, regular computations. Implementing the programmable sparse accelerators described herein communicatively coupled to separate processors can provide a flexible means for accelerating operations with unpredictable complexity at compile time. The accelerators described herein can target memory-bound operations involving complex data movement, reorganization, and summarization. This type of operation can include scatter-gather operations, filtering, sorting, uniquing, etc. The accelerators are targetable according to a compiler stack, e.g., a compiler configured to translate source code into instructions executable by the accelerator and its coprocessors.

[0033] The accelerator described herein can accelerate embedded generation. Embeddings can generally be characterized as involving irregular memory accesses with low computational intensity. This is due, at least in part, to the need to perform on-chip table lookups of tables representing embedding maps and variable-width vector calculations, given the sparse and irregular nature of the input vectors for embedding generation. Embeddings are mappings from discrete objects, e.g., vectors or other data structures of values, to vectors of numerical values, such as real values. Embeddings generally have lower dimensionality and complexity than their corresponding pre-embedded objects. For example, an embedding of one or more words in English may be a vector of real values. Embeddings can be used to represent and identify potentially salient features of pre-embedded inputs; for example, embeddings of different words can be compared by measuring the distance between them to quantify the similarity between two pre-embedded inputs.

[0034] The cooperative prefetching described herein enables improved performance and energy efficiency by reducing instruction fetch bandwidth requirements between tiles. The accelerator's tiled architecture exposes multiple program, multiple data programming models while additionally providing a configurable subset of tiles that operate on a single program, multiple data models.

[0035] The stream instructions described herein enable data-dependent scatter-gather at high bandwidth. Each vector processing unit of the accelerator can generate data-dependent addresses to any level of the memory hierarchy at high bandwidth. The stream instructions described herein provide an architecturally visible construct, e.g., an ISA, that enables the expression of data-dependent access patterns in software. These access patterns include indirect memory accesses with multiple addressing modes, configurable transfer sizes, and atomic operations. The stream instructions allow software to specify the "start" address, "size," and "sequence pattern" for memory accesses. All of this is accomplished while each tile of the accelerator implements a separate core for accessing data and servicing requests using a scatter-gather engine, according to aspects of the present disclosure.

[0036] Data may flow through various components of the processor and other off-chip components and through data streams, which are data values ​​that correspond to addresses in an address stream. A data stream may have a stream descriptor that provides metadata characterizing the stream. As part of the metadata, a data stream may have a stream identifier that can be used to identify on which execution thread of the processor the stream is currently being processed. Multiple streams, for example, 8, 16, or 32 streams, may be active and flowing through the processor simultaneously.

[0037] The circular buffer instructions described herein can enable dynamic data streams of variable size. Circular buffer instructions are architecturally visible (e.g., via an ISA) constructs that allow software to fetch and operate on statically unknown size data streams without explicitly allocating a buffer at compile time. Thus, circular buffer instructions allow the transfer and access of dynamically sized data streams over statically sized regions of memory. A scatter-gather engine tracks the runtime buffer sizes of the various data streams and manages flow control. Circular buffer instructions provide a structural first-in-first-out (FIFO) abstraction for the high-speed common case without excluding software from non-FIFO access patterns, since the underlying memory is also accessible via standard loads and stores. In another example, vector cores in each of multiple tiles can perform out-of-order transfers to the same region of tile-local scratchpad memory. The present invention is configured to provide run-time configuration and access of tile-local scratchpad memory as in-order circular first-in-first-out (FIFO) access without precluding identical accesses.

[0038] The stream and circular buffer instructions enable a general-case FIFO abstraction without preventing software from "randomly" accessing memory as needed. Software can FIFO-fetch buffers, but access data multiple times by reusing the fetched buffer / window as needed. This contrasts with the software approach of requiring multiple pops to reuse data and pushing those pops into a queue. This control balances the situation when software issues prefetches with the scatter-gather engine, which fetches and enforces flow control at a granular level. Software gets a higher-order bit right regarding roughly when to issue streams, and a scatter-gather engine implementing the stream ordering and instructions described herein accommodates fine-grained variations in latency, buffer occupancy, etc.

[0039] Microarchitectures, such as the scatter-gather engine described herein, enable high-performance, irregular memory accesses. The scatter-gather engine is implemented per tile and manages the issuing, fetching, tracking, and ordering of software-defined address streams, such as stream instructions and circular buffer instructions. The engine can maintain a number of read memory requests per tile, such as 256 requests in-flight. The scatter-gather engine tracks and updates buffer occupancy to rate-control address requests. Software can structurally probe buffer occupancy without having to explicitly manage variations in latency when processing individual address requests.

[0040] The combination of stream instructions, circular buffer instructions, and the scatter-gather engine allows for decoupled access execution and can effectively scale long memory access latencies, including irregular memory access streams. This gives software the flexibility to create data-dependent memory accesses and the option to schedule them at a "coarse" granularity, where the scatter-gather engine is configured to control the flow and accommodate the requests.

[0041] Aspects of the present disclosure provide an accelerator configured to execute multiple threads of dynamic small vector tasks across a set of computational tiles. Dynamic tasks include tasks involving data-dependent control and memory access. Small vector tasks may include tasks performed on relatively small vectors (e.g., eight-element vectors). These tiles are managed by a single task management core called a sequencer. The accelerator's computation-to-bandwidth ratio is tailored for irregular and sparse access and computation of large data sets stored in off-processor, high-bandwidth memory.

[0042] The accelerator may include multiple tiles, each including a respective access core and execution core. The access core may be a scalar unit used to decouple data movement from computation. The execution core may be another scalar unit attached to a vector unit with multiple SIMD lanes configured to process data fetched by the corresponding access core. Each tile may also include a respective XPU for performing cross-lane reduce, shuffle, sort, prefix sum, etc. The accelerator may also include a sequencer, which may be a scalar unit for task management across tiles and for communicating with other cores. The accelerator may also include a scratchpad memory. For example, the accelerator may include 8 megabytes of shared memory, although in various examples the size of the shared memory may vary. As described herein, the accelerator may implement a streaming memory access interface to keep transactions outstanding to the off-tile memory. The accelerator may also include a high-bandwidth crossbar for connecting the tiles to the shared scratchpad memory as well as to each other.

[0043] Exemplary System 1A is a block diagram of a hardware circuit 101 for accelerating data-dependent operations according to an aspect of the present disclosure. The hardware circuit 101 may include a sparse accelerator 103, a coprocessor 104, a high-bandwidth memory 107, and an on-chip interconnect 108. The sparse accelerator 103 may include one or more tiles 102A-102F. Each tile implements a respective vector processing unit (VPU) and includes a respective cross-lane processing unit (XPU) 101A-101F. The sparse accelerator 103 may include a tile sequencer 106 configured to coordinate input and output data across the tiles 102A-102F.

[0044] The tiles 102A-120F can be interconnected according to a wide variety of topologies, such as a multidimensional ring or torus. The interconnect can include, for example, a crossbar, which is described in more detail with reference to FIG. 1B. The crossbar or interconnect for the processor can receive and transmit data between, for example, the tiles 102A-120F, on-chip memory, and off-chip memory every clock cycle.

[0045] The tile sequencer 106 is a component of the sparse accelerator 103 that is configured to receive and distribute instructions for performing operations on the tiles 102A-120F in a coordinated manner. This coordination can be managed at least in part by exploiting the distribution relationships of the processing components on the sparse accelerator 103, for example, to exploit different types of data or instruction parallelism. The tile sequencer 106 is described in more detail with reference to FIG. 4.

[0046] The sparse accelerator 103 is configured to perform data-dependent operations using tiles 102A-102F. As shown and described in more detail with reference to FIG. 3A, each tile can implement several data processing lanes for streaming data through a vector processing unit (VPU) and a cross-lane processing unit (XPU). The tiles can retrieve the streamed data from on-chip memory 105, which can be any of a variety of memory devices, including main memory, cache, or persistent storage, such as solid-state storage or hard disk storage. The streamed data can also be retrieved from coprocessor 104, from high-bandwidth memory 107 serving one or both of coprocessors 103 and 104, and / or from another data source connected to the hardware circuit 101 via on-chip interconnect 108.

[0047] The on-chip memory 105 may be scratchpad memory physically distributed across each tile 102A-102F. The scratchpad memory, as opposed to a hardware cache, may be programmatically managed, e.g., storing data according to software instructions. The on-chip memory 105 may be globally addressable via a variety of interfaces, such as direct memory access and / or stream interfaces.

[0048] The coprocessor 104 can be any core, such as a CPU, a high-density core, etc. The coprocessor 104 may be configured for acceleration of specific operations, such as matrix-matrix multiplication and matrix-vector multiplication. An example operation may include dense matrix-matrix calculations, where a large proportion of the elements in the multiplied matrices (e.g., more than 50 percent, in some examples) have non-zero values. The complexity of the calculation may be approximated as a function of the dimensions of the multiplied matrices. In some examples, the coprocessor 104 is on a different device than the rest of the hardware circuit 101 and communicates data to the hardware circuit via an on-chip interconnect 108. The on-chip interconnect 108 may be a data bus or any form of interconnect according to any of various communication standards, such as PCIe. The on-chip interconnect may also implement a core memory network. The core memory network may be an on-chip network connecting the coprocessor 104 and the sparse accelerator 103.

[0049] Exemplary features of the sparse accelerator 103 are directed to improving the computation of sparse operations (e.g., operations involving operands or inputs that generally have more zero-valued elements than non-zero-valued elements). Features of this type may include a combination of using the programmable XPUs 101A-101F to perform sparse operations, cooperative memory prefetching, and / or instruction stream or instruction ordering, as described in more detail herein.

[0050] While examples are provided in the context of sparse computation, it should be understood that in some examples, the sparse accelerator 103 may be used to accelerate other types of operations commonly associated with accelerating machine learning model processing, including linear algebraic operations such as vector / matrix multiplication, computing activation function outputs, pooling layer outputs, normalizing layer outputs, etc. The coprocessor 104 and sparse accelerator 103 implemented as part of a common hardware circuit 101 can facilitate the distribution of various tasks suitable for either, but not limited to, the two. Acceleration of non-sparse operations may include dense computations, e.g., computations on non-sparse inputs. Examples of dense computations may include linear or strided access of arrays. Other examples include dense matrix multiplication, fully connected layers, and convolutional layers of deep neural networks.

[0051] An example of an input to the hardware circuit 101 may be data structured as a tensor. For example, a tensor may represent input data and / or model parameter values ​​for a machine learning model to be executed using the hardware circuit 101. A tensor is a data structure that generalizes various other common data structure types of different dimensions. A tensor may contain zero or more elements, which may be one or more of various data types, such as integers, floating-point values, Boolean values, etc. Within each data type, the data type may be parameterized according to a particular level of precision, for example, 8-bit, 16-bit, or 32-bit integer or floating-point values. The dimension of a tensor is referred to as its "rank." A rank-zero tensor has a single element, also referred to as a scalar. A rank-1 tensor is also referred to as a vector. A rank-2 tensor is also referred to as a matrix. Vectors and matrices may also be referred to as having different ranks. For example, a rank-2 vector is equivalent to a matrix. A non-zero-rank tensor can be described as a collection of tensors one rank lower. For example, a vector or rank 1 is a collection of scalar values, and a matrix of rank 2 is a collection of vectors of rank 1.

[0052] The hardware circuit 101 may at least partially implement a processing pipeline for training a neural network. The pipeline may include generating embeddings for input training examples. The feature tensors for various input training examples may have different degrees of sparsity, which affects the amount of computational effort required to generate the corresponding embeddings. The sparse accelerator 103 receives tensors of feature values ​​representing the training input examples and generates embeddings as tensors of lower rank than the feature tensors. The device may be configured to generate only

[0053] To generate the embedding, the sparse accelerator 103 is configured to implement various data-dependent operations for efficient sparse data computation on the XPUs 101A-101F, etc., or more generally on the VPUs. These operations include sorting or summing sparse vectors, operations for summarizing the contents of input vectors, and operations for converting sparse matrices from one sparse matrix storage format to another.

[0054] Instead of physical predefined circuits for accelerating the execution of data-dependent operations, the VPUs, including XPUs 101A-101F, can be configured, e.g., programmed, to perform a wide variety of data-dependent operations. Sparse accelerator 103 enables general-purpose support for processing sparse data while allowing complementary coprocessor 104 to perform other types of operations.

[0055] The hardware circuit 101 may be, for example, a central processing unit (CPU). CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), or , application-specific integrated circuit (ASIC), e.g. The hardware circuitry 101 may be implemented on a computing device that may itself be part of a system of one or more devices.

[0056] 1B is a block diagram of an exemplary data path implemented as part of a hardware circuit according to an aspect of the present disclosure. Scratch pad memory 162B and task instruction memory 160B can form part of on-chip memory 105.

[0057] Various data paths 150B, 152B, 154B, and 156B are shown. These data paths may or may not physically share DMA path 150B, which includes circuit interconnections between tile sequencer 106 and tiles 102A-120F for direct memory access (DMA) of scratchpad memory 162B. Instruction data path 152B includes circuit interconnections between tiles 102A-120F, task instruction memory 160B, scratchpad memory 162B, and memory transport interface 163B. Scratchpad (spmem) data path 154B illustrates a potential path for data from task instruction memory 160B to tiles 102A-120F. A control data path between tile sequencer 106 and tiles 102A-120F illustrates an exemplary path for control signals generated by tile sequencer 106. When received by tiles 102A-120F, the control signals can cause the tiles to perform one or more primitive or complex operations, such as reading data, writing data, and / or processing data, according to one or more functions specified by the control signals.

[0058] Task instruction memory 160B is shared by tiles 102A-120F. Task instruction memory 160B holds programs executable by the tile access cores and tile execution cores. In some examples, task instruction memory 160B is 468 bits wide and 16,000 words deep, and may be organized as banks of 2,000 words, for example, containing error correction code. Multiple banks allow for multiple reads from or writes to task instruction memory 160B per block cycle. DMA descriptors for task instruction memory 160B may include a length field that is a multiple of 64 bytes. When instruction bundles are stored in high-bandwidth memory, they are optimally routed. The most significant bits may be zero-padded to a 512-bit boundary. The sparse accelerator 103 may drop the padded bits before writing to task instruction memory 160B. Each DMA descriptor can transfer one task instruction memory bundle.

[0059] The tile sequencer 106 may be a scalar core primarily responsible for dispatching tasks to tiles and / or initiating DMA transfers. For example, the tile sequencer may receive instructions as bundles. The instructions in a bundle may execute in parallel across tiles 102A-102F, simultaneously updating architectural state. When a bundle executes, it undergoes scalar or vector issuance before the instructions in the bundle are executed. Scalar or vector issuance may be held for one or more cycles due to various conditions documented in the hold-scalar-issue condition for each scalar instruction or the hold-vector-issue condition for each vector instruction. The ISA for the sparse accelerator 103 may define several global scalar-hold-issue conditions or global vector-hold-issue conditions that can hold a bundle unconditionally. During scalar or vector issuance, several actions may be performed: bundle predicates are evaluated and updated; register values ​​to be used may be recorded; and branches within the instruction may be executed, causing registers to be updated.

[0060] The task instruction memory banks may implement one or more of the following interfaces: Each bank may include a prefetch request and prefetch response broadcast bus. The prefetch request and response bus architecture is tailored to the SPMD (single program, multiple data) mode of operation. Read responses may be broadcast to all tiles on a prefetch response broadcast bus. Bundled broadcasting allows the hardware to duplicate requests originating from different tiles, if possible, reducing overall bandwidth demands.

[0061] 2 is a block diagram of an exemplary environment 200 for implementing the hardware circuit 101. The hardware circuit 101 may be implemented on a device having one or more processors in one or more locations, such as within a server computing device 215. The user computing device 212 and the server computing device 215 may be communicatively coupled to one or more storage devices 230 via a network 260. The storage device 230 may be a combination of volatile and non-volatile memory and may be in the same physical location as the computing devices 212, 215 or in a different physical location. For example, the storage device 230 may include any type of non-transitory computer-readable medium capable of storing information, such as a hard drive, a solid-state drive, a tape drive, optical storage, a memory card, ROM, RAM, DVD, CD-ROM, writable memory, and read-only memory.

[0062] The server computing device 215 may include one or more processors 213 and memory 214. The memory 214 may store information accessible by the processor 213, including instructions 221 executable by the processor 213. The memory 214 may also include data 223 that may be retrieved, manipulated, or stored by the processor 213. The memory 214 may be a type of non-transitory computer-readable medium capable of storing information accessible by the processor 213, such as volatile and non-volatile memory. The processor 213 may include one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), and / or application specific integrated circuits (ASICs), such as tensor processing units (TPUs), etc. The processor 213 may be any of the components described herein with reference to FIGS. 1A-1B. As described herein, it may include co-processors and sparse accelerators implemented as part of the hardware circuitry.

[0063] Instructions 221 may include one or more instructions that, when executed by processor(s) 213, cause the one or more processors to perform the operations defined by the instructions. Instructions 221 may be stored in object code format for direct processing by processor(s) 213, or may be stored in other formats, including interpretable scripts or collections of independent source code modules that are interpreted on demand or pre-compiled. Instructions 221 may include instructions for configuring stream transfers in accordance with aspects of the disclosure. Server computing device 215 and / or user computing device 212 may implement a compiler or other program to generate and send instructions to hardware circuit 101 as control signals to configure tiles of the circuit.

[0064] Data 223 may be retrieved, stored, or modified by processor 213 according to instructions 221. Data 223 may be stored in a relational or non-relational database, in a computer register, as a table with multiple and varied fields and records, or as a JSON, YAML, proto, or XML document. The data 223 may be stored in a computer-readable format, such as, but not limited to, binary values, ASCII, or Unicode. Additionally, the data 223 may include sufficient information to identify associated information, such as numbers, descriptive text, proprietary code, pointers, references, or information used by a function to calculate associated data, for example, for data stored in other memories, including other network locations.

[0065] The user computing device 212, like the server computing device 215, may also be configured with one or more processors 216, memory 217, instructions 218, and data 219. The user computing device 212 may also include a user output 226 and a user input 224. The user input 224 may include any suitable mechanism or technology for receiving input from a user, such as a keyboard, a mouse, a mechanical actuator, a soft actuator, a touchscreen, a microphone, and a sensor.

[0066] The server computing device 215 may be configured to transmit data to the user computing device 212, and the user computing device 212 may be configured to display at least a portion of the received data on a display implemented as part of the user output 226. The user output 226 may also be used to display an interface between the user computing device 212 and the server computing device 215. The user output 226 may alternatively or additionally include one or more speakers, transducers or other audio outputs, haptic interfaces or other tactile feedback that provide non-visual and non-audible information to a platform user of the user computing device 212.

[0067] 2 depicts the processors 213, 216 and memories 214, 217 as being within the computing devices 215, 212, the components described herein, including the processors 213, 216 and memories 214, 217, may include multiple processors and memories that may operate in different physical locations rather than within the same computing device. For example, some of the instructions 221, 218 and data 223, 219 may be stored on a removable SD card, within a read-only computer chip, or the like. Some or all of the instructions and data may be stored physically separate from the processors 213, 216. The computing devices 215, 212 may each include one or more internal clocks that provide timing information usable for timekeeping for operations and programs executed by the computing devices 215, 212.

[0068] The server computing device 215 may be configured to receive requests to process data from the user computing device 212. For example, the environment 200 may be part of a computing platform configured to provide various services to users through various user interfaces and / or APIs that expose platform services. One or more services may be a machine learning framework or set of tools for generating a neural network or other machine learning model according to a specified task and training data. The user computing device 212 may receive and transmit data specifying the type of workload or complex operation that the XPU of the sparse accelerator 103 is to be configured to perform. The user computing device 212 may transmit instructions directly to the hardware circuit 101 or may have the server computing device 215 generate and transmit instructions as control signals to the hardware circuit 101, as described herein.

[0069] The devices 212, 215 may be capable of direct and indirect communication over the network 260. The devices 215, 212 may establish listening sockets that may accept initiating connections for sending and receiving information. The network 260 itself may include a variety of configurations and protocols, including the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local network, and a private network that uses one or more company-specific communication protocols. The network 260 may support a variety of short-range and long-range connections. Short-range and long-range connections are generally related to the Bluetooth® standard. The communication may be performed over various bandwidths, such as 2.402 GHz to 2.480 GHz, commonly associated with the Wi-Fi® communication protocol, 2.4 GHz and 5 GHz, commonly associated with the Wi-Fi® communication protocol, or using various communication standards, such as the LTE® standard for wireless broadband communication. Network 260 may additionally or alternatively support wired connections between devices 212, 215, including via various types of Ethernet® connections.

[0070] 2, it should be understood that aspects of the present disclosure may be implemented according to a wide variety of configurations and quantities of computing devices, in paradigms for serial or parallel processing, or via a distributed network of multiple devices. In some implementations, aspects of the present disclosure may be performed on a single device, and any combination thereof.

[0071] 3A is a block diagram of an exemplary tile 102. XPU 101 is coupled to cross-lane controller 310, which provides a separate thread of control that enables cross-lane instructions on XPU 101. As described herein, XPU 101 can receive a first instruction, for example, through one or more control signals, which can be translated into one or more second and third instructions and provided to the processing cells and crossbar of XPU 101, respectively, to perform the complex operation specified by the first instruction. Instructions for XPU 101 are conveyed through control signals. The processing cells and crossbar of XPU 101 may be configured to interpret the corresponding primitive operations to perform them. An exemplary instruction may be an opcode of an instruction set architecture (ISA).

[0072] 1A , the tile 102 can receive data from the on-chip interconnect 108 and also from the on-chip memory 105. The XPU can also receive instructions from an instruction interface 324, for example, from the tile sequencer 106 through the scalar core 312 or the scalar core 320. The scatter-gather engine 322 of the tile 102 can receive incoming data and control which data is passed to the memory 306 via the memory scheduler 314. In some examples, instead of a scatter-gather engine, the scatter-gather engine 322 may be referred to as a read-write engine 322.

[0073] The memory scheduler 314 can coordinate how data is accessed and retrieved from the memory 306. The memory 306 is dedicated to the tile 102 and cannot be accessed by other components connected to the tile 102, such as other tiles. The arbiter 304 is configured to manage which of the vector processing units (VPUs) 302A-302H access the memory 306, for example, on a clock cycle basis. The tile 102 can maintain a task queue 308 of tasks to be executed by the tile 102, which are sent to the scatter-gather engine 322 via the scalar core 320. The tile 102 can also maintain a register of tile synchronization flags 318 and / or memory flags 316 to synchronize the tile 102 with the hardware circuitry and other tiles in the memory 306, respectively.

[0074] The VPUs 302A-302H are connected to the XPU 101 via data processing lanes, indicated by solid lines between the XPU 101 and the VPUs 302A-302H. The dashed lines between the XPU 101 and the VPUs 302A-302H represent control signals that can be received by a control cell within the XPU 101 to configure the XPU 101 to perform a composite operation corresponding to the received control signal. The vector processing units are configured for efficient operations on input vectors. The length of a vector processed by a tile 102 at one time may depend on the number or width of the VPUs implemented by the tile. For example, eight VPUs 302A-302H are eight wide. The VPUs 302A-302H can process data along the same data processing lane. The VPUs 302A-302H may be configured to perform scalar operations on elements of a received vector from the memory 306. VPUs 302A-302H can receive data from XPU 101, which can process the data across data processing lanes rather than just along the lanes executed by each VPU 302A-302H, as described herein.

[0075] 3B is a block diagram of another example tile 392 that implements XPU 101 for stream transfers. Tile 392 can receive data from on-chip interconnect 108 as described with reference to FIG. 1, as well as from on-chip memory 105. XPU 101 can also receive instructions from instruction interface 324, for example, from tile sequencer 106. Scatter-gather engine 322 of tile 392 can receive incoming data and control which data is passed to memory 306.

[0076] This exemplary tile 392 is based on a decoupled access / execution architecture, where a program (and associated instruction sequences) can be separated into two streams. The first stream can be an access stream for fetching operands and storing results. The second stream can be an execution stream that consumes operands, performs computations, and produces results. These streams are based on a decoupled access / execution architecture. The VPU executes on two separate cores, the Tile Access Core (TAC) 332 and the Tile Execution Core (TEC) 330, which form part of the VPU execution architecture. The TAC 332 is a scalar unit used to decouple data movement from data computation, e.g., decoupling fetching data from memory from processing the fetched memory. The TEC 330 is a scalar unit attached to the VPU with multiple SIMD lanes (e.g., eight SIMD lanes) used to process vectors.

[0077] The tile access core 332 is based on the scalar core complex 320 and may be responsible for prefetching operands to be executed from memory 306 or from high-bandwidth memory outside the tile 392. The tile execution core 330 is based on the scalar complex core 312 and includes the XPU 101 and the VPU 302. The tile execution core 330 may be responsible for performing computational operations on the prefetched operands to generate results. The VPU 302 is connected to the memory 306 via a load-store unit 328. The load-store unit 328 is configured to perform gather and scatter operations on data passing through the tile 303. The gather-scatter operations may be performed at a granular level, e.g., at a 4-byte granularity. The load-store unit 328 may implement load and store queues to manage bank conflicts. The LSU 328 may provide load / store access to a subset of the scratchpad memory.

[0078] The TAC 332 and the TEC 330 have independent instruction streams and together form a producer-consumer pair. The tile 102 may maintain a task queue 308 of tasks to be executed by the tile 102, which are sent to the TAC 332 and the TEC 330.

[0079] The task queue 308 may include one or more queues for pushing and popping instructions. For example, the task queue 308 may have a first queue (e.g., a first-in, first-out (FIFO) queue) for pushing values ​​from the TAC 332 to the TEC 330. The TEC 330 may pop and process the enqueued values. As another example, the task queue 308 may include additional queues for connecting the TAC 332 and the TEC 330 in the reverse direction.

[0080] The TAC 332 and the TEC 330 communicate with each other through a tile-local scratchpad memory (SpMEM), such as memory 306. The TAC 332 and TEC 330 can also communicate through scalar memory 334, instruction buffer 326, and tile synchronization flag 318. Memory 306 can be used by the TAC 332 and TEC 330 to exchange data and can be used as a software-managed circular buffer to pass data between the TAC 332 and TEC 330 in first-in-first-out order. The tile synchronization flag 318 can be used as a counting semaphore between the TAC 332 and TEC 330. For example, if a circular first-in-first-out order is used between the two cores, the producer core increments the synchronization flag 318 by the number of bytes after each push and stops when the count reaches the maximum size of the first-in-first-out order. Similarly, the consumer decrements the synchronization flag 318 after each pop and stops if there is no data in the buffer. Because the amount of data prefetched can be dynamic, a done bit is used to indicate the end of the stream.

[0081] The tile synchronization flags 318 may, in some examples, include a set of 32 synchronization flag registers. Each register can store 32 bits of data plus a "done" bit and an "enable_public_access" bit. The tile synchronization flags 318 registers may be implemented as a monolithic flop array, allowing simultaneous access from all sources. In the event of an address conflict between writes, the priority of access between sources can be specified. For example, scalar miscellaneous instructions may have absolute priority. In some instances, only one of the scalar miscellaneous instructions may be issued. Read-modify-write operations are pipelined, and therefore, regardless of any synchronization flags, Back-to-back read-modify-write operations can be supported.

[0082] DMA updates, stream updates, and remote writes may be combined onto a single (external) interface using round-robin arbitration. For example, the external interface from the tile to the external source may have a separate access path to the synchronization flag, e.g., a data path as shown and described with reference to FIG. 1B.

[0083] Each bank of task instruction memory 160B can arbitrate on a cycle-by-cycle basis among the tile-requested data stored in the bank. These banks can use a distributed arbitration scheme to select a winner from the TACs 332 and TECs 330 in sparse accelerator 103 that gains access to the bank. The arbitration can ensure that the requested bandwidth is divided equally among the requested tiles. The requests can be, for example, to prefetch data by each TAC per tile. For example, the winning prefetch request is given the highest priority for access to any of the accessed banks.

[0084] Accesses from the sparse accelerator 103 via the control status register can have the next highest priority, and are only delayed by prefetch read requests. The control status register (CSR) is used by the sparse accelerator The sparse accelerator 103 may be implemented as part of the processor for storing additional information results of machine instructions executed by the sparse accelerator 103. The sparse accelerator 103 may maintain an indirect CSR timeout status bit to determine whether a CSR access is being prevented from completing. DMA reads or writes may be prioritized last, after bank accesses and CSR accesses. A DMA operation, e.g., a read or write, may be delayed if it accesses a busy bank. In some examples, the priority of how access operations are arbitrated and resolved on the sparse accelerator 103 may differ.

[0085] Decoupling data access from data execution has at least the following advantages: Long (e.g., 600 cycles) memory latencies are tolerated more effectively because the TAC 332 can perform address calculations and prefetch the required data. The architecture described above also has a high tolerance for control latencies. Dynamic dependencies can make loop conditions difficult to resolve, which can prevent effective software deployment. The TAC 332 can run ahead to resolve these dependencies if the conditions can be determined outside of the TEC 330, providing a way to hide control latencies.

[0086] FIG. 4 is a block diagram of a tile sequencer 400 according to aspects of the present disclosure. Sequencer memory 410 (Simem) may be one VLIW instruction bundle wide and approximately 8000 bundles deep, although the width and number of bundles in sequencer memory 410 may vary from implementation to implementation. Sequencer memory 410 may be read or written through direct memory access or through indirect access. This can be done whether sequencer 400 is currently executing or not. Sequencer memory DMA memory descriptors may have length fields that are multiples of 32 bytes. When each bundle is stored in high-bandwidth memory, it may be zero-padded from the most significant bit to a 256-bit boundary. The padded bits are used as RAZ / WI (read-as-zero and write) bits. write-ignored: read and write ignored as zero). Each memory descriptor can transfer one instruction bundle.

[0087] The sequencer memory 410 may be organized as two banks with consecutive addresses between the banks. Each bank may be read or written once per clock cycle. Interleaving allows a consecutive instruction sequence to consume only half of the bandwidth of each bank. An abnormally small loop may consume only three-quarters of the bandwidth of a given bank, still leaving enough bandwidth for accesses to proceed forward.

[0088] Each sequencer memory bank in sequencer memory 410 may follow the following interfaces: Sequencer memory 410 may receive read requests from an instruction data path (e.g., the instruction data path shown and described with reference to FIG. 1B). Banks of sequencer memory 410 may also receive DMA writes and reads, as well as control status register accesses via an indirect register interface. Reads from the instruction data path may have the highest priority for accesses to each bank in sequencer memory 410. DMA read, write, and CSR accesses may have equal priority and may be subject to least recently used (LRU) arbitration for non-busy banks performing instruction fetches.

[0089] The sequencer 400 can fetch its instructions from the sequencer memory 410. The sequencer 400 executes the program's thread of control. This involves generating descriptors that are dispatched by a descriptor dispatch unit 413 (for task and stream descriptors) and a DMA unit 414 (for DMA descriptors), respectively. The task descriptors are provided to a tile FIFO 416, which is then executed by the respective tile. The DMA descriptors can be passed to other components of the hardware circuit 101 and to other off-chip components. The sequencer communicates with other cores in the system and coordinates tasks across the accelerator's tiles.

[0090] The current state of sequencer 400 can be determined by reading the corresponding status register. Other registers relevant to the fetching and execution of instruction bundles are the program counter and branch state. Tile sequencer 400 can issue DMA descriptors that can be throttled by hardware. The hardware can leave outstanding a predetermined number of DMAs issued by the sequencer with synchronization flags stored among synchronization flags 418. Throttle throttling can be enabled or disabled as needed.

[0091] Synchronization flags can appear in more than one type. Synchronization flags 418 can appear in the tile sequencer 400. Other synchronization flags can be stored in the TAC and TEC of each tile. All of the synchronization flags can collectively reside in a single address space accessible to DMA operations, atomic remote set / add instructions, and atomic tile set / add instructions. There are several interfaces that can be implemented between synchronization flags in tiles and synchronization flags in the sequencer 400. DMA operations can atomically add values ​​to synchronization flags while they are running and can set a "Done" bit upon completion. Stream operations can atomically add values ​​to synchronization flags while they are running and can set a "Done" bit upon completion. Remote write instructions generate single-word control writes that can atomically set or add values ​​to synchronization flags. These can be from atomic remote set / add instructions used to update remote synchronization flags. They can also be from atomic tile set / add instructions used to update synchronization flags in sparse accelerators. Another interface that can be implemented is a write interface, where a set synchronization flag and an add synchronization flag instruction cause an atomic update to the synchronization flag, optionally including changing the "Done" bit. Another interface that can be implemented is a read interface.

[0092] The synchronization flags 418 may be organized in memory as several banks. Each entry may contain several bits of data, e.g., 32 bits of data, plus a "Done" bit and an "enable_public_access" bit. The banks may perform cycle-by-cycle arbitration separately for the read and write ports, e.g., to prioritize scalar miscellaneous instructions over DMA, stream, and remote write updates. The synchronization flag registers in the TAC and TEC may also store several bits, a "Done" bit, and an "enable_public_access" bit.

[0093] The synchronization flags in the TAC or TEX can be implemented as a monolithic flop array that allows simultaneous access from all sources. Like the sequencer synchronization flags, the tile synchronization flags can be managed according to a priority scheme to avoid conflicts between writes.

[0094] 5 is a block diagram of an example scratch pad memory 500 having memory across multiple tiles of a sparse accelerator according to aspects of this disclosure. Each tile has a scratch pad memory 500, referred to as a tile scratch pad memory or TileSpmem. The tile scratchpad memory 502 may include portions of tile_scratchpad_memory_02. The tile scratchpad memory 502 may be accessed by tiles via a load / store interface implemented as one or more circuits. The tile scratchpad memory 502 may also be used as a buffer local to each tile for moving data in and out of the tile using stream instruction transfers.

[0095] As shown in FIG. 5, each tile scratch pad memory 502 may include several banks, e.g., 32 banks labeled as banks 0 through 31. Each bank may hold a certain amount of data, e.g., 16 kilobytes. Each bank may hold several words, e.g., 4-byte words and a 7-bit error correction code. In some examples, a bank may hold 4096 words. It should be understood that the scratch pad memory 500 may be implemented with a different number of tile scratch pad memories 502, each capable of holding different banks of different sizes. It should also be understood that each bank may be implemented in various examples to store a varying number of words of different sizes.

[0096] Words and banks in the tile scratchpad memory can be accessed through 17-bit instructions and stream addresses, although the exact size and format of the addresses can vary from implementation to implementation. Each bank can implement one or more of the following interfaces: In response to received instructions, the tile scratchpad memory 502 enqueues access requests in a per-bank load queue (not shown) to load or store and add data in the bank and word specified by the received address or address range. To store data, the tile scratchpad memory 502 enqueues access requests in a per-bank load queue from received store or store add instructions. Each bank can also receive read accesses from one or more external sources, such as from DMA or stream requests to read / write data. Exemplary instructions to the tile scratchpad memory 502 include a vector load instruction and a vector store instruction. These and other instructions can be specified in the ISA for the hardware circuit 101 and are stored in the hardware circuit 101 in response to receiving instructions. A vector load instruction may be issued to the bank load queue. Similarly, a vector store instruction may be issued to the bank store queue. A vector store add instruction may refer to a type of instruction that causes the hardware circuit 101 to perform a read-modify-write operation on a target range of addresses within one or more banks of the tile scratch pad memory 502.

[0097] To handle the prioritization of loads and stores on banks of tile scratchpad memory, a store access at the head of a per-bank store queue may have the highest priority for accessing the write port of its respective bank (not shown in FIG. 5). A load access at the head of a per-bank load queue may have the highest priority for accessing the bank's read port. If a load is to the same address as a queued store and the store was from an instruction bundle that issued before the load, the bank may read data from the store queue in order to load the data instead.

[0098] Accesses from sources external to the tile hosting a bank may undergo multilevel arbitration before being presented to the bank. For example, writes or appends from different sources may first undergo per-bank least recently used (LRU) arbitration among those enqueued in the target bank's per-bank write queue. The write queue may be four entries deep, as an example. Similarly, read access requests from various sources may also undergo LRU arbitration. The winning read request may then undergo a second level of arbitration with stream append accesses to read the bank and read requests resulting from load accesses. Stream append accesses perform a read-modify-write operation, which is also enqueued in the per-bank write queue. If a stream append access is at the head of the write queue for a bank, its read request may undergo round-robin arbitration with winning external read requests. The winning read request has the second-highest priority for the bank's read port, behind load accesses. Write requests at the head of a per-bank write queue may have the second-highest priority for the bank's write port, behind store accesses. In the absence of bank contention, a bank can sustain a throughput of at least one stream add operation per cycle.

[0099] Each bank of the tile scratch pad memory 502 includes ports (not shown in FIG. 5) for reading from and writing to the bank. These ports can generate warnings to detect exhaustion. External sources of read and write requests can run out when accessing a bank that is continually accessed by loads, stores, or store adds. The probability of exhaustion can be mitigated by specifying a maximum access of the total number of banks in the tile scratch pad memory 502, for example, allowing access of only a maximum of 8 of the 32 banks in a given clock cycle.

[0100] The warning generated due to exhaustion may be based on a predetermined threshold, for example, the predetermined threshold may be the number of consecutive clock cycles during which an external source does not access the bank for a read or write request.

[0101] When read port exhaustion is detected by a bank in the tile scratchpad memory 502, one or more actions may be performed by the scatter-gather engine. A hold may be issued if the bundle has a load, store, or store add instruction. The per-bank load and store queues may be drained normally. A hold may be issued until a predetermined maximum number of read requests have been serviced by the scatter-gather engine or the scatter-gather engine read queue is exhausted. may continue until is empty.

[0102] When write port exhaustion is detected by a bank in the tile scratchpad memory 502, the following sequence is executed: Issue a hold if the instruction bundle has a store or store add instruction. Each per-bank store queue is drained normally. Issues can continue to be held until a maximum threshold of held issue requests has been serviced by the scatter-gather engine. The threshold and cycle count may vary by implementation.

[0103] In some examples, the synchronization flag memory 412 may be organized as four banks (not shown). Each entry may have 32 bits of data. Each bank in the synchronization flag memory 412 may perform separate cycle-by-cycle arbitration for the read and write ports among the following sources: Scalar miscellaneous instructions have absolute priority; in some examples, only one of the scalar miscellaneous instructions may be issued in any given cycle; read-modify-write operations may be pipelined, so back-to-back read-modify-write operations to any location may be supported; after the scalar miscellaneous instructions, DMA updates, stream updates, and remote writes may be combined onto a single (external) interface via round-robin arbitration and may have the next highest priority; access from the host to the hardware circuit 101 via the control status register has the lowest priority.

[0104] An access request to a bank can come from multiple sources: A core memory network read is a read access dispatched on the core memory network that can originate from a DMA or stream request; an access request can be a core memory network write for a write access from the core memory network that results from a DMA or stream request; an access request can be from a stream address, e.g., an indirect address read from a local tile; an access request can be from an internal stream read or write.

[0105] FIG. 6 is an example block diagram of a scalar core complex 600 of a tile sequencer according to aspects of the present disclosure. The scalar core complex may be a 32-bit scalar VLIW core. The scalar execution pipeline 601 may be one or more circuits configured to execute scalar and miscellaneous instructions, such as fetching, decoding, and executing. The scalar core complex 600 executes prefetched instruction bundles. During execution, one bundle is fetched per block as addressed by the next PC register. Each bundle may include two scalar instructions and one miscellaneous instruction that are decoded and executed simultaneously.

[0106] Scalar execution pipeline 601 may include a 32-bit computation unit including registers 604 and ALU 606. The computation unit provides address calculation, loop control, and scalar operands used to build descriptors. Complex 600 has memory 608 accessible through a load / store interface. Memory 608 is used by complex 601 to store intermediate scalar values ​​during program execution. The core type determines the depth of memory 608, depending, for example, on whether it is part of a tile sequencer or a TAC or TEC of a tile.

[0107] Complex 600 can build descriptors 609 and use descriptor scratchpad registers 610. If an instruction uses descriptor scratchpad registers 610, then at issue, complex 600 will write the specified descriptor scratchpad register address in the instruction. The descriptor is then enqueued into the descriptor issue FIFO 612.

[0108] Cross-lane Processing Unit (XPU) An exemplary implementation of the XPU is described below with reference to the following discussion and accompanying figures. Distributed embedded training is difficult to accelerate because effective acceleration relies on utilizing provisioned computation, available on-chip main memory bandwidth (HBM), and inter-chip communication bandwidth. Exploiting these computations is challenging because they are dynamic and irregular. In addition, performance bottlenecks vary significantly from model to model, and problem spaces are rapidly evolving with new algorithms, optimizers, model architectures, etc. Flexibility in expressing new algorithms and optimizing for different performance bottlenecks is essential.

[0109] The embeddings can be fed into a machine learning model, e.g., a neural network, to perform a machine learning task, e.g., a natural language processing task, or other task that may benefit from using the embeddings. An embedding layer is a layer of a neural network trained to generate an embedding from an input. Once generated, the embedding can then be processed downstream, e.g., by a later layer of the neural network that implements the embedding layer. Models with embedding layers pose unique computational challenges, e.g., due to low computational density, high stress on memory bandwidth, and a large memory footprint. Furthermore, to accelerate these types of models, there can be wide variations in performance bottlenecks from model to model.

[0110] The embedding function or map can be implemented, for example, as a lookup table or as a sparse vector-dense matrix multiplication. For example, the embedding function can be implemented as a matrix that, when multiplied by an input vector, generates a corresponding embedding for the input vector. For example, the input vector can be a bit vector representing the presence or absence of various natural language words in an input sentence to a machine learning model. Because the bit vector can contain elements for a large vocabulary of potential natural words that can form the input sentence, the bit vector will generally be a sparse vector, with, for example, fewer than 50 percent of the elements in the vector having zero values. The output vector obtained by multiplying the input vector by the embedding function matrix is ​​an embedding representing the input sentence.

[0111] The embedding functions can be very large tables, e.g., hundreds of gigabytes in size. As a result, the embedding functions cannot fit into the main memory of a single accelerator or processor, and therefore, embedding generation is distributed across multiple nodes, each with one or more accelerators. The embedding table may be partitioned across multiple devices, e.g., multiple accelerators in a data center pod.

[0112] Such distribution complicates the processing of these embedding layers that implement these embedding functions. Aspects of the present disclosure provide for accelerating various operations for scattering, gathering, uniquely summing, and summing various input values ​​to facilitate the generation of embeddings for individual or batches of input samples. Large embedding tables can be partitioned among multiple accelerators. For ease of explanation, the following description will focus on a single accelerator, e.g., hardware circuit 101. Furthermore, while the examples provided herein describe embeddings for natural language processing tasks such as machine translation, it should be understood that aspects of the present disclosure can provide acceleration for any type of machine learning model that relies at least in part on embedding generation to perform a respective machine learning task. Other examples include recommendation systems, such as content recommendation systems found in domains such as multimedia recommendation, search result ranking, and advertising. include.

[0113] While an example is provided for embedding generation, it should be understood that the same primitive operations described herein can be composed in other ways to address other sparse problems, such as sparse matrix multiplication. This flexibility allows for the acceleration of a variety of sparse problems. In addition to machine learning and deep neural networks, sparse computing can be employed in other problem spaces, such as scientific computing and / or graph analysis.

[0114] On a forward pass of a neural network processed according to one or more accelerators as described herein, the input may be a batch of one or more input samples. The input samples may be processed by one or more accelerators that perform the operations of one or more embedding layers of the neural network. The output of the embedding layer is a batch of one or more embeddings, one for each sample in the input batch. Note that the input samples share a common length, e.g., the number of potential features attributed to each input sample, but the input samples may have some empty or zero-valued feature values.

[0115] In the forward pass, the input batch is partitioned across multiple different accelerators, for example across one or more host devices.

[0116] An input batch can be represented as two vectors: a vector of values ​​and a vector of indices. The vector of values ​​corresponds to the values ​​for each identifier in the input samples of the batch. The indices may point to the location of the values ​​for each identifier in the tensor representing the input batch. The input batch is partitioned so that portions of the input batch are sent to different accelerators. When an accelerator receives the partitioned input batch, it can "uniquify" the input to remove duplicate identifiers across the input batch. Uniquifying the input means removing multiple instances of the same identifier. One reason for such removal is to reduce inter-chip network usage and conserve bandwidth by avoiding redundant accesses to the same identifier. Uniquification avoids redundant lookups on the embedding table. After uniquification, the uniquified input batch is distributed to multiple devices with portions of the embedding table to generate output embeddings. The generated embeddings can be scattered, gathered from the various devices, and returned to other devices requesting embedding.

[0117] During training, gradients representing the error rate between the ground truth embedding and the predicted embedding may similarly be scattered to devices storing partitions of the embedding table to update the respective partitions.

[0118] Aspects of the present disclosure are directed to an XPU for performing data-dependent operations across multiple data processing lanes of a processor. Rather than implementing physically fabricated operation-specific circuitry for each data-dependent operation, the XPU may be configured to perform various operations in response to input signals that configure individual operations performed by processing cells and crossbars arranged as a stacked network in the XPU. The XPU operates across values ​​of multiple SIMD data processing lanes. The XPU may be implemented as part of a coprocessor complementary to a second coprocessor configured for SIMD parallel processing. The coprocessor implementing the XPU may be configured to perform data-dependent operations.

[0119] Aspects of the present disclosure provide an XPU for initially processing sparse data before passing the data to downstream coprocessors in a processing pipeline, enabling a wider range of workloads for more efficient computation than was previously possible without the XPU. Because the XPU can handle a wide variety of data-dependent operations, it can also reduce the complexity of the processing pipeline and corresponding processors. The XPU can be designed without the constraint of predefining input data for processing on existing SIMD architectures. Without the XPU, existing SIMD architectures cannot efficiently accelerate data-dependent operations such as generating embeddings from a sparse set of features into a machine learning model.

[0120] An exemplary data-dependent operation includes generating an embedding for input training examples. The embedding may be a vector or some other data structure mapped from the input having a higher dimensionality than the embedding. The embedding generation may be performed as part of a workload processed according to a pipeline. As another example, an XPU may perform vector scatter or gather operations, segment sums, and / or partition sparse feature tensors. The XPUs described herein may be complementary processing units to other or connected components of a processor, such as vector processing units built according to the SIMD parallel processing paradigm. One or more XPUs may be connected in each processor core of a larger processor, which may itself include other components for accelerating the performance of specific workloads, such as training neural networks.

[0121] Furthermore, the XPU is not limited to performing a specific type of data-dependent operation; therefore, a processor can be designed to include an XPU to complement other types of processing units for multiple different pipelines. Because the XPU can be configured for each workload, the physical footprint of the XPU is reduced compared to other approaches in which specialized circuits are physically fabricated on the processor as complementary units for sparse data computation. The functionality of the XPU can also be expanded by using instruction sets or by extending the host processor's existing instruction set, further improving the adaptability of various data-dependent operations as pipeline data reception changes. Instructions can be provided as signals to the XPU's components, which translate the instructions to configure the XPU's individual processing cells and crossbar. The XPU can be configured using a program compiled by a corresponding compiler for hardware circuitry that implements the XPU.

[0122] The XPU includes a network of individual processing cells, each processing data passing through one or more data processing lanes through a crossbar connection between the processing cells. Each data processing lane may include one or more registers for temporarily storing data during processing. Each processing cell is configured to perform one or more primitive operations on multiple sets of operands. A first set of operands is provided as input from a data processing lane of a processor shared by the processing cells. A second set of operands is provided from a crossbar configured to coordinate data transmission across the multiple data processing lanes of the XPU.

[0123] The XPU can be divided into multiple pipeline stages, each including a crossbar, one or more processing cells, and a control cell corresponding to each processing cell. The number of stages can vary based on, for example, the complex operations the XPU is configured to perform for a current workload.

[0124] An XPU performs complex operations by executing multiple primitive operations across pipeline stages of a stacked network of processing elements and crossbars. A complex operation is an operation performed by an XPU on inputs to produce an output. Primitive operations are operations that individual processing cells of an XPU are configured to perform, which, when executed by an XPU, cause the XPU to perform a complex operation. Performing a complex operation may require the execution of other complex operations, for example, performing a vector sort. To achieve this, the XPU may perform a prefix addition, another operation composed of multiple primitive operations. Exemplary primitive operations include comparison of input data, arithmetic operations, or operations for bypassing. The XPU performs the compound operation by configuring each of multiple individual processing cells and crossbars arranged according to one of multiple pipeline stages for the XPU.

[0125] The primitive operations performed by each stage of an XPU can be programmatically defined and can vary from workload to workload. The primitive operations a processing cell is configured to perform are determined by one or more control signals or instructions received by a respective control cell for the processing cell. The exact primitive operations performed by a processing cell can depend, for example, on the complex operation the XPU is currently configured to perform. In other examples, processing cells in various lanes or stages of an XPU can be configured to always perform one or more predetermined primitive operations. After the XPU generates output, the output can be passed along multiple data processing lanes to another processing unit or memory unit of the processor implementing the XPU.

[0126] Two exemplary composite operations that the XPU can perform are vector sort and vector duplicate count. Vector sort is an in-place, stable sort of (key, value) tuples of an input vector, sorted by key. Vector duplicate count returns a running duplicate count of values ​​of (key, value) tuples of an input vector. The XPU is configured to perform both vector sort and duplicate count according to the same configuration of processing cells and crossbar, as described herein. By using the same configuration, the XPU can perform both composite operations more efficiently, at least because there is no need to reconfigure the XPU between performing vector sort and vector duplicate count for a given input vector. Other composite operations that the XPU is configured to perform include parallel prefix sum, vector split, vector histogram, vector compact, vector replace, vector reduce, vector shift insert, vector gather, vector scatter, etc. Performing vector duplicate count allows the presence of duplicate values ​​to be identified, which can be used to uniqueize vector inputs and avoid redundant processing.

[0127] Aspects of the present disclosure may provide the following technical advantages: A hardware circuit implementing an XPU may provide more flexible and programmable hardware for embedded class workloads and other data-dependent operations that cannot be efficiently parallelized. The XPU provides an acceleration path for different classes of data-dependent operations for different workloads, without the XPU having to be fixed to efficiently perform only certain operations. By providing a programmable unit as described herein, the implementing hardware circuit can robustly adapt to the demands of various workloads and can complement parallelizable data-independent SIMD operations that may otherwise be inefficient or ineffective for workloads requiring data-dependent operations.

[0128] Hardware circuits, such as application specific integrated circuits, can be designed with varying amounts of XPUs to further scale and distribute workloads. The XPUs described herein also enable efficient execution of multiple operations using the same configuration, further reducing processing and configuration time. For example, an XPU may be configured to perform both vector sort and vector duplicate count, instead of separate configurations of the XPU and / or separate instances of dedicated circuitry to accelerate those operations.

[0129] 7 is a block diagram of an exemplary XPU 700. XPU 700 includes processing cells 701- 709, crossbars 703-711, and control cell 750 (represented by hatched blocks in the block diagram of FIG. 7), with data traveling from bottom to top along data processing lanes 700A-700H, starting at stage 1 and ending at stage 6. Stage 1 includes processing cell 701 and crossbar 702. Stage 2 includes processing cell 703 and crossbar 704. Stage 3 includes processing cell 705 and crossbar 706. Stage 4 includes processing cell 707 and crossbar 708. Stage 5 includes processing cell 709 and crossbar 711. Stage 6 includes processing cell 711 and crossbar 712. In another example, an XPU may include more or fewer stages. An XPU may also include a crossbar 799.

[0130] For purposes of discussion, earlier stages are considered to be "upstream" of later stages, and later stages are considered to be "downstream" of earlier stages. For example, stage 1 is upstream of stage 5, and stage 4 is downstream of stage 3.

[0131] The crossbar in each stage of the XPU can be any type of circuit configured to permute various input values ​​from each lane to various other processing lanes according to the current configuration of the crossbar. The crossbar can receive one or more control signals from control cells for each processing cell in the same stage as the crossbar. The crossbar is configured to permute input values ​​from each processing cell in the same stage according to a fixed pattern. The pattern depends on the complex operation the XPU is currently configured to perform and does not necessarily cause the crossbar to permute every processing cell output. In other words, some processing cell outputs may bypass the crossbar and proceed to the next stage along the same processing lane.

[0132] To configure the processing cells, each processing cell of XPU 700 has a respective control cell 750 configured to receive one or more control signals along the respective processing lane in which the processing cell resides. The processing cells are configured with circuitry for performing a wide variety of primitive operations and executing those operations in accordance with received control signals or instructions, as described in more detail with reference to FIG. 7 . The control cells receive instructions along the data processing lanes, for example, as one or more signals that can be interpreted by the control cell to determine which primitive operation its corresponding processing cell will perform. The control cells can forward control signals to the processing cells, or they can process the received instructions or signals and forward generated control signals that the processing cells are configured to receive to enable or disable execution of the specified primitive operations.

[0133] The processing cells can also be configured to bypass input data received from their respective processing lanes. When bypassed, the received input is passed unmodified from the processing cell to the crossbar at the same stage as the processing cell. Inputs received by the bypass processing cell from the crossbar of a previous stage may be associated with zero or ignored. The actual behavior of the bypass processing cell may depend on the pipeline stage in which the cell resides and / or the processing lane in which the processing cell resides. Figure 7 shows an example processing cell configured to perform a comparison, an arithmetic operation, and / or a bypass primitive operation.

[0134] XPU 700 may be configured to receive instructions defined as part of an instruction set architecture or extensions of an instruction set architecture that a processor implementing XPU 700 is configured to apply and execute. These instructions may specify various complex and / or primitive operations that the XPU and individual processing cells are configured to execute as corresponding operations. Control cell 750 is configured to receive data representing instructions defined as part of an instruction set or extension and / or to execute instructions. into control signals for configuring corresponding processing cells. For example, control cell 750 may receive signals as opcodes of an instruction set corresponding to the processor or hardware circuit that implements XPU 700, i.e., codewords for operations that the XPU is configured to perform. When XPU 700 receives an instruction to perform a complex operation such as a vector sort or vector duplicate count, XPU 700 may configure each processing cell to perform a predetermined respective primitive operation that causes the XPU to perform the commanded complex operation.

[0135] The operations performed by the XPU may be synchronized by clock cycles. For example, the operations performed by the processing cells per stage may be performed in one or more cycles. For example, the operations per stage may be performed in a single cycle. Various compound operations performed by the XPU may take different amounts of clock cycles to perform. For example, a vector sort may be performed by the XPU in 6 cycles, by a vector prefix add in 4 cycles, and by a vector compact in 2 cycles.

[0136] As will be described in more detail with respect to Figure 7, the processing cells may be configured to perform arithmetic operations such as addition between operands of various types, including floating-point values ​​and signed or unsigned integers. Arithmetic operations such as addition may form part of the compound operations performed by the XPU for scan operations.

[0137] Exemplary instructions include instructions for resetting the XPU and retrieving information about clock synchronization and primitive operations executed by the XPU. Other instructions include instructions for retrieving one or more both operands, mask values, and / or segment markers from each processing lane. These instructions may include instructions for accessing data structures stored by the XPU along with control information specifying each of the wide variety of compound operations supported by the XPU. In a further example, these instructions may include instructions for causing the XPU to push data into various registers, latches, or flip-flops and determine whether the contents are valid. The pushed data may include, for example, a value being processed as part of the execution of a compound operation and / or a mask value.

[0138] A configured XPU 700 is said to implement a processing network for performing a particular complex operation. For example, XPU 700 includes 48 XPU cells that can be configured as follows: 18 cells can be configured for arithmetic operations, 38 cells can be configured to compare input values ​​(one cell may be configured for both arithmetic and comparison operations), and 10 cells can be configured to bypass inputs. In response to a new instruction, XPU 700 can reconfigure itself with a new processing network to perform another complex operation.

[0139] The XPU may be configured to operate in a wide variety of operation modes, which may be specified as various instructions in the instruction set or extensions. The various operation modes may include various composite operations for sorting and counting duplicates, scanning and partitioning data, and / or identifying unique values ​​within data input to the XPU. Additionally, these instructions may include operands that specify the type of comparison or arithmetic operation to perform, e.g., an unsigned integer compare or floating-point add for sorting or scanning. Other operands to instructions for performing composite operations include specifying which processing lane the output of the composite operation should be output from for output from the XPU 700. Other received operands may include segment markers for performing composite operations on segments of input data, for example, to sort each of multiple segments of data received by the XPU 700 across data processing lanes.

[0140] To perform vector sorting and / or vector duplicate counting, the XPU 700 is configured to include an odd / even merge network and a value shuffle network. The network configuration includes one or more stages of the XPU, with each cell and crossbar in each stage configured to perform one or more primitive operations.

[0141] XPU 700 may include register files 760A and 760B. Register files 760A and 760B may be coupled to data processing lanes 700A-H between different stages and may be used to store and retrieve data. For example, some data may be stored in register file 760B after processing cell 707 in stage 4, while data output by XPU 700 is stored in register file 760A.

[0142] Stream Ordering and Ordering Aspects of the present disclosure provide a hardware or software interface for asynchronous data movement between off-core memory and core-local memory, where the memory-to-memory movement is referred to as a "stream transfer." Stream transfers may include stream descriptors that allow software to express common data movement patterns, such as those found in sparse workloads. Data may be referred to as streams or data streams. Stream transfers may be initiated by a stream instruction. The stream instruction may encode information necessary to perform the stream transfer. Each stream may have a related relationship, indicated by a data synchronization flag ("sync flag") associated with the stream instruction. Stream instructions issued by cores with the same stream ID may at least partially form a single stream.

[0143] A stream descriptor is an internal data structure that can represent information necessary to perform a stream transfer. For example, the information can include source and destination addresses, stream opcodes, and control information such as linear or circular buffers.

[0144] A stream transfer may only move data to or from core local memory. Additionally, a core local synchronization flag may be used to track the progress of the stream. The synchronization flag can track partial progress of the stream transfer. For example, depending on whether the flag is cleared or set according to a predetermined configuration, the synchronization flag tracks reads from core local memory when core local memory is the source and tracks writes to core local memory when core local memory is the destination. The progress of reads and writes to off-core memory may not be tracked, but a scalar fence instruction can be used to allow selection of which memory accesses the barrier to ensure that outstanding writes to off-core memory are committed.

[0145] Stream transfers may include indirect scatter-gather memory accesses in either off-core memory or core-local memory. The address for the source or destination of the access may be stored in a different memory location relative to the address that is initially read. As an example, the indirect address may be sourced from a register file or from memory with masking support. Indirect scatter-gather memory accesses may further include various addressing modes, such as row addresses or word addresses, as examples. Stream transfers may include support for ScatterAdd / GatherAdd modes directly on memory words. Memory words may be updated atomically. As examples, 32-bit floating point, 32-bit integer, 16-bit floating point, and 16-bit integer addresses may be used. The byte integer data type can be supported.

[0146] Stream transport may include support for circular buffers within the source or destination buffer, which can simplify the buffer allocation problem for software since the buffer size is not known during software compilation.

[0147] Additionally, a stream ordering model is generally disclosed herein. Synchronization primitives for these transfers allow data transfers to be processed in order, while the actual data transfers are out of order. Discrete stream instructions issued by a core with the same stream ID form a single stream. Hardware guarantees ordering for transfers within a single stream, which may span multiple stream instructions.

[0148] Stream instructions belonging to a stream are processed in order. For indirect stream instructions, the offset list is ordered; for example, the offset elements in the offset list are processed in order. Writes are issued to the destination memory in order and can be committed out of order by the destination memory. Reads are issued to the source memory in order and can be serviced out of order by the source memory.

[0149] The sync flag is updated to indicate monotonic incremental progress for the stream. When core local memory is the source, the sync flag tracks reads to the core local memory. A sync flag value of N when core local memory is the source of a read indicates that the first N chunks of data can be overwritten in core local memory, where a chunk of data is a predetermined unit of size for measuring data in core local memory. When core local memory is the destination, writes to the core local memory are tracked by the sync flag. A sync flag value of N in the example where core local memory is the destination indicates that a subsequent read to the first N chunks of data in core local memory will return the requested data.

[0150] A stream can be terminated when the data for the request preceding and including the last stream descriptor is fully committed to memory. As an example, if core local memory is the source, the stream can be terminated when all reads are complete. If core local memory is the destination, the stream can be terminated when all writes are committed.

[0151] Aspects of the present disclosure enable software to more efficiently express common data movement patterns, particularly those found in sparse workloads. Aspects of the present disclosure can also provide effective solutions to the complexity for hiding long memory access latencies while maintaining an in-order core compute core and software programming model.

[0152] Stream transfers allow tiles 102 and tile sequencer 106 to move data between tile-local memory, such as memory 306 or scalar memory 334, and off-tile memory, such as memory 105 or high-bandwidth memory 107. Tile-local memory is an example of core-local memory, such as memory 306, scalar memory 334, as memory that is physically local to sparse accelerator 103. Off-tile memory is an example of off-core memory, as memory physically remote from TEC 330 or TAC 332 can include memory 105 and / or high-bandwidth memory 107. Each stream has an associated stream ID, indicated by a synchronization flag 318 associated with the stream instruction. Discrete stream instructions with the same stream ID are synchronized. Create a single stream with a shared stream ID.

[0153] Stream transfers allow data to be moved to or from tile-local memory. Tile-local synchronization flags 318 are used to track the progress of the stream. The synchronization flags 318 track the progress of portions of the stream transferred to and from various memories through the tiles 102. For example, the synchronization flags 318 track read operations (or “reads”) from the tile-local memory when the tile-local memory is the source, or the synchronization flags 318 track write operations (or “writes”) to the tile-local memory when the tile-local memory is the destination. The progress of reads and writes to off-tile memories may not be tracked. To ensure that all outstanding writes to off-tile memories are committed, a scalar fence instruction can be used to select which memory accesses the barrier. The scatter-gather engine 322 tracks the status of issued stream transfers for each specific memory and communicates this status to the scalar core 320. When a scalar fence is issued for a barrier on a particular memory, the scatter-gather engine 322 waits for a status indicating that all outstanding stream transfers targeting that memory (read or write) are fully committed. Once that condition is met, the fence wait state is released on the scalar core 320.

[0154] Stream transfers can support efficient scatter-gather operations using strided streams to access off-tile memory and indirect streams to access off-tile memory from tile-local memory or register files. The choice of strided or indirect streams can be based on software access patterns. If software wants to access every Nth element in a tensor, strided streams are preferred, but indirect streams can still work. However, if software wants to access a random set of elements in a tensor, indirect streams should be used. Stream transfers can also support circular buffer semantics on tile-local memory.

[0155] Stream transfers support the following data movements, where the granularity and alignment of the data movement depend on the memory source and destination pair. Data can be transferred from memory 306 to on-chip memory 105 and from on-chip memory 105 to memory 306. Data can also be transferred from memory 306 to high-bandwidth off-chip memory 107 and from off-chip memory 107 to memory 306. Data can also be transferred from scalar memory 334 to on-chip memory 105 and from on-chip memory 105 to scalar memory 334. As an example, the minimum granularity, source alignment, and destination alignment can be 4 bytes. As another example, 32-byte accesses can be used to support 4-byte accesses to off-chip memory 107. As yet another example, a 32-byte alignment and a minimum length of 128 bytes can ensure performance for streams to or from off-chip memory 107.

[0156] Each tile in the processor may implement its own Scatter-Gather Engine (SGE) to coordinate the movement of data from the tile to scratchpad memory and / or between various memories. These various memories may include scratchpad memory, off-tile memory including high-bandwidth memory, and on-tile memory. An SGE can support multiple outstanding stream requests that may originate from either or both the TEC and TAC of the tile that implements the SGE. Requests to issue may be processed as gather operations by the SGE, and requests to write data may be processed as scatter operations. The SGE may also be implemented in a tile sequencer to handle reads and writes to data streams between the sequencer and tiles and / or memory.

[0157] 8 is an example functional diagram of the scatter-gather engine 1500. The SGE 1500 can support a variable number of execution threads, e.g., eight threads. A thread may be selected based on the stream identifier for the incoming request and the address generator thread and stream type, e.g., availability of high-bandwidth memory and / or scratchpad memory. In some instances, a stream request with a stream identifier in-flight on the address generator thread needs to be mapped to that same thread.

[0158] The SGE 1500 can ensure ordering for a particular request of the same type (e.g., gather or scatter requests) across multiple streams that belong to the same stream and target the same external interface (e.g., a processor's crossbar or other interconnect) originating from the same tile core. These requests can be identified as belonging to the same stream if they have the same stream identifier. A synchronization flag managed on the tile can be used to track ordering between requests, as described herein.

[0159] The SGE1500 receives scatter / gather requests, unpacks the requests, moves the data on the processor interconnect to a remote Spmem slice or the core memory network CMN to high bandwidth memory, and sends synchronization flag updates to the tile synchronization flag memory of one of the cores, updating it as the transaction progresses.

[0160] The SGE 1500 also services DMA requests received from the CMN interface to write to and read from tile Spmem slices, and handles reads and writes originated by SGEs of remote tiles that target Spmem slices local to this tile.

[0161] Stream requests can be one of several different types. Stream requests can include scatter-gather requests from a tile core. These requests can be processed by a scatter-gather engine, which itself can be implemented on the tile and / or tile sequencer. One type of stream request is a linear request, in which the SGE can expand the request into multiple smaller requests based on the length of the stream request. Another type of stream request is a strided request, in which the SGE expands the request into multiple requests based on the stride and length. Another type of stream request is an indirect request, in which the SGE expands a list of addresses of the same length, which are expanded into separate requests. Another type of stream request is an indirect request, in which the SGE receives a list of addresses.

[0162] The SGE 1500 may implement several stages, for example, a descriptor dispatch stage 1500A, an address generator stage 1500B, and a data transfer stage 1500C. Each stage will be described in turn.

[0163] The SGE can interface directly with the descriptor generators in each tile core, e.g., the TAC and TEC. The descriptor generators are configured to enqueue stream descriptors generated by the cores. Stream descriptors can be sent to the descriptor generator, where metadata related to the descriptor is enqueued into a descriptor issue FIFO for execution. The actual descriptor is written to the descriptor RAM. The contents of the FIFO may include a pointer into the descriptor RAM where the descriptor resides, the memory type, and the stream identifier attached to the corresponding stream. If valid stream descriptor metadata is available at the head of the TAC or TEC FIFO, the following sequence may occur:

[0164] First, the SGE may perform a resource check on the metadata of each core. Next, the SGE may perform least recently used arbitration to select between the two cores, assuming both have the required resources as determined by the resource check. If only one of the cores has the required resources, it will be the next core to be serviced. The stream identifier attached to the metadata is looked up in the stream identifier to address generator map, and an associated counter is incremented or a new entry is created in the map to select the address generator thread in the address generator stage to which the descriptor should be sent. The metadata is then queued in a descriptor metadata queue associated with the selected thread. A resource check may be performed on the descriptor FIFO associated with the descriptor metadata queue. Least recently used arbitration selects one of the descriptor metadata queues. In this case, the winning entry is popped and its metadata is forwarded to the descriptor RAM request FIFO. The request is popped and sent to the descriptor RAM, and the data response stored in the descriptor FIFO is forwarded to the address generator stage.

[0165] The stream identifier-to-address generator map is used to calculate the address generator thread in address generator stage 1500B to which descriptor metadata should be sent, as well as to ensure that ordering is maintained between subsequent stream requests belonging to the same stream. This structure can hold active stream identifiers, the address generator threads to which these identifiers are mapped, a bit indicating that the mapped stream identifier to the thread identifier entry is valid, and a count of the number of descriptors belonging to this stream in the pipeline from the queue in the stream FIFO.

[0166] The above structure can be sized to hold various maximum stream identifiers, for example 16. Each time a stream descriptor is issued with a currently inactive stream identifier, available space in the descriptor metadata queue, and / or available space in the stream identifier-address generator map, the SGE 1500 can perform one or more actions: The SGE selects the next available thread, stores this value in the stream identifier-address generator map, and increments the counter associated with this thread by one. Any subsequent results for the same stream will now be sent to the same address generator thread and will also increment the counter.

[0167] If there is no space available in the address generator thread's descriptor metadata queue for this request, or if there is no space to allocate a new entry in the address generator map in the stream identifier (if this is a new stream identifier), or if the counters in the map are nearly full, the FIFO will not be dequeued until space is free in all required resources. This check is done before arbitration between the two core FIFOs.

[0168] Address Generator Thread in Address Generator Stage Between the threads, four are associated with the CMN interface and the other two are associated with the SC data crossbar. The CMN interface address generator threads deploy requests that target the HBM, while the SC data crossbar interface address generator threads target remote Spmem or tile SpmemN. To determine the next available thread to store with its thread ID, the stream interface metadata determines which address generator thread to choose from, and LRU arbitration is used among the associated threads that also have space in their corresponding descriptor metadata queues.

[0169] When a stream request is expanded by the address generator thread in the address generator stage and the synchronization flag tracking structure in the data transfer stage is updated, a counter in the stream identifier to address generator map can be decremented. The decrement updates to this counter from the data transfer stage come from four parts of the data transfer pipeline: stream scatter to core memory network (xl), stream gather to core memory network (xl), stream scatter to processor data crossbar (x2), and stream gather to processor data crossbar (x2). This is done by the last request to be expanded into a descriptor by the address generator thread.

[0170] When the counter associated with a stream identifier in the stream identifier to address generator map reaches zero, the map entry can be invalidated and any subsequent requests to the same stream can be remapped to any of the available address generator threads for that interface (CMN / Data Crossbar) since they will eventually reach the same synchronization flag tracking structure which maintains ordering between them.

[0171] For example, a stream ID is initially mapped to thread_iD#2 over a window of time when the descriptor is in-flight (up to the data transfer stage). An entry will be invalidated if no more new descriptors are created with the same stream ID for a while (the associated counter goes to zero). If a new descriptor is created with the same stream ID after the mapped entry has been invalidated, the new map created may have the stream ID mapping thread_id #1, but this is not likely to be a problem since space has already been allocated in the synchronization flag tracking structure in the data transfer stage that is responsible for maintaining ordering. Note that this assumes that thread_id#1 and thread_id#2 belong to the same stream interface type (CMN / data crossbar) and are tracking updates to the same memory (scatter vs. gather).

[0172] Not all information from the stream FIFO needs to be enqueued into the descriptor metadata queue, as some of it is only used in the early part of the descriptor dispatch stage 1500A. Only the addresses to the descriptor ram are needed in the later part of the stage to issue reads to the descriptor ram. All other metadata carried in the FIFO is used only to map descriptors to specific address generators. After this point, this metadata can be discarded. The descriptor metadata queue only carries a pointer to the descriptor ram.

[0173] At the output of the descriptor metadata queue, there is LRU arbitration among all address generator threads to gain access to the descriptor RAM in a manner that maintains fairness.

[0174] Referring to the address generator stage 1500B, descriptors from the descriptor FIFO are popped by the stream descriptor manager logic within the address generator stage and then passed to the address resolution state machine for further resolution into requests to remote memory. Each sub-block of the address generator is described in further detail below.

[0175] The stream descriptor manager logic is a state machine that expands the address list of the indirect stream before passing the descriptor to the address expansion state machine. Note that although this specification describes the logic in the form of a state machine, an actual implementation would not explicitly label the states in this way. This was done to simplify the code structure. However, the actual functions performed by the logic do not change and remain the same as described herein. It performs the following tasks in each state:

[0176] Idle State: In this state, the logic pops from the descriptor data FIFO and checks the off_tile_stream_type field of the descriptor. In the Expand Address List state, this state is reached only if there are more than 4x addresses to expand into the address list of an indirect stream, for example.

[0177] The Address Resolution State Machine is used to resolve each stream descriptor presented to it into one or more read / write requests to the Core Memory Network Interface (directed towards HBM) or the Data Crossbar Interface (directed towards remote Spmem).

[0178] Note that the address unwrapping state machine must indicate to the data transfer stage the last stream scatter or gather request unwrapped from each stream descriptor. This information is passed to the address unwrapping state machine from the stream descriptor manager logic as part of the descriptor metadata by indicating the last entry in the address unwrapping input FIFO associated with a particular stream descriptor. This information is needed by the data transfer stage to track the descriptor "committed" and "retired" states for fence instructions.

[0179] In the data transfer stage 1500C, the SGE 1500 handles scatter and gather formatting of output streams to match the interface requirements of the CMN and data crossbar. It manages DMA access to Spmem and synchronization flag tracking for scatter and gather raw streams. This pipeline stage also handles incoming read and write accesses from remote tiles.

[0180] The SGE 1500 increments transaction counters to track the status of fence instructions based on the source core ID and target remote memory. For fences, for example, there may be six counters per core type [12 total] - Spmem (Retired Write, Committed Write, Retired Read), HBM (Retired Write, Committed Write, Retired Read). Each transaction that wins arbitration to the data transfer stage will increment both the retired and committed counters of the associated memory and core type.

[0181] The SGE 1500 sends a decrement to the descriptor dispatch stage for the fence descriptor counter present there if this is the last transaction associated with the descriptor being deployed. This can also be done for the first transfer associated with the descriptor, but since the "last transfer" status is already available, if the decrement is based on the last transfer, no additional information needs to be tracked. This also ensures that an incorrect status is not provided to the core complex. The decrement is performed at the granularity of the next interface transfer.

[0182] SGE 1500 sends a decrement to the descriptor dispatch stage for the synchronization flag id to the address generator map if this is the last transaction associated with the descriptor being deployed.

[0183] Because the descriptor committed state for a stream gather is the same as the descriptor retired state, stream gathers need to be tracked by only one type of state counter. For stream scatters, the two counters will be updated at different points in the flow. Also, because the granularity of updates in the pipeline that update the synchronization flags may be different from the granularity at which requests are tracked to remote memory, logic needs to maintain state in these two separate parts of the data transfer stage.

[0184] The data transfer stage 1500C maintains a counter per source core per remote memory type that is incremented by each instance of the synchronization flag tracking logic. The counter is incremented when a synchronization flag message is enqueued to the message router interface by the synchronization flag tracker. Synchronization flag updates sent to the message router associate the remote memory and source core with the transaction to which the update belongs and are sent back to the SGE once the synchronization flag update is completed to decrement these counters in the data transfer stage.

[0185] Each synchronization flag tracker also maintains status as to whether there are any ongoing transactions being tracked by it (per source core per memory type).

[0186] As long as there is something "in flight" in the synchronization flag tracker for a stream scatter to a particular interface, all descriptors for that memory type are not yet "committed" or "retired." Note that if the stream scatter "retired" counter for a particular memory type is 0, but the associated synchronization flag tracker still has transactions "in flight," the fence status for that memory will still indicate that not all are "committed" or "retired."

[0187] In the case of stream gathers, the current status to the descriptor dispatch stage cannot be set to "committed" or "retired" as long as there are "in flight" transactions in the synchronization flag tracker, even if all "retired" counters have been decremented. When the synchronization flag tracker for a particular memory reports that nothing is being tracked there, the status can be updated to the descriptor dispatch stage.

[0188] The descriptor tracking logic in the data transfer stage 1500C sets the running fence status to the descriptor dispatch stage 1500A.

[0189] As described herein, the SGE 1500 may be implemented as part of the tile sequencer as well as in each tile. The scatter-gather engine within the tile sequencer will be a parameterized version of the SGE subsystem described above, with some differences detailed in this paragraph. The "local" memory in the case of the tile sequencer is always shared memory, which has lower read / write bandwidth requirements compared to scratchpad memory.

[0190] A stream descriptor is a data structure that can represent all the information for the scatter-gather engine 322 to perform a stream transfer. The fields for a stream descriptor can be fully encoded. Below is an example of the fields for a stream descriptor:

[0191] For stream opcodes, the gather stream reads the off-tile memory and stores or adds data to the tile local memory. The scatter stream reads from the tile local memory and stores or adds data to the off-tile memory. The off-tile memory and tile local memory are determined by fields such as the off-tile memory type and the tile local memory type, respectively.

[0192] Additional variants of the stream instructions may support both floating point and signed integer add operations. Gather signed integer add and gather variants of floating point add may be supported for tile local memory. Variants of scatter signed integer add and scatter floating point add may be supported for off-tile and tile local memory. If an illegal combination is detected, a program error may be signaled by the engine.

[0193] The tile-local stream type indicates the address pattern used to access the tile-local memory. For example, a linear stream facilitates several consecutive words starting from the tile-local start offset. The number of consecutive words may have a length of 4 bytes. As another example, a circular buffer stream allows software to build a logical circular buffer in the tile-local memory. In this exemplary access pattern, the base, size, and offset fields in the circular buffer metadata are used to generate addresses for several words. The number of words may have a length of 4 bytes. If the effective length of the granularity is larger than the size field in the circular buffer metadata, a program error may be presented.

[0194] The off-tile stream type indicates the address pattern used to access off-tile memory. A linear stream facilitates accessing several consecutive locations starting from an off-tile starting offset. The actual word size depends on the off-tile memory type. A strided stream facilitates converting strided access patterns to multi-dimensional arrays stored in off-tile memory types. Stream transfers can support a single level of stride. An indirect stream allows for a random scatter-gather access pattern to a table. Here, an indirect offset list is used, and each entry in the list accesses the same length of data.

[0195] The source of the indirect offset list can be a tile local memory or a register file. If the source is a tile local memory, the indirect offset field contains a starting offset into the tile local memory where several offsets are stored. If the source is a register file, the indirect offset field contains the register file and indicates several lanes containing valid offsets. These offsets are used to perform a scatter or gather operation, as indicated by the stream opcode.

[0196] The core type indicates the type of core in the tile 102 that generated the stream descriptor, such as a tile executor core 330 or a tile access core 332. The sync flag core type indicates the core type of the sync flag 318 that tracks the progress of the stream. The encoding may be the same for a core type that allows for stream initiation by a tile access core tracked by a tile executor core and a core type that allows for stream initiation by a tile executor core tracked by a tile access core.

[0197] The sync flag ID indicates an offset in the target sync flag memory. The sync flag ID can also be used as a stream ID to ensure ordering, as described further below. The set done bit indicates that the current descriptor is the last in the stream. The done bit is set after all data for the current and preceding descriptors in the stream has been fully committed to tile-local memory.

[0198] The sync flag count type indicates the type of count that the sync flag 318 is tracking, whether it is the number of words or the number of descriptors. In either case, the sync flag 318 tracks the monotonic stepwise progression of the stream at different granularities.

[0199] The tile local memory type indicates the type of tile local memory involved in the stream transfer, and may include scalar memory 334 or a local bank of memory 306.

[0200] The tile-local starting offset field is used when the tile-local stream type is linear. It indicates the aligned starting offset word, such as a 4-byte word, in tile-local memory that is accessed by this transfer. The actual access type depends on the stream opcode.

[0201] The tile-local stride encodes the stride size and number of bytes accessed per stride, which is used to access the tile-local memory selected by the tile-local memory type. The length, which may be 4 bytes as an example, does not need to be a multiple of the number of bytes accessed per stride. The last request of the strided access will access the remaining transfer words, which may be less than the per-stride length. The stride calculation may be the same for both linear and circular buffer stream types.

[0202] If the tile-local stream type is a circular buffer, the circular buffer metadata field is used. The size of the circular buffer can be a multiple of the granularity of the off-tile memory type, and the offset can be aligned. If the circular buffer wraps around, the request is split into multiple requests, and an error can be signaled if the resulting requests are not a multiple of the granularity of the off-tile memory type. An error can also be signaled if the total length of the stream transfer is greater than the size of the circular buffer.

[0203] The off-tile memory type indicates the type of off-tile memory involved in the transfer. This includes on-chip memory 105 and high-bandwidth memory 107. A high-bandwidth memory view is also available, which allows access at 4-byte granularity and 4-byte alignment. If the sequencer 106 is the initiator of the stream transfer, this field may not have high-bandwidth memory 107 encoded.

[0204] The Tile ID field can be used to select a tile ID for the memory slice. The Off-Tile Starting Offset contains the starting offset word within the off-time memory 105 indicated by the associated off-tile memory type. The units of the offset may be equal to the value indicated in the Offset Alignment column within the off-tile memory type. For example, for high-bandwidth memory 107, an offset value of 1 would translate to byte address 32. If the off-tile stream type is indirect, this field may serve as a base address that is added to the offset read from the indirect offset list before accessing the memory.

[0205] The indirect offset can be used when the off-tile stream type is indirect. If the source is a tile local memory, the indirect offset provides the word starting offset within the tile local memory that stores the indirect offset list. If the source is a register file, the indirect offset provides the file register index that is the source of the indirect offset list. The register file can be read when a stream instruction is issued.

[0206] If the off-tile stream type is indirect, the indirect list size can be used. If the source is tile-local memory, the number of elements in the offset list is stored in tile-local memory. If the source is a register file, the number of lanes containing valid offsets is stored. Completion of transfers is kept in order with the remaining descriptors in the stream.

[0207] The indirect list type can be used when the off-tile stream type is indirect and indicates the type of offsets stored in the offset list. This may include word offsets and row offsets. The indirect list stride is used when the off-tile stream type is indirect and indicates the distance between two address words in the offset list stored in the tile local memory. This may be a signed integer.

[0208] The indirect filter field is used when the off-tile stream type is indirect, and if this field is set, indirect memory addresses that match the indirect filter value will be filtered out. The indirect filter value indicates the value of an element in the indirect access list that needs to be filtered out. This value is of the type indicated by the indirect list type. If the indirect filter field is set for an indirect stream and / or if the value of an element in the indirect offset list matches this field, filtering can be enabled. Off-tile accesses and tile-local accesses that correspond to the filtered elements will be dropped, but the tile-local buffer will still be advanced by the size of the filtered access.

[0209] The length, e.g., a length of 4 bytes or a multiple of 512 bytes, indicates the total number of words accessed by the stream. If the off-tile stream type is linear or strided, this field indicates the total number of words accessed by the stream. If the off-tile stream type is indirect, this field indicates the number of words accessed from each address in the indirect offset list. A program error may also be indicated if the actual value of this field is not a multiple of the granularity of the off-tile memory type. A program error may also be indicated if the generated address exceeds the boundaries of the off-tile memory 105.

[0210] The stride size field indicates the stride size in units of the granularity of the off-tile memory type. It may be a signed integer. The per-stride length, e.g., a per-stride length of 4 bytes or a multiple of 512 bytes, indicates the number of words accessed per stride. Although this is a signed field, it should contain a non-negative value. The length does not need to be a multiple of this field. This field should be a multiple of the granularity of the off-tile memory type selected by this stream descriptor. The final request for a strided access will access the remaining transfer words, which may be less than the per-stride length. If the per-stride length is 0, negative, or not a multiple of the off-tile memory access granularity, a program error may be indicated. A program error may also be indicated if the generated address exceeds the boundaries of the off-tile memory 105.

[0211] The trace field indicates whether the stream transfer should be traced. The trace may include logging information about actions taken during the stream transfer as part of debugging.

[0212] 9 is a flow diagram of an example process 1600 for expanding a stream descriptor into a configuration off-tile stream request or a tile-local stream request. The example process 1600 may be executed on a system of one or more processors in one or more locations. For example, the hardware circuit 101 may perform the process 1600, as described above.

[0213] As shown in block 1610, the process includes a process for receiving a size of the off-tile memory, such as receiving the size in 4 bytes. Additionally, the process includes a process for receiving a maximum chunk size for stream requests targeting the off-tile memory type. As shown in block 1620, the process further includes a process for converting an indirect offset read from a file register or tile local memory to an offset, such as a 4-byte offset, to the off-tile memory based on the indirection list type.

[0214] As shown in block 1630, the process also includes generating strided and / or indirect requests. For strided requests, the process may include partially expanding a strided stream descriptor into a set of requests, each accessing consecutive addresses in the off-tile memory. For indirect tile local memory requests, the process may include obtaining an indirect stream descriptor and generating a list of offsets to the off-tile memory type selected by the descriptor. For indirect file register memory requests, the process may include generating a list of offsets from a file register that is read upon issuance of the indirect stream instruction.

[0215] As shown in block 1640, the process includes generating a list of unrolled off-tile memory requests, where each unrolled request accesses a set of consecutive addresses in the off-tile memory. These requests are used to generate both tile-local memory requests and off-tile memory requests. The tile-local stride, tile-local stream type, and alignment are considered while unrolling the requests. The process further includes generating a list of partially unrolled requests, where each partially unrolled request accesses a set of consecutive addresses in the off-tile memory. These requests are further unrolled to generate a set of requests aligned to the memory granularity selected by the off-tile memory type.

[0216] As indicated at block 1650, the process includes expanding the stream descriptor into a set of off-tile and tile-local memory requests.

[0217] FIG. 10 is a flow diagram of an example process 1700 for ordering stream transfers. The example process 1700 may be executed on a system of one or more processors in one or more locations. For example, hardware circuitry 101 may execute process 1700 as described above. Although discrete stream instructions issued by cores with the same stream ID form a single stream, ordering may not be guaranteed between separate streams. The scatter-gather engine 1722 includes multiple threads that can process these requests in parallel. Ordering may be guaranteed for transfers within a single stream, which may span multiple stream instructions.

[0218] As shown in block 1710, stream instructions belonging to a stream are processed in order. Their corresponding requests are processed in order by the scatter-gather engine 322. It will be issued soon.

[0219] As shown in block 1720, for indirect stream instructions, the offset list is ordered. Offset elements in the offset list are processed in order. Writes are issued to the destination memory in order, but the writes may be committed out of order by the destination memory. Reads are issued to the source memory in order, but the reads may be serviced out of order by the source memory.

[0220] As shown in block 1730, the scatter-gather engine 322 updates the sync flag 318 to indicate monotonic incremental progress for the stream. If the tile local memory is a source, the sync flag 318 tracks reads from it. The sync flag value indicates the first chunk of data that may be overwritten in the tile local memory. If the tile local memory is a destination, the sync flag 318 tracks writes to it. Here, the sync flag value indicates that a subsequent read to the first chunk of data in the tile local memory will return the requested data.

[0221] As shown in block 1740, the done bit in the synchronization flags 318 may be updated at the end of the stream. This is indicated by a set done bit in the stream descriptor. The done bit may be set after all data for the requests preceding and including the last stream descriptor has been fully committed to memory. If tile-local memory is the source, all reads have completed, and if tile-local memory is the destination, all writes have been committed.

[0222] Figure 11 is an example diagram of stream ordering. Consider stream descriptor A and stream descriptor B that make up a stream. Stream descriptor B has the done bit set. The partial progress of the stream is tracked by a synchronization flag. When A0 is committed to memory, either by reading or writing, the synchronization flag is updated to a value of 1. Even though A2 and B1 are committed before A0, the synchronization flag value is not updated to 3. When A1 is committed to memory, five consecutive chunks of data in the stream—A0, A1, A2, B0, and B1—are committed, as indicated by a synchronization flag value of 5. The done bit is not set at this point because stream descriptor A is not the last in the stream. When B2 is committed, the synchronization flag value is set to 6. Now, the done bit can be set when all data chunks for the stream have been committed and stream descriptor B becomes the end of the stream.

[0223] Cooperative Prefetching Aspects of the disclosed technology provide an instruction prefetch pipeline architecture that can be used with sparse accelerator tiles, which provides good performance without the complexity of full cache coherent solutions deployed in conventional CPUs.

[0224] Aspects of the disclosed technology provide methods and systems related to creating a prefetch pipeline around the SPMD aspect of the programming model to reduce cold cache miss overhead. Prefetch responses from any core are broadcast to all cores in the sparse accelerator. These prefetch responses are committed to the core's local cache. This allows other non-requesting cores to obtain bundles of instructions or data before the core would be available to process them, avoiding missing process cycles entirely. Additionally, there may be prefetch request filtering on the arbitration path, which is a logic and / or hardware-based path to arbitrate between requests and provide task instruction memory, thereby facilitating task execution by avoiding redundant request fetches. This boosts instruction memory bandwidth.

[0225] The Task Instruction Memory holds a set of programs that can be executed by the Tile Access Core (TAC) and the Tile Execute Core (TEC). The program counter (PC) in each core is used to A task instruction memory is a physical offset into the instruction memory. The task instruction memory is a software-managed memory exposed to a direct memory access system. Software can use direct memory access to populate a program in the task instruction memory and use the appropriate program counter while issuing tasks to tiles. Because tiles operate in single-program, multiple-data mode, at any point in the execution of the sparse accelerator, statistically most tiles are likely to be executing the same program. These programs may also consist of a single instruction loop or a compact instruction loop. A compact instruction loop may refer to a size of instructions in memory that is small enough to fit into the tile memory. The program itself may be small in size, e.g., a bundle of several hundred instructions, and the program may have multiple branches that can branch within the loop or branch to other loops.

[0226] These and other potential features may be utilized by an instruction pipeline such as that described herein with reference to, for example, Figures 12 and 13. When an instruction bundle is received by the sparse accelerator, the sparse accelerator is configured to broadcast a prefetch response for the received instruction to all of the tiles within the sparse accelerator. The prefetch response is committed to each core's local cache, allowing non-requesting tiles to obtain the bundle in advance, avoiding misses entirely. Additionally, prefetch request filtering on the arbitration path leading to the task instruction memory may be implemented in some examples to boost task instruction memory bandwidth by avoiding redundant request fetches.

[0227] FIG. 12 illustrates a logical diagram of the connections between tiles 1901 and 1902 of an exemplary sparse accelerator in accordance with aspects of the disclosed technology. For clarity, not all of the components, modules, or software blocks associated with FIG. 12 are labeled. Generally, as will become apparent from the following description, prefetching instructions, aggregating requests for instructions, and filtering and retaining instructions or references in memory locations closest to the requesting processing unit may provide instructions to processing units or processing cores more quickly, increasing system efficiency. Additional aspects of the components associated with FIG. 12 are further described below in connection with FIG. 13.

[0228] In a schematic diagram, Figure 12 illustrates aspects of the task instruction memory (Timem Bank or TimemBank), instruction buffer ("iBuf"), prefetch unit, and instruction router. Figure 12 shows a tile 1901 including a tile access core (TAC) 1910 and a tile execution core (TEC) 1920. The TAC 1910 may include a prefetch 1911 and an iBuf 1912. Similarly, the TEC may include a prefetch unit 1921 and an iBuf 1922. Figure 12 also shows a tile core 1902 including a TAC 1930 and a TEC 1940, which include a prefetch 1911 and an iBuf 1932, and a prefetch unit 1941 and an iBuf 1942, respectively.

[0229] 12 also shows Timem 1951 and Timem 1952 and an instruction router 1960, which may be logically or physically contained within floorplan block 1999. Timem 1951 and Timem 1952 may store instructions locally for faster access by each tile core compared to a tile core requesting instructions from a location further downstream from Timem. Also, floorplan block Also shown is an instruction broadcast bus that can broadcast instruction bundles downstream to the Time Bank 1999 and the Time Banks therein. An instruction request bus 1992 can aggregate requests for instructions from various components before requesting those instructions. Deserializers and serializers can deserialize or serialize instructions for transmission along the various buses, such as for receiving instructions from the instruction broadcast bus 1991 or for serializing instructions sent to the instruction request bus 1992.

[0230] A prefetch unit such as the prefetch unit 1911 or the prefetch unit 1912 corresponding to a core receives a miss program counter (PC) (and A read request to Timem can be made starting from the current time (and overlay / task ID) until the end of the prefetch window. The prefetch window is a software-selectable period with a register or other memory area. For example, the prefetch window may be defined by a prefetch depth variable. Prefetch read requests from other tiles may be forwarded by adjacent floorplan block 1999. These forwarded requests may be arbitrated with prefetch requests made by prefetch units in adjacent tile cores. For example, tile 1901 and tile 1902 may be adjacent to each other. In some examples, pairs of cores may be assigned to a single instruction request bus or a single instruction broadcast bus.

[0231] Several prefetch instruction request banks may exist within a tile. In some instances, there may be one bus per Timem bank, but these may be arbitrated independently of each other. Independent arbitration of the buses may allow for the avoidance of head-of-line blocking across independent banks.

[0232] Requests sent from the prefetch window can be received at the instruction router 1960. The instruction router 1960 can filter selected requests to remove duplicates before forwarding them to another instruction router or to the target Timem bank. Filtering can potentially increase instruction request bandwidth when the core is operating in SPMD mode.

[0233] Instructions read from the Timem bank can be broadcast to all tiles on the instruction broadcast bus. For example, there can be as many instruction broadcast buses as there are Timem banks. In some examples, instructions can be transmitted as instruction bundles. An instruction group consists of the instructions contained in a bundle. A bundle can be a sequence of instructions starting on an aligned "boundary." Instruction bundles can be serialized on the corresponding instruction broadcast bus for a certain number of cycles of the processor or core. In some examples, such as during "steady-state" operation of the system, the total bandwidth of the instruction broadcast bus can be two bundles per cycle. In such an embodiment, the instruction broadcast bus 1992 is never backpressured.

[0234] Instructions received on the broadcast bus can be deserialized by the instruction router, and one instruction is forwarded to each of the iBufs. In steady state, the system can be required to maintain up to two writes from the prefetch interface and one read from the instruction fetch interface. The prefetch processes the incoming instruction and determines whether it should be committed to an iBuf or dropped.

[0235] Figure 13 illustrates additional exemplary aspects of the instruction router 1960. Figure 13 shows a round robin (RR) arbiter 1910, a daisy-chain round robin arbiter Arbiter 1920, Round Robin Arbiter 1930, Filter 1940, Serializer The 1950 and 1951, demultiplexer (demux) 1960 , and deserializers 1971 and 1972. Other aspects and components are also shown in FIG. 13 that are not labeled for simplicity.

[0236] The instruction router 1960 may have an independent read request bus for each Timem bank in the system. The router 1960 may throttle instruction bundles at a rate that matches the bandwidth of the instruction broadcast bus before being forwarded to adjacent instruction routers. In the following description, it may be assumed that deserialization and serialization can be performed before a request is presented to the instruction router 1960.

[0237] The instruction router 1960 can arbitrate depending on the location of the core relative to the Timem bank. The instruction router 1960 can be parameterized to select the source and destination based on the instance for which the instruction router 1960 is arbitrating. The demultiplexer 1960 shown in Figure 13 can be designed according to the number of time banks or serializers it is communicating with.

[0238] The instruction router 1860 can arbitrate between the following exemplary sources: prefetch reads forwarded by instruction routers upstream or above the instruction router 1860, prefetch reads forwarded by instruction routers downstream of the instruction router 1860, and prefetch reads originating from cores connected to the instruction router 1860.

[0239] The demux (select is a design parameter) selects top_pre_req or bottom_pre_req to arbitrate among requests originating from cores connected to the instruction router. Arbitration uses a daisy-chain RR arbitration scheme. The daisy-chain round-robin arbiter 1920 can grant a request every "x" cycles to match the bandwidth of the instruction broadcast bus. If a PC matches a PC seen on the instruction broadcast bus, it can drop a request waiting to be arbitrated. This can be considered a first level of filtering.

[0240] The winner of the daisy-chain arbitration may be treated differently based on the position of the instruction router 1860 relative to the Timem bank. For example, if the Timem bank is below the instruction router, the winner of the daisy-chain arbitration may be forwarded to the "bottom" instruction router after passing through filter 1940. If the Timem bank is above the instruction router 1860, the winner of the daisy-chain arbitration is forwarded to the top instruction router 1860 after passing through filter 1940.

[0241] If the Timem bank is within the instruction router 1860, the winner of the daisy-chain arbitration will undergo one or more levels of arbitration with requests forwarded by the bottom instruction router. In this case, there may be two daisy-chain networks arbitrating to reach the Timem bank. Depending on the location of the instruction router 1860, the chains may be unbalanced. To ensure that fair access is provided to cores on both sides of the chain, a modified RR arbiter can be used. As with the first level of arbitration, any requests matching the PC on the broadcast bus will be dropped here. This can be considered a second level of filtering.

[0242] The overall winner of the above is passed to filter 440, which compares the incoming request with one of the other outstanding requests. If the request matches any of the outstanding requests, the request is dropped. This can be considered a third level of filtering.

[0243] Furthermore, the programmability of the system can be ensured in that filtering at each time point can be enabled / disabled with individual programmable or software controllable switches. The Timem access bus can be a bus that connects systems to all Timem banks and allows them to read and write instruction bundles to the Timem banks. The Timem access bus can have four buses: a read request bus, a read response bus, a write request bus, and a write response bus, as further described below.

[0244] The read request bus can be a daisy-chain bus that runs up to the Timem bank. Each Timem bank can forward a request to an adjacent Timem bank if the request does not address it. If the request addresses a Timem bank, the request is served by the Timem bank.

[0245] The read response bus may be a daisy-chain bus that can transmit instruction bundles read from the Timem bank. At each Timem bank, there may be round-robin arbitration between the incoming instructions from adjacent banks and the instruction bundle from the current bank. The instruction bundles are serialized over 'n' cycles, so bus grant is held for 'n' cycles.

[0246] The write request bus can be a daisy-chain bus that runs up to the Timem bank. Write requests can be serialized, for example, over two cycles. Each Timem bank forwards a flit to an adjacent bank if the request does not address it. If the request addresses the Timem bank, the request is deserialized by the bank before being written to the Timem bank.

[0247] The write response bus can be a daisy-chain bus that relays write responses from the Timem banks. At each Timem bank, there is arbitration between the incoming response and the response from the current bank. A simple round-robin arbitration can be used to allow one of the responses to be granted or served.

[0248] A maximum of two read and write requests can be made. ∧ It may have a "q" bit tag to encode q outstanding read and write requests that are returned in a response by the bank and may be used by the overall system or component to provide instructions to identify the request that corresponds to the response.

[0249] If an endpoint cannot accept a request or response, the bus can be "backpressured"; if the bus cannot transfer the commands or data it contains, a backlog builds up to be sent over the bus. In addition, the bus can be backpressured by arbitration losses. This may be tolerable system-wide because Timem access is generally a low-bandwidth access.

[0250] The tile instruction memory (Timem) can be shared by the tile cores described in FIG.

[0251] Aspects of the present disclosure may be implemented as one or more computer programs in a digital circuit, a computer-readable storage medium, or as a combination of one or more of the above. The computer-readable storage medium may be non-transitory, for example, as one or more instructions executable by a cloud computing platform and stored on a tangible storage device.

[0252] Calculation overview As used herein, the phrase "configured to" is used in various contexts relating to a computer system, hardware, or part of a computer program, engine, or module. When a system is described as being configured to perform one or more operations, this means that the system has appropriate software, firmware, and / or hardware installed thereon that, during operation, causes the system to perform one or more operations. When a piece of hardware is described as being configured to perform one or more operations, this means that the hardware includes one or more circuits that, during operation, receive inputs and generate outputs corresponding to the one or more operations according to the inputs. When a computer program, engine, or module is described as being configured to perform one or more operations, this means that the computer program includes one or more program instructions that, when executed by one or more computers, cause the one or more computers to perform one or more operations.

[0253] While the operations illustrated in the accompanying drawings and recited in the accompanying claims are shown in a particular order, it is understood that these operations may be performed in an order different from that illustrated, and that some operations may be omitted, performed multiple times, and / or performed in parallel with other operations. Furthermore, the separation of various system components configured to perform various operations should not be understood as requiring the components to be separated. The described components, modules, programs, and engines may be integrated together as a single system or may be part of multiple systems.

[0254] When substantially any plural and / or singular term (where the term "element" is a substitute for any system, component, data, etc.) is used herein, e.g., "one / the element," "one or more elements," "composite element," "plural elements," "at least one element," etc., one of ordinary skill in the art can convert from plural to singular and / or from singular to plural as appropriate for the described context and / or application. The various singular / plural permutations may be expressly set forth herein without limitation for the sake of clarity, unless expressly specified otherwise.

[0255] Aspects of the present disclosure include methods, systems, and apparatus that use an instruction prefetching pipeline architecture that provides good performance without the complexity of full cache coherent solutions deployed in conventional CPUs.

[0256] Aspects of the disclosed technology relate to components that can be used to build an instruction prefetch pipeline, including an instruction memory (TiMem), an instruction buffer (iBuf), a prefetch unit, and an instruction router.

[0257] Aspects of the present disclosure may relate to certain characteristics that may exist in conjunction with the expected or known behavior of tiles of an XPU, such as, for example, that tiles may be expected to operate in Single Program Multiple Data (SPMD) mode, that at any given time, statistically most tiles may be expected to be running the same program, that programs may consist of one or more compact loops, that program sizes may be small, e.g., hundreds of bundles, or that programs may have multiple branches that may branch off. The disclosed technology can exploit these or related characteristics and provide lower complexity than full-cache-based solutions.

[0258] Aspects of the disclosed technology include a hardware circuit. The hardware circuit may include a plurality of tiles. Each tile is configured to operate in parallel with other tiles in the plurality of tiles, and each tile of the plurality of tiles includes a processing core, a prefetch unit, and an instruction buffer. The hardware circuit further includes a plurality of data processing lanes configured to stream respective data from an upstream input to a downstream destination, and a plurality of task instruction memories, each of the plurality of task instruction memories arranged in sequence and coupled to one or more tiles of the plurality of tiles via an instruction router. The task instruction memories may be arranged in downstream sequence. Each tile may include a tile access core, and the prefetch unit included in each tile may be included in the tile access core. Each tile may include a tile execution core, and the prefetch unit included in each tile may be included in the tile execution core. The hardware circuit may include an instruction broadcast bus and an instruction request bus. The instruction broadcast bus may include independent data lanes. The number of independent data lanes may correspond to the number of task instruction memories.

[0259] The instruction request bus may include independent data lanes. The number of independent data lanes corresponds to the number of task instruction memories. Instructions received by the task instruction memories may be broadcast to all linked tiles over an instruction broadcast bus. The prefetch may be configured to make requests to at least one task instruction memory during a prefetch window. The prefetch window may be software selectable or adjustable. The hardware circuit may further include an instruction router. The instruction router may include a round-robin arbiter configured to arbitrate requests, including prefetch read requests. The instruction buffer may store instructions for the tile access core or the tile execution core. The hardware circuit may be configured as a single instruction multiple data processor. The hardware circuit may be configured as a multiple instruction multiple data processor. The hardware circuit may include a task instruction memory access bus. The task instruction memory access bus may include a read request bus, a read response bus, a write request bus, and a write response bus.

[0260] Aspects of the disclosed technology include a TPU. The TPU may include a hardware circuit and an instruction broadcast bus coupled to the hardware circuit. The instruction broadcast bus is configured to push instructions to the hardware circuit. The hardware circuit may include a plurality of tiles, each configured to operate in parallel with other tiles in the plurality of tiles, and each tile of the plurality of tiles may include a processing core, a prefetch unit, and an instruction buffer. The hardware circuit may further include a plurality of data processing lanes configured to stream respective data from an upstream input to a downstream destination, and a plurality of task instruction memories. Each task instruction memory of the plurality of task instruction memories is arranged in sequence and coupled to one or more tiles of the plurality of tiles via an instruction router. The TPU may further include an instruction request bus coupled to the hardware circuit, the instruction request bus being configured to receive requests for instructions.

[0261] Aspects of the disclosed technology include a method for prefetching or providing instructions by a single instruction multiple data (SIMD) processing unit. The method may include receiving requests for instructions from multiple tiles of the SIMD processing unit, filtering the requests for instructions to eliminate duplicate requests for the same instruction to generate a first set of requests, generating a set of instructions in response to the first set of requests, providing the set of instructions from a compute unit to a task instruction memory of the SIMD processing unit, storing the set of instructions in the task instruction memory, and accessing instructions from the set of instructions by a prefetch unit via an instruction router. The SIMD processing unit may include multiple tiles, each tile configured to operate in parallel with other tiles in the multiple tiles, and each tile of the multiple tiles: The processor includes a processing core, a prefetch unit, and an instruction buffer, wherein the receiving step may occur in a first processing clock cycle and the providing step may occur in a second processing clock cycle, and the first processing clock cycle may occur before the second processing clock cycle.

[0262] Generally, disclosed herein is a hardware / software interface and stream ordering model for asynchronous data movement between off-core memory and core-local memory, referred to as "stream transfer." Stream transfer allows software to more efficiently express common data movement patterns, particularly those found in sparse workloads. Stream instructions belonging to a stream are processed in order. For indirect stream instructions, offset elements in the offset list are processed in order. A synchronization flag is updated to indicate monotonic incremental progress for the stream.

[0263] One aspect of the present disclosure provides a method including steps of identifying, by one or more processors, the progress of data being transferred between off-core memory and core local memory; and identifying, by the one or more processors, reads from the core local memory when the core local memory is a source of the data, the reads being issued to the source in order and serviced out of order by the source; the method further includes steps of identifying, by the one or more processors, writes to the core local memory when the core local memory is a destination for the data, the writes being issued to the destination in order and committed out of order by the destination; and the method further includes steps of accessing, by the one or more processors, the off-core memory based on indirect scatter / gather memory accesses for reads from the off-core memory when the off-core memory is a source of the data and for writes to the off-core memory when the off-core memory is a destination for the data.

[0264] In one example, identifying progress of data being transferred further includes using a core-local synchronization flag. In another example, the method further includes selecting, at one or more processors, a memory access to a barrier based on a scalar fence instruction. In yet another example, accessing off-core memory based on an indirect scatter-gather memory access further includes deriving a source of the indirect address from a register file or core-local memory. In yet another example, the method further includes circular buffering, at one or more processors, in core-local memory.

[0265] In yet another example, the method further includes updating a core-local synchronization flag at one or more processors to indicate monotonic incremental progress for the data transfer. In yet another example, the method further includes terminating the data transfer at one or more processors when all reads from core-local memory have been issued. In yet another example, the method further includes terminating the data transfer at one or more processors when all writes to core-local memory have been committed.

[0266] Another aspect of the present disclosure provides a system including one or more processors and one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for transferring data between an off-core memory and a core local memory, the operations including: identifying progress of data being transferred between the off-core memory and the core local memory; and identifying a read from the core local memory when the core local memory is a source of data, the reads being issued to the source in order and serviced out of order by the source; and further, identifying when the core local memory is The operations include operations for identifying a write to core local memory when it is a destination for the data, the write being issued to the destination in order and committed out of order by the destination, and further including operations for accessing the off-core memory based on indirect scatter / gather memory access to read from the off-core memory when it is a source of the data and to write to the off-core memory when it is a destination for the data.

[0267] In one example, the act of identifying progress of data being transferred further includes an act of using a core-local synchronization flag. In another example, the act further includes an act of selecting a memory access to a barrier based on a scalar fence instruction. In yet another example, the act of accessing off-core memory based on an indirect scatter-gather memory access further includes an act of deriving the source of the indirect address from a register file or core-local memory. In yet another example, the act further includes an act of circular buffering in core-local memory.

[0268] In yet another example, the operations further include updating a core-local synchronization flag to indicate monotonic incremental progress for the data transfer. In yet another example, the operations further include terminating the data transfer when all reads from the core-local memory have been issued. In yet another example, the operations further include terminating the data transfer when all writes to the core-local memory have been committed.

[0269] Yet another aspect of the present disclosure provides a non-transitory computer-readable storage medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for transferring data between an off-core memory and a core local memory, the operations including: identifying progress of data being transferred between the off-core memory and the core local memory; and identifying a read from the core local memory when the core local memory is a source of the data, the read being issued in-order to the source and serviced out-of-order by the source; the operations further including identifying a write to the core local memory when the core local memory is a destination for the data, the write being issued in-order to the destination and committed out-of-order by the destination; and the operations further including accessing the off-core memory based on indirect scatter-gather memory accesses for the read from the off-core memory when the off-core memory is a source of the data and for the write to the off-core memory when the off-core memory is a destination for the data.

[0270] In one example, the act of accessing off-core memory based on the indirect scatter-gather memory access further includes an act of deriving the source of the indirect address from a register file or core local memory, in another example, the act of circular buffering in the core local memory, and in yet another example, the act of updating a synchronization flag to indicate monotonic incremental progress of the data transfer.

[0271] Aspects of the present disclosure are directed to a cross-lane processing unit (XPU) for performing single instruction, multiple data (SIMD) data-dependent operations across multiple data processing lanes of a processor. Rather than physically fabricating operation-specific circuitry for each data-dependent operation, the XPU may be configured to perform various operations in response to input signals, comprising processing cells for performing individual operations and a crossbar arranged as a stacked network in the XPU. Each processing cell is capable of receiving and processing data across multiple data processing lanes. Aspects of the present disclosure include configuring the XPU to perform a vector sort while calculating a duplicate count of duplicate elements in an input vector received for sorting, and configuring the XPU to perform the sort and duplicate count separately for the sort and the crossbar. This eliminates the need to construct U. The XPU can be implemented as a piece of hardware circuitry that complements computations on dense data structures, such as dense matrices, with accelerated processing of sparse data structures, such as sparse vectors or sparse matrices.

[0272] An aspect of the present disclosure includes a hardware circuit including a plurality of stages, each stage including a crossbar and two or more cells, the hardware circuit further including a plurality of data processing lanes for streaming respective data from an upstream input to a downstream destination via the plurality of cells of the plurality of stages and the plurality of crossbars, the hardware circuit configured to receive input data from the upstream input along the plurality of data processing lanes and first instructions for performing a first operation, and configured, in response to receiving the first instructions, for each stage to transmit respective second instructions to respective processing cells of the stage, each cell configured to perform a respective second operation in response to receiving the input from the respective data processing lane, the hardware circuit further configured to transmit respective third instructions to respective crossbars for the stage, the crossbars configured to substitute output from each cell of a stage to a cell of a next stage along the plurality of data processing lanes, and configured to perform the first operation by processing the received input data along the plurality of data processing lanes and the plurality of cells configured to perform the respective second operations.

[0273] An aspect of the present disclosure includes a system including a hardware circuit, the hardware circuit including a plurality of stages, each including a crossbar and two or more cells, and a plurality of data processing lanes that stream respective data from an upstream input to a downstream destination via the plurality of cells of the plurality of stages and the plurality of crossbars, the hardware circuit being configured to receive input data from the upstream input along the plurality of data processing lanes and a first instruction to perform a first operation, and being configured, in response to receiving the first instruction, to send, for each stage, a respective second instruction to a respective processing cell of the stage, each cell being configured to perform a respective second operation in response to receiving the input from the respective data processing lane, the hardware circuit being further configured to send a respective third instruction to a respective crossbar for the stage, the crossbar being configured to substitute output from each cell of the stage to a cell of a next stage along the plurality of data processing lanes, and being configured to perform the first operation by processing the received input data along the plurality of data processing lanes and the plurality of cells configured to perform the respective second operations.

[0274] Aspects of the present disclosure include a computer-implemented method including: receiving, via a hardware circuit including a plurality of stages each including a crossbar and two or more cells and a plurality of data processing lanes that stream respective data from an upstream input to a downstream destination, input data from an upstream input along the plurality of data processing lanes and a first instruction for performing a first operation, via the plurality of cells of the plurality of stages and the plurality of crossbars; and, in response to receiving the first instruction, transmitting, for each stage, by the hardware circuitry, respective second instructions to respective processing cells of the stage, each cell configured to perform a respective second operation in response to receiving the input from the respective data processing lane; the computer-implemented method further includes transmitting, by the hardware circuitry, respective third instructions to respective crossbars for the stage, the crossbar being configured to substitute output from each cell of the stage to a cell of a next stage along the plurality of data processing lanes; and performing the first operation by processing, by the hardware circuitry, the received input data along the plurality of data processing lanes and the plurality of cells configured to perform the respective second operations.

[0275] Aspects of the disclosure may include one or more of the following features: In some examples, aspects of the disclosure include all of the following features in combination.

[0276] Each cell is configured to receive a respective first input operand from a respective data processing lane passing through the cell, and a respective second input operand from a respective crossbar of a stage upstream from the cell.

[0277] The downstream destination of the data from the multiple data processing lanes is a vector processing unit configured to perform single instruction multiple data vector operations on the output data of the hardware circuit.

[0278] Each of the cells is configured to perform one or more of a plurality of predetermined primitive operations in response to the one or more received instructions. The hardware circuit further includes a plurality of control cells, and upon sending a respective second instruction to a respective processing cell, the hardware circuit is configured to generate and send, by each control cell, a respective control signal to the respective processing cell based on the first operation specified by the first instruction.

[0279] In generating and transmitting the respective control signals by each control cell, the hardware circuitry is configured to generate the respective control signals to cause each processing cell to perform one of a respective arithmetic operation, a comparison, and a bypass operation based on at least one of the stage in which the processing cell is located and the data processing lane passing through the processing cell.

[0280] The plurality of cells and the plurality of crossbars form a processing network of connected cells across the plurality of stages and the plurality of data processing lanes, the processing network of connected cells being configured to receive input data and to generate respective output data according to performing a first operation on the input data.

[0281] The processing network of the connected cells is configured to perform a combined vector sort and duplicate count operation, the combined operation including receiving, by the processing network, an input vector of elements, and generating, by the processing network, as output, a sorted output vector and data specifying a count of duplicate elements in the input vector. The input data includes sparse vector data, and the hardware circuitry is configured to perform one of a vector scan, a vector addition, a vector sort, or a vector duplicate count after transmitting the respective second and third instructions.

[0282] Unless otherwise specified, the above-described alternatives are not mutually exclusive and may be implemented in various combinations to achieve particular advantages. Because these and other variations and combinations of the above-described features can be utilized without departing from the subject matter defined by the appended claims, the above description of examples should be construed as illustrative, rather than limiting, of the subject matter defined by the appended claims. Additionally, the presentation of examples described herein, as well as terms such as "such as," "including," and the like, should not be construed as limiting the claimed subject matter to any particular example; rather, the above example is intended to illustrate only one of many possible implementations. Furthermore, the same reference numbers in various drawings may identify the same or similar elements.

Claims

1. 1. A processor, comprising: a plurality of tiles, each of the plurality of tiles comprising: A vector core, a slice of shared software-controlled scratchpad memory; The processor further comprises: a scalar core configured to dispatch tasks to the plurality of tiles; a memory coupled to the plurality of tiles and the scalar core.

2. The processor of claim 1 , wherein each tile is configured to perform an independent calculation.

3. The vector core in each of the plurality of tiles includes a plurality of single instruction, multiple data (SIMD) processing lanes. The processor of claim 1 , comprising:

4. The processor of claim 1 , wherein multiple tiles of the plurality of tiles issue memory requests to a main memory in parallel.

5. The processor of claim 1 , wherein the vector core in each of the plurality of tiles is configured to generate data-dependent address streams to any level of a memory hierarchy.

6. 6. The processor of claim 5, wherein each data-dependent address stream corresponds to a sequence of addresses, the length and particular values ​​of the addresses in the sequence being data-dependent and known only at run-time.

7. 6. The processor of claim 5, wherein the vector core in each of the plurality of tiles is configured to represent a data-dependent address stream while decoupling high performance servicing of the data-dependent address stream to a microarchitecture.

8. 8. The processor of claim 7, wherein the microarchitecture includes a scatter-gather engine for the high performance servicing of the data-dependent address streams.

9. The processor of claim 7 , wherein the data-dependent address stream includes multiple addressing modes, runtime configurable transfer sizes, and indirect memory accesses with atomic arithmetic updates.

10. 2. The processor of claim 1, wherein the vector core in each of the plurality of tiles includes a circular buffer instruction that enables transfer and access of dynamically sized data streams over a statically sized region of memory.

11. The processor of claim 10 , further comprising a microarchitecture configured to track runtime buffer sizes for the dynamically sized data stream.

12. 11. The processor of claim 10, wherein the vector core in each of the plurality of tiles is configured to provide runtime configuration and access areas of the tile-local scratch pad memory as in-order circular first-in-first-out (FIFO) access without precluding out-of-order access to the same area of ​​tile-local scratch pad memory.

13. 13. The processor of claim 12, wherein the in-order circular FIFO access, in conjunction with the microarchitecture, enables the dynamically sized data stream on a statically sized region of the tile-local scratch pad memory.

14. The processor of claim 1 , wherein each tile includes a scatter-gather engine configured to manage issuing, fetching, tracking, and ordering of data streams.

15. 15. The processor of claim 14, wherein each scatter-gather engine is further configured to maintain at least 256 outstanding read requests in-board per tile.

16. 15. The processor of claim 14, wherein each scatter-gather engine is further configured to track and update buffer occupancy to manage flow control.

17. The processor of claim 1 , wherein a subset of the plurality of tiles each further comprises a prefetch unit configured to cooperatively prefetch data stream instructions.

18. The processor of claim 1 , further comprising a cross-lane processing unit configured to accelerate at least one of irregular control flow sequences or intra-vector dependent operations.

19. 2. The processor of claim 1, wherein each tile is configured to support scatter from an off-chip memory to its scratchpad memory and gather from its scratchpad memory to the off-chip memory.

20. The processor of claim 1 , wherein the subset of tiles is grouped based on a logically configurable vector width.

21. 21. The processor of claim 20, wherein the logically configurable vector width comprises a logical SIMD width.

22. The processor of claim 1 , wherein the processor is part of a machine learning accelerator configured to execute a neural network layer that exhibits semantic sparsity.

23. 23. The processor of claim 22, wherein the neural network layer comprises an embedded neural network or a graph neural network.

24. 23. The processor of claim 22, wherein the processor is connected to several other processors via a network configured to perform distributed scatter-gather and computations required by neural network layer computations that are dynamic, irregular, and memory-bound.

Images