A programmable accelerator for data-dependent irregular calculations.

Abstract

To provide an accelerator which can accelerate data-dependent calculation, irregular calculation, and / or memory-bound calculation.SOLUTION: A hardware circuit 101 includes: a sparse accelerator 103; a coprocessor 104 for accelerating predictable calculation for calculation load and behavior on the coprocessor during design and manufacturing; a high-band width memory 107; and an on-chip interconnection 108. The sparse accelerator 103 includes one or more tiles 102A to 102F, and each of the tiles is equipped with its vector processing unit (VPU) and has cross lane processing units (XPU) 101A to 101F therefor. The sparse accelerator has a tile sequencer 106 for coordinating input data and output data for the tiles 102A to 120F.SELECTED DRAWING: Figure 1A

Description

TECHNICAL FIELD

[0001] Cross-Reference to Related Applications This application is a continuation of U.S. Patent Application No. 17 / 981,617, filed on November 7, 2022, which claims the benefit of the filing dates of U.S. Provisional Patent Application No. 63 / 357,281 filed on June 30, 2022, No. 63 / 322,285 filed on March 22, 2022, No. 63 / 281,960 filed on November 22, 2021, and No. 63 / 279,262 filed on November 15, 2021, the disclosures of which are incorporated herein by reference. This application also relates to U.S. Patent Application No. 17 / 972,681 filed on October 25, 2022, No. 17 / 972,663 filed on October 25, 2022, and No. 17 / 722,782 filed on April 18, 2022, the disclosures of which are incorporated herein by reference.

BACKGROUND ART

[0002] Background Hardware acceleration uses computer hardware to perform certain types of operations more efficiently. Exemplary types of operations that may be accelerated include linear algebra operations, such as matrix-to-matrix multiplication or matrix-to-vector multiplication. A device or processor constructed to perform operations accelerated by hardware is sometimes referred to as an accelerator.

[0003] Accelerators are designed and fabricated to accelerate only a small portion of desired operations. During the accelerator design and fabrication process, assumptions are made regarding the nature of the operations that are desired to be accelerated, such as the size and type of inputs received by the accelerator, the regularity with which the accelerator receives inputs, or the computational requirements for performing the operations. As a result, accelerators may often become highly specialized and can only accelerate a small class of predetermined operations, and may be unable to efficiently perform other operations, if any.

[0004] Operations outside this class include data-dependent operations where the computational load on the accelerator cannot be determined before the operation is performed. Multiple instances of this type of accelerated operation can vary depending on various factors, and a given accelerator design may be inefficient in accelerating at least some of these instances. Other types of operations that are difficult to accelerate include memory-rebound operations with low computational intensity and limited data reuse. Yet another type of operation that is difficult to accelerate includes irregular operations that can be characterized by random memory access, complex code patterns, and diverse use of parallel execution of multiple sub-operations performed simultaneously.

[0005] In reality, processing pipelines, such as those used to train or deploy machine learning models, require the execution of a wide variety of computations. Integrating accelerators into the pipeline to speed up only a few types of computations, while relying on devices to perform other types of computations without hardware acceleration, imposes unacceptable latency and memory bandwidth stress on the links and interconnects between accelerators and non-accelerators, limiting overall performance. Designing and fabricating accelerators to cover all types of computations is almost always impossible or impractical. Data-dependent computations do not contribute to acceleration, and the logistical effort required to accelerate other types of computations may not justify the investment in designing, fabricating, and deploying the corresponding accelerators. [Overview of the project]

[0006] overview Aspects of this disclosure provide accelerators capable of accelerating data-dependent, irregular, and / or memory-rebound operations. The accelerators described herein include a programmable engine for efficiently performing on-chip computations that are dynamic, irregular, and / or memory-rebound, along with a coprocessor configured to accelerate operations that are predictable in terms of computational load and behavior on the coprocessor during design and fabrication.

[0007] Dynamic operations are those in which the calculation performed depends on data or input, meaning the input is unknown before the operation is performed. The irregularity of the operation can stem from random memory access, complex code patterns, and varying amounts of computational resources and parallelism required to perform different instances of the operation on various input data. Memory-rebound operations are often low-intensity operations, for example, where a small number of operations are performed per unit of data transferred during acceleration of the operation, and data reuse is limited.

[0008] The accelerators described herein can coordinate and distribute cross-chip data scatter and gather operations of varying sizes to scale acceleration on host devices or data centers implementing multiple accelerators and other processors. As described herein, the accelerators leverage configurable architectural elements to adapt to accelerating various types of data-dependent operations, random operations, and / or memory rebound operations without requiring physical redesign or modification of the hardware circuitry implementing the accelerator itself.

[0009] The aspect of this disclosure provides an accelerator that can accelerate computations in sparsity neural network layers, for example, in an embedded format. Sparse computation refers to computations in which the fractional part of the computed data values ​​(e.g., input values, output values, or median values) is zero. The fractional part can vary, for example, between 0.1% and 50%. The aspect of this disclosure provides embedded training and processing acceleration as part of a machine learning processing pipeline.

[0010] The aspect of this disclosure provides a processor, which comprises a plurality of tiles, each of which comprises a vector core and a slice of shared software-controlled scratchpad memory. The processor further comprises a scalar core configured to dispatch tasks to the plurality of tiles. The processor also comprises memory coupled to the plurality of tiles and the scalar core.

[0011] In one example, each tile is configured to perform independent calculations. In another example, the vector core in each of those tiles includes multiple single-instruction, multiple data (SIMD) processing lanes. In yet another example, the multiple tiles in question issue parallel memory requests to main memory.

[0012] In yet another example, the vector core in each of the multiple tiles is configured to generate a data-dependent address stream to any level of the memory hierarchy. In yet another example, each data-dependent address stream corresponds to a sequence of addresses, the length and specific values ​​of the addresses in the sequence are data-dependent and known only at runtime. In yet another example, the vector core in each of the multiple tiles is configured to represent the data-dependent address stream while keeping the high-performance service of the data-dependent address stream isolated to the microarchitecture. In one example, the microarchitecture includes a scatter-gather engine for the high-performance service of the data-dependent address stream. In yet another example, the data-dependent address stream includes multiple addressing modes, runtime-configurable transfer sizes, and indirect memory access with atomic arithmetic updates.

[0013] In yet another example, the vector core in each of the multiple tiles includes circular buffer instructions that enable the transfer and access of dynamically sized data streams on a statically sized region of memory. In yet another example, the processor further includes a microarchitecture configured to track the runtime buffer size of the dynamically sized data stream. In yet another example, the vector core in each of the multiple tiles is configured to provide the runtime configuration and access region of the tile-local scratchpad memory as sequential circular first-in-first-out (FIFO) access without excluding unsequential access to the same region of the tile-local scratchpad memory. In yet another example, the sequential circular FIFO access, in relation to the microarchitecture, enables dynamically sized data-dependent address streams on the statically sized region of the tile-local scratchpad memory.

[0014] In yet another example, each tile includes a scatter-gather engine configured to manage the issuance, fetching, tracking, and ordering of data streams. In yet another example, each scatter-gather engine is further configured to maintain at least 256 pending read requests within the machine per tile. In yet another example, each scatter-gather engine is further configured to track and update buffer occupancy to manage flow control.

[0015] In yet another example, each subset of the tiles further includes a prefetch unit configured to cooperatively prefetch data stream instructions. In yet another example, the processor further includes a cross-lane processing unit configured to accelerate at least one of irregular control flow sequences or vector-dependent operations. In yet another example, each tile is configured to support scatter from off-chip memory to its scratchpad memory and gather from its scratchpad memory to off-chip memory.

[0016] In yet another example, a subset of these tiles is grouped based on a logically configurable vector width. In yet another example, this logically configurable vector width includes a logical SIMD width.

[0017] In yet another example, the processor is part of a machine learning accelerator configured to run a neural network layer exhibiting semantic sparsity. In yet another example, the neural network layer includes an embedded neural network or a graph neural network. In yet another example, the processor is connected to several other processors via a network configured to perform distributed scatter-gather and computations required by the neural network layer computation, which is dynamic, random, and memory-rebounding. [Brief explanation of the drawing]

[0018] [Figure 1A] This is a block diagram showing a hardware circuit for accelerating data-dependent computations in accordance with the context of this disclosure. [Figure 1B] A block diagram showing an exemplary data path implemented as part of a hardware circuit, in accordance with the aspects of this disclosure. [Figure 2] This is a block diagram illustrating an exemplary environment for implementing hardware circuits in accordance with the aspects of this disclosure. [Figure 3A]FIG. 2 is a block diagram illustrating an exemplary tile in accordance with aspects of the present disclosure. [Figure 3B] FIG. 3 is a block diagram illustrating another exemplary tile implementing an XPU for stream transfer in accordance with aspects of the present disclosure. [Figure 4] FIG. 4 is a block diagram illustrating a tile sequencer in accordance with aspects of the present disclosure. [Figure 5] FIG. 5 is a block diagram illustrating an exemplary scratchpad memory together with memory across multiple tiles of a sparse accelerator in accordance with aspects of the present disclosure. [Figure 6] FIG. 6 is an exemplary block diagram illustrating a scalar core complex of a tile sequencer in accordance with aspects of the present disclosure. [Figure 7] FIG. 7 is a block diagram illustrating an exemplary XPU in accordance with aspects of the present disclosure. [Figure 8] FIG. 8 is an exemplary functional diagram illustrating a scatter-gather engine in accordance with aspects of the present disclosure. [Figure 9] FIG. 9 is a flow diagram illustrating an exemplary process for expanding a stream descriptor into a configured off-tile stream request or a tile-local stream request in accordance with aspects of the present disclosure. [Figure 10] FIG. 10 is a flow diagram illustrating an exemplary process for ordering stream transfers in accordance with aspects of the present disclosure. [Figure 11] FIG. 11 is an exemplary diagram illustrating stream ordering in accordance with aspects of the present disclosure. [Figure 12] FIG. 12 is a logic diagram illustrating connectivity between tiles and an exemplary sparse accelerator in accordance with aspects of the present disclosure. [Figure 13] FIG. 13 is a diagram illustrating additional exemplary aspects of an instruction router in accordance with aspects of the present disclosure. DETAILED DESCRIPTION OF EMBODIMENTS

[0019] Detailed Description Overview This disclosure provides accelerators capable of accelerating data-dependent operations, random operations, and / or memory-rebound operations. An accelerator configured to accelerate data-dependent operations on sparse inputs may be referred to herein as a sparse accelerator. A sparse accelerator can be configured to efficiently and effectively execute neural network layers exhibiting semantic sparsity, such as embedded neural networks or graph neural networks (GNNs). A graph neural network may correspond to a neural network for processing data that can be represented as a graph.

[0020] A sparse accelerator can be organized as a tile-type processor. A sparse accelerator may include a tile sequencer used to dispatch tasks to tiles. Each tile of the sparse accelerator has a vector core extended by a cross-lane processing unit (XPU) and a slice of shared software-controlled scratchpad memory. Both the tiles and the sequencer can be connected to high-bandwidth main memory.

[0021] The organization and structure of sparse accelerators improve data-dependent, random, and / or memory-rebound operations through various combinations of the distinct features described herein. Accelerator tiles may include one or more of the following features to more efficiently utilize the computations that are active while data-dependent, random, and / or memory-rebound operations are being performed: Cross-lane processing units (XPUs) within each tile accelerate common random control flow sequences and / or intravector-dependent operations. Each XPU enables the provision of a custom high-speed data path for common intravector-dependent operations.

[0022] Other features that can be implemented on the accelerators described herein may include cooperative prefetching and stream instructions and ordering. Cooperative prefetching enables improvements in performance and energy efficiency by reducing instruction fetch bandwidth requirements between tiles. Stream instructions described herein enable high-bandwidth data-dependent scatter-gathering. Each vector processing unit of the accelerator may generate high-bandwidth data-dependent addresses to any level of the memory hierarchy.

[0023] Tile-local scatter and gather operations can enable uninterrupted computation in the presence of data irregularities. Each accelerator tile can inherently support high-performance scatter and gather operations to its local memory, exposed by the instruction set architecture (ISA), along with available instructions for indirect vector load, store, and store append operations.

[0024] The implemented XPU's software-controlled tile grouping can provide flexible amortization of control overhead. The software can flexibly group tiles to exhibit a logically configurable vector width, such as a logical SIMD width. This can help amortize control overhead, such as eliminating redundant instruction fetches between tiles within the same group or enabling combined memory access between tiles. This can, as a result, lead to better bandwidth efficiency; for example, high-bandwidth memory (HBM) has a finer access granularity. Different cases can result in different bandwidth efficiencies. Data-dependent operations (also called "input-dependent operations") are operations where the amount of computational work required to perform the operation is not known in advance and depends on the nature of the data. A data-dependent operation can be represented as a data-dependent address stream, which corresponds to a sequence of addresses where the length and specific values ​​of the addresses in the sequence are known at runtime. The computational work can be measured, for example, by the number of operations or processing cycles required to perform the data-dependent operation. Exemplary data-dependent operations include operations for vector sorting, operations for counting duplicate values ​​in a vector, and operations for handling the shapes or sizes of vectors of varying lengths. Data-dependent operations are irregular, at least because there are differences in the random memory access patterns for performing the same type of operation on different inputs. As a result, data-dependent operations are difficult to optimize for performance, in contrast to other types of operations where the computational work does not change based on the nature of the input data, such as its shape, degree, or sparsity.

[0025] Data-dependent operations include operations performed on sparse data. The sparsity of a data structure is a measure of the ratio of non-empty elements to empty elements. Depending on the data structure, empty elements may be zero, reserved words indicating the absence of a value for an element, or have values ​​so small that they contribute only negligibly to operations performed on the data structure as input. A data structure is sparse if there are more empty elements than non-empty elements. Some data structures may be more or less sparse than others.

[0026] An exemplary data-dependent operation involves generating an embedding for input training examples. The embedding may be a vector or some other data structure mapped from an input having a higher dimension than the embedding itself. Embedding generation can be performed as part of a workload processed according to a pipeline.

[0027] Each tile of the sparse accelerator's XPU may also perform other data-dependent operations such as vector scatter or gather operations, segment sums, and / or Sparse data structures such as nsols may be partitioned. The XPUs described herein may be complementary processing units to other components or connected components of the processor, such as vector processing units built according to the SIMD parallel processing paradigm. One or more XPUs may be connected in each processor core of a larger processor, which itself may contain other components to accelerate the performance of specific workloads, such as training neural networks.

[0028] Furthermore, XPUs are not limited to performing specific types of data-dependent operations; therefore, a processor may be designed to include XPUs to complement other types of processing units for multiple different pipelines. Because XPUs can be configured per workload, the physical footprint of the XPU is reduced compared to other approaches where specialized circuitry is physically fabricated on the processor as complementary units for sparse data computation. The functionality of the XPU can also be extended by using or extending the existing instruction set of the host processor, further improving its adaptability to various data-dependent operations in response to changes in pipeline data reception. Instructions can be provided as signals to the components of the XPU, which play a role in translating the instructions to configure the individual processing cells and crossbars of the XPU. The XPU can be configured using a program compiled by a corresponding compiler for the hardware circuitry implementing the XPU.

[0029] The aspects of this disclosure offer at least the following technical advantages: The accelerators described herein provide scalable distributed training of large-scale machine learning models that require performing data-dependent operations where operands are generally unknown until runtime.

[0030] The accelerator, alone and in combination with other processors, enables flexible computational configurations of tiles implementing the XPU within the accelerator to achieve various forms of parallelism, such as task, data, pipeline, and model parallelism. In task-level parallelism, each tile may execute independent computations (tasks) in parallel. For memory-level parallelism, tiles may issue memory requests in parallel to high bandwidth to absorb available memory bandwidth. This allows sparse accelerators to handle varying amounts of work in separate tasks.

[0031] The accelerators described herein may be dedicated coprocessors running alongside other accelerators, or they may be general-purpose processors not configured to accelerate data-dependent computations. The accelerators described herein may accelerate sparse computations or other data-dependent computations to achieve performance gains over approaches that rely on host memory to retrieve data for performing data-dependent computations, for example, by increasing processing speed.

[0032] The accelerators described herein can function as an architectural counterpart to a coprocessor configured to perform high-density, regular computations. By implementing the programmable sparse accelerators described herein, which are communicatively coupled to separate processors, a flexible means of accelerating computations at unpredictable complexity at compile time can be provided. The accelerators described herein can target memo-rebound operations involving complex data movement, reorganization, and summarization. This type of operation may include scatter-gather operations, filtering, sorting, uniqueness, etc. The accelerators are targetable according to the compiler stack, for example, according to a compiler configured to translate source code into instructions executable by the accelerator and its coprocessor.

[0033] The accelerators described herein can accelerate embedded generation. Embedding can generally be characterized as involving irregular memory access with low computational intensity. This is due, at least, to the need to perform on-chip table lookups of tables representing embedding maps and variable-width vector calculations, assuming the sparse and irregular nature of the input vectors for embedding generation. Embedding is a mapping from discrete objects, for example, from vectors or other data structures of values, to vectors of numbers, such as real numbers. Embeddings generally have lower dimensionality and complexity than their corresponding pre-embedded objects. For example, embeddings of one or more English words may be vectors of real numbers. Embeddings can be used to represent and identify potentially prominent features of pre-embedded inputs; for example, the similarity between two pre-embedded inputs can be quantified by comparing these embeddings by measuring the distance between them.

[0034] The cooperative prefetch described herein enables performance and energy efficiency improvements by reducing instruction fetch bandwidth requirements between tiles. The accelerator's tiled architecture exposes a multiple-program multiple-data programming model, while additionally providing a configurable subset of tiles that operate with a single-program multiple-data model.

[0035] The stream instructions described herein enable high-bandwidth, data-dependent scatter-gather operations. Each vector processing unit of the accelerator can generate data-dependent addresses to any level of the memory hierarchy with high bandwidth. The stream instructions described herein provide structurally visible constructs, for example, by ISA, that enable data-dependent access pattern representations in software. These access patterns include indirect memory access, including multiple addressing modes, configurable transfer sizes, and atomic operations. The stream instructions enable software to specify a “start” address, “size,” and “sequence pattern” for memory access. All of this is done while, according to the aspects of this disclosure, each tile of the accelerator implements a separate core for accessing data and responding to requests using a scatter-gather engine.

[0036] Data can flow through various components of the processor and other off-chip components, and through data streams, which are data values ​​corresponding to addresses in an address stream. A data stream may have a stream descriptor that provides metadata characterizing the stream. As part of the metadata, a data stream may have a stream identifier, which can be used to identify which execution thread of the processor is currently processing the stream. Multiple streams, for example, 8, 16, or 32 streams, may be active and flowing through the processor simultaneously.

[0037] The circular buffer instructions described herein can enable variable-size dynamic data streams. A circular buffer instruction is a structurally (e.g., by ISA) visible construct that allows software to statically fetch data streams of unknown size and perform operations on them without explicitly allocating buffers at compile time. Thus, circular buffer instructions enable the transfer and access of dynamically sized data streams over statically sized regions of memory. The scatter-gather engine tracks runtime buffer sizes for various data streams and manages flow control. Since the underlying memory is also accessible via standard load and store operations, circular buffer instructions provide a structural first-in, first-out (FIFO) abstraction for fast common-use cases without excluding software from non-FIFO access patterns. In another example, a vector core in each of multiple tiles accesses the same region of tile-local scratchpad memory in an unordered manner. It is configured to provide runtime configuration and access areas for tile local scratchpad memory as sequential, cyclic, first-in, first-out (FIFO) access without excluding such access.

[0038] Stream and circular buffer instructions enable a FIFO abstraction in the general case without preventing software from accessing memory "randomly" as needed. Software can FIFO fetch buffers, but can access data multiple times by reusing the fetched buffer / window as needed. This is in contrast to the software approach of requiring multiple pops to reuse data and queuing those multiple pops. Such control balances the scatter-gather engine's granular flow control when software issues prefetches. Software gains higher-order bit rights regarding roughly when to issue streams, and the scatter-gather engine, implementing the stream ordering and instructions described herein, responds to finer-grained variations such as latency and buffer occupancy.

[0039] The microarchitecture, for example, the scatter-gather engine described herein, enables high-performance, irregular memory access. The scatter-gather engine is implemented per tile and manages the issuance, fetching, tracking, and ordering of software-defined address streams, such as stream instructions and circular buffer instructions. The engine can maintain several read memory requests per tile, for example, 256 requests within a machine. The scatter-gather engine tracks and updates buffer occupancy to rate-control address requests. Software can structurally scrutinize buffer occupancy without the need to explicitly manage changes in latency when processing individual address requests.

[0040] The combination of stream instructions, circular buffer instructions, and a scatter-gather engine enables isolated access execution and can effectively increase long memory access latency, including irregular memory access streams. This gives software the flexibility to generate data-dependent memory and the option to schedule them at a "coarse" granularity, in which case the scatter-gather engine is configured to control the flow and respond to requests.

[0041] Aspects of this disclosure provide an accelerator configured to execute multiple threads of dynamic small-vector tasks across sets of computation tiles. Dynamic tasks include tasks involving data-dependent control and memory access. Small-vector tasks may include tasks performed on relatively small vectors (e.g., eight-element vectors). These tiles are managed by a single task management core referred to as a sequencer. The accelerator's computation-to-bandwidth ratio is tuned to accommodate the irregular and sparse access and computation of large datasets stored in off-processor high-bandwidth memory.

[0042] An accelerator may contain multiple tiles, each containing its own access core and execution core. An access core may be a scalar unit used to decouple data movement from computation. An execution core may be another scalar unit mounted on a vector unit with multiple SIMD lanes, configured to process data fetched by the corresponding access core. Each tile may also contain its own XPU for performing cross-lane reduction, shuffling, sorting, prefix summing, etc. An accelerator may also include a sequencer, which may be a scalar unit for task management across tiles and for communicating with other cores. An accelerator may also include scratchpad memory. For example, it may include 8 megabytes of shared memory, but the size of the shared memory may vary in various examples. As described herein, the accelerator can implement a streaming memory access interface to keep transactions pending for off-tile memory. The accelerator may also include tiles in shared scratchpad memory, as well as high-bandwidth crossbars for connecting them to each other.

[0043] Exemplary System Figure 1A is a block diagram of a hardware circuit 101 for accelerating data-dependent computations, according to the aspects of this disclosure. The hardware circuit 101 may include a sparse accelerator 103, a coprocessor 104, a high-bandwidth memory 107, and an on-chip interconnect 108. The sparse accelerator 103 may include one or more tiles 102A to 102F. Each tile implements its own vector processing unit (VPU) and includes its own cross-lane processing unit (XPU) 101A to 101F. The sparse accelerator 103 may include a tile sequencer 106 configured to coordinate input and output data across tiles 102A to 120F.

[0044] Tiles 102A-120F can be interconnected according to a wide variety of topologies, such as a multidimensional ring or torus. The interconnections may include crossbars, as described in more detail with reference to Figure 1B. Crossbars or interconnections for the processor can receive and transmit data between, for example, tiles 102A-120F, on-chip memory, and off-chip memory on a per-clock cycle basis.

[0045] The tile sequencer 106 is a component of the sparse accelerator 103 and is configured to receive and distribute instructions for performing calculations on tiles 102A to 120F while coordinating them. This coordination can be managed, at least partially, by utilizing the distributed relationships of processing components on the sparse accelerator 103, for example, to leverage different types of data or instruction parallelism. The tile sequencer 106 will be described in more detail with reference to Figure 4.

[0046] The sparse accelerator 103 is configured to perform data-dependent operations using tiles 102A to 120F. As illustrated and described in more detail with reference to Figure 3A, each tile can implement several data processing lanes for streaming data through a vector processing unit (VPU) and a cross-lane processing unit (XPU). The tiles can retrieve the streamed data from on-chip memory 105, which may be any of the following: main memory, cache, or persistent storage, such as solid-state storage or hard disk storage. The streamed data can also be retrieved from the coprocessor 104, from high-bandwidth memory 107 that serves one or both of the coprocessors 103 and 104, and / or from another data source connected to the hardware circuit 101 via the on-chip interconnect 108.

[0047] The on-chip memory 105 may be a scratchpad memory physically distributed across each tile 102A to 120F. In contrast to a hardware cache, the scratchpad memory can be programmatically managed and, for example, can store data according to software instructions. The on-chip memory 105 may be globally addressable via a wide variety of interfaces, such as direct memory access and / or stream interfaces.

[0048] The coprocessor 104 can be any core, such as a CPU or a high-density core. For example, the coprocessor The processor 104 may be configured to accelerate specific operations such as matrix-to-matrix multiplication and matrix-to-vector multiplication. Examples of operations may include dense matrix-to-matrix calculations, in which case the majority of elements in the multiplied matrices (e.g., more than 50 percent in some examples) have non-zero values. The complexity of the calculation can be approximated as a function of the dimension of the multiplied matrices. In some examples, the coprocessor 104 resides on a different device from the rest of the hardware circuitry 101 and communicates data to the hardware circuitry via an on-chip interconnect 108. The on-chip interconnect 108 may be a data bus or any form of interconnection according to any of various communication standards, e.g., PCIe. The on-chip interconnect may also implement a core memory network. The core memory network may be an on-chip network connecting the coprocessor 104 and the sparse accelerator 103.

[0049] Exemplary features of the sparse accelerator 103 are directed towards improving the computation of sparse operations (e.g., operations on operands or inputs where there are generally more zero-value elements than non-zero-value elements). This type of feature may include performing sparse operations using programmable XPUs 101A-101F, cooperative memory prefetching, and / or a combination of instruction streams or instruction ordering, as will be described in more detail herein.

[0050] While examples are provided in the context of sparse computation, it should be understood that in some examples, the sparse accelerator 103 can be used to accelerate other types of computations commonly associated with accelerating machine learning model processing, including linear algebra operations such as vector / matrix multiplication, computation of activation function outputs, layer output pooling, and layer output normalization. The coprocessor 104 and sparse accelerator 103, implemented as part of the common hardware circuit 101, can facilitate the distribution of various tasks suitable for either of these two, though not limited to these two. Acceleration of non-sparse computations may include dense computation, e.g., computations on non-sparse inputs. Examples of dense computation may include linear or strided access to arrays. Other examples include dense matrix multiplication, fully connected layers, and convolutional layers in deep neural networks.

[0051] One example of input to the hardware circuit 101 is data constructed as a tensor. For example, a tensor may represent input data and / or model parameter values ​​for a machine learning model to be executed using the hardware circuit 101. A tensor is a data structure that generalizes various other common data structure types of different dimensions. A tensor can contain zero or more elements, which can be one or more different data types such as integers, floating-point values, and Boolean values. Within each data type, the data type can be parameterized according to a specific level of precision, e.g., 8-bit, 16-bit, or 32-bit integer or floating-point values. The dimension of a tensor is called its "rank". A rank zero tensor is a single element, also called a scalar. A rank one tensor is also called a vector. A rank two tensor is also called a matrix. Vectors and matrices can also be said to have different ranks. For example, a rank two vector corresponds to a matrix. A non-zero rank tensor can be described as a set of tensors one rank lower. For example, a vector or rank 1 is a set of scalar values, and a rank 2 matrix is ​​a set of rank 1 vectors.

[0052] The hardware circuit 101 may at least partially implement a processing pipeline for training a neural network. The pipeline may include generating embeddings for input training examples. Feature tensors for various input training examples will have varying degrees of sparsity, affecting the amount of computational work required to generate the corresponding embeddings. The sparse accelerator 103 receives a tensor of feature values ​​representing a training input example and embeds it as a tensor of lower rank than the feature tensor. It can be configured to generate mina.

[0053] To generate the embedding, the sparse accelerator 103 is configured to implement a variety of data-dependent operations for efficient sparse data computation on XPUs 101A-101F, etc., or more generally on VPUs. These operations include sorting or summing sparse vectors, operations for summarizing the contents of input vectors, and operations for converting sparse matrices from one sparse matrix storage format to another.

[0054] Instead of predetermined physical circuits to accelerate the execution of data-dependent operations, VPUs, including XPU101A~101F, can be configured, for example, to perform a wide variety of data-dependent operations, through programming. The sparse accelerator 103 enables general-purpose support for processing sparse data while allowing the complementary coprocessor 104 to perform other types of operations.

[0055] Hardware circuit 101 is, for example, a central processing unit: CPU), graphics processing unit (GPU), field-programmable gate array (FPGA), or Application-specific integrated circuits (ASICs), for example For example, it could be any of the many different types of processing units, such as a tensor processing unit (TPU). The hardware circuit 101 can be implemented on a computing device that itself may be part of a system of one or more devices.

[0056] Figure 1B is a block diagram of an exemplary data path implemented as part of a hardware circuit, according to the aspects of this disclosure. The scratchpad memory 162B and the task instruction memory 160B may form part of the on-chip memory 105.

[0057] Various data paths 150B, 152B, 154B, and 156B are shown. These data paths may or may not physically share the DMA path 150B, which includes the circuit interconnection between the tile sequencer 106 and tiles 102A-120F, for direct memory access (DMA) of the scratchpad memory 162B. The instruction data path 152B includes the circuit interconnection between tiles 102A-120F, the task instruction memory 160B, the scratchpad memory 162B, and the memory transport interface 163B. The scratchpad (spmem) data path 154B shows a potential path for data from the task instruction memory 160B to tiles 102A-120F. The control data path between the tile sequencer 106 and tiles 102A-120F shows an exemplary path for control signals generated by the tile sequencer 106. When a control signal is received by tiles 102A to 120F, the tile can be made to perform one or more primitive or complex operations, such as reading data, writing data, and / or processing data, according to one or more functions specified by the control signal.

[0058] Task instruction memory 160B is shared by tiles 102A-120F. Task instruction memory 160B holds programs that can be executed by the tile access core and the tile execution core. In some examples, task instruction memory 160B is 468 bits wide and 16,000 words deep, and may be organized as banks of 2,000 words, for example, error correction codes. Multiple banks allow for multiple reads from or multiple writes to task instruction memory 160B per block cycle. The DMA descriptor for task instruction memory 160B may contain fields of length that are multiples of 64 bytes. When instruction bundles are stored in high-bandwidth memory, they are top-of-the-line. The bits can be zero-padding from the first bit to the 512-bit boundary. The sparse accelerator 103 may drop the padded bits before writing to the task instruction memory 160B. Each DMA descriptor can transfer one task instruction memory bundle.

[0059] The tile sequencer 106 may be a scalar core primarily responsible for dispatching tasks to tiles and / or initiating DMA transfers. For example, the tile sequencer may receive instructions as bundles. Instructions within a bundle can be executed in parallel across tiles 102A-120F, simultaneously updating the architecture state. Once a bundle is executed, it receives scalar or vector issuances before the instructions within it are executed. Scalar or vector issuances may be held over one or more cycles due to various conditions documented in hold-scalar-issue conditions for each scalar instruction, or hold-vector-issue conditions for each vector instruction. The ISA for the sparse accelerator 103 may define several global scalar-hold-issue or global vector-hold-issue conditions that can unconditionally hold a bundle. During a scalar or vector issuance, several actions may be performed: The bundle's predicates are evaluated and updated; the values ​​of registers to be used can be recorded; branches within the instruction can be executed, and registers are updated.

[0060] A task instruction memory bank can implement one or more of the following interfaces. Each bank may include a prefetch request and prefetch response broadcast bus. The prefetch request and response bus architecture is adapted to the SPMD (single program, multiple data) arithmetic mode. The read response may be broadcast to all tiles on the prefetch response broadcast bus. Bundle broadcasting allows the hardware to duplicate requests originating from other tiles, if possible, reducing overall bandwidth demand.

[0061] Figure 2 is a block diagram of an exemplary environment 200 for implementing the hardware circuit 101. The hardware circuit 101 may be implemented on a device having one or more processors in one or more locations, such as within a server computing device 215. The user computing device 212 and the server computing device 215 may be communicably coupled to one or more storage devices 230 via a network 260. The storage device 230 may be a combination of volatile and non-volatile memory and may be in the same physical location as the computing devices 212 and 215 or in a different physical location. For example, the storage device 230 may include any type of non-temporary computer-readable medium capable of storing information, such as hard drives, solid-state drives, tape drives, optical storage, memory cards, ROMs, RAMs, DVDs, CD-ROMs, writable memory, and read-only memory.

[0062] The server computing device 215 may include one or more processors 213 and memory 214. Memory 214 can store information accessible by the processor 213, including instructions 221 that can be executed by the processor 213. Memory 214 may also include data 223 that can be retrieved, manipulated, or stored by the processor 213. Memory 214 may be a non-temporary computer-readable medium of a type capable of storing information accessible by the processor 213, such as volatile memory and non-volatile memory. The processor 213 may include one or more central processing units (CPUs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), and / or application-specific integrated circuits (ASICs), such as tensor processing units (TPUs). The processor 213 is described with reference to Figures 1A and 1B. As detailed in the document, this may include coprocessors and sparse accelerators implemented as part of the hardware circuitry.

[0063] Instruction 221 may include one or more instructions. When executed by processor 213, the instruction causes one or more processors to perform the operation defined by the instruction. Instruction 221 may be stored in an object code format for direct processing by processor 213, or in other formats including interpretable scripts or a set of independent source code modules that are interpreted on demand or pre-compiled. Instruction 221 may include instructions for configuring stream transfers in accordance with aspects of this disclosure. Server computing device 215 and / or user computing device 212 may implement a compiler or other program for generating and sending instructions to hardware circuit 101 as control signals for tiling the circuit.

[0064] Data 223 can be retrieved, stored, or modified by processor 213 according to instruction 221. Data 223 can be stored in computer registers as a table with multiple different fields and records, or as a JSON, YAML, proto, or XML document in relational or non-relational databases. It can be stored in a database. Data 223 can also be formatted in a computer-readable format such as binary values, ASCII, or Unicode, but is not limited to these. Furthermore, data 223 may contain enough information to identify related information, such as numbers, descriptive text, proprietary codes, pointers, criteria, or information used by functions to compute related data, relating to data stored in other memory, including other network locations.

[0065] The user computing device 212 may also be configured with one or more processors 216, memory 217, instructions 218, and data 219, similar to the server computing device 215. The user computing device 212 may also include user outputs 226 and user inputs 224. User inputs 224 may include any suitable mechanism or technology for receiving input from the user, such as a keyboard, mouse, mechanical actuator, soft actuator, touchscreen, microphone, and sensor.

[0066] The server computing device 215 may be configured to transmit data to the user computing device 212, and the user computing device 212 may be configured to display at least a portion of the received data on a display implemented as part of the user output 226. The user output 226 may also be used to display the interface between the user computing device 212 and the server computing device 215. The user output 226 may optionally or additionally include one or more speakers, transducers or other audio outputs, haptic interfaces or other haptic feedback that provide non-visual and non-auditory information to the platform user of the user computing device 212.

[0067] Figure 2 shows processors 213, 216 and memories 214, 217 located within computing devices 215, 212. However, the components described herein, including processors 213, 216 and memories 214, 217, may include multiple processors and memories capable of operating in different physical locations rather than within the same computing device. For example, some of the instructions 221, 218 and data 223, 219 may be stored on a removable SD card in a read-only computer chip, for instance. Some or all of the instructions and data may be physically separated from processors 213, 216. The information present can be stored in a location accessible to the processors 213 and 216. Similarly, processors 213 and 216 may include a set of processors capable of performing concurrent and / or sequential operations. Each computing device 215 and 212 may include one or more internal clocks that provide timing information available for timing measurements for operations and programs performed by computing devices 215 and 212.

[0068] The server computing device 215 may be configured to receive requests for processing data from the user computing device 212. For example, the environment 200 may be part of a computing platform configured to provide various services to the user through various user interfaces and / or APIs that expose platform services. One or more services may be a machine learning framework or set of tools for generating neural networks or other machine learning models according to specified tasks and training data. The user computing device 212 may receive and transmit data specifying the type of workload or complex operation to be configured for the XPU of the sparse accelerator 103 to perform. The user computing device 212 may send instructions directly to the hardware circuit 101, or, as described herein, may have the server computing device 215 generate instructions and send them to the hardware circuit 101 as control signals.

[0069] Devices 212 and 215 may be able to communicate directly and indirectly via network 260. Devices 215 and 212 may be able to configure listening sockets that can accept initiating connections for sending and receiving information. Network 260 itself may include a variety of configurations and protocols, including the Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local networks, and private networks using communication protocols specific to one or more companies. Network 260 may support a variety of short-range and long-range connections. Short-range and long-range connections generally relate to the Bluetooth® standard. This may be done across various bandwidths such as the connected 2.402GHz to 2.480GHz, 2.4GHz and 5GHz commonly associated with the Wi-Fi® communication protocol, or using various communication standards such as the LTE® standard for wireless broadband communication. Network 260 may also additionally or alternatively support wired connections between devices 212 and 215, including via various types of Ethernet® connections.

[0070] While a single server computing device 215 and a user computing device 212 are shown in Figure 2, it should be understood that the aspects of this disclosure can be implemented in a paradigm for sequential or parallel processing, or through a distributed network of multiple devices, according to a wide variety of configurations and quantities of computing devices. In some implementation examples, the aspects of this disclosure can be performed on a single device, or any combination thereof.

[0071] Figure 3A is a block diagram of an exemplary tile 102. The XPU 101 is coupled to a crosslane controller 310. The crosslane controller 310 provides a separate thread of control that enables crosslane instructions on the XPU 101. As described herein, the XPU 101 may receive a first instruction, for example, through one or more control signals, which may be translated into one or more second and third instructions to perform the complex operation specified by the first instruction and provided to the processing cells and crossbars of the XPU 101, respectively. Instructions to the XPU 101 are carried through control signals. The processing cells and crossbars of the XPU101 are configured to interpret this and perform the corresponding primitive operations. An example of such an instruction is an opcode in an instruction set architecture (ISA).

[0072] Tile 102 can receive data from the on-chip interconnect 108 and even from the on-chip memory 105, as illustrated with reference to Figure 1A. The XPU can also receive instructions from the instruction interface 324, for example from the tile sequencer 106, through the scalar core 312 or scalar core 320. The scatter-gather engine 322 of tile 102 can receive incoming data and control which data is passed to memory 306 via the memory scheduler 314. In some examples, the scatter-gather engine 322 may be referred to as the read-write engine 322 instead of the scatter-gather engine.

[0073] The memory scheduler 314 can coordinate how data is accessed and retrieved from memory 306. Memory 306 is dedicated to tile 102 and cannot be accessed by other components connected to tile 102, such as other tiles. The arbiter 304 is configured to manage, for example, which of the vector processing units (VPUs) 302A to 302H accesses memory 306 per clock cycle. Tile 102 can maintain a task queue 308 of tasks to be performed by tile 102, which are sent to the scatter-gather engine 322 via the scalar core 320. Tile 102 can also maintain registers for tile synchronization flags 318 and / or memory flags 316 to synchronize tile 102 with hardware circuits and other tiles in memory 306, respectively.

[0074] VPUs 302A-302H are connected to XPU 101 via data processing lanes indicated by solid lines between XPU 101 and VPUs 302A-302H. The dashed lines between XPU 101 and VPUs 302A-302H represent control signals that can be received by control cells within XPU 101 to configure XPU 101 to perform complex operations corresponding to received control signals. The vector processing unit is configured for efficient operations on input vectors. The length of a vector processed at one time by tile 102 may depend on the number or width of VPUs implemented by the tile. For example, eight VPUs 302A-302H have a width of eight. VPUs 302A-302H can process data along the same data processing lane. VPUs 302A-302H may be configured to perform scalar operations on the elements of received vectors from memory 306. VPU302A-302H can receive data from XPU101, and XPU101 can process the data across data processing lanes, not just along the lanes executed by each VPU302A-302H, as described herein.

[0075] Figure 3B is a block diagram of another exemplary tile 392 implementing the XPU 101 for stream transfer. Tile 392 can receive data from the on-chip interconnect 108, as well as from the on-chip memory 105, as described with reference to Figure 1. The XPU 101 can also receive instructions from the instruction interface 324, for example, from the tile sequencer 106. The scatter-gather engine 322 of tile 392 can receive incoming data and control which data is passed to memory 306.

[0076] This exemplary tile 392 is based on a detached access / execution architecture, where a program (and associated instruction sequence) can be separated into two streams. The first stream may be an access stream for fetching operands and storing the results. The second stream may be an execution stream that consumes operands, performs computations, and produces results. These streams are detached access streams. It runs on two separate cores, the Tile Access Core (TAC) 332 and the Tile Execution Core (TEC) 330, which form part of the execution architecture. The TAC 332 is a scalar unit used to decouple data movement from data computation, for example, by separating the fetching of data from memory from the processing of the fetched memory. The TEC 330 is a scalar unit mounted on the VPU with multiple SIMD lanes (e.g., eight SIMD lanes) used for processing vectors.

[0077] The tile access core 332 is based on the scalar core complex 320 and may be responsible for prefetching operands to be executed from memory 306 or from high-bandwidth memory outside of tile 392. The tile execution core 330 is based on the scalar complex core 312 and includes the XPU 101 and VPU 302, and may be responsible for performing computations on the prefetched operands to produce results. The VPU 302 is connected to memory 306 via the load / store unit 328. The load / store unit 328 is configured to perform gather and scatter operations on data passing through tile 303. Gather and scatter operations can be performed granularly, for example, at a 4-byte granularity. The load / store unit 328 may implement load and store queues to manage bank contention. The LSU 328 may provide load / store access to a subset of scratchpad memory.

[0078] TAC332 and TEC330 have independent instruction streams and together form a producer-consumer pair. Tile 102 can maintain a task queue 308 of tasks to be executed by tile 102, and such tasks are sent to TAC332 and TEC330.

[0079] The task queue 308 may include one or more queues for push and popping instructions. For example, the task queue 308 may have a first queue (e.g., a first-in, first-out (FIFO) queue) for pushing values ​​from the TAC332 to the TEC330. The TEC330 can then pop and process the enqueued values. As another example, the task queue 308 may include an additional queue for connecting the TAC332 and TEC330 in the reverse direction.

[0080] TAC332 and TEC330 communicate with each other via tile-local scratchpad memory (SpMEM), such as memory 306. The TAC332 and TEC330 can also communicate through scalar memory 334, instruction buffer 326, and tile synchronization flag 318. Memory 306 can be used by the TAC332 and TEC330 to exchange data, and can also be used as a software-managed circular buffer to pass data between the TAC332 and TEC330 in a first-in, first-out (FIFO) order. The tile synchronization flag 318 can be used as a counting semaphore between the TAC332 and TEC330. For example, if a circular FIFO order is used between the two cores, the producer core increments the synchronization flag 318 by the number of bytes after each push and stops when the count reaches the maximum size of the FIFO order. Similarly, the consumer decrements the synchronization flag 318 after each pop and stops if there is no data in the buffer. Since the amount of data prefetched can be dynamic, a done bit is used to indicate the end of the stream.

[0081] The tile synchronization flag 318 may, in some examples, contain a set of 32 synchronization flag registers. Each register can store 32 bits of data, plus a "done" bit and an "enable_public_access" bit. The tile synchronization flag 318 register may be implemented as a monolithic flop array, allowing simultaneous access from all sources. This enables access. If there is an address conflict between writes, the priority of access between sources can be specified. For example, scalar miscellaneous instructions may have absolute priority. In some examples, only one of the scalar miscellaneous instructions may be issued. The read-modify-write operation is pipelined and therefore, against any synchronization flag. Back-to-back read-modify-write operations may be supported.

[0082] DMA updates, stream updates, and remote writes may be combined on a single (external) interface using round-robin arbitration. For example, the external interface from the tile to the external source may have a separate access path to the synchronization flag, for example, a data path as illustrated and described with reference to Figure 1B.

[0083] Each bank of the task instruction memory 160B can arbitrate cycle by cycle among the tile request data stored in the bank. These banks can use a distributed arbitration scheme to select a winner from the TAC332 and TEC330 in the sparse accelerator 103 that are vying for access to the bank. The arbitration ensures that the request bandwidth is equally divided among the request tiles. A request could be, for example, to prefetch data by each TAC for each tile. For example, a winning prefetch request is given the highest priority for access to one of the banks that was accessed.

[0084] Access from the sparse accelerator 103 via the control status register may have the next highest priority. This is only delayed by prefetch read requests. The control status register (CSR) is accessed by the sparse accelerator It can be implemented as part of the processor to store additional information results of machine instructions executed by 103. The sparse accelerator 103 can maintain an indirect CSR timeout status bit to determine whether the completion of a CSR access is being hindered. After bank access and CSR access, DMA read or write operations can be prioritized last. DMA operations, such as read or write operations, can be delayed if they access a busy bank. In some examples, the priority of how access operations are arbitrated and resolved on the sparse accelerator 103 may differ.

[0085] Decoupling data access from data execution offers at least the following advantages: Since the TAC332 can perform address calculations and prefetch necessary data, long memory latency (e.g., 600 cycles) is more effectively tolerated. The aforementioned architecture also has high tolerance for control latency. Dynamic dependencies can make resolving loop conditions difficult, potentially hindering effective software deployment. The TAC332 can pre-process these dependencies to resolve them if the conditions can be determined outside the TEC330, providing a way to hide control latency.

[0086] Figure 4 is a block diagram of a tile sequencer 400 according to the aspects of this disclosure. The sequencer memory 410 (Simem) may have a width of VLIW instruction bundles and a depth of approximately 8000 bundles, although the width and number of bundles in the sequencer memory 410 may vary depending on the implementation example. The sequencer memory 410 may be read or written through direct memory access or through indirect access. This can be done whether the sequencer 400 is currently running or not. The sequencer memory DMA memory descriptor may have a field of length that is a multiple of 32 bytes. Each bundle, when stored in high-bandwidth memory, can be zero-padded from the most significant bit to a 256-bit boundary. The padded bits are RAZ / WI (read-as-zero and w It has a write-ignored (read and write ignored as zero) property. Each memory descriptor can transfer one instruction bundle.

[0087] The sequencer memory 410 can be organized as two banks with contiguous addresses between them. Each bank can be read from or written to once per clock cycle. Interleaving allows a continuous instruction sequence to consume only half of the bandwidth of each bank. An unusually small loop can consume only three-quarters of the bandwidth of a given bank, thus leaving sufficient bandwidth to advance access in the forward direction.

[0088] Each sequencer memory bank within the sequencer memory 410 may follow the following interfaces. The sequencer memory 410 can receive read requests from the instruction data path (for example, the instruction data path illustrated and described with reference to Figure 1B). The banks of the sequencer memory 410 can also receive control status register access via the indirect register interface, as well as DMA writes and reads. Reads from the instruction data path may have the highest priority for access to each bank within the sequencer memory 410. DMA reads, writes, and CSR accesses may have equal priority and may be subject to least recently used (LRU) arbitration for banks that are not busy performing instruction fetches.

[0089] The sequencer 400 can fetch its instructions from the sequencer memory 410. The sequencer 400 executes the program's control thread. This involves generating descriptors that are dispatched by the descriptor dispatch unit 413 (for task and stream descriptors) and the DMA unit 414 (for DMA descriptors), respectively. Task descriptors are provided to the tile FIFO 416, which is then executed by the respective tile. DMA descriptors can be passed to other components of the hardware circuit 101 and other off-chip components. The sequencer communicates with other cores in the system and coordinates tasks across accelerator tiles.

[0090] The current state of the sequencer 400 can be determined by reading the corresponding status register. Other registers related to instruction bundle fetching and execution are the program counter and branch state. The tile sequencer 400 can issue DMA descriptors that can be adjusted and throttled by hardware. Hardware can leave a predetermined number of DMAs issued by the sequencer unprocessed, provided that the synchronization flags stored between the synchronization flags 418. Adjustment and throttling can be enabled or disabled as needed.

[0091] Synchronization flags can be displayed in two or more types. Synchronization flag 418 can be displayed in the tile sequencer 400. Other synchronization flags can be stored in the TAC and TEC of each tile. All synchronization flags together can be placed in a single address space accessible by DMA operations, atomic remote set / append instructions, and atomic tile set / append instructions. There are several implementable interfaces between synchronization flags in tiles and synchronization flags in the sequencer 400. DMA operations can atomically add values ​​to synchronization flags during execution and set the "Done" bit upon completion. Stream operations can atomically add values ​​to synchronization flags during execution and set the "Done" bit upon completion. Remote write instructions generate single-word control writes that can atomically set or add values ​​to synchronization flags. These may be from atomic remote set / append instructions used to update remote synchronization flags. These may also be from atomic tile set / append instructions used to update synchronization flags in sparse accelerators. Yes, it is possible. Another possible interface is a write interface, in which atomic updates to the synchronization flags are performed by the set synchronization flags and the synchronization flag append instruction, including optionally changing the "Done" bit. Another possible interface is a read interface.

[0092] The synchronization flag 418 may be organized in memory as several banks. Each entry may contain several bits of data, for example 32 bits of data, in addition to a "Done" bit and an "enable_public_access" bit. The banks may perform separate cycle-by-cycle arbitration for read and write ports to prioritize scalar miscellaneous instructions over, for example, DMA, streams, and remote write updates. The synchronization flag registers in TAC and TEC may also store several bits, a "Done" bit and an "enable_public_access" bit.

[0093] Synchronization flags within TAC or TEX can be implemented as a monolithic flop array enabling simultaneous access from all sources. Similar to sequencer synchronization flags, tile synchronization flags can be managed according to a priority scheme to avoid contention between writes.

[0094] Figure 5 is a block diagram of an exemplary scratchpad memory 500 having memory across multiple tiles of a sparse accelerator, according to the aspects of this disclosure. Each tile is a scratchpad memory 5 referred to as a tile scratchpad memory or TileSpmem. It may include each part of 02. The tile scratchpad memory 502 may be accessed by tiles via load / store interfaces implemented as one or more circuits. The tile scratchpad memory 502 can also be used as a local buffer for each tile to move data in and out of the tiles using stream instruction transfers.

[0095] As shown in Figure 5, each tile scratchpad memory 502 may contain several banks, for example, 32 banks labeled as banks 0 to 31, respectively. Each bank can hold a certain amount of data, for example, 16 kilobytes. Each bank can hold several words, for example, 4-byte words and 7-bit error correction codes. In some examples, a bank can hold 4096 words. It should be understood that the scratchpad memory 500 can be implemented with a different number of tile scratchpad memories 502, each capable of holding different banks of different sizes. It should also be understood that each bank can be implemented in various examples to store a different number of words of different sizes.

[0096] Words and banks in the tile scratchpad memory can be accessed through 17-bit instructions and stream addresses, although the exact size and format of the addresses may vary depending on the implementation. Each bank can implement one or more of the following interfaces. The tile scratchpad memory 502 loads or stores and adds data in the bank and word specified by the received address or address range by enqueuing access requests to a per-bank load queue (not shown) in response to an received instruction. To store data, the tile scratchpad memory 502 enqueues access requests to a per-bank load queue from a receive store or store append instruction. Each bank can also receive read access from one or more external sources, such as DMA or stream requests for reading / writing data. Exemplary instructions to the tile scratchpad memory 502 include vector load and vector store instructions. These and other instructions may be specified in the ISA for the hardware circuit 101, and in response to the receipt of an instruction, the hardware circuit 101 Instructions can be defined to cause the system to perform specific predetermined operations. A vector load instruction can be issued to the bank load queue. Similarly, a vector store instruction may be issued to the bank store queue. A vector store append instruction may refer to a type of instruction that causes the hardware circuit 101 to perform read-modify-write operations on a target range of addresses in one or more banks of the tile scratchpad memory 502.

[0097] To handle the prioritization of loads and stores on banks of tiled scratchpad memory, a store access at the top of the store queue for each bank may have the highest priority for accessing the write port of that bank (not shown in Figure 5). A load access at the top of the load queue for each bank may have the highest priority for accessing the read port of that bank. If a load is for the same address as the queued store, and the store is from an instruction bundle issued before the load, the bank may instead read data from the store queue to load the data.

[0098] Access from sources outside the tile hosting a bank may undergo multi-level arbitration before the access is presented to the bank. For example, writes or appends from different sources may first undergo bank-wide Longest Unused (LRU) arbitration among those enqueued in the target bank's per-bank write queue. The write queue may be four entries deep, for example. Similarly, read access requests from various sources may also undergo LRU arbitration. The winning read request may then undergo a second level of arbitration with read requests arising from stream append access and load access for reading the bank. Stream append access performs a read-modify-write operation, which is also enqueued in the per-bank write queue. If the head of the write queue for a bank is a stream append access, its read request may undergo round-robin arbitration with winning external read requests. The winning read request has second-highest priority over the bank's read ports, behind load access. A write request at the top of the write queue for each bank may have the second highest priority for the bank's write ports, behind store access. In the absence of bank contention, a bank can maintain a throughput of at least one stream append operation per cycle.

[0099] Each bank of the tile scratchpad memory 502 includes ports for reading from and writing to the bank (not shown in Figure 5). These ports can generate warnings to detect exhaustion. External sources of read and write requests may run out when accessing banks that are continuously accessed by loads, stores, or store appends. The probability of exhaustion can be mitigated by specifying a maximum number of accesses to the total number of banks in the tile scratchpad memory 502, for example, allowing access to a maximum of 8 banks out of 32 banks in a given clock cycle.

[0100] Warnings generated due to depletion may be based on a predetermined threshold. For example, the predetermined threshold could be the number of consecutive clock cycles that correspond to the period during which an external source does not access the bank for read or write requests.

[0101] If read port exhaustion is detected by a bank in the tile scratchpad memory 502, one or more actions may be performed by the scatter / gather engine. A hold can be issued if the bundle has a load instruction, store instruction, or store append instruction. The load and store queues per bank may be drained normally. A hold is issued until a predetermined maximum number of read requests are handled by the scatter / gather engine, or the scatter / gather engine read queue This can continue until it runs out.

[0102] If the write port exhaustion is detected by a bank in the tile scratchpad memory 502, the following sequence is executed: A hold is issued if the instruction bundle has a store instruction or a store append instruction. Each store queue for each bank is drained normally. Issues can continue to be held until the maximum threshold of held issue requests is met by the scatter-gather engine. The threshold and cycle count may vary depending on the implementation.

[0103] The synchronization flag memory 412 may, in some examples, be organized as four banks (not shown). Each entry may have 32 bits of data. Each bank in the synchronization flag memory 412 can perform separate cycle-by-cycle arbitration for read and write ports between the following sources. Scalar miscellaneous instructions have absolute priority. In some examples, only one of the scalar miscellaneous instructions may be issued in any given cycle. Read-modify-write operations may be pipelined, thus supporting back-to-back read-modify-write operations to any location. After scalar miscellaneous instructions, DMA updates, stream updates, and remote writes may be combined on a single (external) interface via round-robin arbitration and may have the next highest priority. Access from the host to the hardware circuit 101 via the control status register has the lowest priority.

[0104] A request to access a bank may originate from multiple sources. A core memory network read is a read access sent over the core memory network, which may be initiated by a DMA or stream request. An access request may be a core memory network write for a write access from the core memory network, which may be initiated by a DMA or stream request. An access request may originate from a stream address, for example, an indirect address read from a local tile. An access request may originate from a read or write to an internal stream.

[0105] Figure 6 is an exemplary block diagram of a scalar core complex 600 of a tile sequencer according to the aspects of this disclosure. The scalar core complex may be a 32-bit scalar VLIW core. The scalar execution pipeline 601 may be one or more circuits configured to execute scalar instructions and miscellaneous instructions, such as fetching, decoding, and executing. The scalar core complex 600 executes prefetched instruction bundles. During execution, one bundle per block is fetched, as addressed by the following PC registers. Each bundle may contain two scalar instructions and one miscellaneous instruction that is decoded and executed simultaneously.

[0106] The scalar execution pipeline 601 may include a 32-bit compute unit, which includes registers 604 and ALU 606. The compute unit provides address calculations, loop control, and scalar operands used to construct descriptors. Complex 600 has memory 608 accessible via a load / store interface. Memory 608 is used by Complex 601 to store intermediate scalar values ​​during program execution. The core type determines the depth of memory 608, for example, whether it is part of a tile sequencer or a TAC or TEC of a tile.

[0107] Complex 600 can construct descriptor 609 and use descriptor scratchpad register 610. If an instruction uses descriptor scratchpad register 610, when issued, Complex 600 uses the specified descriptor scratchpad register in the instruction. Fetch an N x 32-bit word starting with the dress. The value of N depends on a specific descriptor type. The descriptor is then enqueued in the descriptor issue FIFO612.

[0108] Cross-lane processing unit (XPU) The following description and attached diagram illustrate an exemplary implementation of the XPU. Distributed embedded training is difficult to accelerate because effective acceleration relies on utilizing provisioned computations, available on-chip main memory bandwidth (HBM), and inter-chip communication bandwidth. These computations are dynamic and irregular, making them difficult to utilize. In addition, performance bottlenecks vary greatly from model to model, and the problem space is rapidly evolving with new algorithms, optimizers, model architectures, etc. Flexibility in representing new algorithms and optimizing them to various performance bottlenecks is essential.

[0109] Embeddings can be fed into machine learning models, such as neural networks, to perform machine learning tasks, such as natural language processing tasks, or other tasks where using embeddings may be beneficial. An embedding layer is a layer of a neural network trained to generate embeddings from inputs. After generation, the embeddings can then be processed downstream by, for example, later layers of the neural network implementing the embedding layer. Models with embedding layers present unique computational challenges, such as low computational density, high stress on memory bandwidth, and a large memory footprint. Furthermore, to accelerate these types of models, there can be a wide range of performance bottlenecks from model to model.

[0110] The embedding function or map can be implemented, for example, as a lookup table or as sparse vector-dense matrix multiplication. For example, the embedding function can be implemented as a matrix that, when multiplied by the input vector, produces a corresponding embedding for the input vector. For example, the input vector may be a bit vector representing the presence or absence of various natural language words in the input sentence to the machine learning model. Since the bit vector may contain elements about a large vocabulary of potential natural words that can form the input sentence, this bit vector will generally be a sparse vector with fewer than 50 percent of its elements having zero values, for example. The output vector obtained by multiplying the input vector by the embedding function matrix is ​​the embedding representing the input sentence.

[0111] The embedded function can be a very large table, for example, several hundred gigabytes in size. As a result, the embedded function cannot fit into the main memory of a single accelerator or processor, and therefore, embedded generation is distributed across multiple nodes, each with one or more accelerators. The embedded table may be partitioned across multiple devices, for example, across multiple accelerators in a data center pod.

[0112] Such dispersion complicates the processing of these embedding layers that implement these embedding functions. Aspects of this disclosure provide acceleration of various operations to scatter, gather, uniqueize, and sum various input values ​​in order to facilitate the generation of embeddings for individual or batch input samples. Large embedding tables can be partitioned across multiple accelerators. For ease of explanation, the following description will focus on a single accelerator, e.g., hardware circuit 101. Furthermore, while the examples provided herein illustrate embeddings for natural language processing tasks such as machine translation, it should be understood that aspects of this disclosure can provide acceleration of any type of machine learning model that relies at least partially on embedding generation to perform its respective machine learning task. Other examples include recommendation systems such as content recommendation systems found in domains such as multimedia recommendation, search result ranking, and advertising. include.

[0113] While examples of embedding generation are provided, it should be understood that the same primitive operations described herein can be constructed in different ways to address other sparse problems, such as sparse matrix multiplication. This flexibility enables acceleration of various sparse problems. Sparse computation can be employed in machine learning and deep neural networks, as well as in other problem spaces such as scientific computing and / or graph analysis.

[0114] On the forward path of a neural network processed according to one or more accelerators as described herein, the input may be a batch of one or more input samples. The input samples may be processed by one or more accelerators that perform operations on one or more embedding layers of the neural network. The output of the embedding layers is a batch of one or more embeddings, one for each sample in the input batch. The input samples share a common length, e.g., the number of latent features attributable to each input sample, but it should be noted that the input samples may have some empty or zero-value feature values.

[0115] In a forward path, the input batch is partitioned across multiple different accelerators, for example, across one or more host devices.

[0116] An input batch can be represented as two vectors: a vector of values ​​and a vector of indices. The vector of values ​​corresponds to the value for each identifier in the input samples of the batch. The indices can point to the position of the value for each identifier in the tensor representing the input batch. The input batch is partitioned so that parts of the input batch are sent to separate accelerators. Upon receiving the partitioned input batches, the accelerators can "uniquify" the inputs to remove duplicate identifiers between input batches. Uniquification means removing multiple instances of the same identifier. One reason for such removal is to conserve bandwidth by reducing network usage between chips and avoiding redundant access to the same identifier. Uniquification avoids redundant lookups on the embedding table. After unification, the unified input batches are distributed to multiple devices, each with a part of the embedding table, to generate output embeddings. The generated embeddings, after scattering, can be gathered from various devices and returned to another device requesting the embeddings.

[0117] During training, the gradient representing the error rate between the ground truth embedding and the predictive embedding can be similarly scattered to the device storing the partitions of the embedding table in order to update each partition.

[0118] The aspect of this disclosure concerns an XPU for performing data-dependent operations across multiple data processing lanes of a processor. Rather than implementing physically fabricated operation-specific circuitry for each data-dependent operation, the XPU may be configured to perform various operations in response to input signals that constitute individual operations performed by processing cells and crossbars arranged as a stacked network within the XPU. The XPU performs operations across values ​​of multiple SIMD data processing lanes. The XPU may be implemented as part of a coprocessor that complements a second coprocessor configured for SIMD parallel processing. The coprocessor implementing the XPU may be configured to perform data-dependent operations.

[0119] The aspect of this disclosure is the provision of an XPU for initially processing sparse data before passing it to downstream coprocessors in the processing pipeline, enabling a wider range of workloads to perform computations more efficiently than previously possible without an XPU. Because the XPU can handle a variety of data-dependent computations, the processing pipeline and corresponding processors... XPU can be designed without the constraint of pre-defining input data for processing on existing SIMD architectures. Without XPU, existing SIMD architectures cannot efficiently accelerate data-dependent operations such as generating embeddings from sparse sets of features into machine learning models.

[0120] An exemplary data-dependent operation includes generating an embedding for an input training example. The embedding may be a vector or some other data structure mapped from the input having a higher dimension than the embedding. Embedding generation can be performed as part of a workload processed according to a pipeline. As another example, the XPU may perform vector scatter or gather operations, segment sums, and / or partition a sparse feature tensor. The XPUs described herein may be complementary processing units to other components or connected components of the processor, such as vector processing units built according to a SIMD parallel processing paradigm. One or more XPUs may be connected in each processor core of a larger processor, which itself may contain other components to accelerate the performance of a particular workload, such as training a neural network.

[0121] Furthermore, XPUs are not limited to performing specific types of data-dependent operations; therefore, processors can be designed to include XPUs to complement other types of processing units for multiple different pipelines. Since XPUs can be configured on a per-workload basis, the physical footprint of an XPU is reduced compared to other approaches where specialized circuitry is physically fabricated on the processor as a complementary unit for sparse data computation. The functionality of an XPU can also be extended by using instruction sets or by extending the host processor to an existing instruction set, further improving the adaptability of various data-dependent operations as pipeline data reception changes. Instructions can be provided as signals to components of the XPU, which play a role in translating instructions to constitute the individual processing cells and crossbars of the XPU. An XPU can be configured using a program compiled by a corresponding compiler for the hardware circuitry implementing the XPU.

[0122] The XPU comprises a network of individual processing cells, each processing data passing through one or more data processing lanes via crossbar connections between processing cells. Each data processing lane may contain one or more registers for temporarily storing data during processing. Each processing cell is configured to perform one or more primitive operations on multiple sets of operands. A first set of operands is provided as input from the processor's data processing lanes shared by the processing cells. A second set of operands is provided from the crossbars configured to coordinate data transmission across the XPU's multiple data processing lanes.

[0123] The XPU can be divided into multiple pipeline stages, each stage containing a crossbar, one or more processing cells, and a control cell corresponding to each processing cell. The number of stages may also vary, for example, based on the complex operations that the XPU is configured to perform for the current workload.

[0124] The XPU performs complex operations by executing multiple primitive operations across pipeline stages of a stacked network of processing elements and crossbars. A complex operation is an operation performed by the XPU on an input to produce an output. Primitive operations are operations configured to be performed by individual processing cells of the XPU, and when executed by the XPU, they cause the XPU to perform a complex operation. Performing a complex operation may require performing other complex operations. For example, performing a vector sort. Therefore, the XPU may perform prefix addition, i.e., another operation consisting of multiple primitive operations. Exemplary primitive operations include comparison of input data, arithmetic operations, or operations for bypass. The XPU performs complex operations by configuring each of multiple individual processing cells and crossbars arranged according to one of multiple pipeline stages for the XPU.

[0125] The primitive operations performed at each stage of the XPU can be programmatically defined and may vary depending on the workload. The primitive operations configured for a processing cell are determined by one or more control signals or instructions received by the respective control cell for that processing cell. The exact primitive operations performed by a processing cell may depend, for example, on the complex operations that the XPU is currently configured to perform. In other examples, processing cells at different lanes or stages of the XPU may be configured to always perform one or more predetermined primitive operations. After the XPU generates output, that output can be passed along multiple data processing lanes to other processing units or memory units of the processor implementing the XPU.

[0126] Two exemplary compound operations that the XPU can perform are vector sort and vector duplicate count. Vector sort is an in-place stable sort of (key, value) tuples of an input vector, sorted by key. Vector duplicate count returns a running duplicate count of the values ​​of the (key, value) tuples of an input vector. The XPU is configured to perform both vector sort and duplicate count according to the same configuration of processing cells and crossbars as described herein. By using the same configuration, the XPU can perform both compound operations more efficiently because, at the very least, it does not need to reconfigure the XPU while performing vector sort and vector duplicate count for a given input vector. Other compound operations that the XPU is configured to perform include parallel prefix sum, vector partition, vector histogram, vector compact, vector permutation, vector shrink, vector shift insert, vector gather, vector scatter, etc. Performing vector duplicate count makes it possible to identify the presence of duplicate values, which can be used to uniqueize vector inputs and avoid redundant processing.

[0127] The aspects of this disclosure can offer the following technical advantages: Hardware circuits implementing the XPU can provide more flexible and programmable hardware for embedded-class workloads, and other data-dependent computations that cannot be efficiently parallelized. The XPU provides an acceleration path for different classes of data-dependent computations per workload, without the need for the XPU to be fixed to efficiently perform only specific computations. By providing programmable units as described herein, the implementing hardware circuits can adapt robustly to the demands of various workloads and complement parallelizable data-independent SIMD computations, which would otherwise be inefficient or ineffective for workloads requiring data-dependent computations.

[0128] Hardware circuits, such as application-specific integrated circuits, can be designed with varying amounts of XPUs to further scale and distribute workloads. The XPUs described herein also enable the efficient execution of multiple operations using the same configuration, further reducing processing and configuration times. For example, an XPU may be configured to perform both vector sorting and vector duplicate counting instead of separate XPU configurations and / or separate instances of dedicated circuitry to accelerate those operations.

[0129] Figure 7 is a block diagram of an exemplary XPU700. The XPU700 consists of processing cells 701~ Including 709, crossbars 703-711, and control cell 750 (represented by hatched blocks in the block diagram of Figure 7), data moves from bottom to top along data processing lanes 700A-700H, starting at stage 1 and ending at stage 6. Stage 1 includes processing cell 701 and crossbar 702. Stage 2 includes processing cell 703 and crossbar 704. Stage 3 includes processing cell 705 and crossbar 706. Stage 4 includes processing cell 707 and crossbar 708. Stage 5 includes processing cell 709 and crossbar 711. Stage 6 includes processing cell 711 and crossbar 712. In another example, the XPU may include more or fewer stages. The XPU may also include crossbar 799.

[0130] For illustrative purposes, earlier stages are considered "upstream" of later stages, and later stages are considered "downstream" of earlier stages. For example, Stage 1 is upstream of Stage 5, and Stage 4 is downstream of Stage 3.

[0131] The crossbars in each stage of the XPU can be any type of circuit configured to substitute various input values ​​from one lane to another, depending on the current configuration of the crossbar. The crossbar can receive one or more control signals from control cells for each processing cell in the same stage as the crossbar. The crossbar is configured to substitute input values ​​from each processing cell in the same stage according to a fixed pattern. This pattern depends on the complex operation that the XPU is currently configured to perform and does not necessarily cause the crossbar to substitute every processing cell output. In other words, some processing cell outputs may bypass the crossbar and proceed to the next stage along the same processing lane.

[0132] To constitute a processing cell, each processing cell of the XPU700 has a control cell 750 configured to receive one or more control signals along the respective processing lane in which the processing cell resides. The processing cell is configured with circuits for performing a wide variety of primitive operations and executing those operations according to the received control signals or instructions, as will be described in more detail with reference to Figure 7. The control cell receives instructions along the data processing lane, for example, as one or more signals that can be translated by the control cell to determine which primitive operation its corresponding processing cell will perform. The control cell can transfer control signals to the processing cell, or it can process the received instructions or signals and transfer generated control signals configured to be received by the processing cell to enable or disable the execution of a specified primitive operation.

[0133] A processing cell can also be configured to bypass input data received from its respective processing lane. When bypassed, the received input is passed from the processing cell to the crossbar in the same stage as the processing cell without modification. Input received by the bypass processing cell from the crossbar in the previous stage may be associated with zero or ignored. The actual behavior of a bypass processing cell may depend on the pipeline stage in which the cell is located and / or the processing lane in which the processing cell is located. Figure 7 shows an exemplary processing cell configured to perform comparisons, arithmetic operations, and / or bypass primitive operations.

[0134] The XPU700 may be configured to receive instructions that are defined as part of or extensions of the instruction set architecture that the processor implementing the XPU700 is configured to apply and execute. These instructions can specify a variety of complex and / or primitive operations that the XPU and individual processing cells are configured to execute as corresponding operations. The control cell 750 is configured to receive data representing instructions defined as part of the instruction set or extension, and / or instructions. The system is configured to convert the signals into control signals for configuring the corresponding processing cells. For example, control cell 750 can receive the signals as opcodes for the instruction set corresponding to the processor or hardware circuit implementing XPU700, i.e., codewords for operations configured for the XPU to perform. When XPU700 receives an instruction to perform a complex operation such as vector sort or vector duplicate count, XPU700 can configure each processing cell to perform a predetermined primitive operation that causes the XPU to perform the instructed complex operation.

[0135] Operations performed by the XPU can be synchronized by clock cycles. For example, operations performed by processing cells at each stage can be completed in one or more cycles. For example, operations at each stage can be completed in a single cycle. Various complex operations performed by the XPU can require varying amounts of clock cycles to execute. For example, vector sorting can be performed by the XPU in 6 cycles, by vector prefix sum in 4 cycles, and by vector compaction in 2 cycles.

[0136] As will be explained in more detail with respect to Figure 7, the processing cell may be configured to perform arithmetic operations such as addition between various types of operands, including floating-point values ​​and signed or unsigned integers. Arithmetic operations such as addition may form part of a complex operation performed by the XPU for scanning operations.

[0137] Exemplary instructions include instructions for resetting the XPU and retrieving information about clock synchronization and primitive operations performed by the XPU. Other instructions include instructions for retrieving one or more operands, mask values, and / or segment markers from each processing lane. These instructions may include instructions for accessing data structures stored by the XPU, along with control information specifying each of the wide variety of complex operations supported by the XPU. In further examples, these instructions may include instructions for causing the XPU to push data into various registers, latches, or flip-flops to determine if the above is valid. The pushed data may include, for example, values ​​being processed as part of the execution of a complex operation, and / or mask values.

[0138] A configured XPU700 is said to implement a processing network for performing specific complex operations. For example, the XPU700 contains 48 XPU cells that can be configured as follows: 18 cells can be configured for arithmetic operations, 38 cells can be configured for comparing input values ​​(one cell may be configured for both arithmetic and comparison operations), and 10 cells can be configured to bypass input. In response to a new instruction, the XPU700 can reconfigure itself with a new processing network to perform another complex operation.

[0139] The XPU can be configured to perform operations in a wide variety of arithmetic modes, which may be specified as various instructions in the instruction set or extensions. These diverse arithmetic modes may include various compound operations for sorting, counting, scanning and partitioning data, and / or identifying unique values ​​in data input to the XPU. Furthermore, these instructions may include operands that specify the type of comparison or arithmetic operation to be performed, e.g., comparison of unsigned integers or floating-point addition for sorting or scanning. Other operands for instructions to perform compound operations include specifying from which processing lane the output of the compound operation should be output so that it is sent out of the XPU700. Other received operands may include, for example, segment markers for performing compound operations on segments of input data to sort each of multiple segments of data received by the XPU700 across data processing lanes.

[0140] When performing vector sorting and / or vector duplicate counting, the XPU700 is configured to include an odd / even merge network and a value shuffle network. The network configuration includes one or more stages of the XPU, and each cell and crossbar in each stage is configured to perform one or more primitive operations.

[0141] The XPU700 may include register files 760A and 760B. Register files 760A and 760B may be combined with data processing lanes 700A to 700H between different stages and may be used to store and retrieve data. For example, some data may be stored in register file 760B after processing cell 707 in stage 4, while data output by the XPU700 is stored in register file 760A.

[0142] Stream Instructions and Ordering This disclosure provides a hardware or software interface for asynchronous data movement between off-core memory and core local memory, where the movement between memory is referred to as “stream transfer”. Stream transfer may include stream descriptors that enable software to represent common data movement patterns, such as those found in sparse workloads. Data may be referred to as streams or data streams. Stream transfer can be initiated by a stream instruction. A stream instruction can encode the information necessary to perform a stream transfer. Each stream is associated with a data synchronization flag ("sync flag") that is indicated by the stream instruction. They may have a series of stream identifiers (IDs). Stream instructions issued by cores with the same stream ID can form at least partially a single stream.

[0143] A stream descriptor is an internal data structure that can represent the information necessary for performing a stream transfer. For example, this information may include the source address, destination address, stream calculation code, and control information such as linear or circular buffers.

[0144] A stream transfer may simply involve moving data to or from core-local memory. Furthermore, a core-local synchronization flag may be used to track the progress of the stream. The synchronization flag can track the partial progress of a stream transfer. For example, depending on whether the flag is cleared or set according to a given configuration, the synchronization flag tracks reads from core-local memory when core-local memory is the source, and tracks writes to core-local memory when core-local memory is the destination. While the progress of reads and writes to off-core memory does not need to be tracked, a scalar fence instruction can be used to allow selection of which memory accesses the barrier, ensuring that any pending writes to off-core memory are committed.

[0145] Stream transfer may include indirect scatter-gather memory access in either off-core memory or core-local memory. The source or destination address of the access can be stored in a different memory location relative to the address being read first. For example, the indirect address may be sourced from a register file or from memory, with masking support. Indirect scatter-gather memory access may further include various addressing modes, such as row addresses or word addresses. Stream transfer may include support for direct ScatterAdd / GatherAdd modes for memory words. Memory words may be updated atomically. For example, 32-bit floating-point, 32-bit integer, 16-bit floating-point, and 16-bit It can support integer data types.

[0146] Stream forwarding may include support for circular buffers in the source or destination buffers, which can simplify buffer allocation problems for software because the buffer size is unknown during software compilation.

[0147] Furthermore, a stream ordering model is schematically disclosed herein. Synchronization primitives for these transfers enable data transfers to be processed in order, even though the actual data transfers are out of order. Discrete stream instructions issued by cores having the same stream ID form a single stream. The hardware ensures ordering for transfers within a single stream, which may span multiple stream instructions.

[0148] Stream instructions belonging to a stream are processed in order. In the case of indirect stream instructions, the offset list is ordered. For example, the offset elements in the offset list are processed in order. Writes are issued to destination memory in order and can be committed in any order by the destination memory. Reads are issued to source memory in order and can be responded to in any order by the source memory.

[0149] The synchronization flag is updated to indicate a monotonous, stepwise progression of the stream. When core-local memory is the source, the synchronization flag tracks reads to core-local memory. A synchronization flag value of N when core-local memory is the source of reads indicates that the first N chunks of data are overwritable in core-local memory, where a chunk of data is a predetermined unit of size for measuring data in core-local memory. When core-local memory is the destination, writes to core-local memory are tracked by the synchronization flag. In the example where core-local memory is the destination, a synchronization flag value of N indicates that subsequent reads to the first N chunks of data in core-local memory will return the requested data.

[0150] A stream can terminate when the data for all requests preceding the last stream descriptor and for the request containing the last stream descriptor has been fully committed to memory. For example, if core-local memory is the source, the stream can terminate when all reads have finished. If core-local memory is the destination, the stream can terminate when all writes have been committed.

[0151] The aspects of this disclosure enable software to more efficiently represent common data movement patterns, specifically those seen in sparse workloads. These aspects can also provide an effective solution to complexity by hiding long memory access latency while maintaining sequential core-to-core computing and software programming models.

[0152] Stream transfer allows tile 102 and tile sequencer 106 to move data between tile-local memory, such as memory 306 or scalar memory 334, and off-tile memory, such as memory 105 or high-bandwidth memory 107. Tile-local memory is an example of core-local memory, such as memory 306, scalar memory 334, etc., which are physically local to the sparse accelerator 103. Off-tile memory is an example of off-core memory, as memory physically distant from TEC330 or TAC332 may include memory 105 and / or high-bandwidth memory 107. Each stream has an associated stream ID, indicated by the synchronization flag 318 associated with the stream instruction. Discrete stream instructions having the same stream ID are: Form a single stream with a shared stream ID.

[0153] Stream transfers allow data to be moved to or from tile-local memory. The tile-local synchronization flag 318 is used to track the progress of the stream. The synchronization flag 318 tracks the partial progress of the stream being transferred to and from various memories via tile 102. For example, if tile-local memory is the source, the synchronization flag 318 tracks read operations (or "reads") from tile-local memory, or if tile-local memory is the destination, the synchronization flag 318 tracks write operations (or "writes") to tile-local memory. Progress of reads and writes to off-tile memory does not need to be tracked. To ensure that all pending writes to off-tile memory are committed, scalar fence instructions can be used to select which memory accesses the barrier. The scatter-gather engine 322 tracks the status of stream transfers issued for each specific memory and communicates this status to the scalar core 320. When a scalar fence is issued against a specific memory barrier, the scatter-gather engine 322 waits for a status to indicate that all pending stream transfers targeting that memory (read or write) are fully committed. Once that condition is met, the fence's waiting state is released on the scalar core 320.

[0154] Stream transfers can support efficient scatter-gather operations using strided streams for accessing off-tile memory and indirect streams for accessing off-tile memory from tile-local memory or register files. Whether a stream is strided or indirect can depend on the software access pattern. A strided stream is preferred if the software wants to access any Nth element in a tensor, although an indirect stream can still function. However, an indirect stream should be used if the software wants to access a random set of elements in a tensor. Stream transfers can also support circular buffer semantics on tile-local memory.

[0155] Stream transfer supports the following data movements, in which case the granularity and alignment of the data movement depend on the source and destination pairs of memory. Data can be transferred from memory 306 to on-chip memory 105 and from on-chip memory 105 to memory 306. Data can also be transferred from memory 306 to high-bandwidth off-chip memory 107 and from off-chip memory 107 to memory 306. Data can also be transferred from scalar memory 334 to on-chip memory 105 and from on-chip memory 105 to scalar memory 334. As an example, the minimum granularity, source alignment, and destination alignment may be 4 bytes. As another example, a 32-byte access can be used to support a 4-byte access to off-chip memory 107. As yet another example, a 32-byte alignment and a minimum length of 128 bytes can ensure performance for streams to or from off-chip memory 107.

[0156] Each tile within the processor may implement its own Scatter-Gather Engine (SGE) to coordinate the movement of data from the tile to the scratchpad memory and / or between different types of memory. These different types of memory may include scratchpad memory, off-tile memory including high-bandwidth memory, and on-tile memory. The SGE can support multiple pending stream requests that may arise from one or both of the TEC and TAC of the tile implementing the SGE. Requests to send data can be processed as gather operations by SGE, while requests to write data can be processed as scatter operations. SGE can also be implemented in a tile sequencer to handle reads and writes to data streams between the sequencer and tiles and / or memory.

[0157] Figure 8 is an illustrative functional diagram of the Scatter-Gather Engine 1500. The SGE1500 can support various numbers of execution threads, for example, eight threads. Threads may be selected based on the stream identifier for an incoming request, the address generator thread, and the availability of stream types, such as high-bandwidth memory and / or scratchpad memory. In some examples, a stream request with an in-machine stream identifier in an address generator thread needs to be mapped to that same thread.

[0158] The SGE1500 can ensure ordering of specific requests of the same type (e.g., gather requests or scatter requests) across multiple streams that belong to the same stream and originate from the same tile core, targeting the same external interface (e.g., a processor crossbar or other interconnect). These requests can be identified as belonging to the same stream if they have the same stream identifier. Synchronization flags managed on the tile can be used to track ordering between requests, as described herein.

[0159] The SGE1500 receives scatter / gather requests, expands the requests, moves data on the processor interconnect to a remote Spmem slice or to the core memory network CMN in high-bandwidth memory, and sends synchronization flag updates to the tile synchronization flag memory of one of the cores, which is updated as the transaction progresses.

[0160] The SGE1500 also services DMA requests received from the CMN interface to perform writes to and reads from tile Spemem slices, and processes reads and writes made by SGEs of remote tiles targeting Spemem slices local to this tile.

[0161] A stream request can be one of several different types. A stream request may include scatter / gather requests from the core of a tile. These requests can be processed by a scatter / gather engine, which may itself be implemented on the tile and / or tile sequencer. One type of stream request is a linear request, in which case the SGE can expand the request into several smaller requests based on the length of the stream request. Another type of stream request is a strided request, in which the SGE expands the request into several requests based on the stride and length. Another type of stream request is an indirect request, in which the SGE expands a list of addresses of the same length, which are then expanded into separate requests. Another type of stream request is an indirect request in which the SGE receives a list of addresses.

[0162] The SGE1500 can implement several stages, such as the descriptor dispatch stage 1500A, the address generator stage 1500B, and the data transfer stage 1500C. Each stage will be described in turn.

[0163] SGE can interface directly with descriptor generators within each tile core, such as TAC and TEC. Descriptor generators are configured to enqueue stream descriptors generated by the core. Stream descriptors can be sent to the descriptor generator, where metadata related to the descriptor is enqueued in the descriptor publishing FIFO. The descriptor is written to the descriptor RAM. The contents of the FIFO may include a pointer in the descriptor RAM where the descriptor resides, the memory type, and the stream identifier attached to the corresponding stream. If valid stream descriptor metadata is available at the beginning of the TAC or TEC FIFO, the following sequence may occur.

[0164] Firstly, the SGE may perform resource checks on the metadata of each core. Next, assuming that both cores have the requested resources determined by the resource checks, the SGE may perform longest-unused arbitration to select between these two cores. If only one of the cores has the required resources, this will be the next core to be serviced. The stream identifier attached to the metadata is looked up in the address generator map, and the associated counter is incremented, or a new entry is created in the map to select an address generator thread at the address generator stage, which is the destination to which the descriptor should be sent. The metadata is then placed in the descriptor metadata queue associated with the selected thread. Resource checks may be performed in the descriptor FIFO associated with the descriptor metadata queue. Longest-unused arbitration selects one of the descriptor metadata queues. In this case, the winning entry is popped, and its metadata is transferred to the descriptor RAM request FIFO. The request is popped and sent to descriptor RAM, and the data response stored in the descriptor FIFO is transferred to the address generator stage.

[0165] The stream identifier / address generator map is used to determine the address generator threads in address generator stage 1500B that should send descriptor metadata, and to ensure that the ordering of subsequent stream requests belonging to the same stream is maintained. This structure can hold active stream identifiers, the address generator threads to which these identifiers are mapped, bits indicating that the mapped stream identifiers to the thread identifier entries are valid, and a count of the number of descriptors belonging to this stream in the pipeline from the queue of the stream FIFO.

[0166] The above structure can be sized to hold various maximum stream identifiers, e.g., 16. Each time a stream descriptor is issued, along with currently inactive stream identifiers, available space in the descriptor metadata queue, and / or available space in the stream identifier-address generator map, the SGE1500 can perform one or more actions. The SGE selects the next available thread, stores this value in the stream identifier-address generator map, and increments a counter associated with this thread by one. Any subsequent results for the same stream will now be sent to the same address generator thread, incrementing the counter.

[0167] If there is no space available in the descriptor metadata queue of that address generator thread for this request, or if there is no space to allocate the new entry in the stream identifier (if this is a new stream identifier) ​​to the address generator map, or if the counter in the map is nearly full, the FIFO will not be dequeued until space becomes available in all the necessary resources. This check is performed before arbitration between the two core FIFOs.

[0168] Address Generator Stage Between the two, 4 is associated with the CMN interface, and the other 2 are associated with the SC data crossbar. The CMN interface address generator thread deploys requests targeting HBM, while the SC data crossbar interface address generator thread targets remote Spemem or tile SpememN. To determine the next available thread to be stored along with the thread ID, the stream interface metadata determines which address generator thread to select, and LRU arbitration is used across the relevant threads, each having space in the corresponding descriptor metadata queue.

[0169] When a stream request is expanded by the address generator thread in the address generator stage and the synchronization flag tracking structure in the data transfer stage is updated, the counter in the stream identifier in the address generator map can be decremented. This decrement update to the counter from the data transfer stage originates from the following four parts of the data transfer pipeline: a stream scatter (xl) to the core memory network, a stream gather (xl) to the core memory network, a stream scatter (x2) to the processor data crossbar, and a stream gather (x2) to the processor data crossbar. This is done by the final request, which is to be expanded into a descriptor by the address generator thread.

[0170] When the counter associated with a stream identifier in the address generator map reaches zero, this map entry can be invalidated, and any subsequent requests to the same stream can be remapped to any of the available address generator threads for its interface (CMN / data crossbar), because this will ultimately lead to the same synchronization flag tracking structure that maintains the order between them.

[0171] For example, the stream ID is initially mapped to thread_iD#2 over a time window, provided the descriptor is still in the machine (up to the data transfer stage). An entry will be invalidated if no new descriptors are created with the same stream ID for a while (the associated counter becomes zero). If a new descriptor is created with the same stream ID after a mapped entry has been invalidated, the new map created may have the stream ID mapping thread_id #1, but this is unlikely to be a problem as space has already been allocated within the synchronization flag tracking structure in the data transfer stage, which is responsible for maintaining ordering. Note that this assumes that thread_id #1 and thread_id #2 belong to the same stream interface type (CMN / data crossbar) and are tracking updates to the same memory (scatter vs. gather).

[0172] All information from the stream FIFO does not need to be enqueued in the descriptor metadata queue, as some of it is used only in the initial part of the descriptor dispatch stage 1500A. Only the address to the descriptor ram is needed in the later part of the stage to send a read to the descriptor ram. All other metadata carried in the FIFO is used only to map the descriptor to a specific address generator. After this point, this metadata can be discarded. The descriptor metadata queue carries only pointers to the descriptor ram.

[0173] In the output of the descriptor metadata queue, LRU arbitration exists across all address generator threads to ensure fairness in obtaining access to descriptor RAM.

[0174] Referring to address generator stage 1500B, descriptors from the descriptor FIFO are popped by the stream descriptor manager logic within the address generator stage and then passed to the address expansion state machine for further expansion into requests to remote memory. Each subblock of the address generator is described in more detail below.

[0175] The stream descriptor manager logic is a state machine that expands the address list of an indirect stream before passing the descriptor to the address expansion state machine. While this specification describes the logic in the form of a state machine, it should be noted that actual implementations do not label states in this way. This is done for the sake of simplifying the code structure. However, the actual functionality performed by the logic remains the same as described herein. It performs the following tasks in each state:

[0176] Idle state: In this state, logic pops from the descriptor data FIFO and checks the descriptor's off_tile_stream_type field. Expanded address list state: This state is reached only if there are more than 4x addresses to expand to the address list, for example, in an indirect stream.

[0177] The address expansion state machine is used to expand each stream descriptor presented to it into one or more read / write requests to the core memory network interface (directed to the HBM) or the data crossbar interface (directed to a remote Spmem).

[0178] The address expansion state machine must also indicate to the data transfer stage the last request expanded from each stream descriptor, which is either a stream scatter or gather. This information is passed from the stream descriptor manager logic to the address expansion state machine as part of the descriptor metadata by indicating the last entry in the address expansion input FIFO associated with a particular stream descriptor. This information is required by the data transfer stage to track the descriptor "committed" and "retired" states for fence instructions.

[0179] In the data transfer stage 1500C, the SGE1500 handles the formatting of the output stream's scatter and gather to meet the interface requirements of the CMN and data crossbar. It manages DMA access to Spmem and synchronization flag tracking for the scatter and gather of the raw stream. This pipeline stage also handles incoming read and write access from remote tiles.

[0180] The SGE1500 increments transaction counters to track the status of fence instructions based on the source core ID and target remote memory. For fences, for example, there may be six counters per core type [total 12] - Spmem (retired write, committed write, retired read), HBM (retired write, committed write, retired read). Each transaction that wins arbitration to the data transfer stage will increment both the retired and committed counters for the associated memory and core type.

[0181] The SGE1500 sends a decrement to the descriptor dispatch stage for any existing fence descriptor counters if this is the last transaction associated with the descriptor being deployed. While this could also be done for the first transfer associated with the descriptor, the "last transfer" status is already available, so if the decrement is based on the last transfer, no additional information needs to be tracked. This also ensures that incorrect statuses are not provided to the core complex. The decrement is then performed at a granularity for the next interface transfer.

[0182] The SGE1500 sends a decrement to the descriptor dispatch stage regarding the synchronization flag ID to the address generator map if this is the last transaction associated with the descriptor being deployed.

[0183] Since the committed descriptor state for a stream gatherer is the same as the retired descriptor state, a stream gatherer only needs to be tracked by one type of state counter. In the case of a stream scatter, the two counters will be updated at different points in the flow. Also, the granularity of updates in the pipeline that updates the synchronization flag may differ from the granularity at which the request is tracked down to remote memory, so logic needs to maintain state in these two separate parts of the data transfer stage.

[0184] The data transfer stage 1500C maintains a counter for each source core per remote memory type, which is incremented by each instance of the synchronization flag tracking logic. The counter is incremented when a synchronization flag message is enqueued to the message router interface by the synchronization flag tracker. Once the synchronization flag update sent to the message router is complete, the data transfer stage decrements these counters, associating the remote memory and source core with the transaction to which the update belongs and which is sent back to the SGE.

[0185] Each synchronization flag tracker also maintains a status (per source core per memory type) indicating whether there are any running transactions being tracked by it.

[0186] As long as there are "running" entries in the synchronous flag tracker for stream scatters to a particular interface, all descriptors for that memory type are neither "committed" nor "retired." Note that if the stream scatter "retired" counter for a particular memory type is 0, but the associated synchronous flag tracker still has "running" transactions, the fence status for that memory will still indicate that not all are "committed" or "retired."

[0187] In the case of stream gatherers, even if all "retired" counters have been decremented, the current status for the descriptor dispatch stage cannot be set to "committed" or "retired" as long as there are "running" transactions in the sync flag tracker. The status can be updated to the descriptor dispatch stage when the sync flag tracker for a particular memory location reports that nothing is being tracked there.

[0188] The descriptor tracking logic in data transfer stage 1500C sets the running fence status to descriptor dispatch stage 1500A.

[0189] As described herein, the SGE1500 may be implemented as part of a tile sequencer and in each tile. The scatter-gather engine within the tile sequencer would be a parameterized version of the SGE subsystem described above, with some differences detailed in this paragraph. In the case of a tile sequencer, the "local" memory is always shared memory and has lower read / write bandwidth requirements compared to scratchpad memory.

[0190] A stream descriptor is a data structure that can represent all the information necessary for the scatter-gather engine 322 to perform stream transfers. A stream instruction is a stream It is possible to fully encode fields for stream descriptors. Below is an example of fields for stream descriptors.

[0191] In stream arithmetic code, a gather stream reads from off-tile memory and stores the data in tile-local memory, or appends data to tile-local memory. A scatter stream reads from tile-local memory and stores the data in off-tile memory, or appends data to off-tile memory. Off-tile memory and tile-local memory are determined by fields such as the off-tile memory type and tile-local memory type, respectively.

[0192] Additional variations of stream instructions can support both floating-point and signed integer addition. Gathering of signed integer additions and gathering of floating-point addition variations can be supported for tile-local memory. Variations of scatter signed integer additions and scatter floating-point additions can be supported for off-tile memory and tile-local memory. If an invalid combination is detected, the engine may present a program error.

[0193] The tile-local stream type indicates the address pattern used to access tile-local memory. For example, a linear stream facilitates several consecutive words starting from a tile-local start offset. The number of consecutive words can be 4 bytes long. As another example, a circular buffer stream allows software to construct a logical circular buffer within tile-local memory. In this exemplary access pattern, the base field, size field, and offset field in the circular buffer metadata are used to generate addresses for several words. The number of words can be 4 bytes long. A program error may be presented if the effective length of the granularity is greater than the size field in the circular buffer metadata.

[0194] The off-tile stream type indicates the address pattern used to access off-tile memory. Linear streams facilitate access to several consecutive locations starting from the off-tile start offset. The actual word size depends on the off-tile memory type. Strided streams facilitate the conversion of strided access patterns into multidimensional arrays stored in off-tile memory types. Stream transfers can support a single level of stride. Indirect streams allow random scatter-gather access patterns to a table. Here, an indirect offset list is used, where each entry in the list accesses data of the same length.

[0195] The source of the indirect offset list can be tile-local memory or a register file. If the source is tile-local memory, the indirect offset field has a starting offset to the tile-local memory where several offsets are stored. If the source is a register file, the indirect offset field has the register file and indicates several lanes containing valid offsets. These offsets are used to perform scatter or gather operations, as indicated by the stream operation code.

[0196] The core type indicates the type of core in tile 102 that generated the stream descriptor, such as the tile executor core 330 or the tile access core 332. The synchronization flag core type indicates the core type of the synchronization flag 318 that tracks the progress of the stream. Encoding may be the same as the core type that enables the stream to be started by the tile access core tracked by the tile access core, and enables the stream to be started by the tile executor core tracked by the tile access core.

[0197] The synchronization flag ID indicates the offset in the target synchronization flag memory. The synchronization flag ID can also be used as the stream ID to ensure ordering, as will be explained further below. The set done bit indicates that the current descriptor is the last in the stream. The done bit is set after all data for the current descriptor and preceding descriptors in the stream has been fully committed to tile local memory.

[0198] The synchronization flag count type indicates the type of count that synchronization flag 318 is tracking, whether it be the number of words or the number of descriptors. In either case, synchronization flag 318 tracks a monotonous, stepwise progression of the stream at different levels of granularity.

[0199] The tile local memory type indicates the type of tile local memory involved in stream transfer and may include local banks of scalar memory 334 or memory 306.

[0200] The tile-local start offset field is used when the tile-local stream type is linear. It indicates the aligned start offset word, such as a 4-byte word, in the tile-local memory accessed by this transfer. The actual access type depends on the stream arithmetic code.

[0201] The tile-local stride encodes the stride size and number of bytes accessed per stride, which is used to access tile-local memory selected by the tile-local memory type. The length, which can be 4 bytes as an example, does not need to be a multiple of the number of bytes accessed per stride. The last request for a strided access will access the remaining transfer word, which may be less than the length per stride. The stride calculation can be the same for both linear and cyclic buffer stream types.

[0202] If the tile-local stream type is a circular buffer, the circular buffer metadata field is used. The size of the circular buffer can be a multiple of the granularity of the off-tile memory type, and the offset can be aligned. If the circular buffer wraps around, the request is split into multiple requests, and an error may be presented if the resulting requests are not a multiple of the granularity of the off-tile memory type. An error may also be presented if the total length of the stream transfer is greater than the size of the circular buffer.

[0203] The off-tile memory type indicates the type of off-tile memory involved in the transfer. This includes on-chip memory 105 and high-bandwidth memory 107. A high-bandwidth memory view is also available, enabling access at 4-byte granularity and 4-byte alignment. If the sequencer 106 is the initiator of the stream transfer, this field may not have an encoded high-bandwidth memory 107.

[0204] The tile ID field can be used to select a tile ID for a memory slice. The off-tile start offset contains the start offset word in the off-time memory 105, as indicated by the associated off-tile memory type. The units of the offset may be equal to the value indicated in the offset alignment column within the off-tile memory type. For example, for high-bandwidth memory 107, an offset value of 1 would translate to byte address 32. If the off-tile stream type is indirect, this field may function as a base address to be added to the offset read from the indirect offset list before accessing the memory.

[0205] Indirect offsets can be used when the off-tile stream type is indirect. If the source is tile-local memory, the indirect offset provides the word start offset in tile-local memory that stores the indirect offset list. If the source is a register file, the indirect offset provides the file register index from which the indirect offset list originates. The register file can be read when a stream instruction is issued.

[0206] If the off-tile stream type is indirect, the indirect list size can be used. If the source is tile-local memory, the number of elements in the offset list is stored in tile-local memory. If the source is a register file, the number of lanes containing valid offsets is stored. Transfer completion is maintained in order with the remaining descriptors in the stream.

[0207] The indirect list type can be used when the off-tile stream type is indirect and indicates the type of offset stored in the offset list. This can include word offsets and row offsets. The indirect list stride is used when the off-tile stream type is indirect and indicates the distance between two address words in the offset list stored in tile-local memory. This can be a signed integer.

[0208] The indirect filter field is used when the off-tile stream type is indirect. When this field is set, indirect memory addresses matching the indirect filter value are filtered out. The indirect filter value indicates the value of the element in the indirect access list that needs to be filtered out. This value is of the type indicated by the indirect list type. Filtering can be enabled when the indirect filter field is set for an indirect stream, and / or when the value of an element in the indirect offset list matches this field. Off-tile and tile-local accesses corresponding to the filtered elements are dropped, but the tile-local buffer will still advance by the size of the filtered access.

[0209] For example, a length that is a multiple of 4 bytes or 512 bytes indicates the total number of words accessed by the stream. If the offtile stream type is linear or strided, this field indicates the total number of words accessed by the stream. If the offtile stream type is indirect, this field indicates the number of words accessed from each address in the indirect offset list. A program error may also be presented if the actual value of this field is not a multiple of the granularity of the offtile memory type. A program error may also be presented if the generated address exceeds the boundary of offtile memory 105.

[0210] The stride size field indicates the stride size in units of granularity for the off-tile memory type. This can be a signed integer. The stride length, such as a stride length that is a multiple of 4 bytes or 512 bytes, indicates the number of words accessed per stride. This is a signed field, but should contain a non-negative value. The length does not need to be a multiple of this field. This field should be a multiple of the granularity for the off-tile memory type selected by this stream descriptor. The last request for a strided access will access any remaining transferred words, which may be less than the stride length. A program error may be presented if the stride length is 0, negative, or not a multiple of the off-tile memory access granularity. A program error may also be presented if the generated address exceeds the boundary of off-tile memory 105.

[0211] The trace field indicates whether the stream transfer should be traced. As part of debugging, the trace may include login information about actions taken during the stream transfer.

[0212] Figure 9 is a flowchart of an exemplary process 1600 for constructing an off-tile stream request or a tile-local stream request from a stream descriptor. The exemplary process 1600 may run on a system of one or more processors at one or more locations. For example, hardware circuit 101 can execute process 1600 as described above.

[0213] As shown in block 1610, the process includes a process that receives the size of the off-tile memory, such as receiving the size as 4 bytes. Furthermore, the process includes a process that receives the maximum chunk size of the stream request targeting the off-tile memory type. As shown in block 1620, the process further includes a process that translates the indirect offset, read from a file register or tile local memory to an offset such as a 4-byte offset, to the off-tile memory, based on the indirect list type.

[0214] As shown in block 1630, the process also includes a process for generating strided and / or indirect requests. In the case of a strided request, the process may include a process for partially expanding a strided stream descriptor into a set of requests, each accessing a contiguous address in off-tile memory. In the case of an indirect tile local memory request, the process may include a process for taking an indirect stream descriptor and generating a list of offsets to the off-tile memory type selected by the descriptor. In the case of an indirect file register memory request, the process may include a process for generating a list of offsets from a file register read when the indirect stream instruction is issued.

[0215] As shown in block 1640, the process includes a process to generate a list of expanded off-tile memory requests, in which case each expanded request accesses a set of contiguous addresses in off-tile memory. These requests are used to generate both tile-local memory requests and off-tile memory requests. The tile-local stride, tile-local stream type, and alignment are considered while expanding the requests. The process further includes a process to generate a list of partially expanded requests, in which case each partially expanded request accesses a set of contiguous addresses in off-tile memory. These requests are further expanded to generate a set of requests aligned to the memory granularity selected by the off-tile memory type.

[0216] As shown in block 1650, the process includes a process that expands the stream descriptor into a set of off-tile memory requests and tile-local memory requests.

[0217] Figure 10 is a flowchart of an exemplary process 1700 for ordering stream transfers. The exemplary process 1700 may run on a system of one or more processors at one or more locations. For example, hardware circuit 101 can run process 1700 as described above. Discrete stream instructions issued by cores having the same stream ID form a single stream, but ordering may not be guaranteed between separate streams. The scatter-gather engine 1722 includes multiple threads that can process these requests in parallel. Ordering can be guaranteed for transfers within a single stream, which may span multiple stream instructions.

[0218] As shown in block 1710, stream instructions belonging to a stream are processed in order. The requests corresponding to them are processed in order by the scatter-gather engine 322. It will probably be published soon.

[0219] As shown in block 1720, in the case of indirect stream instructions, the offset list is ordered. The offset elements in the offset list are processed in order. Writes are issued to destination memory in order, but writes may be committed in no particular order by the destination memory. Reads are issued to source memory in order, but reads may be served in no particular order by the source memory.

[0220] As shown in block 1730, the scatter-gather engine 322 updates the synchronization flag 318 to indicate monotonous, stepwise progress over the stream. If the tile local memory is the source, the synchronization flag 318 tracks reads from it. The synchronization flag value indicates the first chunk of data that may be overwritten in the tile local memory. If the tile local memory is the destination, the synchronization flag 318 tracks writes to it. Here, the synchronization flag value indicates that subsequent reads to the first chunk of data in the tile local memory will return to the requested data.

[0221] As shown in block 1740, the done bit in the synchronization flag 318 can be updated at the end of the stream. This is indicated by the set done bit in the stream descriptor. The done bit can be set after all data for requests preceding the last stream descriptor and requests containing the last stream descriptor have been fully committed to memory. If tile local memory is the source, all reads are complete, and if tile local memory is the destination, all writes are committed.

[0222] Figure 11 is an illustrative diagram of stream ordering. Consider the case where stream descriptors A and B constitute a single stream. Stream descriptor B has a set done bit set. Partial progress of the stream is tracked by the synchronization flag. When A0 is committed to memory, either by reading or writing, the synchronization flag is updated to a value of 1. Even if A2 and B1 are committed before A0, the synchronization flag value is not updated to 3. When A1 is committed to memory, five consecutive chunks of data in the stream A0, A1, A2, B0, B1 are committed, as indicated by the synchronization flag value of 5. The done bit is not set at this point because stream descriptor A is not the end of the stream. When B2 is committed, the synchronization flag value is set to 6. Here, the done bit can be set when all data chunks of the stream have been committed and stream descriptor B is the end of the stream.

[0223] Cooperative prefetching The disclosed technology provides an instruction prefetch pipeline architecture where sparse accelerator tiles are available, which delivers good performance without the complexity of the full cache coherent solutions deployed in conventional CPUs.

[0224] The disclosed aspects of the technology provide methods and systems related to creating a prefetch pipeline around the SPMD aspect of the programming model to reduce cold cache miss overhead. Prefetch responses from any core are broadcast to all cores in the sparse accelerator. These prefetch responses are committed to the core's local cache. This allows other non-requesting cores to obtain bundles of instructions or data before the time the core would become available to process the instructions or data, completely avoiding missing process cycles. In addition, there may be prefetch request filtering on the arbitration path, which is a logical and / or hardware-based path for arbitrating between requests, bringing task instruction memory into play, thereby avoiding redundant request fetches. The memory bandwidth of the instruction is boosted.

[0225] The task instruction memory holds a set of programs that can be executed by the Tile Access Core (TAC) and the Tile Execute Core (TEC). This is possible. The program counter (PC) within each core is used for tasks. This is a physical offset to instruction memory. Task instruction memory is software-managed memory exposed to the direct memory access system. Software can use direct memory access to populate programs in task instruction memory and use appropriate program counters while issuing tasks to tiles. Since tiles operate in a single-program multiple-data mode, statistically, most tiles may be running the same program at any point in the execution of the sparse accelerator. These programs may further consist of one instruction loop or a compact instruction loop. A compact instruction loop can refer to a size of instructions in memory small enough to fit into tile memory. The program itself may be small in size, for example, a bundle of several hundred instructions, and the program may have multiple branches that can branch within a loop or branch into other loops.

[0226] These and other potential features can be utilized by instruction pipelines as described herein, for example with reference to Figures 12 and 13. When an instruction bundle is received by a sparse accelerator, the sparse accelerator is configured to broadcast a prefetch response for the received instruction to all tiles within the sparse accelerator. The prefetch response is committed to each core's local cache, allowing unrequested tiles to obtain the bundle in advance and completely avoiding misses. In addition, prefetch request filtering on the arbitration path leading to task instruction memory can be implemented in some examples to boost task instruction memory bandwidth by avoiding redundant request fetches.

[0227] Figure 12 shows a logical diagram of the connections between tiles 1901 and 1902 of an exemplary sparse accelerator, according to the aspects of the disclosed technology. For clarity, not all components, modules, or software blocks in Figure 12 are referenced. In general, as will become clear from the following description, instructions can be delivered more quickly to processing units or processing cores by prefetching instructions, aggregating requests for instructions, and filtering instructions or references to hold them in the memory location closest to the request processing unit, thereby increasing system efficiency. Additional aspects of the components related to Figure 12 are described further below in relation to Figure 13.

[0228] In the schematic diagram, Figure 12 shows the task instruction memory (Timem bank or Timem bank), instruction buffer ("iBuf"), prefetch unit, and instruction router. Figure 12 shows tile 1901, which includes tile access core (TAC) 1910 and tile execution core (TEC) 1920. TAC 1910 may include prefetch 1911 and iBuf 1912. Similarly, TEC may include prefetch unit 1921 and iBuf 1922. Figure 12 also shows tile core 1902, which includes TAC 1930 and TEC 1940, which include prefetch 1911 and iBuf 1932, and prefetch unit 1941 and iBuf 342, respectively.

[0229] Figure 12 further shows Timem1951 and Timem1952, as well as instruction router 1960, which may be logically or physically contained within floorplan block 1999. Timem1951 and Timem1952 can locally store instructions for faster access by each tile core compared to tile cores that request instructions from locations further downstream from Timem. Also, floorplan block Instruction broadcast buses are also shown that can broadcast instruction bundles downstream to instruction bank 1999 and its Timem bank. Instruction request bus 1992 can aggregate requests for instructions from various components before those instructions are requested. Deserializers and serializers can deserialize or serialize instructions for sending them along various buses, for example, to receive them from instruction broadcast bus 1991, or to serialize instructions sent to instruction request bus 1992.

[0230] Prefetch units such as prefetch unit 1911 or prefetch unit 1912 corresponding to the core include a misprogram counter (program counter: PC) (approximately Read requests to Timem can be made starting from the overlay / task ID and continuing until the end of the prefetch window. The prefetch window is a software-selectable period that has registers or other memory areas. For example, the prefetch window may be defined by a prefetch depth variable. Prefetch read requests from other tiles can be forwarded by adjacent floorplan block 1999. These forwarded requests can be arbitrated by prefetch requests made by prefetch units in adjacent tile cores. For example, tiles 1901 and 1902 may be adjacent to each other. In some examples, a pair of cores can be assigned to a single instruction request bus or a single instruction broadcast bus.

[0231] Several prefetch instruction request banks may exist within a tile. In some examples, there may be one bus per Timem bank, but these can be arbitrated independently of each other. Independent arbitration of buses can allow for the avoidance of head-of-line blocking across independent banks.

[0232] Requests sent from the prefetch window can be received by instruction router 1960. Instruction router 1960 can filter selected requests to remove duplicates before forwarding them to another instruction router or target Timem bank. Filtering can potentially increase instruction request bandwidth when the core is operating in SPMD mode.

[0233] Instructions read from the Timem bank can be broadcast to all tiles on the instruction broadcast bus. For example, there may be as many instruction broadcast buses as there are Timem banks. In some examples, instructions can be sent as instruction bundles. An instruction group consists of the instructions contained in a bundle. A bundle can be a sequence of instructions that start on an aligned "boundary". An instruction bundle can be serialized on the corresponding instruction broadcast bus over a certain number of cycles of the processor or core. In some examples, during the "steady state" operation of the system, the total bandwidth of the instruction broadcast bus may be two bundles per cycle. In such a configuration, the instruction broadcast bus 1992 is never backpressurized.

[0234] Instructions received on the broadcast bus can be deserialized by the instruction router, and one instruction is forwarded to each iBuf. In a steady state, the system may be required to maintain up to two writes from the prefetch interface and one read from the instruction fetch interface. Prefetch processes incoming instructions and determines whether they should be committed to an iBuf or dropped.

[0235] Figure 13 shows an additional exemplary aspect of the instruction router 1960. Figure 13 shows a round-robin (RR) arbiter 1910 and a daisy-chain round-robin arbiter. -Bitter 1920, Round Robin Arbiter 1930, Filter 1940, Serially The 1950s and 1951, Demultiplexer (demux) 1960 Figure 13 also shows deserializers 1971 and 1972. Other aspects and components that are not labeled for simplification are also shown in Figure 13.

[0236] The instruction router 1960 may have an independent read request bus for each Timem bank in the system. Router 1960 may adjust and throttle instruction bundles to match the bandwidth of the instruction broadcast bus before forwarding them to neighboring instruction routers. In the following description, it can be assumed that deserialization and serialization are possible before requests are presented to instruction router 1960.

[0237] The instruction router 1960 can arbitrate according to the core's location relative to the Timem bank. The instruction router 1960 can be parameterized to select source and destination based on the instance it is arbitrating. The demultiplexer 1960 shown in Figure 13 can be designed according to the number of time banks or serializers it is communicating with.

[0238] The instruction router 1860 can arbitrate between the following exemplary sources, including prefetch reads forwarded by instruction routers upstream or above the instruction router 1860, prefetch reads forwarded by instruction routers downstream of the instruction router 1860, and prefetch reads originating from cores connected to the instruction router 1860.

[0239] demux(select(select) is a design parameter) selects either top_pre_req or bottom_pre_req to arbitrate requests originating from cores connected to the instruction router. Arbitration uses a daisy-chain round-robin arbitration scheme. The daisy-chain round-robin arbiter 1920 can grant permissions every "x" cycles to match the bandwidth of the instruction broadcast bus. If a PC matches another PC found on the instruction broadcast bus, the request waiting to be arbitrated can be dropped. This can be considered a first level of filtering.

[0240] The winner of a daisy-chain arbitration can be processed in various ways based on the position of the instruction router 1860 relative to the Timem bank. For example, if the Timem bank is below the instruction router, the winner of the daisy-chain arbitration may be forwarded to the "lowest" instruction router after passing through filter 1940. If the Timem bank is above instruction router 1860, the winner of the daisy-chain arbitration will be forwarded to the top instruction router 1860 after passing through filter 1940.

[0241] If the Timem bank is located within instruction router 1860, the winner of the daisy-chain arbitration will undergo one or more levels of arbitration on requests forwarded by the lowest instruction router. In this case, there may be two daisy-chain networks arbitrating to reach the Timem bank. Depending on the location of instruction router 1860, the chains may not be balanced. A modified RR arbiter can be used to ensure that fair access is provided to the cores on both sides of the chain. Similar to the first level of arbitration, any requests matching a PC on the broadcast bus will be dropped here. This can be considered a second level of filtering.

[0242] The overall winner of the above is passed to filter 440, which compares the incoming request with one of the other pending requests. If the request matches any of the pending requests, the request is dropped. This can be considered a third level of filtering.

[0243] Furthermore, the programmability of this system can be ensured by the fact that filtering at each point in time can be enabled / disabled by individual programmable or software-controllable switches. The Timem access bus may be a bus that connects the system to all Timem banks, allowing them to read and write instruction bundles to and from the Timem banks. The Timem access bus may have four buses, as will be further described below: a read request bus, a read response bus, a write request bus, and a write response bus.

[0244] The read request bus can be a daisy-chain bus that can reach Timem banks. Each Timem bank can forward a request to an adjacent Timem bank if the request does not address it. If the request does address a Timem bank, the request is handled by that Timem bank.

[0245] The read-response bus may be a daisy-chain bus capable of transmitting instruction bundles read from Timem banks. In each Timem bank, round-robin arbitration may exist between incoming instructions from neighboring banks and instruction bundles from the current bank. Since the instruction bundles are serialized over "n" cycles, bus permission is maintained over "n" cycles.

[0246] The write request bus can be a daisy-chain bus that can execute up to the Timem bank. A write request can be serialized, for example, over two cycles. Each Timem bank forwards the frit to the adjacent bank if the request is not addressed. If the request addresses a Timem bank, the request is deserialized by the bank before being written to the Timem bank.

[0247] The write response bus may be a daisy-chain bus that relays write responses from Timem banks. In each Timem bank, arbitration exists between the incoming response and the response from the current bank. Simple round-robin arbitration may be used to allow or provide one of the responses.

[0248] Read and write requests are limited to a maximum of 2 ∧ It may have a "q" bit tag for encoding unprocessed read and write requests for q. These unprocessed read and write requests are returned in the bank's response and may be used by the overall system or component to provide instructions for identifying the requests corresponding to the response.

[0249] If an endpoint is unable to accept a request or response, the bus can be "backpressurized," and a backlog accumulates of instructions or data that should be sent through the bus if the bus is unable to transfer them. In addition, the bus can be backpressured due to arbitration losses. This may be acceptable system-wide because Timem access is generally low-bandwidth access.

[0250] The tile instruction memory (Timem) can be shared by the tile core, as explained in Figure 12.

[0251] The aspects of this disclosure may be implemented in digital circuits, computer-readable storage media, as one or more computer programs, or as one or more combinations thereof. The computer-readable storage media may be non-temporary, for example, as one or more instructions that are executable by a cloud computing platform and stored on a tangible storage device.

[0252] Calculation overview In this specification, the phrase “configured to perform” is used in various contexts relating to computer systems, hardware, or parts of computer programs, engines, or modules. When a system is described as being configured to perform one or more operations, it means that the system has appropriate software, firmware, and / or hardware installed on the system that causes the system to perform one or more operations while it is running. When some hardware is described as being configured to perform one or more operations, it means that the hardware includes one or more circuits that, while running, receive inputs and produce outputs corresponding to one or more operations according to the inputs. When a computer program, engine, or module is described as being configured to perform one or more operations, it means that the computer program includes one or more program instructions that, when executed by one or more computers, cause one or more computers to perform one or more operations.

[0253] The operations shown in the accompanying drawings and described in the accompanying claims are shown in a specific order, but it should be understood that these operations may be performed in an order different from the illustrated order, and that some operations may be omitted, performed multiple times, and / or performed in parallel with other operations. Furthermore, the separation of various system components configured to perform various operations should not be understood as requiring the separation of components. The components, modules, programs, and engines described may be integrated together as a single system or may be part of multiple systems.

[0254] Wherever substantially any plural and / or singular term is used herein (the term “element” is a substitute for any system, component, data, etc.), for example, “one / the element,” “one or more elements,” “compound element,” “multiple elements,” “at least one element,” etc., a person skilled in the art can convert from plural to singular and / or singular to plural as appropriate to the context and / or use described herein. Various singular / plural substitutions may be expressly described herein without limitation for clarity unless otherwise explicitly specified.

[0255] The aspect of this disclosure includes methods, systems, and apparatus that employ an instruction prefetch pipeline architecture that provides good performance without complicating the full cache coherent solutions deployed in conventional CPUs.

[0256] The disclosed aspects of the technology relate to components that may be used to construct an instruction prefetch pipeline, including an instruction memory (TiMem), an instruction buffer (iBuf), a prefetch unit, and an instruction router.

[0257] Aspects of this disclosure may relate to certain characteristics that may exist in conjunction with the expected or known behavior of XPU tiles, such as: tiles may be expected to operate in Single Program Multiple Data (SPMD) mode; statistically, most tiles may be expected to be running the same program at any given time; programs may consist of one or more compact loops; program sizes may be small, for example, hundreds of bundles; or programs may have multiple branches that can branch out. The disclosed technology can leverage these or related characteristics and is less complex than a full cache-based solution.

[0258] The disclosed aspects of the technology include hardware circuits. The hardware circuits may include multiple tiles. Each tile is configured to operate in parallel with other tiles within the multiple tiles, and each tile in the multiple tiles includes a processing core, a prefetch unit, and an instruction buffer. The hardware circuits further include multiple data processing lanes configured to stream their respective data from an upstream input to a downstream destination, and multiple task instruction memories. Each task instruction memory is arranged in sequence and coupled to one or more tiles in the multiple tiles via an instruction router. Task instruction memories may be arranged in downstream sequences. Each tile may include a tile access core, and the prefetch unit contained within each tile may be contained within the tile access core. Each tile may include a tile execution core, and the prefetch unit contained within each tile may be contained within the tile execution core. The hardware circuits may include an instruction broadcast bus and an instruction request bus. The instruction broadcast bus may include independent data lanes. The number of independent data lanes may correspond to the number of task instruction memories.

[0259] The instruction request bus may include independent data lanes. The number of independent data lanes corresponds to the number of task instruction memories. Instructions received by a task instruction memory may be broadcast to all linked tiles on the instruction broadcast bus. Prefetch may be configured to make requests to at least one task instruction memory during a prefetch window. The prefetch window may be selectable or adjustable by software. The hardware circuitry may further include an instruction router. The instruction router may include a round-robin arbiter configured to arbitrate requests, including prefetch read requests. Instruction buffers can store instructions for tile access cores or tile execution cores. The hardware circuitry may be configured as a single-instruction multiple data processor. The hardware circuitry may be configured as a multiple-instruction multiple data processor. The hardware circuitry may include a task instruction memory access bus. The task instruction memory access bus may include a read request bus, a read response bus, a write request bus, and a write response bus.

[0260] The disclosed aspects of the technology include a TPU. The TPU may include hardware circuitry and an instruction broadcast bus coupled to the hardware circuitry. The instruction broadcast bus is configured to push instructions to the hardware circuitry. The hardware circuitry may include multiple tiles, each tile may be configured to operate in parallel with other tiles in the multiple tiles, and each tile in the multiple tiles may include a processing core, a prefetch unit, and an instruction buffer. The hardware circuitry may further include multiple data processing lanes configured to stream their respective data from an upstream input to a downstream destination, and multiple task instruction memories. Each task instruction memory in the multiple task instruction memories is arranged in sequence and coupled to one or more tiles in the multiple tiles via an instruction router. The TPU may further include an instruction request bus coupled to the hardware circuitry, the instruction request bus may be configured to receive requests for instructions.

[0261] Aspects of the disclosed technology include a method for prefetching or providing instructions by a single instruction multiple data (SIMD) processing unit. The method may include: receiving requests for instructions from multiple tiles of the SIMD processing unit; filtering the requests for instructions to deduplicate requests for the same instruction to generate a first set of requests; generating a set of instructions in response to the first set of requests; providing the set of instructions from a compute unit to the task instruction memory of the SIMD processing unit; storing the set of instructions in the task instruction memory; and accessing instructions from the set of instructions by a prefetch unit via an instruction router. The SIMD processing unit may include multiple tiles. Each tile is configured to operate in parallel with other tiles in the multiple tiles, and each tile in the multiple tiles is: It includes a processing core, a prefetch unit, and an instruction buffer. The receiving step may occur in a first processing clock cycle, and the providing step may occur in a second processing clock cycle. The first processing clock cycle may occur before the second processing clock cycle.

[0262] In general, this specification discloses a hardware / software interface and a stream ordering model for asynchronous data movement between off-core memory and core-local memory, referred to as “stream transfer.” Stream transfer enables software to more efficiently represent common data movement patterns, specifically those observed in sparse workloads. Stream instructions belonging to a stream are processed in order. In the case of indirect stream instructions, the offset elements in the offset list are processed in order. A synchronization flag is updated to indicate monotonous, stepwise progress over the stream.

[0263] A method provided by one aspect of the present disclosure comprises the steps of identifying, by one or more processors, the progress of data being transferred between off-core memory and core local memory, and identifying, by the one or more processors, reads from the core local memory when the core local memory is the source of the data, wherein the reads are issued to the source in order and responded to out-of-order by the source, the method further comprising identifying, by the one or more processors, writes to the core local memory when the core local memory is the destination for the data, wherein the writes are issued to the destination in order and committed out-of-order by the destination, and the method further comprising accessing, by the one or more processors, the off-core memory based on indirect scatter / gather memory accesses for reads from the off-core memory when the off-core memory is the source of the data and writes to the off-core memory when the off-core memory is the destination for the data.

[0264] In one example, the step of identifying the progress of the data being transferred further comprises using a core local synchronization flag. In another example, the method further comprises the step of selecting, by the one or more processors, a memory access to a barrier based on a scalar fence instruction. In yet another example, the step of accessing the off-core memory based on indirect scatter / gather memory accesses further comprises obtaining the originator of an indirect address from a register file or the core local memory. In a further example, the method further comprises the step of performing circular buffering by the one or more processors within the core local memory.

[0265] In yet another example, the method further comprises updating a core local synchronization flag at the one or more processors to indicate a monotonic stepwise progress of the data transfer. In yet another example, the method further comprises terminating the data transfer at the one or more processors when all reads from the core local memory have been issued. In yet another example, the method further comprises terminating the data transfer at the one or more processors when all writes to the core local memory have been committed.

[0266] Another aspect of the present disclosure provides a system comprising one or more processors and one or more storage devices coupled to the one or more processors and storing instructions. The instructions, when executed by the one or more processors, cause the one or more processors to perform operations for transferring data between off-core memory and core local memory. The operations include an operation of identifying the progress of data being transferred between the off-core memory and the core local memory, and an operation of identifying reads from the core local memory when the core local memory is the source of the data, wherein the reads are issued to the source in order and responded to by the source out of order, and the operations further comprise, when the core local memory is the destination of the data, an operation of identifying writes to the core local memory, wherein the writes are issued to the destination in order and committed by the destination out of order, and the operations further comprise accessing the off-core memory based on indirect scatter / gather memory accesses for reads from the off-core memory when the off-core memory is the source of the data and for writes to the off-core memory when the off-core memory is the destination for the data.

[0267] In one example, the operation to identify the progress of data being transferred further includes the use of a core-local synchronization flag. In another example, the operation further includes selecting memory access to a barrier based on a scalar fence instruction. In yet another example, the operation to access off-core memory based on indirect scatter / gather memory access further includes obtaining the source of the indirect address from a register file or core-local memory. In yet another example, the operation further includes circular buffering within core-local memory.

[0268] In yet another example, the operation further includes updating the core-local synchronization flag to indicate a monotonous, stepwise progression of the data transfer. In yet another example, the operation further includes terminating the data transfer when all reads from core-local memory have been issued. In yet another example, the operation further includes terminating the data transfer when all writes to core-local memory have been committed.

[0269] A further aspect of the present disclosure provides a non-temporary computer-readable storage medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for transferring data between off-core memory and core-local memory. The operations include operations that identify the progress of data being transferred between off-core memory and core-local memory, and operations that identify reads from core-local memory when core-local memory is the source of data, the reads being issued sequentially to the source and responded to in no particular order by the source, and the operations further include operations that identify writes to core-local memory when core-local memory is the destination for data, the writes being issued sequentially to the destination and committed in no particular order by the destination, and the operations further include operations that access off-core memory on the basis of indirect scatter / gather memory access for reads from off-core memory when off-core memory is the source of data, and for writes to off-core memory when off-core memory is the destination for data.

[0270] In one example, an operation to access off-core memory based on indirect scatter / gather memory access further includes obtaining the source of the indirect address from a register file or core-local memory. In another example, the operation further includes circular buffering within core-local memory. In yet another example, the operation further includes updating a synchronization flag to indicate a monotonous, stepwise progression of data transfer.

[0271] Aspects of this disclosure relate to a cross-lane processing unit (XPU) for performing single-instruction multiple data (SIMD) data-dependent operations across multiple data processing lanes of a processor. Rather than physically fabricating operation-specific circuitry for each data-dependent operation, the XPU may be configured to perform various operations in response to input signals, comprising processing cells for performing individual operations and a crossbar arranged as a stacked network within the XPU. Each processing cell can receive and process data across multiple data processing lanes. Aspects of this disclosure include configuring the XPU to perform vector sorting while calculating a duplicate count of duplicate elements in an input vector received for sorting, with separate XPPs for sorting and duplicate counting. This eliminates the need to construct a U. The XPU can be implemented as part of hardware circuitry that complements the computation of dense data structures such as dense matrices, while accelerating the processing of sparse data structures such as sparse vectors or sparse matrices.

[0272] Aspects of the present disclosure include hardware circuits. The hardware circuits include a plurality of stages, each stage including a crossbar and two or more cells, and the hardware circuits further include a plurality of data processing lanes that stream their respective data from an upstream input to a downstream destination through the plurality of cells and the plurality of crossbars of the plurality of stages, and the hardware circuits are configured to receive input data and a first instruction for performing a first operation from an upstream input along the plurality of data processing lanes, and in response to receiving the first instruction, each stage is configured to send its respective second instruction to the respective processing cell of the stage, each cell is configured to perform its respective second operation in response to receiving input from its respective data processing lane, and the hardware circuits are further configured to send its respective third instruction to the respective crossbar for the stage, the crossbar is configured to replace the output from each cell of the stage with the cell of the next stage along the plurality of data processing lanes, and is configured to perform a first operation by processing the received input data along the plurality of data processing lanes and the plurality of cells configured to perform their respective second operations.

[0273] The systems included in the aspects of the present disclosure include hardware circuitry, which includes a plurality of stages, each including a crossbar and two or more cells, and a plurality of data processing lanes, which stream their respective data from an upstream input to a downstream destination through the plurality of cells of the plurality of stages and the plurality of crossbars, wherein the hardware circuitry is configured to receive input data from an upstream input along the plurality of data processing lanes and a first instruction for performing a first operation, and in response to receiving the first instruction, it is configured to send a second instruction to each processing cell of each stage, each stage, in response to receiving input from its respective data processing lane, and each cell is configured to perform its respective second operation, wherein the hardware circuitry is further configured to send a third instruction to each crossbar for each stage, and the crossbars are configured to replace the output from each cell of a stage with the cell of the next stage along the plurality of data processing lanes, and are configured to perform a first operation by processing the received input data along the plurality of data processing lanes and the plurality of cells configured to perform their respective second operations.

[0274] Aspects of the present disclosure include a method implemented by a computer. The computer-implemented method includes receiving input data from an upstream input along a plurality of data processing lanes via a plurality of cells and a plurality of crossbars, and a first instruction for performing a first operation, by hardware circuitry including a plurality of stages, each including a crossbar and two or more cells, and a plurality of data processing lanes for streaming data from an upstream input to a downstream destination; in response to receiving the first instruction, the hardware circuitry transmits a respective second instruction to each processing cell of the stage, for each stage, wherein each cell is configured to perform its respective second operation in response to receiving input from its respective data processing lane; the computer-implemented method further includes transmitting a respective third instruction to each crossbar for the stage, wherein the crossbar is configured to replace the output from each cell of the stage with the cell of the next stage along the plurality of data processing lanes; and the computer-implemented method further includes performing a first operation by processing the received input data along the plurality of data processing lanes and the plurality of cells configured to perform their respective second operations.

[0275] Aspects of this disclosure may include one or more of the following features. In some examples, aspects of this disclosure may include a combination of all of the following features.

[0276] Each cell is configured to receive its respective first input operand from each data processing lane passing through the cell, and its respective second input operand from each crossbar of the stage upstream of the cell.

[0277] The downstream destination of the data from multiple data processing lanes is a vector processing unit, which is configured to perform single-instruction multiple-data vector operations on the output data of the hardware circuit.

[0278] Each cell is configured to perform one or more predetermined primitive operations in response to one or more received instructions. The hardware circuit further includes multiple control cells, and when each second instruction is transmitted to its respective processing cell, the hardware circuit is configured to have each control cell generate and transmit its respective control signal to its respective processing cell, based on the first operation specified by the first instruction.

[0279] When each control cell generates and transmits its respective control signal, the hardware circuitry is configured to generate the respective control signal for each processing cell to perform one of the following operations: arithmetic, comparison, and bypass operation, based on at least one of the stage in which the processing cell resides and the data processing lane through which the processing cell passes.

[0280] Multiple cells and multiple crossbars form a processing network of connected cells spanning multiple stages and multiple data processing lanes, and the processing network of connected cells is configured to receive input data and generate output data by performing a first operation on said input data.

[0281] The connected cell processing network is configured to perform combined vector sort and duplicate count operations, the combined operation including the processing network receiving an input vector of elements and the processing network generating, as output, a sorted output vector and data specifying the count of duplicate elements in the input vector. The input data includes sparse vector data, and the hardware circuit is configured to perform one of the following after sending the respective second and third instructions: vector scan, vector addition, vector sort, or vector duplicate count.

[0282] Unless otherwise specified, the above-described alternatives are not mutually exclusive and may be implemented in various combinations to achieve specific advantages. Since these and other variations and combinations of the above-described features can be used without departing from the subject matter defined by the appended claims, the above-described examples should be interpreted as illustrative, not limiting, the subject matter defined by the appended claims. Furthermore, the presentation of examples described herein, as well as phrases such as "such as" and "including," should not be interpreted as limiting the subject matter of the claims to specific examples; rather, the above examples are intended to illustrate only one of many possible implementations. Additionally, the same reference numerals in various drawings may identify the same or similar elements.

Claims

1. It is a hardware circuit, It includes multiple tiles, and each of the multiple tiles is Processing unit and Memory unit and A vector core configured to generate a data-dependent address stream for the processing unit, A scalar core configured to dispatch tasks to the aforementioned processing unit, Hardware circuitry including a software-controlled scratchpad memory, the software-controlled scratchpad memory being shared among the plurality of tiles.

2. The hardware circuit according to claim 1, wherein each tile is configured to perform an independent calculation.

3. The hardware circuit according to claim 1, wherein each of the plurality of tiles includes a plurality of single instruction, multiple data (SIMD) processing lanes.

4. The hardware circuit according to claim 1, wherein the multiple tiles of the plurality of tiles issue memory requests in parallel to the main memory.

5. The hardware circuit according to claim 1, wherein the data-dependent address stream is for any level of the memory hierarchy.

6. The hardware circuit according to claim 1, wherein each data-dependent address stream corresponds to a sequence of addresses, and the length and specific value of the addresses in the sequence are data-dependent and determined at runtime.

7. The hardware circuit according to claim 1, wherein the processing unit in each of the plurality of tiles includes a circular buffer instruction that enables the transfer and access of a dynamically sized data stream on a statically sized area of ​​memory.

8. The hardware circuit according to claim 7, further comprising a microarchitecture configured to track the runtime buffer size of the dynamically sized data stream.

9. The hardware circuit according to claim 8, wherein the processing unit in each of the plurality of tiles is configured to provide the runtime configuration and access area of ​​each of the memory units as sequential cyclic first-in-first-out (FIFO) access without excluding unsequential access to the same area of ​​each of the memory units.

10. The hardware circuit according to claim 1, wherein one or more of the plurality of tiles each further comprises a prefetch unit configured to cooperatively prefetch data stream instructions.

11. The hardware circuit according to claim 1, wherein each tile is configured to support gathering from off-chip memory to the scratchpad memory and scattering from the scratchpad memory to the off-chip memory.

12. A machine learning accelerator configured to execute a neural network layer exhibiting semantic sparsity, A machine learning accelerator comprising a hardware circuit as described in any one of claims 1 to 11.

13. A system comprising a plurality of hardware circuits, wherein each of the plurality of hardware circuits is a hardware circuit described in any one of claims 1 to 11.

Images