Streaming Transfer and Ordering Model

Abstract

In general, disclosed herein is a hardware / software interface and stream ordering model for asynchronous data movement between off-core memory and core local memory, referred to herein as "stream transfer." Stream transfer allows software to more efficiently represent common data movement patterns, particularly those found in sparse workloads. Direct stream instructions belonging to a stream are processed in order. For indirect stream instructions, the offset elements in the offset list are processed in order. A sync flag is updated to indicate monotonically incremental progress of the stream.

Description

[Background technology]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. Patent Application No. 17 / 722,782, filed April 18, 2022, which claims the benefit of the filing date of U.S. Provisional Patent Application No. 63 / 322,285, filed March 22, 2022, the disclosures of which are incorporated herein by reference.

[0002] background Deep neural networks (DNNs) typically generate significant memory accesses, and the magnitude of these accesses can be determined at compile time. The sparse accelerator architecture aims to accelerate embedding layer computations while providing a clean architectural foundation for accelerating other sparse computations. Sparse workloads typically use scatter-gather type memory accesses, especially embeddings that exhibit small scatter-gather memory accesses from a large parameter space. Furthermore, memory access patterns and access sizes can be determined at runtime by the input sparsity pattern.

[0003] However, these characteristics can lead to random memory accesses and non-utilitarian utilization of memory bandwidth, making efficient prefetching impossible. The limited ability to reuse data for each computation can make it impossible for a processor to function when simply waiting for data to be fetched. Long latencies for accessing off-chip memory, irregularities in control flow and data flow further exacerbate this effect. The dynamic nature of random memory accesses also does not lend itself to efficient software techniques such as double buffering. Summary of the Invention

[0004] A brief overview In general, disclosed herein is a hardware / software interface and stream ordering model for asynchronous data movement between off-core memory and core-local memory, referred to herein as "stream transfer." Stream transfer allows software to more efficiently represent common data movement patterns, particularly those found in sparse workloads. Stream instructions belonging to a stream are processed in order. For indirect stream instructions, offset elements in the offset list are processed in order. A sync flag is updated to indicate monotonic incremental progress of the stream.

[0005] One aspect of the present disclosure provides a method, including one or more processors identifying a progress of data being transferred between off-core memory and core local memory, and one or more processors identifying a read from the core local memory when the core local memory is a source of the data, the reads being issued in order to the source and serviced out of order by the source, the method further including one or more processors identifying a write to the core local memory when the core local memory is a destination of the data, the writes being issued in order to the destination and committed out of order by the destination, the method further including one or more processors accessing the off-core memory based on indirect scatter / gather memory accesses for reads from the off-core memory when the off-core memory is a source of the data and for writes to the off-core memory when the off-core memory is a destination of the data.

[0006] In one example, identifying progress of the data being transferred further includes using a core-local synchronization flag. In another example, the method further includes the one or more processors selecting memory accesses to barrier based on a scalar fence instruction. In yet another example, accessing off-core memory based on an indirect scatter-gather memory access further includes sourcing the indirect address from a register file or from core-local memory. In yet another example, the method further includes the one or more processors circular buffering in core-local memory.

[0007] In yet another example, the method further includes one or more processors updating a core-local synchronization flag to indicate monotonic incremental progress of the data transfer. In yet another example, the method further includes one or more processors terminating the data transfer when all reads from the core-local memory have been issued. In yet another example, the method further includes one or more processors terminating the data transfer when all writes to the core-local memory have been committed.

[0008] Another aspect of the present disclosure provides a system including one or more processors and one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for transferring data between an off-core memory and a core local memory, the operations including identifying a progress of data being transferred between the off-core memory and the core local memory and identifying a read from the core local memory when the core local memory is a source of the data, the reads being issued in order to the source and serviced out of order by the source, the operations further including identifying a write to the core local memory when the core local memory is a destination of the data, the writes being issued in order to the destination and committed out of order by the destination, and the operations further including accessing the off-core memory based on indirect scatter / gather memory accesses for the read from the off-core memory when the off-core memory is a source of the data and for the write to the off-core memory when the off-core memory is a destination of the data.

[0009] In one example, identifying progress of the data being transferred further includes using a core-local synchronization flag. In another example, the operations further include selecting a memory access to barrier based on a scalar fence instruction. In yet another example, accessing off-core memory based on an indirect scatter-gather memory access further includes sourcing the indirect address from a register file or from core-local memory. In yet another example, the operations further include circular buffering in core-local memory.

[0010] In yet another example, the operations further include updating a core-local synchronization flag to indicate monotonic incremental progress of the data transfer. In yet another example, the operations further include terminating the data transfer when all reads from the core-local memory have been issued. In yet another example, the operations further include terminating the data transfer when all writes to the core-local memory have been committed.

[0011] Yet another aspect of the present disclosure provides a non-transitory computer-readable storage medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for transferring data between an off-core memory and a core local memory, the operations including identifying a progress of data being transferred between the off-core memory and the core local memory and identifying a read from the core local memory when the core local memory is a source of the data, the reads being issued in order to the source and serviced out of order by the source, the operations further including identifying a write to the core local memory when the core local memory is a destination of the data, the writes being issued in order to the destination and committed out of order by the destination, and the operations further including accessing the off-core memory based on indirect scatter-gather memory accesses for the read from the off-core memory when the off-core memory is a source of the data and for the write to the off-core memory when the off-core memory is a destination of the data.

[0012] In one example, accessing off-core memory based on the indirect scatter-gather memory access further includes sourcing the indirect address from a register file or from a core local memory. In another example, the operation further includes circular buffering in the core local memory. In yet another example, the operation further includes updating a synchronization flag to indicate monotonically incremental progress of the data transfer.

[0013] Yet another aspect of the present disclosure provides a hardware / software interface and stream ordering model for asynchronous data movement between off-core memory and core-local memory, referred to as "stream transfer." Stream transfer can enable software to more efficiently represent common data movement patterns, particularly those found in sparse workloads. Direct stream instructions belonging to a stream may be processed in order. For indirect stream instructions, offset elements in an offset list may be processed in order. A sync flag may be updated to indicate monotonic incremental progress of the stream. The methods according to the present disclosure may in particular be computer-implemented methods. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 2 is a block diagram of a hardware circuit for performing stream transfer according to an aspect of the present disclosure. [Figure 2] FIG. 1 is a block diagram of an exemplary environment for implementing hardware circuitry according to aspects of the present disclosure. [Figure 3A] FIG. 2 is a block diagram of an exemplary tile of hardware circuitry for performing stream forwarding according to aspects of the present disclosure. [Figure 3B] FIG. 10 is a block diagram of another example tile of hardware circuitry for performing stream forwarding according to aspects of the present disclosure. [Figure 4] 10 is a flow diagram of an example process for unrolling a stream descriptor into constituent off-tile or tile-local stream requests, according to aspects of the disclosure. [Figure 5] FIG. 1 is a flow diagram of an example process for ordering stream transfers according to aspects of the disclosure. [Figure 6] FIG. 1 is an exemplary diagram of stream ordering according to aspects of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0015] Detailed Description Generally, disclosed herein is a hardware / software interface for asynchronous data movement between off-core memory and core-local memory, referred to herein as a "stream transfer." Stream transfers can include stream descriptors that allow software to more efficiently represent common data movement patterns, particularly those found in sparse workloads. A stream descriptor is an internal data structure that represents information necessary to perform a stream transfer. The information may include, by way of example, source addresses, destination addresses, stream operation codes, and control information such as linear or circular buffers. Data movement, referred to as a "stream," can be initiated by a stream instruction that encodes the information necessary to perform the stream transfer. Each stream can have an associated stream identification (ID), indicated by a data synchronization flag (sync flag) associated with the stream instruction. Stream instructions issued by cores with the same stream ID can form a single stream.

[0016] Stream transfers can only move data to or from core-local memory. Furthermore, only core-local sync flags can be used to track the progress of streams. Sync flags track partial progress of stream transfers. They track reads from core-local memory when core-local memory is the source, or writes to core-local memory when core-local memory is the destination. The progress of reads and writes to off-core memory does not need to be tracked, but a scalar fence instruction can be used to allow selection of which memory accesses to barrier to ensure that outstanding writes to off-core memory are committed.

[0017] Stream transfers may include indirect scatter / gather memory accesses, where the addresses for the source or destination are stored in separate memory locations that are read first. As an example, the indirect addresses are sourced from a register file with masking support or from memory. Indirect scatter / gather memory accesses may further include different addressing modes, such as row addresses or word addresses, as examples. Stream transfers may include support for ScatterAdd / GatherAdd modes directly on memory words. Memory words can be updated atomically. As an example, f32, i32, bf16, and i16 data types may be supported.

[0018] Stream transfers can include support for circular buffers in the source or destination buffer, which simplifies the buffer allocation problem in software since the buffer size is not known during compilation.

[0019] Furthermore, generally, a stream ordering model is disclosed herein. These transfer synchronization primitives allow data transfers to be processed in order, while the actual data transfers are out of order. Individual stream instructions issued by cores with the same stream ID form a single stream. The hardware provides ordering guarantees for transfers within a single stream, which may span multiple stream instructions.

[0020] Stream instructions belonging to a stream are processed in order. For indirect stream instructions, the offset list is ordered, i.e., the offset elements in the offset list are processed in order. Writes are issued to the destination memory in order and may be committed by the destination memory out of order. Reads are issued to the source memory in order and may be serviced by the source memory out of order.

[0021] The sync flag is updated to indicate the monotonically incremental progress of the stream. When core local memory is the source, the sync flag tracks reads from core local memory. A sync flag value of N indicates that the first N chunks of data may be overwritten in core local memory. When core local memory is the destination, the sync flag tracks writes to core local memory. A sync flag value of N indicates that subsequent reads to the first N chunks of data in core local memory will return the requested data.

[0022] A stream can terminate when the data for the request preceding and including the last stream descriptor is fully committed to memory. As an example, if core-local memory is the source, the stream can terminate when all reads have completed. If core-local memory is the destination, the stream can terminate when all writes have committed.

[0023] Aspects of the present disclosure enable software to more efficiently express common data movement patterns, particularly those found in sparse workloads. Aspects of the present disclosure can also provide a low-complexity solution for hiding long memory access latencies while preserving the software programming model of the compute core and the in-order core.

[0024] 1 is a block diagram of a hardware circuit 101 for performing stream forwarding according to an aspect of the present disclosure. The hardware circuit 101 may include a coprocessor 103, a coprocessor 104, a high-bandwidth memory 107, and an on-chip interconnect 108. The coprocessor 103 may include one or more tiles 102A-F, each implementing a respective vector processing unit (VPU) that includes a cross-lane processing unit (XPU) 101A-F. The coprocessor 103 may include a tile sequencer 106 configured to coordinate input and output data across the tiles 102A-F.

[0025] Coprocessor 103 is configured to perform data-dependent operations using XPUs 102A-F, or more generally, VPUs. XPUs 102A-F are each connected to a series of data processing lanes, each lane streaming data through XPUs 102A-F. The streamed data may be retrieved from on-chip memory 105, which may be any of a variety of different memory devices, including main memory, cache, or persistent storage such as solid-state or hard disk storage. The streamed data may also be retrieved from coprocessor 104, high-bandwidth memory 107 serving one or both of coprocessors 103 and 104, and / or another data source connected to hardware circuit 101 through on-chip interconnect 108.

[0026] Coprocessor 104 can be configured to accelerate specific operations, such as matrix-matrix multiplication, matrix-vector multiplication, etc. In some examples, coprocessor 104 is on a different device than the rest of hardware circuit 101 and communicates data to the hardware circuit through on-chip interconnect 108. On-chip interconnect 108 can be a data bus or any form of interconnect according to any of a variety of communication standards, such as PCIe.

[0027] An exemplary input to the hardware circuit 101 is a tensor that represents input data and / or model parameters for a machine learning model executed using the hardware circuit 101. A tensor is a data structure that generalizes various other common data structure types with different dimensions. A tensor can contain zero or more elements, which can be of one or more different data types, such as integers, floating-point values, Boolean values, etc. Within each data type, the data type can be parameterized according to a particular level of precision, such as, for example, 8-bit, 16-bit, or 32-bit integers or floating-point values. The dimension of a tensor is referred to as its "rank." A tensor of rank 0 has a single element and is also called a scalar. A tensor of rank 1 is also called a vector. A tensor of rank 2 is also called a matrix. Vectors and matrices can also be referred to as having different ranks. For example, a vector of rank 2 is equivalent to a matrix. A tensor of non-zero rank can be described as a collection of tensors of one rank lower. For example, a rank-1 vector is a collection of scalar values, and a rank-2 matrix is ​​a collection of rank-1 vectors.

[0028] The hardware circuit 101 can, at least in part, execute a processing pipeline for training a neural network. The pipeline can include generating embeddings for input training examples. The embeddings can represent features of the input training examples using less data, allowing for more efficient processing. The feature tensors of different input training examples will have different degrees of sparsity, which affects the amount of computational effort required to generate the corresponding embeddings. The hardware circuit can be configured to receive tensors of feature values ​​representing the training input examples and generate the embeddings as tensors having a lower rank than the feature tensors.

[0029] To generate the embeddings, coprocessor 103 is configured to perform various data-dependent operations for efficient sparse data computation on XPUs 102A-F, or more generally on VPUs. These operations include sorting or adding sparse vectors, summarizing the contents of input vectors, and converting sparse matrices from one sparse matrix storage format to another.

[0030] Instead of physical predefined circuits for accelerating the performance of data-dependent operations, the VPUs, including XPUs 102A-F, can be configured, e.g., programmed, to perform a variety of different data-dependent operations. Coprocessor 103 enables generalized support for processing sparse data while still allowing complementary coprocessor 104 to perform highly efficient, parallelizable operations.

[0031] The hardware circuitry 101 may be any of a variety of different types of processing units, such as, for example, a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC), such as a tensor processing unit (TPU). The hardware circuitry 101 may be implemented on a computing device, which may itself be part of a system of one or more devices.

[0032] 2 is a block diagram of an exemplary environment 200 for implementing the hardware circuit 101. The hardware circuit 101 may be implemented on a device having one or more processors at one or more locations, such as a server computing device 215. The user computing device 212 and the server computing device 215 may be communicatively coupled to one or more storage devices 230 via a network 260. The storage device 230 may be a combination of volatile and non-volatile memory and may be in the same or a different physical location as the computing devices 212, 215. For example, the storage device 230 may include any type of non-transitory computer-readable medium capable of storing information, such as a hard drive, a solid-state drive, a tape drive, an optical storage device, a memory card, ROM, RAM, DVD, CD-ROM, writable memory, and read-only memory.

[0033] The server computing device 215 may include one or more processors 213 and a system memory 214. The memory 214 may store information accessible by the processor 213, including instructions 221 that may be executed by the processor 213. The memory 214 may also include data 223 that may be retrieved, manipulated, or stored by the processor 213. The memory 214 may be any type of non-transitory computer-readable medium accessible to the processor 213, such as volatile and non-volatile memory. The processor 213 may include one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), and / or application-specific integrated circuits (ASICs), such as a tensor processing unit (TPU). The processor 213 may include a coprocessor implemented as part of a hardware circuit, such as those described herein with reference to FIG. 1.

[0034] The instructions 221 may include one or more instructions that, when executed by the processor(s) 213, cause one or more processors to perform the actions defined by the instructions. The instructions 221 may be stored in object code format for direct processing by the processor(s) 213 or in other formats, including interpretable scripts or collections of independent source code modules that are interpreted on-demand or pre-compiled. The instructions 221 may include instructions for configuring stream transfers consistent with aspects of the present disclosure. The server computing device 215 and / or the user computing device 212 may execute a compiler or other program to generate and send the instructions as control signals to the hardware circuit 101 to configure the VPU of the circuit.

[0035] Data 223 may be retrieved, stored, or modified by processor 213 according to instructions 221. Data 223 may be stored in a computer register, a relational database, or a non-relational database, as a table with multiple different fields and records, or as a JSON, YAML, proto, or XML document. Data 223 may also be formatted in a computer-readable format, such as, but not limited to, binary values, ASCII, or Unicode. Furthermore, data 223 may include sufficient information to identify related information, such as numbers, descriptive text, unique codes, pointers, references to data stored in other memory, including other network locations, or information used by a function to calculate related data.

[0036] Similar to the server computing device 215, the user computing device 212 may also be configured with one or more processors 216, memory 217, instructions 218, and data 219. The user computing device 212 may also include a user output 226 and a user input 224. The user input 224 may include any suitable mechanism or technique for receiving input from a user, such as a keyboard, a mouse, a mechanical actuator, a soft actuator, a touchscreen, a microphone, and a sensor.

[0037] The server computing device 215 may be configured to transmit data to the user computing device 212, which may be configured to display at least a portion of the received data on a display implemented as part of the user output 226. The user output 226 may also be used to display an interface between the user computing device 212 and the server computing device 215. The user output 226 may alternatively or additionally include one or more speakers, transducers or other acoustic outputs, a haptic interface or other tactile feedback that provides non-visual and non-auditory information to a platform user of the user computing device 212.

[0038] While Figure 2 depicts the processors 213, 216 and memories 214, 217 as being within the computing devices 215, 212, the components described herein, including the processors 213, 216 and memories 214, 217, may include multiple processors and memories that may operate in different physical locations not within the same computing device. For example, some of the instructions 221, 218 and data 223, 219 may be stored on a removable SD card and others within a read-only computer chip. Some or all of the instructions and data may be stored in locations physically separate from the processors 213, 216 but still accessible by the processors. Similarly, the processors 213, 216 may include a collection of processors that may perform simultaneous and / or sequential operations. The computing devices 215, 212 may each include one or more internal clocks that provide timing information that may be used to time operations and programs run by the computing devices 215, 212.

[0039] The server computing device 215 can be configured to receive requests to process data from the user computing device 212. For example, the environment 200 can be part of a computing platform configured to provide various services to users through various user interfaces and / or APIs that expose platform services. One or more services can be a machine learning framework or toolset for generating a neural network or other machine learning model according to a specified task and training data. The user computing device 212 can receive and transmit data specifying a workload or type of configured operation that the VPU of the hardware circuit 101 is to be configured to perform. The user computing device 212 can transmit instructions directly to the hardware circuit 101 or can have the server computing device 215 generate and transmit instructions as control signals, as described herein, to the hardware circuit 101.

[0040] The devices 212, 215 may be able to communicate directly and indirectly via the network 260. The devices 212, 215 may set up listening sockets capable of accepting initiating connections for sending and receiving information. The network 260 may itself include a variety of configurations and protocols, including the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local network, and a private network using one or more proprietary communication protocols. The network 260 may support a variety of short- and long-range connections. The short- and long-range connections may occur over various bandwidths, such as 2.402 GHz to 2.480 GHz, commonly associated with the Bluetooth® standard, 2.4 GHz and 5 GHz, commonly associated with the Wi-Fi® communication protocol, or various communication standards such as LTE® for wireless broadband communications. The network 260 may additionally or alternatively support wired connections between the devices 212, 215, including connections via various types of Ethernet® connections.

[0041] 2, it is understood that aspects of the present disclosure can be performed according to a variety of different configurations and quantities of computing devices, including in paradigms for serial or parallel processing, or via a distributed network of multiple devices. In some embodiments, aspects of the present disclosure can be implemented on a single device, or any combination thereof.

[0042] FIG. 3A is a block diagram of an example tile 102 for performing stream transfers. The XPU 101 is coupled to a cross-lane controller 310. The cross-lane controller 310 provides a separate control thread to enable cross-lane instructions to the XPU 101. As described herein, the XPU can receive a first instruction, for example, through one or more control signals, which can be translated into one or more second and third instructions, each of which can be provided to the processing cells and crossbar of the XPU 101 to perform the configured operation specified by the first instruction. Instructions to the XPU 101 can be conveyed via control signals, which the processing cells and crossbar of the XPU 101 are configured to interpret to perform the corresponding primitive operation. An example instruction can be an opcode of an instruction set architecture (ISA).

[0043] The tile 102 can receive data from the on-chip interconnect 108 and on-chip memory 105, as described with reference to FIG. 1 . The XPU can also receive instructions from an instruction interface 324, e.g., from the tile sequencer 106, through the scalar core 312 or the scalar core 320. The scatter / gather controller 322 of the tile 102 can receive incoming data and control which data is passed into the memory 306 through the memory scheduler 314. The memory scheduler 314 coordinates how data is accessed and retrieved from the memory 306. The memory 306 is private to the tile 102 and is not accessible to other components connected to the tile 102, such as other tiles. The arbiter 304 is configured to manage which of the vector processing units (VPUs) 302A-H access the memory 306, e.g., every clock cycle. A tile 102 may maintain a task queue 308 of tasks to be performed by the tile 102, which are sent to a scatter / gather controller 322 through a scalar core 320. A tile 102 may also maintain registers of tile synchronization flags 318 and / or memory flags 316 for synchronizing the tile 102 with other tiles of the hardware circuit and memory 306, respectively.

[0044] The VPUs 302A-H are connected to the XPU 101 through data processing lanes, indicated by the solid lines between the XPU 101 and the VPUs 302A-H. The dashed lines between the XPU 101 and the VPUs 302A-H represent control signals that can be received by a control cell within the XPU 101 to configure the XPU 101 to perform a configured operation corresponding to the received control signal. The vector processing units are configured for efficient operation on input vectors. The length of a vector processed by a tile 102 at a time may depend on the number (or width) of VPUs implemented by the tile. For example, eight VPUs 302A-H have a width of eight. The VPUs 302A-H can process data along the same data processing lane. The VPUs 302A-H can be configured to perform scalar operations on elements of an incoming vector from memory 306. VPUs 302A-H can receive data from XPU 101, which can process data across multiple data processing lanes, rather than processing data only along a lane as performed by each VPU 302A-H, as described herein.

[0045] 3B is a block diagram of another example tile 102 that implements the XPU 101 for stream transfers. The tile 102 can receive data from the on-chip interconnect 108 and on-chip memory 105 as described with reference to FIG. 1. The XPU can also receive instructions from an instruction interface 324, for example, from the tile sequencer 106. A scatter / gather controller 322 of the tile 102 can receive incoming data and control which data is passed into the memory 306.

[0046] This example tile 102 is based on an access / execution separation architecture, which can separate a program (and associated instruction sequences) into two streams. The first stream can be an access stream that fetches operands and stores results. The second stream can be an execution stream that consumes operands, performs computations, and produces results. These streams run on two separate cores: a tile access core (TAC) 332 and a tile execution core (TEC) 330.

[0047] The tile access core 332 is based on the scalar core complex 320 and may be responsible for prefetching operands for execution from the memory 306. The tile execution core 330 is based on the scalar complex core 312 and includes the XPU 101 and the VPU 302 and may be responsible for performing computational operations on the prefetched operands to generate results. The VPU 302 is connected to the memory 306 through a load-store unit 328. The TAC 332 and the TEC 330 have independent instruction streams and together form a producer-consumer pair. The tile 102 may maintain a task queue 308 of tasks to be performed by the tile 102 that are sent to the TAC 332 and the TEC 330.

[0048] The TAC 332 and TEC 330 communicate with each other through scalar memory 334, instruction buffer 326, and tile sync flag 318. The scalar memory 334 can be used by the TAC 332 and TEC 330 to exchange data and can be used as a software-managed circular buffer to pass data between the TAC 332 and TEC 330 in a first-in-first-out order. The tile sync flag 318 can be used as a counting semaphore between the TAC 332 and TEC 330. For example, when a circular first-in-first-out order is used between the two cores, the producer core increments the sync flag 318 by the number of bytes after each push and stops when the count reaches the maximum size of the first-in-first-out order. Similarly, the consumer decrements the sync flag 318 after each pop and stops when there is no more data in the buffer. The amount of data prefetched can be dynamic, and a done bit is used to indicate the end of the stream.

[0049] Stream transfers allow tiles 102 and tile sequencer 106 to move data between tile-local memory, such as memory 306 or scalar memory 334, and off-tile memory, such as memory 105 or high-bandwidth memory 107. Data movement can be initiated by a stream instruction and can be referred to as a stream. Each stream has an associated stream ID, indicated by a sync flag 318 associated with the stream instruction. Separate stream instructions with the same stream ID form a single stream.

[0050] Stream transfers can move data to or from tile-local memory, and the tile-local sync flag 318 is used to track the progress of the stream. The sync flag 318 tracks the partial progress of the stream being transferred. For example, the sync flag 318 tracks reads from the tile-local memory when the memory is the source, or the sync flag 318 tracks writes to the tile-local memory when the memory is the destination. The progress of reads and writes to off-tile memory does not need to be tracked. To ensure that all outstanding writes to off-tile memory are committed, a scalar fence instruction can be used to allow selection of which memory accesses to barrier. The scatter-gather engine 322 tracks the status of outstanding stream transfers for each particular memory and communicates this status to the scalar core 320. When a scalar fence is issued for a barrier to a particular memory, it waits for the status to indicate that all outstanding stream transfers (reads or writes) targeting that memory are fully committed. Once that condition is met, the fence wait is released on the scalar core 320 .

[0051] Stream transfers can support efficient scatter-gather operations using strided streams to access off-tile memory and indirect streams to access off-tile memory from tile local memory or register files. The choice of strided or indirect streams can be based on the software access pattern. If software wants to access every Nth element in a tensor, strided streams are preferred, but indirect streams can still work. However, if software wants to access a random set of elements in a tensor, an indirect stream should be used.

[0052] Stream transport can also support circular buffer semantics for tile-local memory.

[0053] Stream transfers support the following data movements, where the granularity and alignment of the data movement depend on the source-destination pair: Data can be transferred from memory 306 to on-chip memory 105 and from on-chip memory 105 to memory 306. Data can also be transferred from memory 306 to high-bandwidth off-chip memory 107 and from off-chip memory 107 to memory 306. Data can also be transferred from scalar memory 334 to on-chip memory 105 and from on-chip memory 105 to scalar memory 334. As an example, the smallest granularity source and destination alignment can be 4 bytes. As another example, 32-byte accesses can be used to support 4-byte accesses to off-chip memory 107. As yet another example, 32-byte alignment and a minimum length of 128 bytes can guarantee performance on streams to or from off-chip memory 107.

[0054] The stream descriptor is an internal data structure that represents all the information for the scatter-gather controller 322 to perform a stream transfer. A stream instruction can fully encode the fields of the stream descriptor. The following are example fields of a stream descriptor:

[0055] For stream operation codes, gather streams read off-tile memory and store or append data to tile local memory. Scatter streams read from tile local memory and store or append data to off-tile memory. Off-tile memory and tile local memory are determined by the Off-Tile Memory Type and Tile Local Memory Type fields, respectively.

[0056] The add variants of the stream instructions support both floating-point and signed integer add operations. The gather signed integer add and gather floating-point add variants can be supported for tile-local memory. The scatter signed integer add and scatter floating-point add variants can be supported for off-tile and tile-local memory. If an illegal combination is detected, a program error may be introduced.

[0057] The tile-local stream type indicates the address pattern used to access the tile-local memory. For example, a linear stream facilitates multiple consecutive words starting at the tile-local start offset. The multiple consecutive words can have a length of 4 bytes. As another example, a circular buffer stream allows software to build a logical circular buffer in the tile-local memory. In this exemplary access pattern, the base, size, and offset fields of the circular buffer metadata are used to generate addresses of the multiple words. The multiple words can have a length of 4 bytes. If the effective length of the granularity is larger than the size field in the circular buffer metadata, a program error may result.

[0058] The off-tile stream type indicates the address pattern used to access the off-tile memory. A linear stream facilitates access to multiple consecutive locations starting at an off-tile starting offset. The actual word size depends on the off-tile memory type. A strided stream facilitates the translation of strided access patterns to multi-dimensional arrays stored in the off-tile memory type. Stream transfers can support a single level of stride. An indirect stream allows a random scatter-gather access pattern to a table, where an indirect offset list is used and each entry in the list accesses the same length of data.

[0059] The source of the indirect offset list can be a tile local memory or a register file. If the source is a tile local memory, the indirect offset field has a starting offset into the tile local memory where the number of offsets is stored. If the source is a register file, the indirect offset field has a register file and indicates the number of lanes that contain valid offsets. These offsets are used to perform the scatter or gather operation as indicated by the stream operation code.

[0060] The core type indicates the core type that generated the stream descriptor, such as a tile execution core 330 or a tile access core 332. The sync flag core type indicates the core type of the sync flag 318 that tracks the progress of the stream. The encoding can be the same as the core type to allow streams initiated by a tile access core to be tracked by a tile execution core and vice versa.

[0061] The sync flag ID indicates an offset in the target sync flag memory. The sync flag ID can also be used as a stream ID and can have ordering guarantees, which are further described below. The set done bit indicates that the current descriptor is the last in the stream. The done bit is set after all data for the current descriptor and the preceding descriptors in the stream have been fully committed to tile-local memory.

[0062] The sync flag count type indicates the type of count that the sync flag 318 is tracking, either the number of words or the number of descriptors. In both cases, the sync flag 318 tracks the monotonically incremental progress of the stream, but at different granularities.

[0063] The tile local memory type indicates the type of tile local memory involved in the stream transfer, which may include scalar memory 334 or a local bank of memory 306.

[0064] The tile-local starting offset field is used if the tile-local stream type is linear. It indicates the aligned starting offset word, such as a 4-byte word, in the tile-local memory accessed by this transfer. The actual access type depends on the stream operation code.

[0065] The tile-local stride encodes the stride size and the number of bytes accessed in each stride, which is used to access the tile-local memory selected by the tile-local memory type. The length, which may be 4 bytes as an example, does not need to be a multiple of the number of bytes accessed in each stride. The final request of the strided access will access the remaining words of the transfer, which may be shorter than the per-stride length. The stride calculation can be the same for both linear and circular buffer stream types.

[0066] The circular buffer metadata field is used when the tile-local stream type is a circular buffer. The size of the circular buffer can be a multiple of the granularity of the off-tile memory type, and the offset can be aligned. If the circular buffer wraps around, the request is broken down into multiple requests, and an error can be generated if the resulting requests are not a multiple of the granularity of the off-tile memory type. An error can also be generated if the total length of the stream transfer is larger than the size of the circular buffer.

[0067] The off-tile memory type indicates the type of off-tile memory involved in the transfer. This includes on-chip memory 105 and high-bandwidth memory 107. A high-bandwidth memory view may also be used, which allows access at 4-byte granularity and 4-byte alignment. If the sequencer 106 is the initiator of the stream transfer, this field may be uncoded to indicate high-bandwidth memory 107.

[0068] The Tile ID field is used to select the tile ID for the memory slice. The off-tile starting offset contains the starting offset word within the off-time memory 105 indicated by the associated off-tile memory type. The units of the offset may be equal to the value indicated in the Offset Alignment column of the off-tile memory type. For example, for high bandwidth memory 107, an offset value of 1 translates to a byte access of 32. If the off-tile stream type is indirect, this field may act as a base address that is added to the offset read from the indirect offset before accessing the memory.

[0069] If the off-tile stream type is indirect, an indirect offset can be used. If the source is tile local memory, the indirect offset provides the word starting offset in the tile local memory that stores the indirect offset list. If the source is a register file, the indirect offset provides the file register index that sources the indirect offset list. The register file can be read when a stream instruction is issued.

[0070] If the off-tile stream type is indirect, the indirect list size can be used. If the source is tile local memory, the number of elements in the offset list is stored in tile local memory. If the source is a register file, the number of lanes containing valid offsets is stored. Completion of the transfer is in order with the remaining descriptors in the stream.

[0071] The indirect list type is used when the off-tile stream type is indirect and indicates the type of offsets stored in the offset list, which can include word offsets and row offsets.

[0072] The indirect list stride is used if the off-tile stream type is indirect and indicates the distance between two address words in the offset list stored in tile local memory. It can be a signed integer.

[0073] If the off-tile stream type is indirect, the indirect filter field is used, and if this field is set, indirect memory addresses that match the indirect filter value are filtered out. The indirect filter value indicates the value of an element in the indirect access list that needs to be filtered out. This value is of the type indicated by the indirect list type. Filtering can be enabled when the indirect filter field is set for an indirect stream and / or the value of an element in the indirect offset list matches this field. The off-tile and tile-local accesses corresponding to the filtered elements are withdrawn, but the tile-local buffer is still advanced by the size of the filtered access.

[0074] The length, e.g., a length of 4 bytes or a multiple of 512 bytes, indicates the total number of words accessed by the stream. If the off-tile stream type is linear or strided, this field indicates the total number of words accessed by the stream. If the off-tile stream type is indirect, this field indicates the number of words accessed from each address in the indirect offset list. If the actual value of this field is not a multiple of the granularity of the off-tile memory type, a program error may result. If the generated address exceeds the boundaries of the off-tile memory 105, a program error may also result.

[0075] The stride size field indicates the stride size in units of granularity for the off-tile memory type, which can be a signed integer.

[0076] The per-stride length, e.g., a per-stride length of 4 bytes or a multiple of 512 bytes, indicates the number of words accessed in each stride. It is a signed field but should contain a non-negative value. The length does not need to be a multiple of this field. This field should be a multiple of the granularity of the off-tile memory type selected by this stream descriptor. The final request of a strided access will access the remaining words of the transfer, which may be shorter than the per-stride length. A program error may result if the per-stride length is 0, negative, or not a multiple of the off-tile memory access granularity. A program error may also result if the generated address exceeds the boundaries of the off-tile memory 105.

[0077] The trace field indicates whether the stream transfer should be traced. Tracing can include logging information about actions taken during the stream transfer as part of debugging.

[0078] 4 is a flow diagram of an example process 400 for unrolling a stream descriptor into constituent off-tile or tile-local stream requests. The example process 400 can be implemented on a system of one or more processors in one or more locations. For example, a hardware circuit 101 such as that described above can implement the process 400.

[0079] As shown in block 410, the process includes receiving a size of the off-tile memory, such as receiving a size of 4 bytes. Additionally, the process includes receiving a maximum chunk size for stream requests targeting the off-tile memory type. As shown in block 420, the process further includes converting an indirect offset read from a file register or tile local memory to an offset, such as a 4-byte offset, into the off-tile memory based on the indirection list type.

[0080] As shown in block 430, the process also includes generating strided and / or indirect requests. For strided requests, the process may include partially unrolling a strided stream descriptor into a set of requests, each accessing consecutive addresses in the off-tile memory. For indirect tile local memory requests, the process may include obtaining an indirect stream descriptor and generating a list of offsets to the off-tile memory type selected by the descriptor. For indirect file register memory requests, the process may include generating a list of offsets from a file register that is read upon issuance of the indirect stream instruction.

[0081] As shown in block 440, the process includes generating a list of unrolled off-tile memory requests, where each unrolled request accesses a set of contiguous addresses in the off-tile memory. These requests are used to generate both tile-local memory requests and off-tile memory requests. The tile-local stride, tile-local stream type, and alignment are considered while unrolling the requests. The process further includes generating a list of partially unrolled requests, where each partially unrolled request accesses a set of contiguous addresses in the off-tile memory. These requests are further unrolled to generate a set of requests that are aligned to a memory granularity selected by the off-tile memory type.

[0082] As indicated at block 450, the process includes unrolling the stream descriptor into a set of off-tile and tile-local memory requests.

[0083] 5 is a flow diagram of an example process 500 for ordering stream transfers. The example process 500 can be implemented on a system of one or more processors in one or more locations. For example, hardware circuitry 101, as described above, can implement the process 500. Individual stream instructions issued by cores with the same stream ID form a single stream, but ordering cannot be guaranteed across different streams. The scatter-gather controller 322 includes multiple threads that can process these requests in parallel. Ordering can be guaranteed for transfers within a single stream, which may span multiple stream instructions.

[0084] As shown in block 510, the stream instructions belonging to a stream are processed in order, and their corresponding requests are issued by the scatter-gather controller 322 in order.

[0085] As shown in block 520, for indirect stream instructions, the offset list is ordered. Offset elements in the offset list are processed in order. Writes are issued to the destination memory in order, but the writes may be committed out of order by the destination memory. Reads are issued to the source memory in order, but the reads may be serviced out of order by the source memory.

[0086] As shown in block 530, the scatter / gather controller 322 updates the sync flag 318 to indicate the monotonically incremental progress of the stream. When the tile local memory is the source, the sync flag 318 tracks reads from the tile local memory. The sync flag value indicates the first chunk of data that can be overwritten in the tile local memory. When the tile local memory is the destination, the sync flag 318 tracks writes to the tile local memory. Here, the sync flag value indicates that subsequent reads to the first data chunk in the tile local memory will return the data that is being requested.

[0087] As shown in block 540, the done bit in the sync flag 318 may be updated at the end of the stream. This is indicated by a set done bit in the stream descriptor. The done bit may be set after all data for the request preceding and including the last stream descriptor has been fully committed to memory. All reads have completed if tile-local memory is the source, and all writes have completed if tile-local memory is the destination.

[0088] FIG. 6 is an example diagram of stream ordering. Consider stream descriptors A and B to form a single stream. Stream descriptor B has a set done bit. The partial progress of the stream is tracked by a sync flag. When A0, which is either a read or a write, is committed to memory, the sync flag is updated to a value of 1. Even if A2 and B1 are committed before A0, the sync flag value is not updated to 3. When A1 is committed to memory, five consecutive data chunks in the stream, A0, A1, A2, B0, and B1, are committed, as indicated by a sync flag value of 5. Because stream descriptor A is not the last in the stream, the done bit is not set at this point. When B2 is committed, the sync flag value is set to 6. Because all data chunks for the stream have been committed and stream descriptor B is the end of the stream, the done bit can now be set.

[0089] Aspects of the present disclosure may be implemented as one or more computer programs in a digital circuit, a computer-readable storage medium, or a combination of one or more of the above. The computer-readable storage medium may be non-transitory, such as executable by a cloud computing platform, and as one or more instructions stored on a tangible storage device.

[0090] The phrase "configured to" is used herein in several different contexts to refer to computer systems, hardware and hardware circuits, or portions of computer programs, engines, or modules. When a system is described as being configured to perform one or more operations, this means that the system has appropriate software, firmware, and / or hardware installed thereon that, when operated, causes the system to perform the one or more operations. When some hardware is described as being configured to perform one or more operations, this means that the hardware includes one or more circuits that, when operated, receive inputs and, in accordance with the inputs, generate outputs corresponding to the one or more operations. When a computer program, engine, or module is described as being configured to perform one or more operations, this means that the computer program includes one or more program instructions that, when executed by one or more computers, cause the one or more computers to perform one or more operations.

[0091] While the operations illustrated in the figures and recited in the claims are shown in a particular order, it is understood that the operations may be performed in an order different from that shown, and that some operations may be omitted, performed more than once, and / or performed in parallel with other operations. Furthermore, the separation of different system components configured to perform different operations should not be understood as requiring the components to be separated. The described components, modules, programs, and engines may be integrated together as part of a single system or multiple systems.

[0092] Unless otherwise stated, the above alternatives are not mutually exclusive and may be implemented in various combinations to achieve inherent advantages. Because these and other variations and combinations of the above-described features can be utilized without departing from the subject matter defined by the claims, the above description of embodiments should be taken as illustrative, and not as limiting, of the subject matter defined by the claims. Additionally, the examples and provision of clauses such as "such as," "including," etc., described herein should not be construed as limiting the subject matter of the claims to any particular examples; rather, the example is intended to illustrate only one of many possible embodiments. Furthermore, the same reference symbols in different drawings may identify the same or similar elements.

Claims

1. A method for transferring data between an off-core memory and a core-local memory, comprising: one or more processors identifying a read from a core local memory when the core local memory is a source of data; Reads are issued to the source in order and serviced by the source out of order; The method comprises: the one or more processors further identifying a write to the core local memory when the core local memory is a destination for the data; writes are issued to the destination in order and committed by the destination out of order; The method comprises: the one or more processors: for reading from the off-core memory when the off-core memory is the source of the data; and for writing to the off-core memory when the off-core memory is the destination of the data; The method further comprising accessing off-core memory based on random scatter / gather memory accesses using indirect addresses from a register file or from the core local memory.

2. The method of claim 1, further comprising the one or more processors tracking the monotonically incremental progress of the data transfer after identifying a read from or a write to the core local memory.

3. The method of claim 2, wherein tracking the monotonically incremental progress of the data transfer further includes incrementing a core local synchronization flag after identifying a read from or a write to the core local memory.

4. The method of claim 1 , further comprising the one or more processors selecting memory accesses to barrier based on a scalar fence instruction.

5. The method of claim 1 , further comprising the one or more processors circularly buffering the data for transfer within the core local memory.

6. 4. The method of claim 1, further comprising the one or more processors terminating tracking progress of the data transfer when all the reads from the core local memory have been issued.

7. 4. The method of claim 1, further comprising the one or more processors finalizing the transfer of the data when all writes to the core local memory have committed.

8. 1. A system comprising: one or more processors; one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for transferring data between an off-core memory and a core local memory; and the operation comprises: identifying a progress of data being transferred between the off-core memory and the core local memory; identifying a read from said core local memory when said core local memory is a source of said data; Including, Reads are issued to the source in order and serviced by the source out of order; identifying a write to the core local memory when the core local memory is a destination for the data; writes are issued to the destination in order and committed by the destination out of order; accessing the off-core memory based on random scatter / gather memory accesses using indirect addresses from a register file or from the core local memory for reading from the off-core memory when the off-core memory is the source of the data, and for writing to the off-core memory when the off-core memory is the destination of the data.

9. The system of claim 8, wherein the operation further includes tracking the monotonically incremental progress of the data transfer after identifying a read from or a write to the core local memory.

10. The system of claim 9, wherein tracking the monotonically incremental progress of the data transfer further includes incrementing a core local synchronization flag after identifying a read from or a write to the core local memory.

11. The system of claim 8 , wherein the operations further include selecting a memory access to barrier based on a scalar fence instruction.

12. The system of claim 8 , wherein the operations further include circular buffering for transfer of the data within the core local memory.

13. 11. The system of claim 8, wherein the operations further include terminating tracking progress of the transfer of the data when all the reads from the core local memory have been issued.

14. 11. The system of claim 8, wherein the operations further include finalizing the transfer of the data when all writes to the core local memory are committed.

15. 1. A program comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations for transferring data between an off-core memory and a core local memory, the operations comprising: identifying a read from said core local memory when said core local memory is a source of said data; Including, Reads are issued to the source in order and serviced by the source out of order; identifying a write to the core local memory when the core local memory is a destination for the data; writes are issued to the destination in order and committed by the destination out of order; accessing the off-core memory based on random scatter / gather memory accesses using indirect addresses from a register file or from the core local memory for reading from the off-core memory when the off-core memory is the source of the data, and for writing to the off-core memory when the off-core memory is the destination of the data.

16. The program of claim 15, wherein the operation further includes tracking the monotonically incremental progress of the data transfer after identifying a read from or a write to the core local memory.

17. 17. The program of claim 15 or 16, wherein the operations further include circular buffering for transfer of the data within the core local memory.

18. The program of claim 16, wherein tracking the monotonically incremental progress of the data transfer further includes updating a core local synchronization flag after identifying a read from or a write to the core local memory.

Images