Cooperative Instruction Prefetching in Multicore Systems

Abstract

Aspects of the present disclosure are directed to methods, systems, and apparatus that use an instruction prefetch pipeline architecture that provides good performance without the complexity of full cache coherent solutions deployed in traditional CPUs. The architecture can include components that can be used to construct an instruction prefetch pipeline, such as an instruction memory (TiMem), an instruction buffer (iBuf), a prefetch unit, and an instruction router.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. Patent Application No. 17 / 972,681, filed October 25, 2022, which claims the benefit of the filing date of U.S. Provisional Patent Application No. 63 / 281,960, filed November 22, 2021, the disclosures of which are incorporated herein by reference. [Background technology]

[0002] background A single instruction multiple data (SIMD) processor is a type of processor for processing multiple data inputs in parallel by performing the same operation on each input. The operations that are accelerated by a SIMD processor are determined in advance when the SIMD processor is designed. Summary of the Invention [Problem to be solved by the invention]

[0003] Adding instruction memory to each tile of a cross-lane processing unit (XPU) is costly, especially when the typical use case is single program multiple data (SPMD) running different programs simultaneously. Full cache coherent solutions, such as those deployed in central processing units (CPUs), are too complex to be cost-effective for XPUs. Providing each core with a private instruction memory that can hold all possible programs, or even a single large program, is impractical.

[0004] An alternative is to share a common instruction memory (TiMem) across all tiles, but have a small instruction buffer (iBuf) in each compute core, and prefetch instructions from TiMem to the iBuf as needed. All compute cores share this TiMem and access it simultaneously.

[0005] overview Aspects of the present disclosure include methods, systems, and apparatus that use an instruction prefetch pipeline architecture that provides good performance without the complexity of a full cache coherent solution deployed in a CPU. The architecture can include components that can be used to construct an instruction prefetch pipeline, such as an instruction memory (TiMem), an instruction buffer (iBuf), a prefetch unit, and an instruction router. [Means for solving the problem]

[0006] One aspect of the disclosure provides a hardware circuit. The hardware circuit includes a plurality of tiles, each tile configured to operate in parallel with other tiles in the plurality of tiles. Each tile of the plurality of tiles includes a processing core, a prefetch unit, and an instruction buffer. The hardware circuit further includes a plurality of data processing lanes configured to stream respective data from an upstream input to a downstream destination. The hardware circuit also includes a plurality of task instruction memories, each of the plurality of task instruction memories being arranged in sequence and coupled to one or more tiles of the plurality of tiles via an instruction router.

[0007] In one example, the task instruction memories are arranged in downstream sequence. In another example, each tile includes a tile access core and a prefetch unit included within each tile is included within the tile access core. In yet another example, each tile includes a tile execution core and a prefetch unit included within each tile is included within the tile execution core.

[0008] In yet another example, the hardware circuitry further includes an instruction broadcast bus and an instruction request bus. In yet another example, the instruction broadcast bus includes separate data lanes, where the number of separate data lanes corresponds to the number of task instruction memories. In yet another example, the instruction request bus includes separate data lanes, where the number of separate data lanes corresponds to the number of task instruction memories. In yet another example, instructions received by the task instruction memories are broadcast to all tiles linked on the instruction broadcast bus.

[0009] In yet another example, the prefetch unit is configured to provide requests to the at least one task instruction memory during a prefetch window. In yet another example, the instruction router includes a round robin arbiter configured to arbitrate requests including the prefetch read requests. In yet another example, the instruction buffer is configured to store instructions for a tile access core or instructions for a tile execution core.

[0010] In yet another example, the hardware circuitry further includes a task instruction memory access bus, the task instruction memory access bus including a read request bus, a read response bus, a write request bus, and a write response bus.

[0011] Another aspect of the disclosure provides a method of providing instructions by a single instruction multiple data (SIMD) processing device, the method including: one or more processors receiving instruction requests from multiple tiles of the SIMD processing device, the one or more processors filtering the instruction requests to eliminate duplicates of identical instruction requests and generating a first set of requests, the one or more processors generating an instruction set in response to the first set of requests, the one or more processors providing the instruction set from a computing device to a task instruction memory, the one or more processors storing the instruction set in the task instruction memory, and the one or more processors accessing instructions from the instruction set by a prefetch unit via an instruction router.

[0012] In one example, the receiving step occurs in a first processing clock cycle and the providing step occurs in a second processing clock cycle. In another example, the first processing clock cycle occurs prior to the second processing clock cycle. In yet another example, each tile of the plurality of tiles includes a processing core, a prefetch unit, and an instruction buffer.

[0013] Yet another aspect of the present disclosure provides a non-transitory computer-readable medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for providing instructions by a single instruction multiple data (SIMD) processing unit, the operations including receiving instruction requests from multiple tiles of the SIMD processing unit, filtering the instruction requests to eliminate duplicates of identical instruction requests and generate a first set of requests, generating an instruction set in response to the first set of requests, providing the instruction set from the computing device to a task instruction memory of the SIMD processing unit, storing the instruction set in the task instruction memory, and accessing instructions from the instruction set by a prefetch unit via an instruction router.

[0014] In one example, the receiving occurs at a first processing clock cycle and the providing occurs at a second processing clock cycle. In another example, the first processing clock cycle occurs prior to the second processing clock cycle. In yet another example, each tile of the plurality of tiles includes a processing core, a prefetch unit, and an instruction buffer. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a block diagram of an example hardware circuit for implementing a cross-lane processing unit (XPU) according to an aspect of the present disclosure. [Diagram 2] FIG. 2 is a block diagram of an example tile implementing an XPU according to an aspect of the disclosure. [Diagram 3] FIG. 2 is a block diagram of an example XPU according to an aspect of the disclosure. [Figure 4] 1 is a block diagram illustrating aspects of an example instruction router according to aspects of the present disclosure. [Diagram 5] 1 is a block diagram of an example XPU configuration for vector scan operations performed by an XPU in accordance with an aspect of the present disclosure. [Figure 6] FIG. 1 is a block diagram of an exemplary environment for implementing hardware circuitry according to aspects of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Detailed Description Generally disclosed herein is an implementation of an instruction prefetch pipeline architecture that can be used in an XPU to provide good performance without the complexity of a full cache coherent solution deployed in a CPU. An XPU can include multiple tiles, and each tile can be based on a decoupled access execute (DAE) architecture, for example, with a tile execute core (TEC) and a tile access core (TAC). For example, an 8-core XPU can run a total of 16 independent cores in parallel.

[0017] Aspects of the disclosed technology provide methods and systems related to creating a prefetch pipeline optimized for the single program multiple data (SPMD) aspect of the programming model to reduce the overhead of cold cache misses. Prefetch responses from any core are broadcast to all cores in the XPU. These prefetch responses are committed to the core's local cache. This allows other non-requesting cores to obtain a bundle of instructions or data before the core is available to process the instructions or data, avoiding missing process cycles altogether. Additionally, prefetch request filtering can be done on the arbitration path. The arbitration path is a logic and / or hardware-based path for arbitrating between requests, leading to Timem. This improves the bandwidth of Timem by avoiding fetching redundant requests.

[0018] 1 is a block diagram of a hardware circuit 101 implementing cross-lane processing units (XPUs) 101A-F according to an aspect of the disclosure. The hardware circuit 101 may include a first co-processor 103, a second co-processor 104, a high-bandwidth memory 107, and an on-chip interconnect 108. The co-processor 103 may include one or more tiles 102A-F, each tile implementing a respective XPU 101A-F. The first co-processor 103 may include a tile sequencer 106 configured to coordinate input and output data across the tiles 102A-F.

[0019] The first co-processor 103 is configured to perform data-dependent operations using the XPUs 102A-F. The XPUs 102A-F are connected to a series of data processing lanes, each lane streaming data through the XPUs 102A-F. The streamed data may be obtained from an on-chip memory 105, which may be any of a variety of different memory devices including main memory, cache, or persistent storage such as solid-state storage or hard disk storage. The streamed data may also be retrieved from the second co-processor 104, a high-bandwidth memory 107 serving one or both of the co-processors 103 and 104, and / or another data source connected to the hardware circuit 101 via an on-chip interconnect 108.

[0020] The second coprocessor 104 can be configured to accelerate certain operations such as matrix-matrix multiplication, matrix-vector multiplication, etc. In some examples, the second coprocessor 104 resides on a different device than the rest of the hardware circuitry 101 and communicates data to the hardware circuitry via an on-chip interconnect 108. The on-chip interconnect 108 may be a data bus or any form of interconnect according to any of a variety of communication standards, such as PCIe.

[0021] An exemplary input to the hardware circuit 101 is a tensor that represents input data and / or model parameters for a machine learning model executed using the hardware circuit 101. A tensor is a data structure that generalizes various other common data structure types of different dimensions. A tensor can contain zero or more elements that can be of one or more different data types, such as integers, floating-point values, Boolean values, etc. Within each data type, the data type can be parameterized according to a particular level of precision, such as 8-bit, 16-bit, or 32-bit integer or floating-point values. The dimension of a tensor is called its "rank." A tensor of rank 0 is a single element and is also called a scalar. A tensor of rank 1 is also called a vector. A tensor of rank 2 is also called a matrix. Vectors and matrices can also be said to have different ranks. For example, a vector of rank 2 is equivalent to a matrix. A tensor of non-zero rank can be described as a collection of tensors one rank lower. For example, a vector of rank 1 is a collection of scalar values, and a matrix of rank 2 is a collection of vectors of rank 1.

[0022] The hardware circuit 101 may at least partially implement a processing pipeline for training a neural network. The pipeline may include generating embeddings of input training examples. The embeddings allow features of the input training examples to be represented using less data, thus enabling more efficient processing. The feature tensors of the various input training examples may have different degrees of sparsity, affecting the amount of computation required to generate the corresponding embeddings. The hardware circuit may be configured to receive a tensor of feature values ​​representing the training input examples and generate the embeddings as a tensor having a lower rank than the feature tensor.

[0023] To generate the embeddings, coprocessor 103 is configured to implement various data-dependent operations to efficiently compute sparse data on XPUs 102A-F. These operations include sorting or summing sparse vectors, operations that summarize the contents of input vectors, and operations that convert sparse matrices from one sparse matrix storage format to another.

[0024] Instead of physical predefined circuits for accelerating the execution of data-dependent operations, XPUs 102A-F can be configured, e.g., programmed, to perform a variety of different data-dependent operations. Coprocessor 103 enables generalized support for processing sparse data, while complementary coprocessor 104 enables highly efficient, parallelizable operations.

[0025] The hardware circuitry 101 may be any of a variety of different types of processing devices, such as, for example, a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), or an application-specific integrated circuit (ASIC) such as a tensor processing unit (TPU). The hardware circuitry 101 may be implemented on a computing device, which itself may be part of a system of one or more devices, as described herein with reference to FIG.

[0026] 2 is a block diagram of an exemplary tile 102 implementing an XPU 101. The XPU 101 is coupled to a cross-lane controller 210. The cross-lane controller 210 provides a separate control thread that enables cross-lane instructions on the XPU 101. As described herein, the XPU can receive a first instruction, for example through one or more control signals, which can be translated into one or more second and third instructions and provided to the processing cells and crossbar of the XPU 101, respectively, to perform a compound operation specified by the first instruction. Instructions to the XPU 101 can be conveyed via control signals, which the processing cells and crossbar of the XPU 101 are configured to interpret to perform a corresponding primitive operation. An exemplary instruction can be an opcode of an instruction set architecture (ISA).

[0027] The tile 102 can receive data from the on-chip memory 105 as well as from the on-chip interconnect 108 as described with reference to FIG. 1. The XPU can also receive instructions from the instruction interface 224, e.g., from the tile sequencer 106 via the scalar core 212 or the scalar core 220. The scatter / gather controller 222 of the tile 102 can receive input data and control which data is passed to the memory 206 via the memory scheduler 214. The memory scheduler 214 coordinates how data is accessed and retrieved from the memory 206. The memory 206 is private to the tile 102 and is not accessible by other components connected to the tile 102, such as other tiles. The arbiter 204 is configured to manage which of the VPUs 202A-H accesses the memory 206, e.g., on a clock cycle basis. The tile 102 can maintain a task queue 208 of tasks to be executed by the tile 102. The tile 102 may also maintain a register of tile synchronization flags 218 and / or memory flags 216 to synchronize the tile 102 with other tiles in the hardware circuitry and memory 206, respectively.

[0028] 3 illustrates a logical diagram of the connections between blocks in an XPU in accordance with aspects of the disclosed technology. Through instruction prefetching, aggregating instruction requests, filtering, and keeping instructions or references in memory locations closest to the requesting processing unit, instructions can be made available to processing units or processing cores more quickly, improving system efficiency.

[0029] FIG. 3 illustrates aspects of a task instruction memory (Timem or Timem bank), an instruction buffer (iBuf), a prefetch unit (prefetch), and an instruction router. FIG. 3 illustrates a tile core 301 including a tile access core (TAC) 310 and a tile execution core (TEC) 320. TAC 310 may include a prefetch 311 and an iBuf 312. Similarly, TEC 320 may include a prefetch 321 and an iBuf 322. Also illustrated in FIG. 3 is a tile core 302 including a TAC 330 and a TEC 340 including a prefetch 311, an iBuf 332, and a prefetch 341 and an iBuf 342, respectively.

[0030] FIG. 3 further illustrates Timems 351 and 352 and an instruction router 360 that may be included within floorplan block 399. Timems 351 and 352 may store instructions locally for quicker access by each tile core, rather than the tile core requesting instructions from a location further downstream from Timem. Also illustrated is an instruction broadcast bus, which may broadcast instruction bundles to downstream floorplan blocks 399 and Timem banks therein. An instruction request bus 392 may aggregate requests for instructions from various components before requesting those instructions. The deserializers and serializers may deserialize or serialize instructions received from instruction broadcast bus 391 or may serialize instructions sent to instruction request bus 392.

[0031] A prefetch unit (such as prefetch 311 or prefetch 312) corresponding to a core can make a read request to Timem starting from the miss program counter (PC) (and overlay / task ID) until the end of the prefetch window. The prefetch window is a time period that can be selected by software using a register or other memory area. For example, the prefetch window can be defined by the prefetch_depth variable. Prefetch read requests from other tiles can be forwarded by adjacent floorplan block 399. These forwarded requests can be arbitrated with prefetch requests made by the prefetch unit of an adjacent tile core. For example, tile core 301 and tile core 302 may be adjacent to each other. In some examples, pairs of cores can be assigned to a single instruction request bus or a single instruction broadcast bus.

[0032] There can be many prefetch instruction request banks in an XPU. In some cases, there can be one bus per Timem bank, which can arbitrate independently of each other. Independent arbitration of the buses avoids head-of-line blocking between independent banks.

[0033] Requests sent from the prefetch window can be received by the instruction router 360. The instruction router 360 can filter selected requests to remove duplicates before forwarding them to another instruction router or to the target Timem bank. Filtering can potentially increase the bandwidth of instruction requests if the core is operating in SPMD mode.

[0034] Instructions read from the Timem bank can be broadcast to all tiles on the instruction broadcast bus. For example, there can be as many instruction broadcast buses as there are Timem banks. In some examples, instructions can be sent as instruction bundles. An instruction group consists of the instructions contained in a bundle. A bundle can be an instruction sequence that starts on an aligned "boundary". An instruction bundle can be serialized on a corresponding instruction broadcast bus for a fixed number of cycles of the processor or core. In some examples, such as during "steady state" operation of the system, the total bandwidth of the instruction broadcast bus can be two bundles per cycle. In this way, there is no backpressure on the instruction broadcast bus 392.

[0035] Instructions received on the broadcast bus are deserialized by the instruction router, which forwards one instruction to each Ibuf. In steady state, the system may need to sustain up to two writes from the prefetch interface and one read from the instruction fetch interface. The prefetch processes the received instruction and decides whether to commit it to an Ibuf or drop it.

[0036] Figure 4 illustrates additional aspects of instruction router 360. Shown in Figure 4 are round robin (RR) arbiter 410, daisy-chained round robin arbiter 420, round robin arbiter 430, filter 440, serializers 450 and 451, demultiplexer (demux) 460, and deserializers 471 and 472.

[0037] The instruction router 360 has an independent read request bus for each Timem bank in the system. Instruction bundles can be throttled at a rate matching the bandwidth of the instruction broadcast bus before being forwarded to an adjacent instruction router. In the following description, it can be assumed that deserialization and serialization can be performed before the request is presented to the instruction router 360.

[0038] The instruction router 360 can arbitrate between sources depending on the location of the core relative to the Timem bank. The instruction router 360 can be parameterized to select sources and destinations based on the instance for which the instruction router 360 arbitrates. The Demux 460 can be designed depending on the number of time banks or serializers it communicates with.

[0039] Instruction router 360 can arbitrate prefetch reads from the following exemplary sources: prefetch reads forwarded by instruction routers upstream or above instruction router 360, prefetch reads forwarded by instruction routers downstream of instruction router 360, and prefetch reads initiated by cores connected to instruction router 360.

[0040] The demux can choose top_pre_req or bottom_pre_req to arbitrate requests originating from cores connected to the instruction router. This arbitration can use a daisy-chained RR arbitration scheme. The daisy-chained round-robin arbiters 420 can grant every "x" cycles to match the bandwidth of the instruction broadcast bus. A request waiting for arbitration can be dropped if its PC matches a PC seen on the instruction broadcast bus. This can be thought of as the first level of filtering.

[0041] The winner of the daisy-chained arbitration may be processed differently based on the position of the instruction router 360 relative to the Timem bank. For example, if the Timem bank is below the instruction router, the winner of the daisy-chained arbitration may be forwarded to the "bottom" instruction router after passing through filter 440. If the Timem bank is above the instruction router 360, the winner of the daisy-chained arbitration is forwarded to the top instruction router 360 after passing through filter 440.

[0042] If the Timem bank is within the instruction router 360, the winner of the daisy-chain arbitration undergoes another level of arbitration, using requests forwarded by the lowest-ranked instruction router. In this case, there may be two daisy-chained networks arbitrating to reach the Timem bank. Depending on the location of the instruction router 360, the chain may become unbalanced. To ensure that fair access is provided to cores on either side of the chain, a modified RR arbiter can be used. As with the first level of arbitration, any requests matching the PC on the broadcast bus are dropped here. This can be thought of as a second level of filtering.

[0043] The overall winner from above is passed to a filter 440 which compares the incoming request to one of the other outstanding requests. If the request matches any of the outstanding requests, the request is dropped. This can be thought of as the third level of filtering.

[0044] Furthermore, the programmability of the system can be ensured since filtering at each point can be enabled / disabled by individual programmable or software controllable switches.

[0045] Referring back to Figure 3, the Timem access bus may be a bus that connects the system to all the Timem banks, allowing instruction bundles to be read and written to the Timem banks. The Timem access bus may include four buses: a read request bus, a read response bus, a write request bus, and a write response bus, as described in more detail below.

[0046] The read request bus can be a daisy-chained bus that can reach the Timem banks. Each Timem bank can forward a request to an adjacent Timem bank if the request is not addressed to it. If the request is addressed to a Timem bank, the request is serviced by the Timem bank.

[0047] The read response bus can be a daisy-chained bus that can transmit instruction bundles read from the Timem banks. Each Timem bank can round-robin arbitrate between the incoming instructions from adjacent banks and the instruction bundles from the current bank. Instruction bundles are serialized over "n" cycles, so bus grant is held for "n" cycles.

[0048] The write request bus can be a daisy-chained bus that can reach the Timem banks. Write requests can be serialized, for example, over two cycles. Each Timem bank forwards a flit to an adjacent bank if the request is not addressed to it. If the request is addressed to a Timem bank, the request is deserialized by the bank before being written to the Timem bank.

[0049] The write response bus can be a daisy-chain bus that relays write responses from the Timem banks. Each Timem bank arbitrates between the received response and the response from the current bank. A simple round-robin arbitration can be used to allow or provide one of the responses.

[0050] Read and write requests can include a "q" bit tag to encode up to 2^q outstanding read and write requests. These requests are returned by the bank as responses and are available to the entire system or to components that provide instructions to identify which request the response corresponds to.

[0051] If an endpoint cannot accept a request or response, the bus may develop "backpressure." If the bus cannot forward the commands or data it contains, a backlog of data may build up to be transmitted over the bus. Additionally, the bus may be backpressured due to arbitration losses. This may be acceptable to the overall system, since time-domain accesses are generally low-bandwidth accesses.

[0052] The tile instruction memory (Timem) may be shared by the tile cores described in Figure 3. An exemplary architecture of a Timem bank is shown in Figure 5. As shown in Figure 5, a bank structure 510 may operate in conjunction with an error correcting code (ECC) memory 520, a deserializer 540, a multiplexer, a demultiplexer, an output first-in-first-out memory space 550, and gates 531 and 532.

[0053] Timem can hold a set of programs that can be executed by the Tiled Access Core (TAC) and Tiled Execution Core (TEC). The Program Counter (PC) of each core is a physical offset into Timem. Timem is a software managed memory exposed to the direct memory access system. Software can enter programs into Timem using direct memory access and use the appropriate PC while issuing tasks to the SCT.

[0054] The entire tiled instruction memory can be sliced ​​into multiple banks that can be distributed among floorplan blocks 399. Each Timem bank uses cycle-by-cycle, last-recently-used (LRU) arbitration across one of three sources: (1) prefetch read requests from the instruction bus connected to the bank, (2) core write requests (arrived by direct memory access writes or Control Status Register (CSR) indirect write requests), and (3) core read requests (arrived by direct memory access reads or CSR indirect read requests).

[0055] The arbiter input is gated by the availability of output first-in first-out (FIFO) space corresponding to the requester. This FIFO holds the bundles read by the requester. In some examples, the instruction prefetch responses stored in the first-in first-out memory space 550 are popped or reset every x cycles, while the read response memory is popped every 2x cycles. In other examples, the depth or size of the FIFO memory can be increased to absorb bursts of information.

[0056] 6 is a block diagram of an exemplary environment 600 for implementing the hardware circuit 101. The hardware circuit 101 may be implemented on a device having one or more processors at one or more locations, such as a server computing device 615. The user computing device 612 and the server computing device 615 may be communicatively coupled to one or more storage devices 630 via a network 660. The storage device 630 may be a combination of volatile and non-volatile memory and may be in the same or different physical location as the computing devices 612, 615. For example, the storage device 630 may include any type of non-transitory computer-readable medium that can store information, such as a hard drive, solid-state drive, tape drive, optical storage, memory card, ROM, RAM, DVD, CD-ROM, writeable memory, read-only memory, etc.

[0057] The server computing device 615 may include one or more processors 613 and memory 614. The memory 614 may store information accessible by the processor 613, including instructions 621 that may be executed by the processor 613. The memory 614 may also include data 623 that may be retrieved, manipulated, or stored by the processor 613. The memory 614 may be a type of non-transitory computer-readable medium, such as volatile and non-volatile memory, that may store information accessible by the processor 613. The processor 613 may include one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), and / or application specific integrated circuits (ASICs), such as tensor processing units (TPUs). The processor 613 may include a coprocessor implemented as part of a hardware circuit, as described herein with reference to FIG. 1.

[0058] The instructions 621 may include one or more instructions that, when executed by the processor(s) 613, cause one or more processors to perform an action defined by the instructions. The instructions 621 may be stored in object code format for direct processing by the processor(s) 613, or in other formats including an interpretable script or collection of independent source code modules that are interpreted on demand or pre-compiled. The instructions 621 may include instructions for configuring an XPU of the hardware circuit 101 consistent with aspects of the disclosure. The server computing device 615 and / or the user computing device 612 may implement a compiler or other program to generate and send as control signals to the hardware circuit 101 instructions for configuring the XPU of the circuit.

[0059] Data 623 may be obtained, stored, or modified by processor 613 according to instructions 621. Data 623 may be stored in a computer register, relational or non-relational database, as a table with multiple distinct fields and records, or as a JSON, YAML, Proto, or XML document. Data 623 may also be formatted in a computer readable format, such as, but not limited to, binary values, ASCII, or Unicode. Additionally, data 623 may include sufficient information to identify related information, such as, for example, numbers, descriptive text, unique codes, pointers, references to data stored in other memory, including other network locations, or information used by a function to calculate the related data.

[0060] The user computing device 612 may be configured similarly to the server computing device 55 with one or more processors 516, memory 517, instructions 518, and data 519. The user computing device 612 may also include a user output 526 and a user input 524. The user input 524 may include any suitable mechanism or technology for receiving input from a user, such as a keyboard, a mouse, a mechanical actuator, a soft actuator, a touch screen, a microphone, and a sensor.

[0061] The server computing device 615 may be configured to transmit data to the user computing device 612, which may be configured to display at least a portion of the received data on a display implemented as part of the user output 526. The user output 526 may also be used to display an interface between the user computing device 612 and the server computing device 615. The user output 526 may alternatively or additionally include one or more speakers, transducers or other audio output, a haptic interface or other haptic feedback that provides non-visual and non-audible information to a platform user of the user computing device 612.

[0062] 6 illustrates the processors 613, 516 and memories 614, 517 as being within the computing devices 615, 612, the components described herein, including the processors 613, 516 and memories 614, 517, may include multiple processors and memories that may operate in different physical locations rather than within the same computing device. For example, some of the instructions 621, 518 and data 623, 519 may be stored on a removable SD card, while others may be stored in a read-only computer chip. Some or all of the instructions and data may be stored in a location physically separate from the processors 613, 516, but still accessible to the processors 613, 516. Similarly, the processors 613, 516 may include a collection of processors that may perform simultaneous and / or sequential operations. The computing devices 615, 612 may each include one or more internal clocks that provide timing information that may be used to time operations and programs executed by the computing devices 615, 612.

[0063] The server computing device 615 can be configured to receive requests to process data from the user computing device 612. For example, the environment 600 can be part of a computing platform configured to provide various services to users through various user interfaces and / or APIs that expose platform services. One or more services can be a machine learning framework or toolset for generating neural networks or other machine learning models according to a specified task and training data. The user computing device 612 can send or receive data specifying a type of workload or synthetic operation that the XPU of the hardware circuit 101 should be configured to perform. The user computing device 612 can send instructions directly to the hardware circuit 101 or can have the server computing device 615 generate and send instructions as control signals to the hardware circuit 101 as described herein.

[0064] The devices 612, 615 can communicate directly and indirectly through the network 660. The devices 615, 612 can set up listening sockets that can accept initiating connections to send and receive information. The network 660 itself can include a variety of configurations and protocols, including the Internet, the World Wide Web, intranets, virtual private networks, wide area networks, local networks, and private networks that use communication protocols proprietary to one or more companies. The network 660 can support a variety of short-range and long-range connections. The short-range and long-range connections can be implemented at different bandwidths, such as, for example, 2.402 GHz to 2.480 GHz (typically associated with the Bluetooth® standard), 2.4 GHz and 5 GHz (typically associated with the Wi-Fi® communication protocol), or using a variety of communication standards, such as the wireless broadband communication standard LTE. The network 660 can additionally or alternatively support wired connections between the devices 612, 615, including various types of Ethernet® connections.

[0065] 6 depicts a single server computing device 615 and user computing device 612, it is understood that aspects of the disclosure can be implemented according to a variety of different configurations and quantities of computing devices, including in serial or parallel processing paradigms, or on a distributed network of multiple devices. In some implementations, aspects of the disclosure can be performed on a single device, and any combination thereof.

[0066] Aspects of the present disclosure may be implemented as one or more computer programs in a digital circuit, a computer-readable storage medium, or as a combination of one or more of the foregoing. The computer-readable storage medium may be non-transitory, for example executable by a cloud computing platform, as one or more instructions stored on a tangible storage device.

[0067] In this specification, the phrase "configured to" is used in various contexts relating to computer systems, hardware and hardware circuits, or portions of computer programs, engines, or modules. When a system is said to be configured to perform one or more operations, this means that the system has appropriate software, firmware, and / or hardware installed and that, during operation, the system performs one or more operations. When a piece of hardware is said to be configured to perform one or more operations, this means that the hardware includes one or more circuits that, during operation, receive inputs and generate outputs corresponding to the one or more operations according to the inputs. When a computer program, engine, or module is said to be configured to perform one or more operations, this means that the computer program includes one or more program instructions that, when executed by one or more computers, cause the one or more computers to perform one or more operations.

[0068] Although the operations illustrated in the figures and recited in the claims are shown in a particular order, it is understood that the operations may be performed in an order different from that shown, and that some operations may be omitted, performed multiple times, or performed in parallel with other operations. Furthermore, the separation of different system components configured to perform different operations should not be understood as requiring the components to be separated. The components, modules, programs, engines described may be integrated as a single system or may be part of multiple systems.

[0069] Unless otherwise stated, the above alternatives are not mutually exclusive and may be implemented in various combinations to achieve unique advantages. These and other variations and combinations of the above-mentioned features may be utilized without departing from the subject matter defined by the claims, and therefore the above description of the embodiments should be construed as illustrative rather than limiting the subject matter defined by the claims. Furthermore, the provision of examples described herein, as well as clauses expressed as "such as," "including," and the like, should not be construed as limiting the subject matter of the claims to a particular example, but rather, the example is intended to illustrate only one of many possible implementations. Furthermore, the same reference numbers in different drawings may identify the same or similar elements.

[0070] With respect to the use of substantially any plural and / or singular term herein, e.g., "one element," "one or more elements," "multiple elements," "plurality of elements," "at least one element," etc. (where the term "element" is a substitute for system, component, data, etc.), one of ordinary skill in the art can translate from the plural to the singular and / or from the singular to the plural as appropriate to the context and / or application being described. The various singular / plural permutations may be expressly set forth herein for the sake of clarity and without limitation unless expressly indicated.

Claims

1. A hardware circuit comprising: a plurality of tiles, each tile configured to operate in parallel with other tiles in the plurality of tiles, each tile of the plurality of tiles: A processing core; A prefetch unit; an instruction buffer, the hardware circuit further comprising: a plurality of data processing lanes each configured to stream data from an upstream input to a downstream destination; a plurality of task instruction memories, each task instruction memory of the plurality of task instruction memories arranged in sequence and coupled to one or more tiles of the plurality of tiles via an instruction router.

2. The hardware circuit of claim 1 , wherein the task instruction memory is arranged in downstream order.

3. The hardware circuit of claim 1 , wherein each tile comprises a tile access core, and a prefetch unit included within each tile is included within the tile access core.

4. The hardware circuit of claim 1 , wherein each tile comprises a tiled execution core, and the prefetch unit contained within each tile is contained within the tiled execution core.

5. The hardware circuit of claim 1 further comprising an instruction broadcast bus and an instruction request bus.

6. The hardware circuit of claim 5 , wherein the instruction broadcast bus includes separate data lanes, the number of separate data lanes corresponding to a number of task instruction memories.

7. The hardware circuit of claim 5 , wherein the instruction request bus includes independent data lanes, the number of independent data lanes corresponding to a number of task instruction memories.

8. The hardware circuit of claim 7 , wherein instructions received by a task instruction memory are broadcast to all linked tiles on the instruction broadcast bus.

9. The hardware circuit of claim 1 , wherein the prefetch unit is configured to provide requests to at least one task instruction memory during a prefetch window.

10. 2. The hardware circuit of claim 1, wherein the instruction router comprises a round robin arbiter configured to arbitrate requests, including prefetch read requests.

11. The hardware circuit of claim 1 , wherein the instruction buffer is configured to store instructions for a tile access core or instructions for a tile execution core.

12. 2. The hardware circuit of claim 1, further comprising a task instruction memory access bus, the task instruction memory access bus including a read request bus, a read response bus, a write request bus, and a write response bus.

13. 1. A method of providing instructions by a single instruction multiple data (SIMD) processing device, the method comprising: receiving instruction requests from multiple tiles of the SIMD processing unit, by one or more processors; the one or more processors filtering the instruction requests to eliminate duplicates of identical instruction requests to generate a first set of requests; generating, by the one or more processors, a set of instructions in response to the first set of requests; providing the one or more processors with the set of instructions from a computing device to a task instruction memory; the one or more processors storing the instruction set in the task instruction memory; and the one or more processors accessing instructions from the instruction set with a prefetch unit via an instruction router.

14. 14. The method of claim 13, wherein the receiving step occurs in a first processing clock cycle and the providing step occurs in a second processing clock cycle.

15. 15. The method of claim 14, wherein the first processing clock cycle occurs before the second processing clock cycle.

16. The method of claim 13 , wherein each tile of the plurality of tiles includes a processing core, a prefetch unit, and an instruction buffer.

17. A computer program storing instructions which, when executed by one or more processors, cause said one or more processors to carry out the method of any one of claims 13 to 16.

Images