Queue Allocation in Machine Learning Accelerators

Abstract

This disclosure generally provides solutions for improving the performance of custom-built packet-switched TPU accelerator-side communication networks. Specifically, a set of solutions is described for improving flow control behavior by tuning packet buffer queues in on-chip routers within distributed training supercomputer networks.

Description

[Technical Field]

[0001] Priority claim This application claims the benefit of priority under 35 U.S.C. §119(e) to U.S. Provisional Application No. 63 / 091,708, filed October 14, 2020.

[0002] Technical Field The present disclosure relates to allocating shared memory for traffic queuing in a machine learning accelerator communication network. [Background technology]

[0003] background An electronic device may be composed of multiple different computing units that need to communicate data between them in order for the electronic device to operate. Data communication between the computing units may be non-deterministic. For example, data communication may be subject to variable latency between the time of transmission at one unit and the time of reception at another unit. That is, the time it takes for data to travel from one unit to another is not constant, but is subject to many different sources of variation in transmission time. Summary of the Invention

[0004] overview Generally, the present disclosure includes a method for memory allocation in a machine learning accelerator communication network. Generally, the method includes accessing metadata associated with multiple communication ports of an application-specific integrated circuit (ASIC). The metadata identifies, for each port, whether the particular port is used in the current configuration and the communication medium associated with the particular port. Based on the metadata, an expected latency is determined for each port, and a portion of shared memory is allocated to each port. The shared memory is allocated by allocating zero memory to unused ports, determining a memory allocation for each port based on the expected latency, and assigning a start address and a stop address of the shared memory to each port. Implementations may optionally include one or more of the following features.

[0005] In some implementations, assigning a start and stop address of the shared memory to each port involves calling an application programming interface (API) by a device different from the ASIC.

[0006] In some implementations, the process is executed on an ASIC using a machine learning accelerator communication network and allocated shared memory. In some implementations, the process is to train a neural network.

[0007] In some implementations, the ASIC is a tensor processing unit (TPU).

[0008] In some implementations, the communication medium identified in the metadata is a copper cable medium, an optical medium, a printed circuit board (PCB) medium, or any combination thereof.

[0009] In some implementations, the memory is

[0010]

number

[0011] It is allocated according to In an alternative implementation, a method for memory allocation in a machine learning accelerator communication network includes determining a network topology for a network of machine learning accelerator ASICs. Metadata associated with a plurality of communication ports of each ASIC in the network is accessed. For each port of the plurality of ports, the metadata identifies whether the particular port is used in the current configuration and the communication medium associated with the particular port. For each port used in the network topology, a round-trip time (RTT) delay is determined. A portion of shared memory is allocated to each port of the plurality of ports by determining a memory allocation for each port proportional to the RTT delay and assigning a start address and a stop address of the shared memory to each port for the determined memory allocation. A process is executed on the machine learning accelerator to send profiling traffic to the network for a predetermined duration. For each port, the number of received traffic packets is determined, and a portion of the shared memory is reallocated to each port of the plurality of ports, the reallocation being performed by determining a memory allocation for each port proportional to the received packet count, and reallocating a start address and a stop address for each port of the determined memory allocation.

[0012] In some implementations, assigning and reassigning the start and stop addresses of the shared memory to each port involves calling an API by a device different from the ASIC.

[0013] In some implementations, the memory allocation proportional to the determined RTT delay for each port is calculated using the formula

[0014]

number

[0015] is determined in accordance with In some implementations, memory allocation proportional to received packet count for each port is calculated using the formula

[0016]

number

[0017] is determined in accordance with In some implementations, the RTT delay is calculated before run time by sending and receiving one or more timing messages to determine the latency.

[0018] In some implementations, the profiling traffic includes all-to-all traffic, nearest neighbor traffic, synthetic traffic profiles, or any suitable combination thereof.

[0019] In an alternative implementation, a method for allocating memory for a machine learning accelerator communication network includes determining a network topology for a network of machine learning accelerator application-specific integrated circuits (ASICs). Metadata associated with a plurality of communication ports of each ASIC in the network is accessed. For each port of the plurality of ports, the metadata identifies whether the particular port is used in a current architecture and a communication medium associated with the particular port. For each port of the plurality of ports, the metadata identifies whether the particular port is used in the current configuration and the communication medium associated with the particular port. For each port used in the network topology, a round-trip time (RTT) delay is determined. A portion of shared memory is allocated to each port of the plurality of ports by determining a memory allocation for each port proportional to the RTT delay and assigning a start address and a stop address of the shared memory to each port for the determined memory allocation. A process is executed on the ASIC, and the process uses the machine learning accelerator communication network with the allocated shared memory. During execution of the process, a number of message packets received at each port of the plurality of ports over a first time period is determined. A desired portion size of the shared memory is determined for each port of the plurality of ports based on the number of message packets received at each port. During a second time period, the process is paused and it is determined whether the shared memory is clear of pending message packets. If it is determined that the shared memory is clear, start and stop addresses of the shared memory for each port are reallocated according to the desired portion size, and execution of the process is resumed.

[0020] Various implementations provide one or more of the following advantages: For example, in some implementations, a machine learning hardware accelerator may run faster and avoid bottlenecks in the accelerator's on-board communication network by more efficiently allocating message buffer queues.

[0021] The details of one or more implementations of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 is a schematic diagram illustrating an exemplary machine learning accelerator according to an implementation of the present disclosure. [Figure 2] FIG. 1 illustrates an exemplary shared memory with buffer queues allocated for port-to-port traffic. [Figure 3] 1 is a flowchart of an example process for allocating shared memory based on expected latency, consistent with an implementation of the present disclosure. [Figure 4] 1 is a flowchart of an example process for allocating shared memory based on profiling traffic execution, consistent with an implementation of the present disclosure. [Figure 5] 1 is a flowchart of an example process for allocating shared memory based on traffic during runtime, consistent with an implementation of the present disclosure. [Figure 6] FIG. 2 is a schematic diagram illustrating an example of a special purpose logic chip that may be used in the multi-chip system of FIG. [Figure 7] FIG. 7 illustrates an exemplary neural network (NN) computational tile 700 that may be used in an ASIC. DETAILED DESCRIPTION OF THE INVENTION

[0023] Detailed Description Generally, the present disclosure relates to allocating shared memory among different channels of communication or ports in a high-speed communication network, for example, a communication network of a machine learning accelerator including multiple application specific integrated circuits (ASICs). Deep learning training requires distributed parallel processing. Distribution may be partitioning large amounts of training data into different replicas (e.g., data parallelism) or partitioning very large models into smaller modules (e.g., model parallelism). The partitioned training data and model parameters are placed on different processing units for simultaneous computation.

[0024] Distributed training is performed in a synchronous, iterative, and incremental loop. Under data parallelism, each processing unit ingests a mini-batch of data at each step, computes local gradients, and then exchanges all local gradients across the network of computational units in an all-reduce manner to compute a final globally consistent gradient, which updates the model weights at the end of the step.

[0025] Under model parallelism, each processing unit obtains model activation inputs from its local training data or from the output of another processing unit operating on its previous hidden layer. The processing unit then computes activation outputs that may be the final model output or may serve as activation inputs for another processing unit. Gradients are computed on the processing unit containing the final layer and sent back to previous layers to update the partitioned submodels. This process can be pipelined to operate on successive mini-batches. Under this approach, intermediate activation outputs are sent around the network, and gradients at model split boundaries are also sent.

[0026] In practice, data and model parallelism can be combined to achieve the highest performance. For an exemplary model with hundreds of billions of weight parameters, enormous amounts of computational resources and communication are required to converge the model to the required level of accuracy.

[0027] To speed up the training process, custom-built accelerator chips, ASICs such as Tensor Processing Units (TPUs), are designed to function as processing units to accelerate deep learning calculations. In some implementations, chip types (e.g., FPGAs, GPGPUs, or CPUs) are used. Meanwhile, co-designed high-speed communication networks between accelerators are also built to speed up communication between the processing units. Together, the training system can provide exaFLOP-level computational performance equivalent to current-state supercomputers.

[0028] The custom-built network of TPU accelerators is designed for simplicity, reducing network processing overhead. Once the basic functions of an interconnected network, such as addressing, error detection and correction, routing and flow control, are completed, the bulk of the network processing is performed on the accelerator chip hardware, accelerating processing.

[0029] This disclosure focuses on improving the performance of custom-built packet-switched TPU accelerator-side communication networks. Specifically, a set of solutions is proposed to improve flow control behavior by tuning packet buffer queues in on-chip routers in distributed training supercomputer networks.

[0030] The solutions disclosed herein apply to the synchronous data-parallel and model-parallel training patterns discussed above, and may also be applied to asynchronous distributed training in accelerator networks in general.

[0031] FIG. 1 is a schematic diagram illustrating an exemplary machine learning accelerator 100 according to an implementation of the present disclosure. This example shows a network of TPUs interconnected through on-chip network stack processing hardware modules. In this example, each TPU 106 has six network ports, which are linked together through buffer queues to form an intra-chip crossbar. In some implementations, each TPU 106 may have more or fewer than six ports (e.g., 10, 3, or other numbers). In some implementations, the links between the ports may be further divided into multiple virtual channels of communication. Furthermore, which ports are connected may be determined at the software level prior to execution of a job within the TPU network. For example, while FIG. 1 shows a number of TPUs 106 in a two-dimensional configuration, it is possible to have a three-dimensional (e.g., spherical) configuration or other geometry based on the connections between the TPU ports.

[0032] The accelerator-side network stack for the TPU 106 is implemented in an on-chip hardware processing module or router 102 on each TPU 106. Communications during distributed training are formatted as multiple network packets. In operation, packets between a source TPU chip and a destination TPU chip pair hop over one or more TPU chips as intermediate steps before reaching their final destination. At each TPU chip hop, the on-chip network stack interfaces with the chip's various network ports and forwards packets received from one port to another as dictated by a user-specified routing algorithm. Port-to-port transmissions within the chip are carried over in buffer queues. The cross-connected buffer queues within the chip form a crossbar for high-speed chip-level network processing.

[0033] Flow control can be implemented between each pair of directly connected TPUs 106 in a supercomputer network or machine learning accelerator 100, preventing, for example, a faster sending TPU 106 accelerator from overwhelming a slower receiving TPU 106 accelerator. As the TPU network begins to forward activation output and weight gradient traffic across different accelerators, buffer queues in each chip's on-chip network stack constantly input packets from input ports to output ports, with buffer space utilization dynamically changing depending on the traffic ingress and egress rates at the TPU chip 106. When a buffer queue corresponding to a port is fully utilized, no packets can be received at that port because there is no buffer space remaining, and the upstream TPU accelerator should stop sending messages. The sender is regulated by timely sending flow control messages that inform the sender of the amount of buffer queue space available at the receiving port, hereinafter referred to as "credit." The sender should not issue messages larger than the available credits to receive. In some implementations, the link between any two connected TPU accelerators 106 is bidirectional, and credit messages are sent bidirectionally as well.

[0034] The TPU accelerator-side communication network may be a heterogeneous network environment. In other words, communication times between ports may vary from port to port and from TPU 106 to TPU 106. In addition, the amount of traffic distributed across ports may not be uniform, and therefore, intelligent and dynamic message queuing may significantly improve communication in the TPU communication network.

[0035] This heterogeneity can be caused by several different factors. For example, for large-scale deep learning training supercomputers scaling to thousands of TPU accelerators 106, a variety of link media may be used to wire the entire network. Exemplary media used in the links include, but are not limited to, PCB traces utilized for TPU accelerators 106 soldered on the same tray to exchange data, copper cables used for lower-cost short-range communications, and optical cables used for long-range communications. Different media materials exhibit heterogeneous link performance in terms of throughput and capacity, which impacts queue utilization in the router.

[0036] In addition, large-scale supercomputer networks inevitably involve links with different lengths. Thousands of TPU accelerators 106 span at least multiple racks, leading to intra-rack cables with shorter distances and lower latencies, and inter-rack cables with longer distances, longer latencies, and greater variations between different links. Latency affects buffer queue utilization. These cable links have different quality levels after a period of use, measured by bit-error-rate (BER). The varying BER causes packet transmission retry behavior to differ across all links, leading to heterogeneous link pressure and queue saturation rates.

[0037] TPU network ports are often not used uniformly depending on the topology of the distributed training job. For example, if a user training job requests TPU resources in a 2D mesh topology among all TPU resources available on a supercomputer, TPUs on the edge of the mesh will not use all ports, and ports that are not inward-facing in the mesh will connect to adjacent accelerators but will not receive traffic compared to other actively transmitting ports.

[0038] Large-scale supercomputer networks can inherently provide fault tolerance by routing around faulty chips, hosts, or inter-TPU link media. Fault-tolerant routing can cause imbalances in link traffic pressure, even when the non-faulty topology is originally homogeneous. TPUs 106 with all ports actively in use and connected to media of the same type, distance, and quality may experience highly heterogeneous traffic, as some ports are on more critical routes that go around the faulty site. Furthermore, accelerators adjacent to the faulty site exacerbate the imbalanced port usage if ports directly connected to the faulty chip or link do not receive traffic.

[0039] The distributed training approach itself introduces heterogeneity. In data-parallel models, all TPUs 106 exchange the same amount of local gradients with each other, but more advanced techniques like model partitioning and model parallelism make the gradients and activation outputs exchanged on the network inherently heterogeneous across partition boundaries. Because inference on partitioned models often depends on the output from another partition, communication latency, rather than throughput, becomes more important in model parallelism. In addition, reinforcement learning training also makes TPU network traffic heterogeneous, with a group of TPU workers 106 acting as agents performing inference and another group of TPU workers 106 running classical iterative training loops. Traffic within and between these groups is inherently different. Therefore, dynamic queue allocation to address this heterogeneity is desirable.

[0040] Each TPU 106 includes multiple communication ports, channels, and on-chip hardware processing modules (routers 102), as well as a shared programmable buffer queue storage structure 104. The buffer queues for each port may be software controlled to different depths and are stored in shared random access memory (RAM), e.g., static RAM (SRAM) or dynamic RAM (DRAM). In some implementations, each port of the TPU 106 is assigned a starting address and an ending address in the shared RAM 104 to identify that particular port's queue region within the shared memory space.

[0041] Sizing the buffer queues associated with the various network ports of the TPU accelerator 106 is critical in tuning the overall training communication throughput. For a given link that receives a large amount of traffic, allocating too little buffer queue space for that port at the receiving TPU 106 can lead to the sending TPU 106 frequently stopping its packet transmission, slowing down traffic and potentially creating a bottleneck for synchronous training communication. Each queue can be further divided into several virtual channels for per-channel queue allocation. While the interconnection links are bidirectional, every queue corresponds to a unidirectional link between two accelerators.

[0042] FIG. 2 illustrates an exemplary shared memory 200 with buffer queues allocated for port-to-port traffic. Traffic in the buffer queues can be egress-to-port traffic (e.g., traffic generated from a TPU and outbound to a different TPU), port-to-ingress traffic (e.g., traffic generated outside a particular TPU and ingested by a particular TPU), or port-to-port traffic (e.g., traffic passing through a TPU in a TPU network). Furthermore, each port can be subdivided into multiple virtual channels, each providing a unique communication path. While FIG. 2 illustrates only port-to-port queues allocated in the shared memory, all three types of traffic, or any suitable combination thereof, can be present and are within the scope of this disclosure. In the illustrated example, each queue's FIFO (First In, First Out) pointer uses its start and end addresses as a reference for pushing and popping data from the queue. The port-to-ingress and egress-to-port queues follow the same addressing scheme, with fewer queues sharing RAM capacity. The same mechanism applies to accelerators with more or fewer ports than shown.

[0043] These start and stop addresses (e.g., address 0 and address 1 as shown) can be controlled or assigned via software. A set of application programming interfaces (APIs) are exposed through the TPU accelerator device driver to write to PCIe config space registers that specify the size of each buffer queue. The APIs convert the queue sizes to the start and end addresses of each queue for hardware operation. The disclosed solution can call these APIs to dynamically configure and allocate queue sizes for all ports in the TPU communication network.

[0044] 3 is a flowchart of an exemplary process for allocating shared memory based on expected latency according to an implementation of the present disclosure. The buffer queue allocation process examines actively used ports in (sub)networks participating in the training process and allocates queue size (credits) according to the weight of each port's upstream round-trip latency. This scheme takes into account heterogeneity of link media, inactive ports in non-homomorphic networks, and inactive ports directly connected to faulty links or faulty TPU accelerator chips. This scheme is divided into two steps: offline link calibration and online credit allocation.

[0045] During offline link calibration, the average round-trip time (RTT) for different link mediums used in the TPU communication network is determined (302). This may be determined based on historical values, ping operations, or other methods to provide a general expected latency for a given link based on the medium of that link. For example, a PCT trace may have a lower expected latency than an optical cable.

[0046] During the online credit allocation phase, topology discovery may be performed to determine the network topology of the TPU communication network needed or required for a particular job to be executed 304. Topology discovery may identify which TPUs will participate in the upcoming job and which ports are available on each TPU.

[0047] After the network topology is determined, metadata providing information about the communication ports used in the TPU communication network for the job may be accessed 306. For example, each TPU may maintain metadata in on-chip memory that identifies the port ID of each port, whether the port is in use in the current topology, the medium (e.g., copper, optical, PCB, etc.) associated with each port, and the cable length associated with the port.

[0048] The accessed metadata, in combination with the expected latency determined during offline calibration, may be used to determine a predicted or expected RTT delay for each port, which may then be used to allocate a queue buffer in shared memory for each port according to the expected RTT delay or latency (308). Queue sizes are calculated for each port in each TPU by the TPU itself. The TPU first assigns a queue size of zero to any unused ports and then allocates the remaining queue size for each port. In some implementations, the queue size calculation process is performed independently and simultaneously for all TPUs for pending jobs in the TPU communication network. It indexes into the offline calibrated link latency table and calculates the queue size for each port proportional to the round trip time of the input port. In some implementations, the queue size is calculated via the following formula, where p is the port being calculated and N is the total number of actively used ports on the chip:

[0049]

number

[0050] Once the queue size for each port and channel is determined, a starting address and an ending address of the shared memory can be assigned to each port according to the size allocation (310). In some implementations, this is accomplished using one or more APIs provided by the TPU architecture. For example, the APIs can write to PCIe config space registers that specify the size of each buffer queue. The APIs convert the queue sizes to starting and ending addresses of each queue for hardware operation. The disclosed solution can call these APIs to dynamically configure and allocate queue sizes for all ports in the TPU communication network.

[0051] Once the allocation of addresses to the shared memory is complete, the TPU communication network may be enabled and the inference process or training run may begin 312. The inference process may be a long running process intended to train or weight a neural network for future machine learning outputs.

[0052] 4 is a flowchart of an example process for allocating shared memory based on a profiling traffic run, according to an implementation of the present disclosure. By performing a profiling traffic run, a more authentic representation of the traffic demands during job execution may be determined. In some implementations, the process for allocating shared memory based on a profiling traffic run begins by completing the process as described with respect to FIG. 3 (e.g., 302-310), where an initial allocation is given to a traffic queue.

[0053] Once the initial buffer queues are allocated, one or more message traffic profiles may be selected for a profiling run on the network (402). Exemplary profiles may include, but are not limited to, all-to-all, nearest-neighbor traffic, or another synthetic pattern generated by a machine learning model (e.g., a model-partitioned ResNet-50 or a bidirectional encoder representation from transformer (BERT) model). In some implementations, all or a combination of previous traffic profiles are combined for the profiling run. Before starting the profiling run, software clears hardware performance counter registers in each TPU, effectively resetting them to zero. The hardware TPU performance counter registers record the number of packets successfully received on each channel of each port of the TPU. The profiling traffic is injected into the TPU communication network, and the number of packets successfully received on each port of the TPU is counted by hardware registers (404). A separate software system reads the counter registers (e.g., via an API) and determines the total number of packets received for each port (and channel) during the profiling run. In some implementations, the profiling run is significantly shorter than the actual training run that is performed. For example, the profiling run may be on the order of minutes or hours, while the training job may take days or weeks.

[0054] Once the profiling run is complete, the number of packets received on each channel of each port can be used to determine the queue size for that port (406). In some implementations, the queue size is determined using the following formula, where p is the port, N is the total number of ports on the ASIC, and packet_num is the number of packets received during the profiling run. Note that the following formula yields a value of zero for unused ports, and therefore unused ports are allocated a queue size of zero.

[0055]

number

[0056] Once the queue size for each port and channel is determined, a starting address and an ending address of the shared memory can be assigned to each port according to the size allocation (310). In some implementations, this is accomplished using one or more APIs provided by the TPU architecture. For example, the APIs can write to PCIe config space registers that specify the size of each buffer queue. The APIs convert the queue sizes to starting and ending addresses of each queue for hardware operation. The disclosed solution can call these APIs to dynamically configure and allocate queue sizes for all ports in the TPU communication network.

[0057] Once the allocation of addresses to the shared memory is complete, the TPU communication network may be enabled and the inference process or training run may begin 312. The inference process may be a long-running process intended to train or weight a neural network for future machine learning outputs.

[0058] Profile-guided queue adjustment can reflect packet retransmissions caused by poor link quality, unbalanced link traffic caused by routing algorithms, or faulty chips / links. Therefore, it may be a more optimal scheme to allocate more credits for ports / links that receive more traffic and require more buffer space to forward messages from one port to another.

[0059] 5 is a flowchart of an example process for allocating shared memory based on traffic during execution, according to an implementation of the present disclosure. The process described in FIG. 5 may begin similarly to the processes of FIGS. 4 and 3 (e.g., 302 through 312). In some implementations, some or all of the processes occur sequentially. For example, the system may first allocate memory based on predicted latency as described with respect to FIG. 3, then perform a profiling run and reallocate memory based on the profiling run as described with respect to FIG. 4, and finally, during the execution of a training run, further reallocate memory according to the process described below with reference to FIG. 5. Any suitable combination of these processes is contemplated by the present disclosure.

[0060] Once the initial queue allocation is achieved (via either or both of the processes described in FIG. 3 or FIG. 4), the training job may begin execution. During execution, the controller may monitor the packet count for each port, as recorded by the hardware TPU performance counter registers as described above. Initially, at the beginning of execution, per-port packet statistics and hardware counters may be cleared (502). The training job may run for a predetermined amount of time, and then the queues may be adjusted based on the packet count for each port during job execution. In some implementations, the message queue is reallocated periodically during execution or at a set number of times (e.g., once per hour for the first five hours of execution). In some implementations, the message queue is reallocated more frequently during the start of the training job and less frequently as execution progresses (e.g., every 10 minutes for the first hour, then every hour for the next five hours, and then once per day thereafter). In some implementations, the time period is configured by the user on a case-by-case basis.

[0061] If a queue should be reallocated after a predetermined time has elapsed, packet statistics for each port are read from the hardware TPU performance counter registers 504. The desired queue size is calculated 506. In some implementations, the desired queue size is calculated using a formula such as that described with reference to FIG.

[0062] Once the queue sizes are determined, execution of the training job may be paused, allowing the TPUs to fully ingest and process any pending message traffic and clear their queues (508). In some implementations, the controller verifies that the queues are clear (e.g., the shared RAM has no pending messages for each TPU) before proceeding (510). In some examples, the pause lasts for a predetermined amount of time, and if the shared RAM is not clear (512), execution of the training job resumes (516). In these examples, the controller may retry the reallocation at a later point in time. If the shared RAM is determined to be clear, the queues in the shared RAM are each assigned new start and stop addresses (e.g., via the API discussed above) (514). The controller may then clear per-port packet statistics stored in hardware TPU performance counter registers, and the training job then resumes execution (502). This process may be repeated as necessary during the execution of the training job.

[0063] Figure 6 is a schematic diagram illustrating an example of a special-purpose logic chip that may be used in the machine learning accelerator of Figure 1. Figure 6 illustrates a block diagram of an ASIC used in the machine learning hardware accelerator as an exemplary computing system 600 for accelerating tensor calculations associated with deep neural networks (DNNs). System 600 may be, for example, ASIC 102 as described with reference to Figure 1. System 600 generally includes a controller 602, a host interface 608, an input / output (I / O) link 610, a plurality of tiles including a first tile set 612 and a second tile set 614, a classifier portion 616, and a data bus identified in a bus map 618 (shown for clarity but not included in system 600). Controller 602 generally includes a data memory 604, an instruction memory 606, and at least one processor configured to execute one or more instructions encoded on a computer-readable storage medium. The instruction memory 606 may store one or more machine-readable instructions that are executable by one or more processors of the controller 602. The data memory 604 may be any of a variety of data storage media for storing and subsequently accessing various data related to the computations occurring within the system 600.

[0064] Controller 602 is configured to execute one or more instructions related to tensor computation within system 600, including instructions stored in instruction memory 606. In some implementations, data memory 604 and instruction memory 606 are volatile memory units. In some other implementations, data memory 604 and instruction memory 606 are non-volatile memory units. Data memory 604 and instruction memory 606 may also be another form of computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration. In various implementations, controller 602 may be called or referred to as core manager 602.

[0065] As shown, host interface 608 is coupled to I / O link 610, controller 602, and classifier portion 616. Host interface 608 receives instructions and data parameters from I / O link 610 and provides the instructions and parameters to controller 602. Generally, instructions may be provided to one or more devices in system 600 via instruction bus 624 (described below), and parameters may be provided to one or more devices in system 600 via ring bus 628 (described below). In some implementations, instructions are received by controller 602 from host interface 618 at an initial time and stored in instruction memory 606 for execution by controller 602 at a later time.

[0066] The classifier portion 616 is similarly coupled to the controller 602 and tile 7 of the second tile set 614. In some implementations, the classifier portion 616 is implemented as a separate tile within the system 600. In alternative implementations, the classifier portion 616 is located or installed within the controller 602 as a subcircuit or subdevice of the controller 602. The classifier portion 616 is generally configured to perform one or more functions on accumulated pre-activation values ​​received as the output of a fully connected layer. The fully connected layer may be partitioned across tiles in the tile sets 612 and 614. Thus, each tile is configured to generate a subset of pre-activation values ​​(i.e., linear outputs) that may be stored in the tile's memory unit. A classification result bus 620 provides a data path from the classifier portion 616 to the controller 602. Data including post-function values ​​(i.e., results) are provided from the classifier portion 616 to the controller 602 via the classification result bus 620.

[0067] The bus map 618 illustrates a data bus that provides one or more interconnected data communication paths between tiles in the first tile set 612 and the second tile set 614. The bus map 618 provides a legend for identifying the classification result bus 620, the CSR / master bus 622, the instruction bus 624, the mesh bus 626, and the ring bus 628, as shown in FIG. 6 . Generally, tiles are the core components within the accelerator architecture of the system 600 and are the focal point for tensor computations occurring in the system. Each tile is an individual computational unit that cooperates with other tiles in the system to accelerate computations across one or more layers of a multilayer neural network. While tiles in the tile sets 612, 614 may share the execution of tensor computations associated with a given instruction, individual computing units are self-contained computational components configured to perform a subset of tensor computations independently of other corresponding tiles in the tile sets 612, 614.

[0068] The CSR bus 622 is a single master multiple slave bus that allows the controller 602 to send one or more commands to set program configurations and read status registers associated with one or more tiles. The CSR bus 622 can be connected in a single daisy chain configuration with one master bus segment and multiple slave bus segments. As shown in Figure 6, the CSR bus 622 provides a communication coupling through a bus data path connecting tiles in the controller 602 and tile sets 612, 614 in the ring to the host interface 610. In some implementations, the host interface 610 is a single master of the CSR bus ring, and the entire CSR bus address space is memory mapped to memory space at the host interface 610.

[0069] The CSR bus 622 may be used by the host interface 610 to perform one or more operations, including, for example, programming memory buffer pointers in the controller 602 to enable the controller 602 to begin fetching instructions from the instruction memory 606, updating / programming various tile settings that remain static during one or more calculations (e.g., coefficient tables for polynomial approximation calculations), and / or loading / reloading firmware into the classification portion 616. In one example, a firmware reload may include a new function to be applied to the linear output (i.e., pre-activation value). Thus, every slave having access to the CSR bus 622 will have a distinct node identifier (node ​​ID) that is tied to and identifies the slave. The node ID is part of the instruction address and will be used, inspected, or otherwise interrogated by the CSR slave (i.e., the controller 602, tiles 612, 614, and classifier 616) to determine whether a CSR packet is addressed to the slave.

[0070] In some implementations, one or more instructions may be transmitted by the host interface 602 through the controller 602. The instructions may be, for example, 32 bits wide, with the first 7 bits containing header information indicating the instruction address / destination where the instruction should be received and executed. The first 7 bits of the header may contain a data parameter representing a specific node ID. Thus, a slave on the CSR bus ring (e.g., each tile) may inspect the header of the instruction to determine whether the request by the master (host interface 610) was addressed to the tile inspecting the header. If the node ID in the header does not indicate that the destination is the inspection tile, the inspection tile copies the incoming CSR instruction packet to the CSR bus input connected to the next tile for inspection by the next tile.

[0071] The instruction bus 624 further provides a communication coupling through a bus data path that originates at the controller 602 and, similar to the CSR bus 622, connects the tiles in the tile sets 612, 614 in the ring back to the controller 602. In one implementation, the controller 602 broadcasts one or more instructions over the instruction bus 624. The instructions broadcast by the controller 602 may differ from the instructions provided over the CSR bus 622. However, the manner in which tiles receive and / or consume or execute instructions received over the bus 624 may be similar to the process for executing instructions received over the CSR bus 622.

[0072] In one example, the header of the instruction (i.e., a bitmap) indicates to the receiving tile that it needs to consume a particular instruction based on the bitmap associated with the instruction. The bitmap may have a particular width defined in terms of bits. Instructions are typically transferred from one tile to the next based on the parameters of the instruction. In one implementation, the width of the instruction bus 624 may be configured to be smaller than the size / width of the instruction. Thus, in such a configuration, the transmission of an instruction spans several cycles, and a bus stop on the instruction bus 624 will cause the decoder to place the instruction received at the tile into the appropriate target instruction buffer associated with that tile.

[0073] As described further below, tiles in tile sets 612, 614 are generally configured to support two broad categories of instructions. The two broad categories may also be referred to as instruction types. The instruction types include tensor operation (TensorOp) instructions and direct memory access (DMAOp) instructions. In some implementations, DMAOp instructions have one or more specializations that are allowed to be simultaneous. The one or more specializations may be referred to as DMAOp instruction subtypes or Op codes. In some cases, each unique and / or valid DMAOp instruction type / subtype tuple will have a separate instruction buffer within a particular tile.

[0074] At a particular one of the tiles 612, 614, the bus stop associated with the instruction bus 624 examines the header bitmap to determine the instruction type / subtype. The instruction may be received by the tile and then written to the tile's instruction buffer prior to execution of the instruction by the tile. The tile's instruction buffer to which the instruction is written may be determined by the instruction's type and subtype indicator / fields. The instruction buffer may include a first-in, first-out (FIFO) control scheme that prioritizes consumption of one or more associated instructions. Thus, under this FIFO control scheme, instructions of the same type / subtype will always be executed in the order in which they arrive on the instruction bus.

[0075] The different instruction buffers within a tile are the TensorOp instruction buffer and the DMAOp instruction buffer. As indicated above, instruction types include TensorOp instructions and DMAOp instructions. With respect to the DMAOp instruction, instruction subtypes (indicating "write" buffer locations) include 1) mesh inbound instruction buffer, 2) mesh outbound instruction buffer, 3) narrow-wide DMA instruction buffer, 4) wide-narrow DMA instruction buffer, and 5) ring-bus DMA instruction buffer. These buffer locations are described in more detail below with reference to FIG. 7. The names wide and narrow are used throughout this specification and generally refer to the approximate size in width (bits / bytes) of one or more memory units. As used herein, "narrow" may refer to one or more memory units each having a size or width of less than 16 bits, and "wide" may refer to one or more memory units each having a size or width of less than 64 bits.

[0076] The mesh bus 626 provides a data communication path distinct from the CSR bus 622, the instruction bus 624, and the ring bus 628 (described below). As shown in FIG. 6, the mesh bus 626 provides a communication path coupling or connecting each tile to its corresponding neighboring tile in both the X and Y dimensions. In various implementations, the mesh bus 626 may be used to transport input activation amounts between one or more narrow memory units in adjacent tiles. As shown, the mesh bus 626 does not allow for direct transfer of input activation data to non-adjacent tiles.

[0077] In various implementations, the mesh bus 626 and the various tiles connected via the mesh bus 626 may have the following configurations: The four corner tiles of the mesh have two outbound ports and two inbound ports. The four edge tiles of the mesh have three inbound ports and three outbound ports. All non-edge, non-corner tiles have four inbound ports and four outbound ports. Generally, considering an exemplary NxN tile layout, edge tiles are tiles with only three neighbors, while corner tiles are tiles with two neighbors. Regarding the data flow method over the mesh bus 626, generally, all input activations arriving via the mesh bus 626 for a particular tile must be committed to one or more narrow memory units of the tile. Furthermore, for tile configurations with fewer than four inbound ports, a DMAOp instruction may write zero values ​​to locations in the tile's narrow memory instead of waiting for data on absent input ports. Similarly, for tile configurations with fewer than four outbound ports, the DMAOp instruction does not perform narrow memory reads and port writes associated with transfers to any absent ports.

[0078] In some implementations, the location or address of a narrow memory unit to which a particular input activation is written or read will be generated by a Tensor Traversal Unit (hereinafter "TTU") based on inbound / outbound DMAOps provided over mesh bus 626. The inbound and outbound DMAOps may be executed simultaneously, with any required synchronization being managed through a synchronization flag control scheme managed by controller 602. The TTU is described in further detail below with reference to FIG. 7.

[0079] The ring bus 628 further provides communication coupling through a bus data path that originates at the controller 602 and connects the tiles 612, 614 in the ring back to the controller 602, similar to the CSR bus 622 and instruction bus 624. In various implementations, the ring bus 628 generally connects or couples all of the wide memory units (described in more detail below with reference to FIG. 7 ) in all of the tiles 612, 614. Thus, the payload width of the ring bus 628 corresponds to the width of the wide memory units located within each tile of the tile set 612, 614. As discussed above, the ring bus 628 further includes a bitmap header that indicates which tiles should consume payload data, including instructions or parameters, communicated over the ring bus 628.

[0080] In response to receiving information regarding data (i.e., payload) to be received at a particular tile via ring bus 628, each tile zeros (i.e., clears) the position data indicated in the bitmap header that is specific to the receiving tile before forwarding the data to another tile. Thus, if the header bitmap does not have the remaining bits set data indicating the particular tile that should receive the payload, forwarding of the payload to another tile will stop. Payload data generally refers to activations and weights used by one or more tiles during tensor computations performed based on execution of deeply nested loops.

[0081] In some implementations, the controller 602 may be described as being part of the ring bus 628. In one example, for a DMAOp instruction executed within a particular tile, the controller 602 may be used to pop data / payload from a ring bus stop in the ring and forward the payload to a ring bus stop in the next tile. The controller 602 may also commit payload data to one or more wide memory units of the tile if such action is required by an instruction in the bitmap header. The addresses of one or more wide memory units to which the data needs to be written may be generated by the DMAOp instruction within a particular tile.

[0082] In various implementations, each tile in the tile sets 612, 614 can be either a producer of payload data or a consumer of payload data. If a tile is a producer of payload data, it reads data from one or more of its wide memory units and multicasts the data over the ring bus 628 for consumption by one or more other tiles. If a tile is a consumer of payload data, it receives the data, writes it to one or more wide memory units within the tile, and forwards the payload data for consumption by one or more other tiles. With respect to the movement of payload data over the ring bus 628, there will typically be only one producer / master of data on the ring bus 628 at any given time. The DMAOp instruction execution order (e.g., a FIFO control scheme) across all tiles ensures that there is only one producer / master of data on the ring bus 628 at a given time.

[0083] In some implementations, the controller 602 uses a synchronization flag control architecture to ensure that there is only one producer / master of payload data on the ring bus 628 at a given time. In one example, each write by a tile to a ring output triggers an increment of a corresponding synchronization flag count. The controller 602 may inspect the payload data to determine the number of data chunks or segments that comprise the payload. The controller 602 then monitors execution by a tile to ensure that the expected number of data segments are transferred and / or consumed by the tile before another tile executes in master mode.

[0084] An exception to ensuring that there is only one producer / master of data on the ring bus 628 at a given time occurs when there are local multicast groups connected via the ring bus 628 that do not have overlapping regions on the ring bus. For example, Tile 0 (master) may multicast (i.e., generate data) to tiles in the Tile 0-Tile 3 grouping, while Tile 4 (master) may do the same for tiles in the Tile 4-Tile 7 grouping. An important requirement of this dual-master multicast method is that different multicast groups must not be allowed to see each other's data packets, as this could result in packet overlap and lead to one or more data calculation errors.

[0085] 6, the controller 602 provides a communication data path coupling or connecting tiles in the tilesets 612, 614 to the I / O 610 and includes several core functions. The core functions of the controller 602 generally include providing one or more I / O input activations to tiles in the tilesets 612, 614, providing tiles with one or more input activations and parameters received from the I / O 610, providing tiles with one or more instructions received from the I / O 610, sending I / O output activations to the host interface 608, and functioning as ring stops for the CSR bus 622 and the ring bus 628. As described in more detail below, the first tileset 612 and the second tileset 614 each include multiple tiles used to perform one or more tensor computations that are executed based on a deep loop nest consisting of an inner loop and an outer loop.

[0086] The system 600 generally operates as follows: The host interface 608 will provide the controller 602 with one or more instructions that define the direct memory access operations (DMAOps) that will occur for a given computation. Descriptors associated with the instructions provided to the controller 602 will contain information needed by the controller to facilitate large-scale dot product computations associated with multidimensional data arrays (tensors). Generally, the controller 602 receives input activations, tile instructions, and model parameters (i.e., weights) for performing tensor computations for a given layer of a neural network from the host interface 608. The controller 602 may then multicast the instructions to tiles 612, 614 in a dataflow manner defined by the instructions. As discussed above, a tile that consumes an instruction may then initiate broadcasting of new / subsequent instructions to another tile based on the bitmap data in the instruction header.

[0087] In terms of data flow, input activations and parameters are sent to tiles in the tile set 612, 614 via the ring bus 628. Each of the tiles 612, 614 stores a subset of the input activations needed to compute the subset of output activations assigned to that particular tile. A DMAOp instruction for a tile moves the input activations from wide memory to narrow memory. Computation within a tile begins when the necessary input activations, parameters / weights, and computation instructions (TTU operations, memory addresses, etc.) are available in the tile. Computation occurring within a tile ends when the MAC operator (described below) within the tile completes all dot product operations defined by the instruction set and the pre-activation function is applied to the results of the multiplication operations (i.e., the output activations).

[0088] The results of one or more tensor computations include writing the output activations of a computational layer to the narrow memory unit of the tile performing the computation. Some tensor computations involve transferring the output edge activations to adjacent tiles via the mesh bus 626. If the computation spans multiple layers, transferring the output edge activations to adjacent tiles is required to compute the output activations for subsequent layers. Once the computations for all layers are complete, the DMAOp transfers the final activations to the classifier tile 616 via the ring bus 628. The controller 602 then reads the final activations from the classifier tile 616 and executes the DMAOp to transfer the final activations to the host interface 608. In some implementations, the classifier portion 616 performs the computations for the output layer (i.e., the last layer) of the NN. In other implementations, the output layer of the NN is a classifier layer, a recurrent layer, or one of another layer types commonly associated with neural networks.

[0089] FIG. 7 illustrates an exemplary neural network (NN) computational tile 700 that may be used in an ASIC (e.g., TPU 106) as described with reference to FIG. 1 . Generally, the exemplary tile 700 may correspond to any of the tiles in the first tile set 612 and the second tile set 614 discussed above with reference to FIG. 6 . In various implementations, the computational tile 700 may also be referred to as or referred to as a computational unit 700. Each computational tile 700 is a self-contained computational unit configured to execute instructions independently of other corresponding tiles in the tile sets 612 and 614. As briefly discussed above, each computational tile 700 executes two types of instructions: TensorOp instructions and DMAOp instructions. Generally, each instruction type includes computational operations associated with a deep loop nest; therefore, each instruction type will generally be executed over multiple time epochs to ensure completion of all loop iterations.

[0090] As discussed in more detail below, different instruction types are executed by independent control units within the computation tile 700 that synchronize on data through synchronization flag control managed within the computation tile 700. The synchronization flag control manages concurrency between the execution of different instruction types within the computation tile 700. Each computation operation associated with each instruction type will be executed in strict order of issue (i.e., first-in, first-out). For two instruction types, TensorOP and DMAOp, no ordering guarantees exist between these different instruction types; each type is treated as a separate thread of control by the computation tile 700.

[0091] With respect to the data flow configuration, the computational tile 700 generally includes a data path 702 and a data path 705, each of which provides a communication path for data flow into and out of the computational tile 700. As described above, the system 600 includes three separate data bus structures laid out in a ring configuration: the CSR bus 622, the instruction bus 624, and the ring bus 628. With reference to FIG. 7 , the data path 705 corresponds to the instruction bus 624, and the data path 702 generally corresponds to one of the CSR bus 622 and the ring bus 628. As shown, the data path 702 includes a ring output 703 that provides an output path for data exiting the computational tile 700, and a ring input 704 that provides an input path for data entering the computational tile 700.

[0092] The compute tile 700 further includes a TensorOp control 706, which includes a TensorOp tensor traversal unit (TTU) 726, and a DMAOp control 708, which includes a DMAOp TTU 728. The TensorOp control 706 generally manages writes to and reads from TensorOp TTU registers 732, managing traversal operations for execution by the TensorOp TTU 726. Similarly, the DMAOp control 708 generally manages writes to and reads from DMAOp TTU registers 734, managing traversal operations for execution by the DMAOp TTU 728. The TTU registers 732 include an instruction buffer for storing one or more instructions, including operations to be performed by the TensorOp TTU 726 upon execution of the instruction by the TensorOp control 706. Similarly, the TTU registers 734 include an instruction buffer for storing one or more instructions containing operations to be performed by the TTU 708 upon execution of the instruction by the DMAOp control 708. As described further below, the TTUs are generally used by the computation tiles 700 to traverse array elements of one or more tensors residing in the narrow memory 710 and the wide memory 712.

[0093] In some implementations, an instruction for execution by the computational tile 700 arrives at the tile via data path 705 (i.e., portion of instruction bus 624). The computational tile 700 examines a header bitmap to determine the instruction type (TensorOp or DMAOp) and instruction subtype (read operation or write operation). Instructions received by the computational tile 700 are then written to a particular instruction buffer depending on the instruction type. Generally, instructions are received and stored (i.e., written to a buffer) prior to execution of the instructions by components of the computational tile 700. As shown in FIG. 7, the instruction buffers (i.e., TensorOp TTU register 732 and DMAOp TTU register 734) may each include a first-in, first-out (FIFO) control scheme that prioritizes consumption (execution) of one or more associated instructions.

[0094] As briefly discussed above, tensors are multidimensional geometric objects, and exemplary multidimensional geometric objects include matrices and data arrays. Algorithms involving deeply nested loops may be executed by the computational tile 700 to perform tensor computations by iterating one or more nested loops to traverse an N-dimensional tensor. In one exemplary computational process, each loop in a loop nest may be responsible for traversing a particular dimension of an N-dimensional tensor. As described herein, the TensorOp control 706 generally manages one or more tensor operations that drive the sequence in which dimensional elements of a particular tensor construct are traversed and accessed to complete the computation defined by the deeply nested loops.

[0095] The computational tile 700 further includes a narrow memory 710 and a wide memory 712. The narrow and wide designations generally refer to the size in width (bits / bytes) of the memory units of the narrow memory 710 and the wide memory 712. In some implementations, the narrow memory 710 includes memory units each having a size or width of less than 16 bits, and the wide memory 712 includes memory units each having a size or width of less than 32 bits. Generally, the computational tile 700 receives input activations via a data path 705, and the DMA control 708 performs operations to write the input activations to the narrow memory 710. Similarly, the computational tile 700 receives parameters (weights) via a data path 702, and the DMA control 708 performs operations to write the parameters to the wide memory 712. In some implementations, narrow memory 710 may include a memory arbiter typically used in shared memory systems to determine, for each memory cycle, which control device (e.g., TensorOp control 706 or DMAOp control 708) is allowed to access that shared memory unit of narrow memory 710.

[0096] The computational tile 700 further includes an input activation bus 716 and a MAC array 714 including multiple cells, each including a MAC operator 715 and a sum register 720. Generally, the MAC array 714 performs tensor computations, including arithmetic operations related to dot-product calculations, using the MAC operators 715 and sum registers 720 across multiple cells. The input activation bus 716 provides a data path along which input activations are provided by the narrow memory 710, one for each access by each MAC operator 715 of the MAC array 714. Thus, based on the broadcast transmission of the input activations one by one, a single MAC operator 715 of a particular cell each receives the input activation. The arithmetic operations performed by the MAC operators of the MAC array 714 generally include multiplying the input activations provided by the narrow memory 710 with parameters accessed from the wide memory 712 to generate a single output activation value.

[0097] During an arithmetic operation, partial sums may be accumulated and stored, for example, in corresponding sum registers 720, or may be written to wide memory 712 and re-accessed by specific cells of MAC array 714 to complete a subsequent multiplication operation. A tensor computation may be described as having a first and a second part. The first part is completed when the multiplication operation generates an output activation, for example, by completing multiplication of an input activation with a parameter to generate the output activation. The second part involves application of a nonlinear function to the output activation, and the second part is completed when the output activation is written to narrow memory 710 after application of the function.

[0098] The computational tile 700 further includes an output activation bus 718, a non-linear unit (NLU) 722 including an output activation pipeline 724, an NLU control 738, and a reference map 730 indicating core attributes of the components in the computational tile 700. The reference map 730 is shown for clarity but is not included in the computational tile 700. The core attributes include whether a particular component is a unit, a storage device, an operator, a control device, or a data path. Generally, upon completion of a first portion of the tensor computation, output activations are provided to the NLU 722 from the MAC array 714 via the output activation bus 718. After arriving at the NLU 722, data specifying an activation function received via the activation pipeline 724 is applied to the output activations, which are then written to the narrow memory 710. In some implementations, the output activation bus 718 includes at least one pipelined shift register 736, and completing the second part of the tensor computation includes using the shift register 736 of the activation bus 718 to shift the output activations toward the narrow memory 710.

[0099] For example, for a dot-product calculation of two multidimensional data arrays, for a single computational tile 700, the MAC array 714 provides robust single instruction multiple data (SIMD) capabilities. SIMD generally means that all parallel units (multiple MAC operators 715) share the same instruction (based on deep loop nesting), but each MAC operator 715 executes the instruction on different data elements. In one basic example, adding arrays [1,2,3,4] and [5,6,7,8] element-wise to obtain array [6,8,10,12] in one cycle would typically require four computational units to perform the operation on each element. By using SIMD, the four units can share the same instruction (e.g., "add") and perform the calculation in parallel. Thus, the system 600 and computational tile 700 provide improved acceleration and parallelism in tensor computations over conventional methods.

[0100] In one example, a single instruction may be provided by the controller 602 to multiple computational tiles 700 (see tilesets 612, 614 in FIG. 6 ) for consumption by multiple MAC arrays 714, as described in more detail below. Generally, a neural network layer may include multiple output neurons, which may be partitioned such that tensor computations associated with a subset of the output neurons may be assigned to a particular tile of the tilesets 612, 614. Each tile of the tilesets 612, 614 may then perform related tensor computations on a different group of neurons for a given layer. Thus, the computational tiles 700 may provide at least two forms of parallelism: 1) one form involving partitioning output activations (corresponding to subsets of output neurons) among multiple tiles of the tilesets 612, 614; and 2) another form involving simultaneous computation (with a single instruction) of multiple subsets of output neurons based on the partitioning among the tiles of the tilesets 612, 614.

[0101] While the disclosed inventive concepts include those defined in the appended claims, it should be understood that the inventive concepts may be further defined according to the following embodiments.

[0102]

[0013] Embodiment 1 is a method for memory allocation in a machine learning accelerator communication network. Generally, the method includes accessing metadata associated with multiple communication ports of an application-specific integrated circuit (ASIC). The metadata identifies, for each port, whether the particular port is used in the current configuration and the communication medium associated with the particular port. Based on the metadata, an expected latency is determined for each port, and a portion of shared memory is allocated to each port. The shared memory is allocated by allocating zero memory to unused ports, determining a memory allocation for each port based on the expected latency, and assigning a start address and a stop address of the shared memory to each port. Implementations may optionally include one or more of the following features.

[0103] Embodiment 2 is the method according to embodiment 1, wherein assigning a start address and a stop address of the shared memory to each port includes calling an application programming interface (API) by a device other than the ASIC.

[0104]

[0013] Embodiment 3 is the method of embodiment 1 or 2, wherein the process is performed on an ASIC using a machine learning accelerator communication network and allocated shared memory. In some embodiments, the process is training a neural network.

[0105] Embodiment 4 is the method according to any one of embodiments 1 to 3, wherein the ASIC is a tensor processing unit (TPU).

[0106]

[0023] Example 5 is the method of any one of Examples 1 to 4, wherein the communication medium identified in the metadata is a copper cable medium, an optical medium, a printed circuit board (PCB) medium, or any combination thereof.

[0107] In a sixth embodiment, the memory comprises a memory element having a formula

[0108]

number

[0109] 6. The method according to any one of embodiments 1 to 5, wherein the allocation is performed according to the following: Embodiment 7 is a system including one or more computers and one or more storage devices that store instructions, which, when executed by the one or more computers, are operable to cause the one or more computers to perform the method described in any one of claims 1 to 6.

[0110] Embodiment 8 is a computer storage medium encoded with a computer program, the program comprising instructions operable, when executed by a data processing device, to cause the data processing device to perform the method of any one of claims 1 to 6.

[0111]

[0014] Embodiment 9 is a method for memory allocation in a machine learning accelerator communication network, the method including determining a network topology for a network of machine learning accelerator ASICs. Metadata associated with a plurality of communication ports of each ASIC in the network is accessed. For each port of the plurality of ports, the metadata identifies whether the particular port is used in the current configuration and the communication medium associated with the particular port. For each port used in the network topology, a round-trip time (RTT) delay is determined. A portion of shared memory is allocated to each port of the plurality of ports by determining a memory allocation for each port proportional to the RTT delay and assigning a start address and a stop address of the shared memory to each port for the determined memory allocation. A process is executed on the machine learning accelerator to send profiling traffic to the network for a predetermined duration. For each port, the number of received traffic packets is determined and a portion of the shared memory is reallocated to each port of the plurality of ports, the reallocation being performed by determining a memory allocation for each port proportional to the received packet count and reallocating a start address and a stop address for each port of the determined memory allocation.

[0112]

[0023] Embodiment 10 is the method according to embodiment 9, wherein allocating and reassigning the start address and stop address of the shared memory to each port includes calling an API by a device other than the ASIC.

[0113] In an eleventh embodiment, a memory allocation proportional to the determined RTT delay for each port is calculated by the formula

[0114]

number

[0115] 11. The method of embodiment 9 or 10, wherein the method is determined according to In a twelfth embodiment, the memory allocation for each port proportional to the received packet count is calculated by the formula

[0116]

number

[0117] 12. The method according to any one of embodiments 9 to 11, wherein the temperature is determined according to the following formula:

[0033] Embodiment 13 is the method according to any one of embodiments 9 to 12, wherein the RTT delay is calculated before execution by sending and receiving one or more timing messages to determine the latency.

[0118]

[0023] Embodiment 14 is the method according to any one of embodiments 9 to 13, wherein the profiling traffic comprises all-to-all traffic, nearest neighbor traffic, a composite traffic profile, or any suitable combination thereof.

[0119] Embodiment 15 is a method according to any one of embodiments 9 to 14, wherein assigning a start address and a stop address of the shared memory to each port includes calling an application programming interface (API) by a device other than the ASIC.

[0120]

[0023] Embodiment 16 is the method of any one of embodiments 9 to 15, wherein the process is performed on an ASIC using a machine learning accelerator communication network and allocated shared memory. In some embodiments, the process is training a neural network.

[0121] Embodiment 17 is the method according to any one of embodiments 9 to 16, wherein the ASIC is a tensor processing unit (TPU).

[0122] Embodiment 18 is a system including one or more computers and one or more storage devices that store instructions, which, when executed by the one or more computers, are operable to cause the one or more computers to perform a method according to any one of claims 9 to 17.

[0123] Embodiment 19 is a computer storage medium encoded with a computer program, the program comprising instructions operable, when executed by a data processing device, to cause the data processing device to perform the method of any one of claims 9 to 17.

[0124]

[0033] Embodiment 20 is a method for allocating memory for a machine learning accelerator communication network, the method including determining a network topology for a network of machine learning accelerator application-specific integrated circuits (ASICs). Metadata associated with multiple communication ports of each ASIC in the network is accessed. For each port of the multiple ports, the metadata identifies whether the specific port is used in a current architecture and a communication medium associated with the specific port. For each port of the multiple ports, the metadata identifies whether the specific port is used in the current configuration and the communication medium associated with the specific port. For each port used in the network topology, a round-trip time (RTT) delay is determined. A portion of shared memory is allocated to each port of the multiple ports by determining a memory allocation for each port proportional to the RTT delay and assigning a start address and a stop address of the shared memory to each port for the determined memory allocation. A process is executed on the ASIC, and the process uses the machine learning accelerator communication network with the allocated shared memory. During execution of the process, a number of message packets received at each port of the plurality of ports over a first time period is determined. A desired portion size of the shared memory is determined for each port of the plurality of ports based on the number of message packets received at each port. During a second time period, the process is paused and it is determined whether the shared memory is clear of pending message packets. If the shared memory is clear, start and stop addresses of the shared memory are reallocated for each port according to the desired portion size, and execution of the process is resumed.

[0125]

[0032] Embodiment 21 is the method of embodiment 20, wherein assigning a start address and a stop address of the shared memory to each port includes calling an application programming interface (API) by a device other than the ASIC.

[0126]

[0023] Embodiment 22 is the method of embodiment 20 or 21, wherein the process is performed on an ASIC using a machine learning accelerator communication network and allocated shared memory. In some embodiments, the process is training a neural network.

[0127] Embodiment 23 is the method according to any one of embodiments 20 to 22, wherein the ASIC is a tensor processing unit (TPU).

[0128] Embodiment 24 is a system including one or more computers and one or more storage devices that store instructions, which, when executed by the one or more computers, are operable to cause the one or more computers to perform a method described in any of claims 20 to 23.

[0129] Embodiment 25 is a computer storage medium encoded with a computer program, the program comprising instructions operable, when executed by a data processing device, to cause the data processing device to perform the method of any one of claims 20 to 23.

[0130] The foregoing description has been provided in the context of one or more specific implementations. Various modifications, variations, and substitutions of the disclosed implementations are possible without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited to only the implementations described or shown, but is to be accorded the widest scope consistent with the principles and features disclosed herein.

[0131] Certain embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, while bus lines are described as “controllable,” not all bus lines need have the same level of control. For example, the degree of controllability may vary; some bus lines may be only controlled, and some bus lines may be limited with respect to the number of tiles from which data can be sent or to which data can be transmitted. In another example, some bus lines may be dedicated to providing data along a single direction, such as north, east, west, or south, as described herein. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order or sequential order shown to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

[0132] Embodiments of the subject matter and functional operations described herein may be implemented in digital electronic circuitry, in computer hardware including the structures disclosed herein and their structural equivalents, or in one or more combinations thereof. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or control of the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to a receiver apparatus suitable for execution by a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof.

[0133] The term "data processing apparatus" encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. Such apparatus may include special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC. In addition to hardware, an apparatus may also include code that creates an execution environment for the computer program, such as code comprising processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0134] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may also be implemented as, special purpose logic circuitry, such as, for example, an FPGA, an ASIC, or a GPGPU (general purpose graphics processing unit).

[0135] While this specification contains details of many specific implementations, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features specific to particular implementations of a particular invention. Certain features described herein in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented separately in multiple implementations or in any suitable subcombination. Furthermore, while features may be described above as working in a particular combination and may even be initially claimed as such, one or more features of a claimed combination may in some cases be deleted from that combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0136] Similarly, although operations are depicted in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown, or in sequential order, or that all of the shown operations be performed, to achieve desirable results. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged in multiple software products.

[0137] In addition to the above, users may be provided with controls that allow them to select both whether and when the systems, programs, or features described herein may enable the collection of user information. Furthermore, certain data may be handled in one or more ways before being stored or used, such that personally identifiable information is removed. For example, the user's identity may be handled so that personally identifiable information cannot be determined for the user, or so that the user's skin data and / or diagnosis cannot be identified when associated with the user. Thus, users may have control over what information is collected about them and how that information is used.

[0138] Thus, specific implementations of the subject matter have been described. Other implementations are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. 1. A computer-implemented memory allocation method for a machine learning accelerator communication network, the method comprising: accessing metadata associated with a plurality of communication ports of an application specific integrated circuit (ASIC), the metadata identifying, for each port of the plurality of ports, whether the respective port is used in a current configuration and a communication medium associated with the respective port; The method further comprises: determining an expected latency for each port of the plurality of ports based on the accessed metadata, the expected latency being a round trip time (RTT) delay; The method further comprises: allocating a portion of the shared memory to each port of the plurality of ports; The allocating step includes: assigning a queue size of zero to unused ports; determining a memory allocation for each port based on the expected latency; assigning a start address and a stop address of the shared memory to each port; A method carried out by.

2. 2. The method of claim 1, wherein assigning the start address and the stop address of the shared memory to each port comprises invoking an application programming interface (API) by a device different from an ASIC.

3. 3. The method of claim 1, further comprising executing a process on the ASIC, the process using the machine learning accelerator communication network and the allocated shared memory.

4. The method of claim 3 , wherein the process includes training a neural network.

5. The method of any one of claims 1 to 4, wherein the ASIC is a tensor processing unit (TPU).

6. The communication medium identified in the metadata is: Copper cable medium, Optical media, or Printed Circuit Board (PCB) Media The method according to any one of claims 1 to 5, comprising at least one of:

7. The memory allocation is [Equation 1] is determined based on 7. The method of claim 1, wherein p is the port being calculated, N is the total number of ports in active use, Queue Size(p) is the queue size allocated to port p, latency p is the latency of port p, latency i is the latency of port i, and Total Size is the capacity of the queue.

8. 1. A computer-implemented memory allocation method for a machine learning accelerator communication network, the method comprising: determining a network topology for a network of machine learning accelerator application specific integrated circuits (ASICs); accessing metadata associated with a plurality of communication ports of each ASIC in the network, the metadata identifying, for each port of the plurality of ports, whether the respective port is used in a current configuration and a communication medium associated with the respective port; The method further comprises: determining a round trip time (RTT) delay for each port used in the network topology; allocating a portion of the shared memory to each port of the plurality of ports; The allocating step includes: determining a memory allocation for each port proportional to the RTT delay; assigning a start address and a stop address of the shared memory to each port for the determined memory allocation; carried out by The method further comprises: running a process on the machine learning accelerator to send profiling traffic to the network for a predetermined duration; determining, for each port, a number of traffic packets received; reallocating a portion of the shared memory to each port of the plurality of ports; The reallocating step includes: determining a memory allocation for each port proportional to the number of traffic packets received; reassigning the start address and the stop address of the shared memory to each port for the determined memory allocation; A method carried out by.

9. 9. The method of claim 8, wherein assigning and reassigning the start and stop addresses of the shared memory to each port includes invoking an application programming interface (API) by a device different from an ASIC.

10. 10. The method of claim 8 or 9, comprising executing a process on the ASIC, the process using the machine learning accelerator communication network and the allocated shared memory.

11. The method of claim 10 , wherein the process includes training a neural network.

12. The method of any one of claims 8 to 11, wherein the ASIC is a tensor processing unit (TPU).

13. The memory allocation proportional to the determined RTT delay is [Equation 2] is determined based on 13. The method of any one of claims 8 to 12, wherein p is the port being calculated, N is the total number of ports actively used, Queue Size(p) is the queue size allocated to port p, latency p is the latency of port p, latency i is the latency of port i, and Total Size is the capacity of the queue.

14. The memory allocation for each port proportional to the number of packets received is [Equation 3] is determined based on 13. The method of claim 8, wherein p is the port being calculated, N is the total number of ports of the ASIC, Queue Size(p) is the queue size allocated to port p, packet_num p is the number of received packets for port p, packet_num i is the number of received packets for port i, and Total Size is the capacity of the queue.

15. The method of any one of claims 8 to 14, wherein the RTT delay is calculated before execution by sending and receiving one or more timing messages to determine the latency.

16. The profiling traffic includes: All-to-all traffic, Nearest neighbor traffic, or Synthetic Traffic Profiles The method according to any one of claims 8 to 15, wherein the method is at least one of the following:

17. 1. A computer-implemented memory allocation method for a machine learning accelerator communication network, the method comprising: determining a network topology for a network of machine learning accelerator application specific integrated circuits (ASICs); accessing metadata associated with a plurality of communication ports of each ASIC, the metadata identifying, for each port of the plurality of ports, whether the respective port is used in the current configuration and a communication medium associated with the respective port; The method further comprises: determining a round trip time (RTT) delay for each port used in the network topology; allocating a portion of the shared memory to each port of the plurality of ports; The allocating step includes: determining a memory allocation for each port proportional to the RTT delay; assigning a start address and a stop address of the shared memory to each port for the determined memory allocation; carried out by The method further comprises: executing a process on an ASIC, the process using the machine learning accelerator communication network with the allocated shared memory; The method further comprises: During the execution of said process, determining a number of message packets received at each port of the plurality of ports over a first time period; determining a desired portion size of the shared memory for each port of the plurality of ports based on the number of message packets received at each port; pausing the process for a second period of time; determining that the shared memory is clear of pending message packets; reassigning the start address and the stop address of the shared memory to each port for the desired portion size; resuming execution of said process; A method comprising:

18. 18. The method of claim 17, wherein assigning and reassigning the starting and stopping addresses of the shared memory to each port comprises invoking an application programming interface (API) by a device different from an ASIC.

19. 19. The method of claim 17 or 18, wherein the ASIC is a tensor processing unit (TPU).

20. The method of any one of claims 17 to 19, wherein the process comprises training a neural network.

21. A program that causes a computer to execute the method according to any one of claims 1 to 20.

Images